Measuring Outcomes and Effectiveness of e-learning

Comparative Review of Group Case Studies

Review Purpose:

In this paper the author conducts a comparative review of three individual case studies chosen by members of the rose btwn 2 thorns group. At least one of these case studies was expected to be from an international perspective. Group members were each required to select a case study addressing the groups focus topic; Measuring Outcomes and Effectiveness of e-learning.

The author initially outlines the focus topic and then moves on to provide a brief overview of each case study. In addition, areas of similarity and discord amongst the studies will be highlighted. Discussion of the various issues raised for educators will be undertaken by the author, and suggestions of possible future research directions made.

Outline of Group Focus Topic:

The group focus topic was Measuring Outcomes and Effectiveness of e-learning. This topic provided considerable challenge to group members who found it difficult to locate current case studies, specifically relevant to the topic, which adequately answered the questions posed. From the outset, group members agreed that measuring outcomes and effectiveness of any type of learning could at times be unreliable due to subjectivity of teacher’s assessment processes, and learners own opinions as to what they had achieved.

Group members wanted to investigate precisely how outcomes are being measured by organizations and exactly what determining factors contribute to making e-learning programs more effective. It was anticipated that effective e-learning programs would ultimately lead to successful outcomes for both learners and educators.

Overview of Selected Case Studies:

Rick Vanderburg’s case study was a telephone interview with Graeme Wilton a Certificate IV Building teacher at Armidale TAFE. The college delivers the course face-to-face, and in virtual classrooms via web conferencing using Adobe Connect ®. The virtual classroom pilot study undertaken this year has experienced unpredicted success. As a result the college will not be offering face-to-face delivery in 2008 (Vanderburg, 2007).

The interview involved a partial comparative analysis of delivery methods; focusing specifically on attrition rates, attendance and assessment results. Vanderburg (2007) reports that attrition rates were significantly lower in the virtual class as opposed to face-to-face and distance delivery methods. 100% attendance of all thirty learners for each session has also been achieved. Nevertheless there is no explanation provided for these outcomes.

It was indicated that students performed poorly in assessment in the virtual classroom when contrasted with the face-to-face cohort. Wilton believes this to be a result of the inexperience of educators utilizing the technology and a lack of non-verbal communication as a result of the delivery method (Vanderburg, 2007).

The case study chosen by Richard Toovey was embedded within a larger research document E-learning in Australia and Korea: Learning from practice (Misko et al, 2004). The micro-study ‘E-Learning at OTEN’ (pp.65-72) discusses various aspects of course delivery at the Open Training and Education Network [OTEN] which is the specialist distance arm of TAFE NSW. OTEN delivers programs via blended delivery both locally and internationally.

Approximately 7% of enrolled learners study courses exclusively online, with remaining learners utilizing print or CD-Rom versions of learning materials (Misko et al, 2004). Rolling enrollments are conducted, and learning is self-paced. Student to teacher ratio average is ‘…200:1…’, and there is no face-to-face interaction between learners and teachers or fellow learners (Misko et al, 2004, p.68).

The study points out challenges associated with distance delivery and discusses ways these can be overcome. Student and teacher readiness, support mechanisms, assessment, evaluation and learning methods are all elucidated. The strengths and weaknesses of the OTEN approach to e-learning are summarized, and the study concludes by identifying contributory success factors.

The Capella University study (Barron, 2006) chosen by Nayomie Baihn reveals factors that contribute to the success of e-learning programs. Barron (2006) hints that success results from deliberate planning, potent course content, and dedicated people who are committed to the cause.

The study discusses the essential ‘ingredients’ required in preparing the ‘recipe’ of successful and effective e-learning (Barron, 2006, p.361). These encompass team orientated visions of learning, high quality tailored courses, clearly detailed marking criteria and descriptive feedback. User choice in group tasks, consistent standards of facilitation, and experienced teachers with the required skill and knowledge levels, who are able to ‘develop a strong sense of community’ (Barron, 2004, p.362).

In addition, the provision of adequate prior training and ongoing support for instructors, documented course parameters outlining clear expectations for learners, and the necessity of learner feedback that is significant and well-timed. Completing the list is the need for continuous improvement and ongoing monitoring and evaluation (Barron, 2006).

Comparison and Contrast:

The Armidale TAFE (Vanderburg, 2007) and Capella University Studies (Barron, 2006) contain elements of bias in that the primary source of information in both is an employee of the organization, and in the case of Vanderburg (2007); a friend of the writer. In comparison the OTEN (Misko et al, 2004) study is conducted by an independent source. Research methodologies are similar in that they are all qualitative in nature. However they differ in the data collection methods used.

Vanderburg (2007) used an informal telephone interview technique. Barron (2006) also used interviews in addition to focus groups, observation and evaluations with both staff and learners involved. Misko et al (2004) similarly used semi-structured interviews involving teachers, learners, managers, administrative and support staff, in addition to analysing information gained from the OTEN website and student focus groups.

The studies indicate dissimilar teacher to learner ratios. Enrolment figures, curriculum offerings, and delivery modes vary notably across the studies. The Armidale study (Vanderburg, 2007) has thirty learners enrolled with an estimated 300% increase in 2008. Geographically learners come from across Australia. The study is limited to one specific course and the delivery modes discussed are face-to-face and web conferencing.

Comparatively OTEN provides training via blended delivery to ’…38,000…’ learners across ‘…250 courses’ including local qualifications and international delivery via partnership arrangements (Misko et al, 2004, p.65). As mentioned earlier, only 7% of learners study entirely online. Whereas Capella University delivers a staggering ‘…650…’ courses, exclusively online to ‘…14, 500 learners from 63 countries…’ (Barron, 2006. p.360).

The Armidale study implies it is vital that adequate teacher training prior to course commencement is undertaken to increase effectiveness of the course and improve assessment outcomes of learners (Vanderburg, 2007). This view is reiterated by both Barron (2006) and Misko et al (2004) who identified the following factors contributing to effective e-learning programs: wide-spread professional development of staff, ongoing strategies in place to support both learners and staff, and effective technological support systems.

Barron (2006, p.361) discusses the need for a ‘shared learning centred vision’. This is similarly referred to in the OTEN study as ‘an organisational approach to delivery’ (Misko et al, p.71). Both studies concur on the importance of ongoing monitoring and evaluation to determine learner’s satisfaction levels and facilitate continual course improvements.

The development, continual maintenance, and timely upgrading of adequate infrastructure and course programs to ensure currency and efficiency is another area of consensus within the Barron (2006) and Misko et al (2004) studies.

Implications for Educators:

Facilitators and learners alike must possess the required technological skills and knowledge to participate in e-learning programs. They must also be provided with training, and ongoing support to remain engaged. Teachers and learners must to be able to gain access to and successfully utilize the required technologies for learning to be effective. This will provide a foundation which is capable of producing measurable learning outcomes. Courses must be monitored and updated continually to ensure that they are meeting the needs of learners which will ultimately lead to successful outcomes for learners, teachers and organizations.

The author believes that further studies specifically focusing on measurement of outcomes should be conducted. It is evident from the case studies presented here and the difficulties experienced by group members in obtaining such documentation that numerous studies tend to focus on success stories without explaining exactly how the perceived successes have been measured. It is far easier to locate contributory factors influencing effectiveness, and information on the perceived benefits and weaknesses of e-learning methods.

Conclusion:

In this paper the author has reviewed two Australian and one international case study. The Australian studies are drawn from Armidale TAFE College and The Open Training and Education Network [OTEN]. Capella University in the USA provided the resource material for the required international study.

The context for further discussion was established by the author through an early outline of the groups focus topic; Measuring Outcomes and Effectiveness of e-learning. An overview of the substance of each case study was also portrayed. Areas of resemblance and dissonance amongst the studies have been examined, and implications for educators gleaned from the review have been highlighted. Potential ideas for future research directions have also been offered.

References:

Barron, J. 2006, ‘Top ten secrets of effective e-learning’, Industrial and Commercial Training, vol. 38, no. 7, pp.360-364.

Misko, J.,  Choi, J., Hong, S., & Lee, I. 2004, E-learning in Australia and Korea: Learning from practice, Korea Research Institute for Vocational Education & Training [KRIVET], Republic of Korea.

Vanderburg, R. 2007, Measuring Outcomes and Effectiveness: Graeme Wilton telephone interview, delivery of Certificate IV Building course at Armidale TAFE College via web conferencing, viewed 20 September, 2007, <http://elexp.wikispaces.com/rose+btwn+2thorns>.