^{Module Three Additional Assignment – Option Two}

^{Developing a Criteria for Evaluating Online Courses}

• What is Evaluation?

Evaluation, as defined by Tovey & Lawlor (2004, p.240) is ‘…the worth or value of the training course in relation to the aims and goals for which it was developed’. Ayoub and Ekong (2007) hold a similar view, stating that ‘evaluation is about making a judgment of the merit or worth of something… based on criteria which are either determined individually or collectively.’

Bramley (1991) proposes that evaluation completes the cycle of training; stating that this is the most common view. The author of this paper disagrees with this statement and believes, as do Tovey and Lawlor (2004, p.241), that evaluation should be viewed as an ‘ongoing’ and continuous improvement process.

Evaluation identifies areas of achievement and non-achievement, explores reasons for both, and assists in developing strategies for improving future performance (Gallasch & Davenport, 1992).

• Why Evaluate?

It is good professional practice to evaluate training programs. It allows us to continually improve effectiveness. Evaluations are carried out for many reasons. Some examples of which are listed below:

1. To determine if the proposed outcomes of training achieve
2. To determine if training is cost effective
3. To decide if the expense involved is considered worthwhile
4. To determine the extent of any changes in behaviour and work performance
5. To find out if pre-existing problems were solved or reduced
6. To find out if anything needs to be changed because it did not work
7. To assess whether additional training is require
8. To find out if changes are required to the program, budget, or time-frames

• Outline and Context of the Evaluation Criteria:

It is envisaged by the author that in the not too distant future e-learning in some form or other will be integrated into the Certificate III Hairdressing Trade Course offered at Nepean TAFE College, Kingswood. Maybe this is wishful thinking on her part; but the author believes that it may soon become a reality. Read on to find out why.

The implementation of the WRH06 Training Package in early 2007 has led to a change from modular-based to unit-based enrolment. Some of the previous modules now consist of as many as four smaller individual units. A significant increase in the nominal hours accompanied the new training package with no corresponding increase in delivery time-frames. As a result, it is now virtually impossible to deliver the first semester units of the course in the traditional face-to-face manner; even if they are delivered in an integrated holistic manner as is presently the case.

Industry and Employer Associations such as the Professional Hairdressers Association are driving the need for more flexibility in program delivery, and there has been serious talk of reducing the delivery time-frame of the Certificate III Trade Course from the current two years, to eighteen months. In view of these issues, the author foresees the necessity of change along with the possibility of developing and introducing a blended learning approach to delivery.

The first of the evaluation tools developed by the author will assist in the evaluation of learner readiness prior to implementation of a pilot program for a proposed online course {hypothetical}. This tool is considered a form of front-end evaluation (Ayoub and Ekong, 2007) which gathers information about the pre-existing knowledge, skills, attitudes and behaviours of learners. The second evaluation tool will enable evaluation of the program after implementation. This is what Ayoub and Ekong (2007) refer to as process evaluation. The tools could also provide educators with insight into some considerations {from a learner’s perspective} that should be made when developing courses.

• What Influenced Development of the Evaluation Tools?

The Evaluation Tools designed by the author have been primarily based on the personal learning experiences of the author during e-Learning Experiences lectures at UTS, and whilst studying the Visual Link Spanish^TM course in Module Three. The information gained from completing Activity 3.4 in the Module Three Activity Guide (Pritchard, 2007) was used as a guide to further develop ideas. It created a starting point to build on. The SWOT Analysis of e-learning conducted in the eLExp lecture on November 24 also aided in determining what criteria to include.

Ayoub and Ekong (2007) provided the following food for thought:

‘evaluation can provide feedback about the program content, learning materials, assessment, presentation, facilities, suggestions for improvement and most importantly the degree to which learning outcomes are being met’.

The author gained background insight into what evaluation entails and why it would be conducted in Tovey and Lawlor (2004). They discuss four main areas that should be looked at in any evaluation strategy: trainers, learners, the course itself, and the costs of the course. They also point out the steps that should be taken to assist in designing evaluations: determine who the evaluation is for, what you want to evaluate, identify what you will do with the results of the evaluation, and then developing a strategy for the evaluation. The next step is to choose a technique, making sure it is suitable for the purpose it was intended.

Donald Kirkpatrick’s Four Levels of Evaluation (Kruse, 2007: Tovey & Lawlor, 2004) also influenced the development of the evaluation criteria. Kirkpatrick’s four levels of evaluation are:

• Level I – Reaction
• Level II – Learning
• Level III – Behaviour
• Level IV – Results

The Reaction level is based on how learners feel about the course, the instructor or facilitator, and the training environment. Reaction level evaluations are usually conducted using ‘smile sheets’ (Kruse, 2007, p.1) or ‘happy sheets’ (Tovey & Lawlor, 2004, p.243). Kruse (2007) suggests that evaluations on this level can produces more valuable information by using questionnaires to survey full student populations. He suggests that such evaluations can reveal information about:

• Relevance of objectives
• Ability of course to maintain learner interest
• Amount and appropriateness of interactive exercises
• Ease of navigation
• Apparent value of transferability to the workplace

The Learning Level is based on results. It is about measuring changes in learner’s knowledge, skills and attitudes. It looks at what has been learned in the course, and simply asks did learners actually achieve what the program intended is usually. Assessment is the most common way of measuring results. Kruse (2007) suggests learners should complete a pre and post test to compare results.

The Behaviour Level is based on the transfer of knowledge, skills and attitudes to the workplace and measures achievement of performance objectives. It looks at the changes that have occurred as a result of training and evaluates the transfer of these skills knowledge and attitudes to the workplace. Ideally this evaluation should be carried out ‘…three to six months after training’… (Kruse, 2007, p.2).

The Outcomes Level assesses the long-term success, bottom-line impact on community and the environment; the ‘business results’ as Kruse (2007, p.2) calls them. It is rarely done.

A further level which the author believes should be addressed would be ROI or return on investment. This would be something that organisations would most certainly be interested in determining through evaluation of training programs.

The templates for the pre-course and post-course evaluation tools developed by the author were adapted from the Pilot Test Evaluation Questionnaire created by Raymond Karsan Associates which was accessed from the e-learning guru; Kevin Kruse. Click on the links below to access these tools.

• Evaluation Tools Developed by the Author:

Pre-Course Evaluation Tool

Post-Course Evaluation Tool

The author has located an abundance of useful information whilst researching for this assignment and references to these sources have been included in a bibliography of additional sources at the end of the assignment.

References:

Ayoub, M. & Ekong, S. 2007, Evaluating e-learning, viewed, 8 November, http://elexp.wikispaces.com/EVALUATING+E-LEARNING

Bramley, P. 1991, Evaluating Training Effectiveness: translating theory into practice, The McGraw-Hill Training Series, Series Editor: Roger Bennett, McGraw-Hill,London.

Gallasch, P. & Davenport, A. 1992, Open Learning and Flexible Delivery National Staff Development Program, Module No.3, Managing Flexible Delivery, TAFE National Staff Development Committee, Adelaide, South Australia.

Kruse, K. 2007, Evaluating e-Learning: Introduction to the Kirkpatrick Model, viewed 26, October, 2007, http://www.e-learningguru.com/articles/art2_8.htm

Prichard, L. 2007, eLExp Reading Guide Mod3 2007, University of Technology, Sydney.

Raymond Karsan Associates. Pilot Test Evaluation Questionnaire, viewed 6 November, 2007, http://www.e-learningguru.com/tools/pilottest.doc

Tovey, M.D, & Lawlor, D.R. 2004, Training in Australia: design, delivery, evaluation, management, 2nd edn, Pearson Education,
Australia.

Bibliography of Additional Resources:

Business Balls. 2007, Kirkpatrick Learning and Evaluation Model, viewed, 4 November, 2007, http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm

Business Balls. 2007, Training Evaluation: Kirkpatrick Model, viewed, 4 November, 2007, http://www.businessballs.com/freematerialsinexcel/trainingevaluationkirkpatrickmodel.xls

Carliner, S. 2002, Tips for Conducting Level 1 Evaluations of Asynchronous e-Courses, viewed, 26 October, 2007, http://saulcarliner.home.att.net/oll/levelone.htm

EvaluTech, 2007, Criteria for Evaluating Online Courses, viewed, 13 October, 2007, http://www.evalutech.sreb.org/criteria/online.asp

Horton, W. 2001a, Evaluating eLearning checklist, viewed, 6 November, 2007, http://www.horton.com/content/spreadsheets/evaluatingelearningchecklist.xls

Horton, W. 2003, Evaluating Webinar, viewed, 6 November, 2007, http://www.horton.com/evaluating/webinar.pdf

Kennedy, G., Petrovic, T. & Keppell, M. 1998, The Development of Multimedia Evaluation Criteria and a Program of Evaluation for Computer Aided Learning, viewed, 26 October, 2007, http://www.ascilite.org.au/conferences/wollongong98/asc98 pdf/kennedypetrovickeppel.pdf

Kruse, K. 2007, Checklist: Evaluating Instructional Design for e-Learning, viewed, 6 November, 2007, http://www.e-learningguru.com/articles/art37.htm

Lambert, P. & MacLean,I. 2007, What Makes e-Learning Work? Online Learning Australia, viewed, 24 October, 2007, http://www.olla.com.au/resourceswhatmakeselearningwork.htm

Leacock, T. L., & Nesbit, J. C. 2007. A Framework for Evaluating the Quality of Multimedia Learning Resources, Educational Technology & Society, vol.10, no.2, pp.44-59, viewed, 26 October, 2007, http://www.ifets.info/journals/10_2/5.pdf

Mayberry, E. 2005, Kirkpatrick’s Level 3: Improving the Evaluation of E-Learning, viewed, 24 October, 2007, http://www.learningcircuits.org/2005/may2005/mayberry.htm

Phillips, J.J., Pulliam Phillips, P., & Krucky Hodges, T. 2004, Resources Needed to Make Evaluation Work, viewed, 24 October, 2007, http://www.learningcircuits.org/2004/nov2004/phillips.htm

Rae, L. 2004, Training Evaluation Tools, viewed, 6 November, 2007, http://www.businessballs.com/trainingevaluationtools.pdf

Sener, J. 2004, Escaping the Comparison Trap: Evaluating Online Learning on Its Own Terms. Innovate vol.1, no.2, viewed, 20 October, 2007, http://www.innovateonline.info/index.php?view=article&id=11

Simpson, C. 2007. Thoughts on: Evaluating online learning: Models and methods, viewed 19 , October, 2007, http://elearningnews.edublogs.org/2007/10/17/thoughts-on-evaluatingonlinelearningmodelsandmethodsgunawardenaetal2000/

Sonwalkar, N. 2001, A New Methodology for Evaluation: The Pedagogical Rating of Online Course, Campus Technology, Nov- Dec, 2001, viewed, 8 November, 2007, http://campustechnology.com/articles/38946/

Wright, C.R. 2003, Criteria for Evaluating the Quality of Online Courses, Alberta Distance Education and Training Association, October, viewed, 8 November, 2007 http://www.imd.macewan.ca/imd/content.php?contentid=36