Weeks five and six: Evaluation methods

Part 1

The networking course I am proposing to part evaluate has been developed by Cisco Systems. It comes with its own LMS. This is a course designed to prepare the student to an external professional certification examination, The Cisco Certified Network Associate.

The students are supplied with a username and a password, then are enrolled in a relevant on line class on the Cisco LMS by the lecturer. Amongst the learning environment supplied, the student can access a message board, his/her present and past classes, the on-line course, a gradebook, a discussion board. The lecturer can activate end of chapter tests which can be use for formative and summative assessment.

At the moment I am using this environment in conjunction with the Blackboard LMS for announcements/emails /message boards (not very successfully)/ storage of resources.

The course content and interfaces are upgraded regularly and there is a bug tracking system enabling to report any bugs in the courses and assessments.

The course alternate theory and lab supporting the understanding of the theory. The student have access to real equipment in the classroom and now to a downloadable simulator which is now getting very close to reality.

From Cisco point of view, to pass a course a student needs to pass the final online theory examination and the practical examination. The minimum level for a pass is left to the discretion of the “academy” (Manukau Institute of Technology). The recommended pass rate is 70%.

Changes

The course is divided in 4 parts from a Cisco point of view, but at the moment the course is mapped to two semester courses. The proposal is to develop the course to 4 semesters next year, changing its delivery mode to Blended delivery. This would involve using other tools to allow easy access to the lecturer and making lectures available online for students to access in their own time.

The present running of the course:

The course is running as a face to face course, and the students have to attend 4 hours weekly.

The students can access the lab to complete the practical work at any time, 7 days per week, when the room is not used by a class.

The students can complete the formative assessments (end of chapter tests) 24*7; they get immediate feedback on their mistakes. To encourage them to complete them, we count the marks towards a small percentage of the final results (10% at the present time).

The assessment structure includes a combination of on-line tests, and a paper examination.

The rational for changing the format of the course:

Students are completing the CCNA course as part of their studies in the Faculty of Engineering and Trade. These two courses are both been reviewed this year and the plan is to divide the CCNA course in four M.I.T. courses.

There are issues due to the amount of work that has to be completed by the students in the present format. The CCNA course and certification have been refreshed in the last 18 months, and its content is much more significant. Our average mark for the MIT course has gone down and the drop out rate increased since we have introduced the new material.

Last year a proposal was submitted to Cisco system in which the course would be taught in a Blended Learning format. There are strict conditions attached to be approved to teach in that format. The proposal failed as Cisco demanded that the minimum pass mark for the online examinations to be increased from the 50% suggested to 70%. There was chance we could comply with this requirement as our student pass rate would have been far too low, due to the amount of work involved in the present format of the course.

Proposed Evaluation of the course

The course content is already continually improved and updated. The course developers are already using a pool of professional educators to review the course. Cisco System is very good to take on board the feed-back of the instructors into consideration to improve future releases of the course.

I can only influence

• the schedule of delivery
• the delivery itself
• the interaction with the students

Therefore the evaluation I am planning to conduct is a formative evaluation where I could use the feedback of the present CCNA students, and of colleagues in MIT or other institutions to identify improvement and adjustment at an early stage in the redevelopment of the course. This would fit perfectly with what is described in chapter seven of the book Interactive Learning Systems Evaluation [1]

A needs assessment or maintenance evaluation would not be suitable as I am modifying the format of delivery of an existing course. A summative evaluation would usually be carried out at the completion of an eLearning program, which is not the case in this situation. [2]

Part 2

I have found two articles that relates to the type of evaluation I plan to use in my project:

Formative vs. Summative Evaluation [3] The column entitled "formative Evaluation" contrast the five definitions of formative evaluation from several authors. The five definitions all reinforced my belief that the formative evaluation is the way ahead within my project.

In the project evaluation toolkit, 8. Developing an evaluation plan [4] has a lot of sections useful to a formative evaluation, starting from "8.4 considering the constraints". I will have to consider carefully if I can ask the students currently enrolled in a course for feedback on potential changes and suggestions for improvement. Sections 8.5 Crystallising the evaluation questions,

8.6 Choosing data gathering techniques will be helpful for developing the content of questionnaire or to guide the interview of colleagues and students when evaluating the changes to the course.

Table 8.2 Methods of obtaining evidence relevant to the formative evaluation of both the learning environment and the contextual learning processes has interesting entries and link in the two first rows about interview/questionnaires and focus group.

This article is based on the Eclectic-Mixed Methods-Pragmatic Paradigm; the use of mixed methods is encouraged to collect data for several sources.

[1] Reeves, Thomas C., Hedberg, John G. (2003). Interactive learning systems evaluation. Educational Technology Publications, Inc. New Jersey. ISBN 0-87778-304-7.

[2] http://www.wikieducator.org/WikiEdProfessional_eLearning_Guidebook/Evaluating_the_impacts_of_eLearning/Evaluation_methods

[3] http://jan.ucc.nau.edu/edtech/etc667/proposal/evaluation/summative_vs._formative.htm

[4] http://www.utas.edu.au/pet/sections/developing.html#before

Week four: Evaluation paradigms and models

I am now catching up with the course; I had to focus on a heavy workload before Easter.

The article by professor Reeves is quite interesting, even if the vocabulary used is not always immediately accessible –especially for a non native speaker.

I have taught technological related subjects for more of my career, and I have met a lot of colleagues working in line with Paradigm 1: Analytic-Empirical-positivist-quantitative paradigm. It is an easy model to work with, and easy to adopt if you are from a scientific background.

Paradigm 2: Constructivist-Hermeneutic-Interpretivist-Qualitative Paradigm feels ‘extreme’ and I have the feeling would not help teaching or assessing a course if used on its own.

I have met in parts of Europe a lot of educators fitting the description of Paradigm 3: Critical Theory-Neomarxsist-Postmodern-Praxis Paradigm. They seem to be prevalent in educational establishments where underprivileged students are present. They do not work: they are on a mission!

The paradigm that I most relate to is Paradigm 4: Eclectic-Mixed Methods-Pragmatic Paradigm. I believe that this would be the most useful method to use in a fast changing environment:

• our courses seem to need changing more and more frequently

• our students come to us with changing general expectations and motivations.

In the document type of evaluation models , the ‘multiple type evaluation model’ seems to be the most logical to use in order to evaluate a technical course which is what I am planning to do.