Part 1
The networking course I am proposing to part evaluate has been developed by Cisco Systems. It comes with its own LMS. This is a course designed to prepare the student to an external professional certification examination, The Cisco Certified Network Associate.
The students are supplied with a username and a password, then are enrolled in a relevant on line class on the Cisco LMS by the lecturer. Amongst the learning environment supplied, the student can access a message board, his/her present and past classes, the on-line course, a gradebook, a discussion board. The lecturer can activate end of chapter tests which can be use for formative and summative assessment.
At the moment I am using this environment in conjunction with the Blackboard LMS for announcements/emails /message boards (not very successfully)/ storage of resources.
The course content and interfaces are upgraded regularly and there is a bug tracking system enabling to report any bugs in the courses and assessments.
The course alternate theory and lab supporting the understanding of the theory. The student have access to real equipment in the classroom and now to a downloadable simulator which is now getting very close to reality.
From Cisco point of view, to pass a course a student needs to pass the final online theory examination and the practical examination. The minimum level for a pass is left to the discretion of the “academy” (Manukau Institute of Technology). The recommended pass rate is 70%.
Changes
The course is divided in 4 parts from a Cisco point of view, but at the moment the course is mapped to two semester courses. The proposal is to develop the course to 4 semesters next year, changing its delivery mode to Blended delivery. This would involve using other tools to allow easy access to the lecturer and making lectures available online for students to access in their own time.
The present running of the course:
The course is running as a face to face course, and the students have to attend 4 hours weekly.
The students can access the lab to complete the practical work at any time, 7 days per week, when the room is not used by a class.
The students can complete the formative assessments (end of chapter tests) 24*7; they get immediate feedback on their mistakes. To encourage them to complete them, we count the marks towards a small percentage of the final results (10% at the present time).
The assessment structure includes a combination of on-line tests, and a paper examination.
The rational for changing the format of the course:
Students are completing the CCNA course as part of their studies in the Faculty of Engineering and Trade. These two courses are both been reviewed this year and the plan is to divide the CCNA course in four M.I.T. courses.
There are issues due to the amount of work that has to be completed by the students in the present format. The CCNA course and certification have been refreshed in the last 18 months, and its content is much more significant. Our average mark for the MIT course has gone down and the drop out rate increased since we have introduced the new material.
Last year a proposal was submitted to Cisco system in which the course would be taught in a Blended Learning format. There are strict conditions attached to be approved to teach in that format. The proposal failed as Cisco demanded that the minimum pass mark for the online examinations to be increased from the 50% suggested to 70%. There was chance we could comply with this requirement as our student pass rate would have been far too low, due to the amount of work involved in the present format of the course.
Proposed Evaluation of the course
The course content is already continually improved and updated. The course developers are already using a pool of professional educators to review the course. Cisco System is very good to take on board the feed-back of the instructors into consideration to improve future releases of the course.
I can only influence
- the schedule of delivery
- the delivery itself
- the interaction with the students
Therefore the evaluation I am planning to conduct is a formative evaluation where I could use the feedback of the present CCNA students, and of colleagues in MIT or other institutions to identify improvement and adjustment at an early stage in the redevelopment of the course. This would fit perfectly with what is described in chapter seven of the book Interactive Learning Systems Evaluation [1]
A needs assessment or maintenance evaluation would not be suitable as I am modifying the format of delivery of an existing course. A summative evaluation would usually be carried out at the completion of an eLearning program, which is not the case in this situation. [2]
Part 2
I have found two articles that relates to the type of evaluation I plan to use in my project:
Formative vs. Summative Evaluation [3] The column entitled "formative Evaluation" contrast the five definitions of formative evaluation from several authors. The five definitions all reinforced my belief that the formative evaluation is the way ahead within my project.
In the project evaluation toolkit, 8. Developing an evaluation plan [4] has a lot of sections useful to a formative evaluation, starting from "8.4 considering the constraints". I will have to consider carefully if I can ask the students currently enrolled in a course for feedback on potential changes and suggestions for improvement. Sections 8.5 Crystallising the evaluation questions,
8.6 Choosing data gathering techniques will be helpful for developing the content of questionnaire or to guide the interview of colleagues and students when evaluating the changes to the course.
Table 8.2 Methods of obtaining evidence relevant to the formative evaluation of both the learning environment and the contextual learning processes has interesting entries and link in the two first rows about interview/questionnaires and focus group.
This article is based on the Eclectic-Mixed Methods-Pragmatic Paradigm; the use of mixed methods is encouraged to collect data for several sources.
[1] Reeves, Thomas C., Hedberg, John G. (2003). Interactive learning systems evaluation. Educational Technology Publications, Inc. New Jersey. ISBN 0-87778-304-7.
[3] http://jan.ucc.nau.edu/edtech/etc667/proposal/evaluation/summative_vs._formative.htm
[4] http://www.utas.edu.au/pet/sections/developing.html#before
Posts
Hi Herve
ReplyDeleteI have been thinking about your evaluation, and the thing that is really striking as you read is the drop out rate since the last material upgrade. It makes me want to ask....what did they do? If you are experiencing such a marked increase in your drop-out rates then there's a suggestion that the effectiveness of the programme has been altered by the changes. Perhaps you would be able to find out what parts of the changes by using the checklist approach (evaluation cookbook, page 16), and see the example at the bottom of the page. Also, I am wondering if you could do confidence logs in regard to the content offered. If you do this pre- and post teaching it may give you a good indication of what doesn't help to meet learning needs and maybe this will also show through in the confidence log.....perhaps the results of these together will give you a way of influencing the delivery as you (hopefully) pin point what doesn't work. I wondered also if you were able to benchmark with other institutions that deliver this course and find out if they are having similar problems.....
All the best!
Sam
Hi Sam,
ReplyDeleteFrom the following up on the students who have withdrawn from the course last year, the amount of work to complete to pass is the most frequently reason cited. We can not change the content of the course as it is linked to an external certification examination.
This is also what is coming up from other institution teaching the same course.
I will take your suggestions on board.
Thank you for your feed-back
Cheers
Hervé
And isn't it lovely to fins it's not just you!!
ReplyDeleteSam
Hello Herve
ReplyDeleteyes I agree with you and Sam it is worrying when an increase in content leads to problems with retention. You might find this article useful: Designing effective communication tools for online computer science tools - the authors mention the difficulties in teaching programming online and suggest four solutions. http://cims.clayton.edu/jchastine/papers/Designing%20Effective.pdf
This could help with your ideas around evaluation of the development changes you are looking at and could also be a good theoretical underpinning for your evaluation project.
The articles you have chosen for the weekly task, although relevant to helping you design your evaluation, do not provide current research on which to support the direction of your evaluation project.
It would be good if you did a search on Google Scholar to find something else which is relevant.
Another article called: Powering E-Learning In the New Millennium: An Overview
of E-Learning and Enabling Technology from Zhang, D. & Nunumaker, J. Information Systems Frontiers 5:2, 207–218, 2003 - I will email to you. It mentions effectiveness evaluation but not formative.