It feels good

I did manage to submit the report last Friday. It felt good to press the send button!
I am still keeping an eye on your blogs.


Cheers!

Summary of the interviews (Part deux)

I have finished the summary of the two interviews I have conducted for the study.
It is located on http://docs.google.com/View?id=dcsr6jkb_91fb3wvqfc

Survey results (part deux)

I have rewritten my analysis of the survey following the useful comments from Bronwyn. I am still open to suggestions, so feel free to suggest!
The document (published properly this time) can be found here.
I will finalise the summary of the two interviews soon and will keep you posted.

Survey Results and Interview Summary

I have shared with you the analysis of the survey on Google documents .
The summary of the interview can be found here.

Analysis of the results from the survey:

A quick entry to let you know that I am still alive... I have almost finished to run my practical exam, and I seem to have a bit more time to work at the course again.

Following Bronwyn suggestion, I have used Survey Monkey to create my survey. It is a great tool to create a survey easily. However, I have not in the past used this information gathering method and the learning curve was steep.

I was amazed at the amount of effort you have to deploy to get just a few people to answer a short survey.

There are several possible formats in which I could download the survey. The full Excel or CSV files were not very user friendly to analyse the data. All the questions were sorted per user horizontally. This did not make it for easy interpretation of the results. My primary source of information was therefore the web interface results page. I am sometimes using the Excel file to link an open answer with the answer to a previous question.

The result should hopefully be available before the week-end.

WEEK NINE

I have (at last) published my evaluation plan on:
http://docs.google.com/Doc?id=dcsr6jkb_85drjmfvcq

WEEK SEVEN AND EIGHT: DRAFT EVALUATION PLAN

Introduction:

This plan is going to describe an evaluation of the future implementation of the CCNA Exploration courses in the blended format at the Manukau Institute of Technology. The study will be conducted by Hervé Carpentier.

Background:

The CCNA course is part of the Cisco Networking Academy Program and is has been preparing the students for the Cisco Certified Network Associate certification examination worldwide for more than ten years. The present implementation of the course at M.I.T. is face to face, with all course materials accessible through a web browser from anywhere with an internet connection. At the present time, the four parts of this course are delivered over two semesters in a face to face environment.

The four part of this course are :

1. Networking Basics

2. Routers and Routing Basics

3. Switching Basics and Intermediate Routing

4. WAN Technologies

The courses are level 5 and 6 courses

The amount of material to study has increased greatly in recent years, and it has been decided at national level to deliver the full course over four semester within the Bachelor of Engineering Technology program. The practical work involved in the course is supporting the theory and vice versa. This course has been described as using

The students can access the laboratories to complete the practical work at any time, 7 days per week, when the room is not used by a class.

The students can complete the formative assessments (end of chapter tests) 24*7; they get immediate feedback on their mistakes. To encourage them to complete them, we are the present time counting the marks towards a small percentage of the final results (10% at the present time).

The assessment structure includes a combination of on-line tests, and a paper examination.

Purposes:

The purpose of this evaluation is to

1. Assess the effectiveness of the present usage of eLearning tools.

2. Consider the use of further tools and activities to enhance and facilitate the student learning.

3. Identify if Blended learning would improve the delivery of the course to the students

I will be using the following two guidelines taken from http://elg.massey.ac.nz:

a. TT6 Are online activities available that will enable students to assess their learning?

There already are several assessment tools I am using for the present format of the course. It would be interesting to see if I could develop or use further tools to enhance student learning. The redevelopment of the course means that there will be more time for the students to work to study the course and would therefore give use the opportunity to introduce more online activities that would enable her to enhance their learning.

b. The second guideline is inspired from TD1 "Is the use of e-learning appropriate to the intended learning outcomes?"
Is the use of blended learning more appropriate to achieve the intended learning outcome?
It would be interesting to find out if using a blended learning format would a progress on the present delivery of the course.It might be that my present use of eLearning tool in this context is not the best match for the learning outcomes.

Audiences:

The primary and secondary audiences of this evaluation are staff and students involved the teaching of the course. This will probably extended to the Cisco Account Manager looking after our academy since he or she will be involved in the approval for changing the delivery of the course to blended learning.

A third audience will be the staff and students of the Evaluation of eLearning for Best Practice course, to which the evaluation will be submitted for assessment.

Decisions:

The findings of this study will influence the decision on how to run the CCna course from the firsty semester of 2010, and the set of eLearning tools used to run it.

Questions

The questions I am attempting to answer with this study are:

-Are the on-line activities used in the delivery of the course in the present format sufficient for students to assess their progress?

-Are their any other online tools I could use to enhance the students learning and experience?

-Would moving from face to face delivery to blended delivery improve learning experience of the students and therefore improve their achievement?

The paradigm that I will be using is the Eclectic-Mixed Methods-Pragmatic Paradigm. The model I will be implementing is is the multiple type evaluation model.

I am planning to use the feedback of CCNA students following the present format of the course, as well as the ideas and suggestions from colleagues in MIT or other institutions teaching the course to identify improvement and adjustment at an early stage in the redevelopment of the course.

Methods:

I am intending on on using focus groups and questionnaires to get the information from students and staff; there may be issues with the publication of the results of the questionnaires. I will need to investigate if the Ethics Committee at M.I.T. needs to be involved.

Sample:

The students involved are the present students involved on the CCNA course; The colleagues I will work with are:

the other lecturer teaching the CCNA course

colleagues from other Cisco Academies (two or three)

Instrumentation:

Questionnaires delivered to the students through the Blackboard-eMIT interface.

Discussions through focus groups will be used to further develop some of the ideas/information gathered via the questionnaires.

I will discuss the proposal with former colleagues involved in the teaching of the same course in an academy located in the U.K. This will probably be done through Skype.

Limitations:

This study will look at the specific implemtation of the CCNA course as taught at MIT.

Depending on how many students are willing to take parts in the questionnaire and discussions, the sample size might be small.

This project is only looking at some aspect of the course.

Logistics:

I will conducts all aspect of this evaluation; I might have to get support from the eMIT support desk to set the questionnaires up for the students.

Time Line:

This evaluation is part of a bigger project whose deadlines are as follow

1. May to August 2009 : collecting the information from students and colleagues and take it into consideration to develop the new format

2. August 2009 : getting approval from the Technical Expert Group at MIT and Management for the Blended format, then write the proposal to be approved to deliver in the Blended format by Cisco Systems.

3. September 2009: sending the proposal to Cisco System.

4. November 2009: reply for the approval; if positive, developing the new format

5. February 2009: delivering the course as blended delivery.

Budget:

My spare time will be used, as I am working full time on the present delivery of various courses in the School of Electrical Engineering and trade.
I am planning to use electronic documents as much as possible, cutting the cost of printing and photocopying.

Weeks five and six: Evaluation methods

Part 1

The networking course I am proposing to part evaluate has been developed by Cisco Systems. It comes with its own LMS. This is a course designed to prepare the student to an external professional certification examination, The Cisco Certified Network Associate.

The students are supplied with a username and a password, then are enrolled in a relevant on line class on the Cisco LMS by the lecturer. Amongst the learning environment supplied, the student can access a message board, his/her present and past classes, the on-line course, a gradebook, a discussion board. The lecturer can activate end of chapter tests which can be use for formative and summative assessment.

At the moment I am using this environment in conjunction with the Blackboard LMS for announcements/emails /message boards (not very successfully)/ storage of resources.

The course content and interfaces are upgraded regularly and there is a bug tracking system enabling to report any bugs in the courses and assessments.

The course alternate theory and lab supporting the understanding of the theory. The student have access to real equipment in the classroom and now to a downloadable simulator which is now getting very close to reality.

From Cisco point of view, to pass a course a student needs to pass the final online theory examination and the practical examination. The minimum level for a pass is left to the discretion of the “academy” (Manukau Institute of Technology). The recommended pass rate is 70%.

Changes

The course is divided in 4 parts from a Cisco point of view, but at the moment the course is mapped to two semester courses. The proposal is to develop the course to 4 semesters next year, changing its delivery mode to Blended delivery. This would involve using other tools to allow easy access to the lecturer and making lectures available online for students to access in their own time.

The present running of the course:

The course is running as a face to face course, and the students have to attend 4 hours weekly.

The students can access the lab to complete the practical work at any time, 7 days per week, when the room is not used by a class.

The students can complete the formative assessments (end of chapter tests) 24*7; they get immediate feedback on their mistakes. To encourage them to complete them, we count the marks towards a small percentage of the final results (10% at the present time).

The assessment structure includes a combination of on-line tests, and a paper examination.

The rational for changing the format of the course:

Students are completing the CCNA course as part of their studies in the Faculty of Engineering and Trade. These two courses are both been reviewed this year and the plan is to divide the CCNA course in four M.I.T. courses.

There are issues due to the amount of work that has to be completed by the students in the present format. The CCNA course and certification have been refreshed in the last 18 months, and its content is much more significant. Our average mark for the MIT course has gone down and the drop out rate increased since we have introduced the new material.

Last year a proposal was submitted to Cisco system in which the course would be taught in a Blended Learning format. There are strict conditions attached to be approved to teach in that format. The proposal failed as Cisco demanded that the minimum pass mark for the online examinations to be increased from the 50% suggested to 70%. There was chance we could comply with this requirement as our student pass rate would have been far too low, due to the amount of work involved in the present format of the course.

Proposed Evaluation of the course

The course content is already continually improved and updated. The course developers are already using a pool of professional educators to review the course. Cisco System is very good to take on board the feed-back of the instructors into consideration to improve future releases of the course.

I can only influence

• the schedule of delivery
• the delivery itself
• the interaction with the students

Therefore the evaluation I am planning to conduct is a formative evaluation where I could use the feedback of the present CCNA students, and of colleagues in MIT or other institutions to identify improvement and adjustment at an early stage in the redevelopment of the course. This would fit perfectly with what is described in chapter seven of the book Interactive Learning Systems Evaluation [1]

A needs assessment or maintenance evaluation would not be suitable as I am modifying the format of delivery of an existing course. A summative evaluation would usually be carried out at the completion of an eLearning program, which is not the case in this situation. [2]

Part 2

I have found two articles that relates to the type of evaluation I plan to use in my project:

Formative vs. Summative Evaluation [3] The column entitled "formative Evaluation" contrast the five definitions of formative evaluation from several authors. The five definitions all reinforced my belief that the formative evaluation is the way ahead within my project.

In the project evaluation toolkit, 8. Developing an evaluation plan [4] has a lot of sections useful to a formative evaluation, starting from "8.4 considering the constraints". I will have to consider carefully if I can ask the students currently enrolled in a course for feedback on potential changes and suggestions for improvement. Sections 8.5 Crystallising the evaluation questions,

8.6 Choosing data gathering techniques will be helpful for developing the content of questionnaire or to guide the interview of colleagues and students when evaluating the changes to the course.

Table 8.2 Methods of obtaining evidence relevant to the formative evaluation of both the learning environment and the contextual learning processes has interesting entries and link in the two first rows about interview/questionnaires and focus group.

This article is based on the Eclectic-Mixed Methods-Pragmatic Paradigm; the use of mixed methods is encouraged to collect data for several sources.

[1] Reeves, Thomas C., Hedberg, John G. (2003). Interactive learning systems evaluation. Educational Technology Publications, Inc. New Jersey. ISBN 0-87778-304-7.

[2] http://www.wikieducator.org/WikiEdProfessional_eLearning_Guidebook/Evaluating_the_impacts_of_eLearning/Evaluation_methods

[3] http://jan.ucc.nau.edu/edtech/etc667/proposal/evaluation/summative_vs._formative.htm

[4] http://www.utas.edu.au/pet/sections/developing.html#before

Week four: Evaluation paradigms and models

I am now catching up with the course; I had to focus on a heavy workload before Easter.

The article by professor Reeves is quite interesting, even if the vocabulary used is not always immediately accessible –especially for a non native speaker.

I have taught technological related subjects for more of my career, and I have met a lot of colleagues working in line with Paradigm 1: Analytic-Empirical-positivist-quantitative paradigm. It is an easy model to work with, and easy to adopt if you are from a scientific background.

Paradigm 2: Constructivist-Hermeneutic-Interpretivist-Qualitative Paradigm feels ‘extreme’ and I have the feeling would not help teaching or assessing a course if used on its own.

I have met in parts of Europe a lot of educators fitting the description of Paradigm 3: Critical Theory-Neomarxsist-Postmodern-Praxis Paradigm. They seem to be prevalent in educational establishments where underprivileged students are present. They do not work: they are on a mission!

The paradigm that I most relate to is Paradigm 4: Eclectic-Mixed Methods-Pragmatic Paradigm. I believe that this would be the most useful method to use in a fast changing environment:

• our courses seem to need changing more and more frequently

• our students come to us with changing general expectations and motivations.

In the document type of evaluation models , the ‘multiple type evaluation model’ seems to be the most logical to use in order to evaluate a technical course which is what I am planning to do.

Week 3: eLearning guidelines for quality

The first task this week was to get familiar with the eLearning Guidelines for New Zealand: http://elg.massey.ac.nz/. Excellent content! It is a great idea to use a cube for identifying the different elements in a 3 entry table. Mind you, the interface could be improved by allowing clicking directly on the “cubelets”. Enough of evaluating of eLearning tools, let’s go to the core of this week’s work.

• In regards with the course evaluation suggested in the week 1 posting, I have chosen two eLearning guidelines relevant to my area of practise.

1. TT6 Are online activities available that will enable students to assess their learning?
2. TD1 Is the use of e-learning appropriate to the intended learning outcomes?
There already are several assessment tools I am using for the present format of the course. It would be interesting to see if I could develop or use further tools to enhance student learning. It might be that my use of eLearning tool in this context is not the best match for the learning outcomes.

• I have identified two issues I think are important related to the quality of eLearning.

1. Issue 1: The confusion that can arise from using too many different tools at the same time at the beginning of a course
   
2. Issue 2: The lack of interaction between students and staff.

Issue 1 can be linked to guidelines one and two. The fact that I am using two different LMS at the same time for the same course made me aware of the potential confusion for the students, at least at the beginning of each semester.
Issue 2 can be resolved by carefully selecting the tool used with the help of guideline two. I believe that frequent feedbacks between students and staff are improving the quality of courses.

It looks like I am able to stay on top of the schedule, if only just. I was not expecting that.

Week 2: Quality and evaluation

The presentation on Plick was very interesting and has given me more ideas for the subsequent parts of the course.
The only problem I have encountered with what is for me a new tool, is that it does not seem to be running on Firefox. Never mind; nothing is perfect.
So, the three questions to answer for this week are:
1. Why is evaluation important to you and how do you define it?
2. What sort of evaluations mentioned on the presentation are familiar to you already and why?
3. Why is quality important in eLearning?
Evaluation is important as a tool giving me feedback on how to improve my work. One aspect to this is the evaluation of the output from the students which in turn helps me to fine-tune my teaching.
But more important, it is needed at all stages of the teaching process, from creating the course to putting resources and material available to the students.
It is difficult to assess what sort of evaluations mentioned in the presentation are familiar to me; nothing came as a surprise. I think that any teacher or lecturer would use the different types mentioned to assess various aspects of a course or student progress.
Quality is much more important eLearning than in traditional teaching as you do not have the same interaction between the participants; I know when presenting a topic to a traditional class that someone is stuck on a concept by observing the class. Further clues appear when they are trying to apply the theory in the subsequent labs.
There are quality issues specific to eLearning, and they cannot be ignored in a context where there is a lot of pressure on staff in the New Zealand tertiary sector to deliver most courses using the blended or remote teaching format. It looks like it is too often considered a cost saving exercise, and the temptation is to deliver the eLearning course without paying attention to its specific challenges.
These are, but not limited to:
-the appropriate use of the specific tools.
-how to engage the students in a remote teaching environment
-how to provide support for these students
-how to conduct summative assessments and provide adequate formative assessments for these students.

Week 1- 23 February to 1 March

My name is Hervé Carpentier and I am a Lecturer in Computer Networking at the Manukau Institute of Technology. I am teaching students from Certificate to BET level, and most of these courses are making heavy use of online tools.
I am very busy at work, and am also reviewing on-line material for a course provider at the moment. (It must be nice to have a life !)
I have managed to create the blog at the beginning of this week, but never found the time to enter anything up to today. I am however keeping an eye on what is happening on the various blogs linked to the course.
I have taught all kinds of subjects, from secondary schools to university, in several countries. I have noticed a healthy appetite form everywhere for courses using eLearning technologies, even when they were not as accessible as today.

My interest in evaluation
I find evaluation an essential tool for getting feedback on students, course design and practises. Being a bit of a perfectionist, I am always on a look out for improvement in my teaching practises and material. It is all about improving the student experience of courses, helping retention on the way.
I hope that the Evaluation of eLearning for best practice course will help me to become more efficient in assessing courses and students using eLearning tools.

Evaluation project
Correct me if I am wrong, but I seem to understand from reading the resources available that we will have to evaluate a course which makes use of eLearning technologies.
I have developed a blended learning computer networking course last year. It was based on an existing course using eLearning. In the proposed new format, I was introducing new tools which would have enabled the students not to have to attend for each session. My evening class students are sometime travelling from far away after a long day at work.
The course did not go through at the end. Five days before the beginning of the semester, the provider of the online course came out with requirements which we could not fulfil at such a short notice.
I am thinking of analysing this proposal, or a new proposal which would enable me to get the course approved for next year.

What do you think?