Assessment and Evaluation of Students and Online Learning Programs
“Curiosity is the very basis of education and if you tell me that curiosity killed the cat, I say only the cat died nobly.”
– Arnold EdinboroughChapter contents
13.2 Online Quizzes and Assignments
13.3 Dealing with Dishonest Conduct
13.4 Proctoring Software
13.5 Assessment of Quality of Online Programs
Although the chapter title is perhaps ambiguous, we will focus on good practice in assessment of students engaged in online programs as well as external reviews of the quality of a particular program.
Figure 13.1: Various methods of assesment
Initially, we will examine online quizzes and assignments. This will be followed by reviewing the approaches to dealing with dishonest conduct, which is increasingly prevalent in both online and residential colleges due to the easy accessibility of resources on the internet. A way of partially addressing the issue of dishonest conduct using a novel approach of remote supervision of tests and examinations using proctoring software will then be examined. Finally, the much-debated issue of quality of online programs will be examined from a number of perspectives.
13.2 Online quizzes and assignments
Formative assessment comprises low stakes formal and informal testing throughout the learning process, requiring the instructor to modify her teaching and learning activities to ensure the student reaches and exceeds the required learning outcomes.
Summative assessment (of learning) on the other hand, summarises the development of the learners for a particular block of learning and hopefully confirms that they mastered the learning outcome. It should be noted that, no matter what testing scenario is conducted, formative assessments would appear to provide greater gains in learning than summative assessments.1 Students who attempt the weekly quizzes and review streaming videos generally did better in the final examinations (comprising a combination of formative or summative forms).2Student assessments
The evaluation of students should be done both formatively and summatively and based around:
• Class participation.
• Student contributions during the delivery.
• Homework assignments.
• Examinations and any online quizzes.
Naturally, the results should be normalized against previous years’ students’ scores so that every year has a similar range of marks. The students should be required to comment during a course on the instructor, quality of the course and delivery mechanisms, allowing for continual and immediate optimization.
Sadly enough, students are really only motivated to do these if they contribute to the final grade, even if it only contributes a small amount. The top students tended to work steadily (e.g. as measured by activity on the LMS), whereas the weaker students tended to work vigorously just before an assessment.Assessments of students done 100% online
This research was undertaken at the Universidad Nacional de Colombia.3
Two issues were considered here: whether it is possible to adequately assess online courses that are completely online and how we can verify who is doing the actual assessment. This was considered in the context of an online Physics course. For a blended course on Physics, the final weighting of an assessment mark was done on the basis of:
• 10% for a self-assessment test before and after a class session.
• 10% for attendance at lectures (typically a minimum of 80% is required).
• 20% for examinations providing 24-to 72-hour window of attempts with multiple attempts allowed and random generation of questions from a large bank.
• 10% for forums based on attendance, quality and relevance of contribution.
• 10% for workshops that involved group work solving a lab problem where measurements are done within a simulation (using Java applets).
• 15% for quizzes done on the basis of the workshops.
• 25% for the final exam which has a maximum of 36 questions with an average maximum permitted time of 2.5 minutes for each open ended question (done in time window of 2 to 3 days).
The key to minimizing the risks of assessment and identification was the creation of large online banks of questions. The use of the online formats of presentation makes for far less congested traditional residential classes.Advantages of online quizzes
The key advantages of online quizzes are:4
• Quick and easy access no matter where the learner is.
• Instantaneous feedback.
• Reuse of quizzes possible.
• Multiple attempts are allowed.
• Automatic corrections are given .
Students were generally positive about quizzes as they assisted with revision but believed they needed to be directly related to current lecture material. Students stated that they were time consuming, and there appeared to be a significant mismatch between instructor and student’s apprehensions about actual duration. Quizzes were considered useful to test basic concepts (e.g. calculate resistance of three resistors in parallel) but the jury was out on testing more sophisticated knowledge.
Online homework modules for an electrical engineering and systems course at Kansas State University ranged from complex numbers and signal graphing to Fast Fourier Transforms.5 These were used for nine semesters with mixed results (weak correlation between online homework scores and final exam results for the 359 students surveyed). Overall, students appreciated the immediacy of answers but the syntax inflexibility and requirement for all answers to be correct to receive full credit were issues they were unhappy with. Instructors found the time saved, consistency in grading and statistics on student study habits useful.Web-based assessment tool
A web-based assessment tool for engineering programming students was instituted at the University of Michigan. Online assessments were taken between the four class examinations conducted every month.6 Each student was given three days in which to undertake the assessment. Collaboration between students was minimized by indicating the actual score of the assessment did not contribute to the final grade (but participation was measured). It was shown that students who performed poorly in the assessment, would have weak results in the subsequent examination. This identification allowed the instructor to assist the student in succeeding in the examinations and thus in achieving a higher grade.Online quizzes improve participation
At the price of a moderate increase in instructor workload, the use of online quizzes (with multiple choice questions) increases student contact with the lecture material.7 This has also allowed removal from lectures of simple rote materials and getting the student to review these elements by going to specific websites and then undertaking confirmation of understanding by doing a quiz; situated throughout the course. This also allowed for formative assessment rather than relying on one final test at the end. The strategy suggested at an introductory biology course at the University of Wisconsin-La Crosse was to link assignments with quizzes. The assignments were not marked by the instructor and the quizzes were scored and recorded on the Desire2Learn LMS. Each quiz comprised five multiple-choice questions with five minutes to complete and two attempts allowed with the score immediately available at completion (but no answers). Limitations in the quiz approach can be minimized by cycling of questions from a bank and providing questions that require analysis of data rather than only in regurgitated memorized content.
Online quizzes should not be merely considered as a method of assessment but also as a vehicle for increased participation in the class and the learning process, hence a substantial portion of 10% towards the final grade should be allocated (and the student should be allowed to retake the quiz).8
At the School of Technology at Michigan Technological University a variation on assessing assignments for the end of course grade was for students to thoroughly work through their weekly homework assignments, submit them and then engage in an online quiz, which was assessed (rather than assessing only an assignment).9 A mid-tern and end of term examination were also used for overall assessment.Online assessments and poor integrity
Considerable unease was expressed by auditors of online assessments (referred to as e-assessments) of vocational courses in Australia.10 62% of instructors using online learning for delivery of units of competency were using online assessments mainly for diagnostics (knowledge and skills before the course commenced) to formative, where feedback is provided to the learner as the course progresses. The main methods of assessment included the use of quizzes, blogs and wikis. Specific concerns were weak validity with e-assessments not validly assessing the skills being tested and weak linkage between required learning outcomes and assessment. The online quizzes also had limited reliability and were often done simply to reduce costs. The use of e-portfolios had grown significantly but there were concerns about the authenticity of the evidence and validity of the assessment tasks. While there are certainly major problems to be addressed, the one workable suggestion was to build up a strong relationship between the instructor and student over many elements of assessment and thus to authenticate the work this way.
One suggestion to improve the integrity of examinations is to require the student to physically attend the campus for a week once during the entire degree program to undertake their final presentation and to write the requisite examinations.11Online assessment coupled with instructor assistance is the ticket
Research revealed that an online assessment system (with quick feedback) against that of only using an instructor, showed no differences in results.12 In addition, providing online assessment and also in providing (20%, in this example) class time to students working on problems demonstrated a significant improvement in achievement. Hence, use of online sessions in itself is not adequate, but supplemented with instructor-led sessions can lead to a significant improvement in results.Preparation for examinations
Old exam papers are widely used by students for preparation for an exam (in addition to lecture notes/slides, tutorials and personal notes).13 A novel scheme was introduced at the University of Portsmouth in the UK to assist students in this respect using what was called Examopedia. This used a wiki (initially Twiki and then latterly Google Sites) to host exam paper questions. Each question was placed on an editable wiki page (with its associated text, images and data) in the top row. To the right of the question were comments from the instructor giving some feedback and guidance. There were then an unlimited number of rows below the exam question (and the instructor’s feedback) with scope for a student to place in their comments and for the instructor to provide a comment to each student’s submission.
Google Talk supplemented this allowing synchronous chatting for one-to-one tutoring where a quick response was required. This was extensively used just before the examination.
This facility was especially used during the 15-20 days before the examination with most students not providing much commentary but using it. This was a source of concern from some students (presumably the active contributors) who felt the passive users should have contributed more.Computer assisted assessments
Computer Assisted Assessments have a number of advantages including:14
• Quicker feedback than the drawn out ones prepared on paper which depend on the markers returning the scripts in a timely fashion.
• Questions can be personalized. This reduces the possibility of plagiarism or cheating as each question can be subtly different.
• Students can submit their assignments anywhere, and don’t have to dart into the university to drop off their assignments at the midnight deadline.
• Comprehensive tracking of assignment submission by student is provided so there is minimal chance of loss of materials.
• This approach is good for the environment as it is a paperless process.
Research has suggested that students perform better in online assessments than in the traditional manner.
Apart from the built in assessment facilities built into the LMS (e.g. Moodle and Blackboard Vista), there are a large variety of computer assisted assessment systems on the market. These include Questionmark Perception, Quiz Factory 2, CyberExam, Test Pilot, Hot Potatoes, Maple TA and WebMCQ.
It is important in setting the assessments up that different numbers are assigned to different students to minimize cheating. Another subtle issue is to give partial marks when the student gets the question partially correct. This can be challenging to set up with some of these assessment packages.Pre-session quizzes
Marks makes an excellent suggestion for pre-session quizzes, based on solid pedagogical research and something that makes excellent intuitive sense.15 Based on our experience, we can confidently say that most of our study in a university course occurs outside the classroom and laboratory. However the focus with a considerable amount of commentary relates to the classroom environment, thus ignoring an area where dramatic improvements are possible. One area where students are expected to put in significant effort is in preparatory reading before a classroom (or indeed, online live web conference session). Inevitably, many tend to do this superficially. This problem is accentuated by the student’s poor study skills especially in reading (and assimilating) an engineering tome that is written in a highly organized fashion as compared to one of those delightful books that are required reading for an English literature course, for example.
Based on Bloom’s hierarchical taxonomy, interaction with instructors and one’s peers should be restricted to working through the higher level cognitive activities such as application, analysis, synthesis and evaluation and the lower levels such as knowledge and comprehension should mainly be done before the class session. Hence, the suggestion is to force the students to do the reading and then to do a self-test at the conclusion of the reading. The test or quiz (a less confrontational word) is constructed out of simple multiple-choice questions interspersed with open-ended questions, as well. The results of these tests are then sent to the instructor so that she can form a profile of the students before the next lecture commences. In the research, many students freely admitted that they wouldn’t have bothered to read the pre-course materials if they hadn’t been forced to do the quiz. Obviously, the instructor should randomly ask the students appropriate questions during the lecture, as this is proven to drive them to do the pre-course reading.The challenges with online examinations
The process of constructing exams for online students is a challenge to ensure that educational quality is not sacrificed and security is not compromised. Care must also be taken to deal with students who fear poor performance in the online exam environment. There is considerable support in the literature for the open book exam approach in engineering and the sciences. The “honest, open book, open mind” approach is gaining some traction. Most online exams comprise multiple choice, true/false or short answer type questions with essay type questions not used extensively. The authors point out the following issues have to be addressed in the online examination environment:16, 17
Figure 13.2: Open book examinations can be effective
Examination security. Where open book exams are used (with no resident invigilator or remote monitoring), the students must be tested on the concepts rather than regurgitating the materials where it is easy to copy from their textbooks. Random questions and problems can be generated from a test bank within the LMS.Interactivity. Students apparently prefer live tests rather than online ones where academic staff are available via phone, messaging and emails. Equity. In constructing banks of questions that are randomly selected, it is vital to keep them to the same level of difficulty to ensure equity in the exam process. The suggestion is that questions contained in the bank should be grouped by difficulty when assigning them. Hands-on labs. Many academics in the engineering and science world would like to see a physical demonstration by the student of handling equipment or dealing with a concept. One approach is to use screen-capturing software that records the movement of the cursor across the screen. Teamwork assessment. Methods for assessment of working in teams should be reviewed and actively used as this is a key activity for engineers and scientists.
One of the challenging parts of engineering and technology exams is the extensive use of graphical-based software. The suggestion is made that many software packages (such as AutoCAD) allow for collaborative work with teams and the contributions from each student member can be easily measured by reviewing the log of each student's activity. The other approach for collaborative work is to provide a discussion (virtual) space that is only accessible by students and the instructor. It is relatively easy to review the contributions from all members of the team in this way.
Preliminary results indicate that students who do their exams synchronously (or live) tend to do better than those undertaken asynchronously.
Ethics. The Accreditation Board for Engineering and Technology (ABET) has placed significant emphasis on the ethical responsibilities of engineers and this needs to be demonstrated by the students especially in an online environment. Thus students must read an ethical guideline and agree to it before they undertake any (online) examinations.
”Open book, open web” exams superior to closed book
Research conducted by U21Global, an online university servicing approximately 4,000 students in around 60 countries indicated that “open book, open web” exams are a superior form of assessment compared to the traditional closed book examination.18 The key to the examination process is considered to be authentic assessment where the student acts in the role of decision-maker in working on a real problem supported with convincing links to websites and video. Students are expected to apply the knowledge they have gained in the course in solving the problem. The possibility of dishonest conduct is minimized by having the submission of the student's work done electronically (so that it can be checked by plagiarism software) and ensuring that the submission contains extensive references to course materials thus making it extensively contextualized and minimizing support from a so-called "cheat site". Students were unwavering in their assessment that overall this approach is considerably superior to the closed book approach and enhanced their learning approach. There is certainly the risk that cheating could occur via chat rooms, copying directly from the internet and discussing solutions through email. Yet students felt that the opportunities were still minimized with this approach. This is based on information solely obtained from students, so one could argue this may not be sufficiently objective.Learning management systems and engineering assessment tools
Although LMSs are widely employed, they do have limitations for engineering education assessments.19 The main reasons advanced for this are the difficulty in manipulating equations and the multi-stage nature of most engineering problems. A solution using a conventional spreadsheet approach (e-Task) was designed. The approach here is to distribute a pre-prepared and password protected copy of e-Task spreadsheet based on Microsoft’s Excel program. Each new dataset was generated randomly based on an instructor’s hidden spreadsheet. In practice mode, instantaneous feedback comments (and marks) are displayed as the student works through the solution. Correct numerical answers could also be provided (but marks are deducted for provision of any solution). For reasons of security, the package was provided in an internet-hosted format based around open LibreOffice. One tricky problem was peak usage inevitably immediately before an assignment was due and this was handled with load balancing allowing more virtual machines to be operated in the 10 hours before the deadline with a class of 160 students accommodated. As a result of using this software, students felt more deeply engaged with the subject and plagiarism was minimized (apart from students helping each other).Formative quiz assessments
The University of Southern Queensl and used a concrete structures course (within the Bachelor of Engineering program) to apply 14 formative quizzes to help with learning the fundamental concepts before undertaking the assignment and examination at the conclusion of the course.20 Each quiz (using the Moodle LMS) contained about 15 multiple choice questions with immediate feedback to students. The quiz had no time limit. It was shown that students who undertook the voluntary quizzes performed better on the summative assessments than those who did not do the quiz.
Assessments support student learning when they are structured so that:21
• They encourage “time on task”.
• They maintain an even pace of workload throughout the semester.
• They drive the student into deep learning (as opposed to surface learning).
• They set clear and high expectations.
Feedback for assignments should:
• Be clear, timely and appropriate.
• Provide clear linkages between content, question, learning outcomes and assessment criteria.
• Focus on learning rather than “working the system” in terms of improved grades.
• Allow students to work towards an improved future submission.In studying retrieval is actually better
Research has shown that in a comparison of traditional studying (with repetitive review of content which is probably the most common), against that of creating a visual concept map of the material or retrieval practice of the content, the optimum approach is the latterly mentioned technique.22 However, students often believe that the most powerful form of learning is the use of concept maps as this is active learning which uses nodes to represent concepts that are linked together based on the relationships between them.
As an example, retrieval practice could involve studying two techniques to tune a process control loop and then practising recalling as much as they could on these two techniques. The students would then review their study materials and attempt to recall the content perfectly the second time. The idea is that the students would be retrieving concepts from their long-term memory, hence the reference to retrieval practice.
The challenge for instructional designers is to create opportunities for retrieval. This is undoubtedly challenging, as it is relatively easy to set up multiple choice quizzes, but it would be difficult to measure the quality of retrieval of content by a student.
13.3 Dealing with dishonest conduct
plagiarism in online environment
Although there is some conjecture about whether the online learning environment has more or similar levels of plagiarism to that of the classroom environment, there is no doubt that it is alive and well in the online learning world.23 Plagiarism is often a violation of copyright, but in learning we are more interested in whether the student is being academically honest. Plagiarism is taking someone else’s work and presenting it as one’s own. All materials or ideas taken from another source should be rigorously acknowledged. There is some debate about what is so-called, “common knowledge”, which may not need to be acknowledged.
Typical areas where the student has to acknowledge the work of others are opinions, formulas, graphs, pictures, quotations, statistics, facts and indeed even collections of materials gathered by others. Where another person paraphrases materials that are copyright of others, this should always be acknowledged.
What is somewhat of a puzzle is that when the student does indeed acknowledge the work of others, he or she gains considerably more credibility in their own work. It also strengthens the credibility of the student’s work considerably.
So why do students plagiarize? A kinder view is that students often do not know the finer intricacies of copying other work and thus do it unwittingly. This can certainly be true, but often the student falls behind due to other commitments (family or work-related), and then has to rapidly produce something useful and thus copies directly without doing any synthesis of their own. As a result of attending high school, where simple attendance is sufficient to be bumped up to the next grade, students may have similar expectations at a tertiary level and expect to pass a subject on the same basis, and may be surprised by not being passed and thus resort to other means to get through a subject.
Some suggestions to minimize plagiarism and increase academic integrity in the online environment are:
• In addition to making everyone very familiar with the relevant academic policies, the instructor should set clear expectations of what is required of the student.
• The instructor should be a good role model and actively cite references and sources of materials.
• Provide students with access to tools such as Turnitin to check for examples of plagiarism.
• Ensure all online conduct is marked by respect for other people’s materials and zero tolerance of plagiarism.Cheating is a way of life
Research showed that when left unsupervised (or unproctored) students will gravitate to cheating.24 According to one report, 50% of those who admitted cheating felt it was acceptable practice.25 This survey was conducted mainly in a traditional classroom environment and online the picture may represent a higher risk (although some research indicates it is actually lower in an online environment). However, it is critical that the online instructor protects the integrity of her course with a clear-cut strategy that demonstrates to the world that there is minimal or no cheating.
At the outset ensure that students know what academic dishonesty is, why it harms society and why it will not be tolerated under any circumstances. They should also be made aware that anti-plagiarism software is used and that it works effectively. At the beginning of the course, all students should submit clear copies of their photo ID with them holding the photo ID to confirm that it is actually them attending the course.
Each student should have a unique password when submitting materials to the LMS.
When conducting examinations using the LMS; a randomized set of questions should be provided for each student from a bank of questions. Alternatively, the simple strategy of providing the same questions but in a random order can throw off the determined cheater. Tight time limits in undertaking an examination also make it difficult to cheat. Examinations that require substantial critical thought should be given a maximum of 24 hours. It would be difficult in this time to get someone else to undertake the task effectively.
Proctored examinations and tests are useful provided the proctor’s bona fides are carefully checked or the examinations are conducted at designated test centers. When using proctors, the safest approach is to take the examination online by logging into the LMS portal in the proctor’s presence. The proctor would have been initially given the password to log in.
When students undertake lab sessions, get them to provide photos of themselves next to their experiment or indeed videos of themselves. This is easy to accomplish with modern phones and cameras. Watch out for similarities in lab submissions and non-native speakers suddenly acquiring incredible proficiency in English.
The LMS should be used to benchmark and profile student activity such as first and last accesses/duration and regularity of accesses/number of email messages sent and read/discussion board messages posted and read.
Cheating has been an ongoing problem with online courses as it is difficult to identify who the individual is who contributed to an assignment.26 However, this can be controlled to some extent by requiring students to attend an approved testing center and more frequent, informal assessments and longer term projects.Online learner authentication
The growth of online education has made assessment of online students’ identity increasingly important.27 There has been added impetus by virtue of the United USA’s Higher Education Opportunity Act (2008) which requires institutions to have processes to establish that the same person who registers for a course, is the one who participates in, completes it and receives the academic credit. At this stage, all that is required is that a student has an ID and password. From a practical point of view, all online colleges should specifically define throughout what academic dishonesty is, what identity fraud is and the
severe penalties should this be detected.
It is accepted that there are three conditions to establish the “fraud triangle”: an opportunity, a reward for the effort and a rationalization. The optimum approach is thus to reduce one or all of these conditions to minimize student fraud. Action is vital in the light of another study that has revealed that that up to 74% of students believe that it is easier to cheat in an online course than in a face-to-face course.
Assessment (in order of decreasing usage) in online courses ranges from assignments, tests and quizzes, discussion forum postings, projects/papers, proctored tests/quizzes, team projects, personal journals and a student portfolio.
Authentication refers to the process of determining whether someone or something actually is what it is purported or stated to be.28Authentication of students needs to be considered in this context.
Assessments in the Blackboard LMS provides numerous options for questions from fill in the blanks, matching, multiple answers, multiple choice, ordering, short answer/essay and finally, true/false.29 There are various options in administering these assessments and these include showing detailed score for each question, showing correct answers, providing instructor feedback for each question, allowing multiple attempts and allocating time for completion of quiz (and showing a running clock). Students do value the opportunity to take a quiz multiple times as well as feedback on whether their answer was correct or not.Solutions for identifying a student correctly
A few solutions suggested for authentication include:30
• Biometrics and web video recording. Methods used here include identifying unique typing style, signature, fingerprint, voice or webcam recording during the testing process.
• Proctored examination at a certified testing center.
• Challenge questions based on third party public and private data (not merely the usual, “What is your mother’s maiden name?”).Websites that do your assignments for you
There are a number of websites that have appeared offering to take an online student’s entire course or degree program.31Examples of this insidious activity are such websites as wetakeyourclass.com, boostmygrades.com ($695 for a graduate class and noneedtostudy.com (entire course for $900). To add insult to injury, the sites often guarantee at least a B grade for the course.
Suggested ways to deal with this problem are to verify a student’s identity and to check the student computer IP addresses. Some institutions currently use a process to ensure the correct person is undertaking the class using an identity check for registering or taking an exam (asking questions about the person’s identity such as addresses where they have lived and who they have lived with using a database service such as Acxiom to confirm results). In addition, a large number of assignments and tests are set so that getting someone else to do them would be horrendously expensive. Finally, a high level of interaction with the student is essential, especially using video and web conferencing systems and the instructors or student coordinators should be able to learn who the “real student” is.Minimizing plagiarism and dishonest conduct by students
A way of maintaining the integrity of your programs is to use a software program to detect plagiarism. Viper is a free download (scanmyessay.com) available to find out whether students have copied content from the internet, from previous work or from other students.32 Students also use this to confirm they have not omitted citations or indeed, copied inappropriately. In order to establish a coherent view of an individual student’s track record in this regard, it is useful to maintain a database of all work submitted by students to all instructors at the institution.
Another package widely used, as mentioned above, is Turnitin (turnitin.com) from iParadigms. This has a fee attached to it, but is widely used worldwide within universities and colleges.
13.4 Proctoring software
Proctoring or remote testing supervision software is used to allow testing of a student’s knowledge with a high level of integrity in terms of preventing her accessing unauthorized sources of support (either another individual/communication channel or materials), or if the student does access these unauthorized sources of support, we have demonstrable evidence of this breach. The operation of a typical proctoring software package is given in Appendix E.
In all our exams and tests we have no way of verifying the integrity of the work done by the students, either in a remote lab, simulation or assignment and test. Certainly, building up our relationship with the students and getting them to sign an honor code will result in fewer transgressions, but there are still a minority of students who will plagiarize and falsify their results. We are unable to determine how the students are answering the questions (use of books, online resources, help from others, etc.).
In addition, the various accreditation authorities expect us to perform a check on the various submissions of assignments and examinations. Normally, in other well-established colleges and universities, their students undertake their examinations at a specific approved test center. For example, Pearson Vue have approximately 400 exam centers over the world, which a student can attend to write their examination.
Western Governors University initially ran proctored exams at approved test sites such as testing centers and libraries.33 The majority of exams are now conducted through a web-based proctoring service provided by a third party company. Those students who want to undertake their exams at home are sent a free webcam. The exam must be taken in an environment totally quiet with no one else present. Facial recognition software confirms the identity of the student and the proctoring service monitors the student.
Industry specific performance assessments are used to confirm practical experience and competencies.A typical proctoring software package
There is a simpler approach that will give an improvement in the integrity of test results, although it should be noted that this is definitely still not fail-safe, with a determined hacker still being able to “break the system”. We believe this simpler and undoubtedly more convenient approach to checking on a student’s work can be achieved with a computer, appropriate software and a webcam (with audio capability) as is discussed below in the following sections.
Figure 13.3: Proctoring software package
To minimize the risk of plagiarism, better-formulated questions need to be written up in the exam paper to bring out what the students have learned in a practical application manner. Rigorous sanctions should be applied to students caught cheating or plagiarizing; this is not an acceptable practice and shows disrespect for the college and other students/instructors.
Students must have a working webcam, microphone and a stable internet connection is required. However, unstable connections can be taken care of with the exam being paused or even frozen during the time the internet is unavailable.
Before the examination commences, the student identifies himself/herself, such as using a finger print scanner (rather expensive and difficult to organize for each student) or pointing the webcam to his/her face and stating their name and in which course they are enrolled (preferably holding up a driver’s license with photo next to the face as well). A possible alternative is for the student to type in her name and student number (or some secret password), to match with the college’s records of the student.
Switching windows (applications) on the computer must be blocked so that the student’s screen is locked on the specific exam paper or remote lab, and no web browsing or other software can be allowed.
Due to the different time zones in which the students came from, the instructor will write up a large number of questions (with three levels of difficulty), and a few of them will be randomly selected for each student. Alternatively the instructor can prepare different exam papers for different time zones.
The student will be timed from the start of the examination period for each question. After a predetermined amount of time, the examination ends and /shuts down. All data received (answers, video, audio and snapshots) are saved on the main server for access by the examining team.
Keystrokes are recorded so that copying and pasting is obvious when the data are verified (only “ctrl+c” and “ctrl+v” will be recorded for copy/pasting instead of the sentence/paragraph found in the answer sheet. Similarly, the student uses his mouse to copy/paste.
It is unlikely that labs will ever be used as examinations but they may be used for certification reasons. Exams in labs are not done at any university that we are aware of. There are two main problems with labs that need to be tracked: the student doesn’t attend the lab or the student is cheating by manipulating the equipment pretending to do the lab and then producing reports obtained from another student. These issues could be dealt with by ensuring the webcam is viewing the student undertaking the lab and the student’s activities (which are recorded) are compared with the lab report she generates.
Naturally, the student can still prepare her lab report offline, paste in her results and upload to Moodle. We will use the lab recording to detect any possible irregularities and possibly (but unlikely) to compare the report with the student’s activities in the lab.
Two types of proctoring sessions could be conducted. In the first, there is a standard exam on a website (such as Survey Monkey) where the student attempts a quiz in a given time. This could also be done with the Moodle learning management system quiz that has random questions for each student so that different time zones can be accommodated, and sharing of answers is deterred. These are adequate for mini-quizzes and short tests, but not assignments and examinations. Assignments and examinations will be conducted using a Word file that the student types into.
The second scenario for assessing a student is within a remote lab and confirming that they have undertaken the correct steps, e.g. configuring a data logger and achieving desired results. It is unlikely that this will be a formal examination. It will be more for confirmation that the student has indeed undertaken this lab and how he undertook the work in the lab (a minute of work or 50 minutes of meaningful activity).Student portfolios
A supporting set of documentation for achievement of applicable graduate attributes required can be recorded in a student’s portfolio, which can be kept online.34 Students can deposit evidence such as written work, videos, photos, audios, presentations, computer programs and photographs in this online portfolio. Deakin University in Australia has been experimenting with a system. A survey revealed some confusion amongst students on the exact purpose of a student portfolio, so some orientation to what and why they are required is needed.
13.5 Assessment of quality for online programs
Poor quality programs
With higher education becoming more mass oriented, there are real concerns about the associated quality with inflated grades and poorly educated graduates.35 In the past, quality assurance was undertaken by looking at the inputs such as student quality, quality of tuition, course materials, course structure, libraries, faculty quality, facilities and duration of courses. This model doesn’t work well for online education with widely scattered students, contract instructors and students participating in courses in (often intermittent) chunks.Online learning evaluation
Kirkpatrick’s 4 levels or Phillips’s 5 levels (including Return on Investment or ROI) can be applied to assessing the value of online learning and these generally indicate a strong ROI based simply on lower costs in developing and conducting online learning programs compared to classroom-based training (for example, by the elimination of travel expenses).36 However, our research reveals that, generally, most of our course participants feel that traditional classroom learning still offers greater interactivity and thus satisfaction than with synchronous online learning. However, we have found that the learning outcomes (as compared to classroom instruction) are probably at least equivalent or better with our synchronous online learning sessions.
Some suggested features to assess your online learning courses are discussed below. Their use will vary depending on whether it is a asynchronous, synchronous or indeed, blended production. These features include:Content. Is the content appropriate for the audience and course outline in terms of depth/technical quality and breadth? Instructional design Will the learners actually gain the appropriate level of knowledge from undertaking this course? Interactive and engaging. Does the course contain sufficient interactivity to make it interesting and does it engage the learner? How deep does the learner get immersed in the course? This can vary from a “book on the web” to a highly interactive game or simulation. Authentic hands-on and experiential. Does the learner get an authentic hands-on and experiential impact from undertaking the practical course exercises? Computer and software compatibility. How did the course run on varied sets of hardware and operating systems ranging from Linux, Windows XP to Windows 7 and the Mac Operating system? Were there any glitches or dropouts in attending or running the course? Navigation and road map Can the user easily find their way around the course and supporting presentation software (and remote labs) and enter/ pause and exit quickly and effectively? Can the irritating course manual be dispensed with? Assessment. Is the learner assessed in a formative (throughout the course) or summative way (at the conclusion of the course)? What is the risk like with the assessment? Is the assessment meaningful, patronizing or too difficult? Was the learner assessed at the beginning of the course to test his / her knowledge before commencing the course? A course with panache Is the course laid out attractively and in an appealing or elegant way? Or is the student confronted with garish colors and irritating sounds? Does the entire course hang together or is It disjointed? Audit trail. Are the learner’s results, time to complete, assessments and performance in the course recorded for later review? Track record. Historically, what were the results from the course in terms of gaining of knowledge and expertise? What was the “stickability” and attrition rate for this course like? Is the course recommended to others? Further improvements. Is there a feedback mechanism from the learners to improve the course and were these actually acted upon by the instructional team? Assessment of programs using balanced scorecard approach
The concept of a balanced scorecard approach was introduced two decades ago by Robert Kaplan and David Norton and contains four overall measures of an organization:37
• How customers perceive us (Customer satisfaction).
• What must we excel at internally to succeed (Internal processes).
• What must we do to continue improving and creating value (Visibility).
• How we are succeeding on a financial basis (Financial Viability).
Appropriate suggested measures at Virginia Tech are shown in the table below.Table 13.1: Balanced scorecard measures at Virginia Tech
(Adapted from Thongsamak, S., Scales, G., & Peed, C.37)
|Customer Satisfaction|| Student Feedback |
Number of graduates
Average degree completion time
|Internal Processes|| Number of Online courses |
Number of videoconferencing courses
|Visibility|| Number of hits on website |
The total amount of publicity
|Financial Viability|| Enrollment |
As with most ventures of this nature, in implementing this scorecard it is vital that there is wholehearted support from all levels in the organization.Accreditation of engineering programs
In the context of distance learning, some caution has been expressed about the current method of accreditation of engineering programs with the need for a genuine outcomes-based system which is based on the attainment of specific graduate attributes.38 Until this is achieved, distance learning programs in engineering will have obstacles to being accredited. For example, Engineers Australia has a policy based around demonstrated outcomes; however they also require (in their policies) that all undergraduates attend on-campus for two weeks per full-time year of their degree for lab-based work. The reason expressed is that hands-on experiential learning is a key part of engineering education and it is considered vital that this is clearly demonstrated.
In terms of engineering accreditation, an outcomes based assessment of graduate competency is now used.39 In Australia there is still a compulsory residency component enforced for distance learning students. It is often suggested that engineering is a peculiar case requiring this residency on-campus component due to the labs requirements. In any event, the difference between on- and off-campus activities is steadily being blurred by residential students now making extensive use of off-campus learning resources (such as recordings of lectures).Professional accreditation issues with distance learning
Although perhaps of peripheral interest to most people involved in online learning, it is important in the overall design of a course for engineering and technology degree programs to be aware of a possible stumbling block in the acceptance of a distance online learning-based course.40 There are very few true Bachelor of Engineering programs presented purely in full off-campus mode, although as pointed out perhaps rather wryly by Carnevale as long ago as 2002, ”..the technology is already there…It’s a matter of legitimizing it.” There are some significant difficulties with achieving professional accreditation of engineering programs presented through distance learning. Engineering education accrediting organizations have slowly migrated to an outcomes-based assessment of graduate competency, but still have a strong affinity to demanding evidence of inputs (such as hours on-campus). Engineers Australia typically requires students to have up to 30 contacts hours per week and a full-time study workload of up to 50 hours per week.
It is suggested that although practical skills have been gained off-campus, staff need to be assured by a student’s presence on-campus of this. However, Engineers Australia do remark that they will consider distance learning if other techniques used to demonstrate this equivalence to on-campus work are equally effective.
There is a good argument that certain skills such as collaborative teamwork on projects and leadership require physical attendance to connect with each other. However, there are some very good solutions for this type of learning using tools such as web conferencing and collaborative project software. One could also comment that with the current globally-based dispersed engineering project teams that working together virtually is also very good experience for the modern student.
This attitude towards distance learning is perhaps somewhat arcane as the typical student today has changed dramatically from a few decades ago. The typical student does not live on campus and is often combining work and study. There is also strong evidence that off-campus students tend to do better in their grades and the most effective learning for on-campus students occurs off-campus.
The other point is that these changes don’t only apply to mature age students but only young school leavers. There is considerable evidence that on-campus students are increasingly using online resources and there is a blurring of the boundaries occurring. This trend can only accelerate with the modern highly mobile student emerging from schools.Remote labs and assessment
Contrary to many expectations, it is suggested that remote labs can actually enhance the chance of an engineering degree programs achieving accreditation.41 One should bear in mind that the engineering profession is rapidly changing and there is considerable work that is done which is purely PC-based (e.g. programming a PLC or writing an engineering program or analyzing a building structure). In addition, with the high costs of labor in remote mining sites (associated with extreme distances, fly-in fly-out workers disconnect with their families and the hazardous environment) there has been a move to working on remote-access technologies. Thus education in this area is enormously beneficial to engineering students and appropriate for “a tick in the box” under accreditation processes. Naturally, it is the judicious mix of the classical face-to-face labs combined with virtual and remote labs that will ensure satisfactory accreditation of an engineering program. The main issue at stake is how the virtual and remote lab fits into the overall thrust of the program rather than its access mode.Are traditional universities and accreditation on the wrong side of history?
With numerous examples of good course materials available on line at very little cost, questions can be asked about the value of a traditional university’s degree.42 Naturally, there have been questions asked about the actual accreditation status of these free (or low-cost) courses. MITx which was launched by the famous Massachusetts Institute of Technology will make available open courseware to anyone for free. It will charge a nominal fee for certificates of completion for successful completion of a course.
Peer 2 Peer University, founded in 2008 and with support from the University of California and other prominent foundations, has volunteer teachers to guide students through their open courseware.
University of the People provides undergraduate programs with open courseware and peer-to-peer teaching conducted by volunteers. This has recently been accredited by the respected, US-based DETC.
Saylor.org offers more than 200 free higher education courses with electronic portfolios to show prospective employers what they have achieved in terms of knowledge gain.
The challenge will be to achieve recognition from businesses about completion of these programs and acceptance by traditional universities of these credentials. This will require tight assessment and acceptable examination processes.
An important point to make with this enormous amount of open or low-cost courseware is that simply provision of information and even knowledge is not learning.43 We have to go well beyond this to achieve a high quality learning experience.Rating the quality of an online education program
There is far more dem and for public accountability of online education. Kaye Shelton from Dallas Baptist University put together a list of measures of quality in online education using a panel of experts suggested by the respected Sloan Consortium.44
The scoring is based on a perfect score of 210 with 189-209 considered exemplary ranging to below 125 that is unacceptable. The rating for each criteria was from 0 (not observed) / 1 (insufficient) / 2 (moderate use) to 3 (meets criteria completely).
The different categories on the scorecard were:
• Institutional Support (e.g. governance/authentication of students/copyright ownership of course materials/defined the value of distance learning).
• Technology Support (e.g. documented technology plan including electronic security / reliable delivery systems with measurable operational metrics / backup / support for staff / students).
• Course Delivery and Instructional Design (e.g. minimum standards guidelines / regular reviews / measurable learning objectives / consistency / engagement / continuous improvement).
• Course Structure (e.g. comprehensive detail provided / access to library and learning resources / expectations explained and widely available / special needs support / peer-to-peer collaboration resources / open access to all materials).
• Teaching and Learning (e.g. student-to-student and student-to-instructor interaction / timely comprehensive feedback / library support and research / strategies to provide presence).
• Social and Student Engagement (e.g. interaction between students).
• Faculty Support (e.g. assistance with transition to online / instructors are keen about online instruction / legal and ethical training / professional development).
• Student support (e.g. suitability review for online education / detailed course information before commencement / training and support in use of online systems / academic and career counseling / encouragement of student engagement).
• Evaluation and Assessment (e.g. continuous program assessment / continuous assessment of course / recruitment and retention assessment / faculty performance review).
A detailed assessment matrix is available as listed in the references.
Key points and applications
The following are the key points and applications from this chapter entitled: Assessment and Evaluation of Students and Online Learning Programs.
1. Formative assessments provide greater gains in learning than merely only summative assessments.
2. Some advantages of online quizzes:
• Quick and easy access.
• Instantaneous feedback.
• Ability to reuse quizzes.
• Possibility of multiple attempts.
3. Online assessment combined with instructor-led sessions can be particularly powerful.
4. There are major concerns with weak validity with e-assessments not validly assessing the skills being tested and a weak linkage between required learning outcomes and assessment. This can be dealt with by establishing a strong relationship between the instructor and student over many elements of assessment and thus to authenticate the work this way.
5. Pre-session quizzes are good to test knowledge and comprehension and to undertake application, analysis, synthesis, and evaluation activities in a classroom setting.
6. Open book assessments are gaining considerable traction (“Honest, open book, open mind”)
7. The following issues need to be dealt with in an online examination environment:
• Security: Test on concepts rather than regurgitating materials.
• Interactivity: Live tests are preferred to online ones.
• Equity: For random questions; it is important that they are all at a similar level.
• Hands-on labs: Physical demonstrations of working with lab software/equipment is preferred and this may be dealt with by using screen capturing software.
• Team work assessment: Working in teams should be assessed as this is a key activity of engineers and scientists.
8. Some suggestions to minimise plagiarism:
• Set clear expectations of the student.
• Ensure that the instructor is a good role model.
• Provide students with checking tools.
• Ensure zero tolerance for plagiarism.
9. Consider use of proctoring software to allow remote testing of a student’s knowledge.
10. Key elements of assessing online programs:
• Instructional Design.
• Content that is interactive and engaging.
• Authentic hands-on and experiential work.
• Computer and software compatibility.
• Navigation and road map around the course.
• Audit trail.
• Track record (in terms of gaining knowledge and expertise).
• Further improvements feedback.
11. Balanced Scorecard approach in assessing quality of a program includes:
• Customer Satisfaction (Student feedback, Faculty feedback, Number of graduates, Average course completion time).
• Internal processes (number of online courses, number of videoconferencing courses).
• Visibility (number of hits on website, total amount of publicity).