11
TESTING AND EVALUATION IN ONLINE COURSES AND PROGRAMS
Contemporary nursing students expect educational institutions to provide flexible instructional methods that help them balance their academic, employment, family, and personal commitments (Jones & Wolf, 2010). Online education has rapidly developed as a potential solution to these demands. The growth rate of online student enrollment in all disciplines has far exceeded the growth rate of traditional course student enrollment in U.S. higher education (Seaman, Allen, & Seaman, 2018). Over 6.3 million students enrolled in at least one college-level online course during the fall 2016 academic term, with the proportion of all students taking at least one online course representing 32.0% of all students (Seaman et al., 2018). In nursing, the American Association of Colleges of Nursing (2019) reported that some of the 219 RN-to-master’s degree programs and more than 600 RN-to-BSN programs were offered at least partially online. Doctor of nursing practice (DNP) programs, which have experienced a huge growth over the past few years, are frequently offered partially or completely online, and online courses are included in prelicensure and other levels of nursing education programs as well.
For the purposes of this chapter, online courses are those in which at least 80% of the course content is delivered online. Face-to-face courses are those in which 0% to 29% of the content is delivered online; this category includes both traditional- and web-facilitated courses. Blended (sometimes called hybrid) courses have between 30% and 80% of the course content delivered online. Examples of various course management systems used for online courses include Blackboard, Desire2Learn (Brightspace), Sakai, and Moodle.
Along with the expansion of online delivery of courses and programs comes concern about how to evaluate their quality. Chapter 18, Program Evaluation and Accreditation, provides standards for evaluating distance education programs. The accreditation criteria from each of the accrediting bodies in nursing address online programs in different ways, and these are presented in Chapter 18, Program Evaluation and Accreditation. This chapter discusses recommendations for assessment of 200learning in online courses, including testing and appraising course assignments, to determine whether course goals and outcomes have been met. It also suggests ways to assess online courses and programs, and to assess teaching effectiveness in online courses and programs.
Assessment of Learning at the Individual Learner Level
Online assessment and evaluation principles do not differ substantially from the approaches used in the traditional classroom environment. As with traditional format courses, assessment of individual achievement in online courses should involve multiple methods such as tests, written assignments, and contributions to online discussions. Technological advances in testing and assessment have made it possible to administer tests on a computer and assess other products of student thinking even in traditional courses (Miller, Linn, & Gronlund, 2013). But courses and programs that are offered only online or in a hybrid format depend heavily or entirely on technological methods to assess the degree to which students have met expected learning targets or outcomes.
Online Testing
The choice to use online testing inevitably raises concerns about academic dishonesty. How can the course instructor be confident that students who are enrolled in the course are the ones who are taking the tests? How can teachers prevent students from consulting unauthorized sources while taking tests or sharing information about tests with students who have not yet taken them? To deter cheating and promote academic integrity, faculty members should incorporate a multifaceted approach to online testing. Educators can employ low- and high-technology solutions to address this problem.
One example of a low-technology solution includes creating an atmosphere of academic integrity in the classroom by including a discussion of academic integrity expectations in the syllabus or student handbook (Conway-Klaassen & Keil, 2010; Hart & Morgan, 2009). When teachers have positive relationships with students, interact with them regularly about their learning, and convey a sense of confidence about students’ performance on tests, they create an environment in which cheating is less likely to occur (Brookhart & Nitko, 2019; Miller et al., 2013). Faculty members should develop and communicate clear policies and expectations about cheating on online tests, plagiarism, and other examples of academic dishonesty (Morgan & Hart, 2013). Unfortunately, students do not always view cheating or sharing as academic dishonesty; they often believe it is just collaboration (Wideman, 2011).
201Another low-technology option is administering a tightly timed examination. This approach may deter students from looking up answers to test items for fear of running out of time to complete the assessment. Other suggestions to minimize cheating on online examinations include randomizing the test items and response options, displaying one item at a time and not allowing students to review previous items and responses, creating and using different versions of the test for the same group of learners, and developing open-book examinations (Conway-Klaassen & Keil, 2010). However, each of these approaches has disadvantages that teachers of online courses must take into consideration before implementing them.
Randomized Sequence of Test Items and Response Options
As discussed in Chapter 10, Assembling, Administering, and Scoring Tests, the sequence of test items may affect student performance and therefore assessment validity. Many testing experts recommend arranging items of each format in order of difficulty, from easiest to most difficult, to minimize test anxiety and allow students to respond quickly to the easy items and spend the majority of testing time on the more difficult ones. Another recommendation is to sequence test items of each format in the order in which the content was taught, allowing students to use the content sequence as a cognitive map by which they can more easily retrieve stored information. A combination of these approaches—content sequencing with difficulty progression within each content area—may be the ideal design for a test (Brookhart & Nitko, 2019). Many testing experts also recommend varying the position of the correct answer to multiple-choice and matching items in a random way to avoid a pattern that may help testwise but uninformed students achieve higher scores than their knowledge warrants. A simple way to obtain sufficient variation of correct answer position is to arrange the responses in alphabetical or numerical order (Brookhart & Nitko, 2019; Gronlund, 2006). Therefore, scrambling the order of test items and response options on an online test may affect the validity of interpretation of the resulting scores, and there is no known scientific evidence to recommend this practice as a way of preventing cheating on online tests.
Displaying One Item at a Time and Not Allowing Students to Review Previous Items
This tactic is appropriate for the computerized adaptive testing model in which each student’s test is assembled interactively as the person is taking the test. Because the answer to one item (correct or incorrect) determines the selection of the next item, there is nothing to be gained by reviewing previous items. However, in teacher-constructed assessments for traditional or online testing, students should 202be permitted and encouraged to return to a previous item if they recall information that would prompt them to change their responses. While helping students develop test-taking skills to perform at the level at which they are capable, teachers should encourage students to bypass difficult items and return to them later to use the available time wisely (Brookhart & Nitko, 2019). Therefore, presenting only one item at a time and not permitting students to return to previous items may produce test scores that do not accurately reflect students’ abilities.
Creating and Using Different Forms of an Examination With the Same Group of Students
As discussed in Chapter 2, Qualities of Effective Assessment Procedures: Validity, Reliability, and Usability, alternate forms of a test are considered to be equivalent if they were developed from the same test blueprint or table of specifications, and if they produce highly correlated results. Equivalent test forms are widely used in standardized testing to ensure test security, but alternate forms of teacher-constructed tests usually are not subjected to the rigorous process of obtaining empirical data to document their equivalence. Therefore, alternate forms of a test for the same group of students may produce results that are not comparable, leading to inaccurate interpretations of test scores.
Developing and Administering Open-Book Tests
Tests developed for use in traditional courses usually do not permit test-takers to consult references or other resources to arrive at correct responses, and most academic honesty codes and policies include expectations that students will not consult such resources during assessments without the teacher’s permission. However, for online assessments, particularly at the graduate level, teachers may develop tests that permit or encourage students to make use of appropriate resources to select or supply correct answers. Commonly referred to as open-book or take-home tests, these assessments should gauge students’ higher order thinking abilities by requiring use of knowledge and skill in novel situations. One of the higher order skills that may be important to assess is the ability to identify and use appropriate reference materials for problem solving, decision making, and clinical reasoning. Teachers can use test item formats such as essay and context-dependent item sets (interpretive exercises) to craft novel materials for students to analyze, synthesize, and evaluate. Because these item formats typically require more time than true–false, multiple-choice, matching, and completion items, teachers should allot sufficient time for online open-book testing. Therefore, administering an open-book assessment as a tightly timed examination to deter cheating will not only produce results that do not accurately reflect students’ true abilities but will likely also engender unproductive feelings of anxiety and anger among students (Brookhart & Nitko, 2019).
203An additional low-technology strategy used to deter cheating may be the administration of tests in a timed synchronous manner, in which students’ test results are not revealed until after all students have finished the examination. Although synchronous online testing may be inconvenient, adequate advance knowledge of test days and times could alleviate scheduling conflicts that some students may encounter.
High-technology solutions to prevent cheating on unproctored tests include browser security programs such as Respondus™ to keep students from searching the Internet while taking the examination (Hart & Morgan, 2009). However, this security feature does not prevent students from using a second computer or seeking assistance from other people during the test. For those wanting to use the best technology available to prevent academic dishonesty, faculty members could use remote proctoring to ensure student identity and monitor student actions (Dunn, Meine, & McCarley, 2010). Remote proctoring, also called virtual proctoring, is a service that works within an online test delivery system to emulate the role of an on-site proctor, confirming the identity of the test-taker and safeguarding the integrity of the exam. Proctoring may be synchronous (in real time) or asynchronous (recording the testing session for later review by a proctor), and may be performed by a live person or by automated monitoring technology. Remote proctoring systems usually incorporate devices such as web cameras, microphones, and even biometric scanners into the learning management system.
Students also may be required to use webcams to confirm their identities to the faculty member. Some course management systems have password-protected access and codes to prevent printing, copying, and pasting. Additional anticheating methods include requiring an online password that is different for each test and changing log-in codes just prior to testing. However, these methods do not prevent students from receiving help from other students. Therefore, a reasonable compromise to these dilemmas may be the use of proctored testing centers (Stonecypher & Wilson, 2014).
Many universities and colleges around the country cooperate to offer students the opportunity to take proctored examinations close to their homes. Proctors should be approved by the faculty in advance to observe students taking the examination online (Hart & Morgan, 2009) and should sign an agreement to keep all test materials secure and maintain confidentiality. Although the administration of proctored examinations is not as convenient as an asynchronous nonproctored test, it offers a greater level of assurance that students are taking examinations independently.
Course Assignments
Course assignments may require adjustment for online learning to suit the electronic medium. Online course assignments can be crafted to provide opportunities for students to develop and demonstrate cognitive, affective, and psychomotor abilities. Table 11.1 provides specific examples of learning products in the cognitive, affective, and psychomotor domains that can be used for formative and summative 204evaluation. Assignments such as analyses of cases and critical thinking vignettes, discussion boards, and classroom assessment techniques may be used for formative evaluation, whereas papers, debates, electronic presentations, portfolios, and tests are more frequently used to provide information for summative evaluation (O’Neill, Fisher, & Newbold, 2009). Online course assignments may be used for formative or summative evaluation of student learning outcomes. However, the teacher should make it clear to the students how the assignments are being used for evaluation. No matter what type of assignment the faculty member assesses, the student must have clearly defined criteria for the assignment and its evaluation.
TABLE 11.1 Examples of Learning Products Used for Online Assessment of Learning
| COGNITIVE DOMAIN | AFFECTIVE DOMAIN | PSYCHOMOTOR DOMAIN |
|---|---|---|
| Discussion boards Online chats Case analysis Term papers Research or evidence-based practice papers Short written assignments Journals Electronic portfolios | Discussion boards Online chats Case analysis Debates Role-play Discussions of ethical issues Interviews Journals Blogs | Video recordings, use of other technologies Virtual simulations Developing web pages Web page presentations Interactive modules Presentations |
Feedback
As in traditional courses, feedback during the learning process and following teacher evaluation of assignments facilitates learning. Students need more feedback in online learning than in the traditional environment because of the lack of face-to-face interaction and subsequent lack of nonverbal communication. Teachers should give timely feedback about each assignment to verify that they are in the process of or have finished assessing it, or to inform the student when to expect more detailed feedback. O’Neill et al. (2009) suggested that feedback should be given within 24 to 48 hours, but it may not be reasonable to expect teachers to give detailed, meaningful feedback to a large group of students or on a lengthy assignment within that time frame. For this reason, the syllabus for an online or a hybrid course should include information about reasonable expectations regarding the timing of feedback from the teacher. For example, the syllabus might state, “I will acknowledge receipt of submitted assignments via e-mail within 24 hours, and I will e-mail [or post as a private message on the learning management system, or other means] more detailed, specific feedback [along with a score or grade if appropriate] within [specify time frame].”
Feedback to students can occur through a variety of methods. Many faculty members provide electronic feedback on written assignments using the Track Changes 205feature of Microsoft Word (or similar feature of other word-processing software) or by inserting comments into the document. Feedback also may occur through email, orally using vodcasting or scheduled phone conferences, or via a telecommunications application such as Skype, FaceTime, or Zoom.
As discussed in Chapter 9, Assessment of Written Assignments, the teacher may also incorporate peer critique within the process of completing an assignment. For example, for a lengthy written formal paper, the teacher may assign each student a peer-review partner, or each student may ask a peer to critique an early draft. The peer reviewer’s written feedback and the resulting revision should then be submitted for the faculty member to assess.
When an assignment involves participation in discussion using the course management system’s discussion forum, the teacher may also assign groups or partners to critique each other’s posted responses to questions posed by the teacher or other students. Although peer feedback is important to identify areas in which a student’s discussion contribution is unclear or incomplete, the course faculty member should also post summarized feedback to the student group periodically to identify gaps in knowledge, correct misinformation, and help students construct new knowledge.
No matter which types of feedback a teacher chooses to use in an online course, clear guidelines and expectations should be established and clearly communicated to the learners, including due dates for peer feedback. Students should understand the overall purpose of feedback to effectively engage in these processes. Structured feedback forms may be used for individual or group work. O’Neill et al. (2009) recommended multidimensional feedback that:
• Addresses the content of the assignment, quality of the presentation, and grammar and other technical writing qualities.
• Provides supportive statements highlighting the strengths and areas of improvement.
• Conveys a clear, thorough, consistent, equitable, constructive, and professional message.
Development of a scoring rubric provides an assessment tool that uses clearly defined criteria for the assignment and gauges student achievement. Rubrics enhance assessment reliability among multiple graders, communicate specific goals to students, describe behaviors that constitute a specific grade, and serve as a feedback tool. Table 11.2 provides a sample rubric for feedback about an online discussion board assignment.
Assessing Student Clinical Performance
Clinical evaluation of students in online courses and programs presents challenges to faculty members and program administrators. When using an online delivery mode, it is critical to ensure the clinical competence of nursing students. Although the didactic component of nursing courses may lend itself well to online delivery, teaching and evaluating clinical skills can prove more challenging in an online context (Bouchoucha, Wikander, & Wilkin, 2013).
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
