Frank Nguyen

frankn.net

It's a right of passage and a privilege to eager 16 year olds everywhere. The driver's license is a much anticipated milestone in a young person's life.

My mom took me down to the motor vehicle office on my 15 year, 7 month birthday to apply for my learner's permit. I had studied the driver's handbook from cover to cover for weeks and was confident. I passed the written test with flying colors. I was ready to drive! Well, at least so I thought.

She immediately took me out to an unoccupied industrial park with perfectly paved, straight roads and virtually no traffic. After about thirty minutes of practice, she then had me drive the 113-mile trip on Interstate 10 from Phoenix to Tucson, Arizona. Those were, and still are to this day, the two most frightening hours of my life. I distinctly recall slowing down to let 45-mph recreational vehicles merge onto the freeway. (Trust me, that doesn't happen anymore!) She then bravely declared that I was ready for the test.

Unlike my written test, I was not confident in my ability to pass the secondary driving test. The epic torturous desert drive had done wonders for my driving skills and traffic awareness. However, one seemingly insurmountable hurdle stood between me and my driver's license: the parallel parking test. People have struggled to master this elusive skill for more than fifty years. It also didn't help that my mom drove a gas guzzling sports utility vehicle—not exactly the most svelte, agile car to attempt such a maneuver.

At the time, I dreaded the parallel parking test and have since often noted that, except for those who live in dense urban areas, few ever parallel park again. Even among those who nobly passed the parallel parking test initially, some will admit that they are no longer confident in their ability to do so now. (Hence the advent of cars that can parallel park themselves.) After a dozen years of working in the learning and performance world, though, the driving and parallel parking tests do make sense. After all, how else would the government know if you can really park a car in a tight spot? Certainly not from scribbled-in bubbles on a 25-item multiple choice test.

Parallel parking for your eLearning course

The irony is that we, as eLearning designers and developers, are often guilty of doing just that. We rely on written tests to tell us if learners know how to do something, but we stop short of asking them to demonstrate that they can do what they've learned. We pride ourselves in building robust and instructionally sound courses. We provide learners with media-rich instruction, diverse examples, interactive practice activities, and a host of other Instructional System Design (ISD) bells and whistles. However, when it comes to measuring whether or not a learner has mastered the objectives of a course, we often fall short. We often rely on multiple choice tests or true and false items. Even worse, sometimes we don't bother testing our learners at all.

Explore the assessment samples in Figures 1 through 3. Figure 1 illustrates a simple quiz used to test learners about the facts about global warming. Figure 2 shows a common approach to assess software procedures, in this case, how to set margins in Microsoft Word®. Figure 3 is a portion of a multiple choice test administered to hotel front desk representatives dealing with customer conflicts.

Figure 1. Before: Sample of a test on global warming.

• Watch the demo: Sample of a test on global warming

Figure 2. Before: Sample of a test on setting margins in Microsoft Word.

• Watch the demo: Sample of a test on setting margins in Microsoft Word

Figure 3. Before: Sample of a test in handling a customer conflict.

• Watch the demo: Sample of a test on setting margins in Microsoft Word

Do any of these examples seem familiar to you? Have you ever taken a course that included test items like these? Have you ever created one yourself? It's second nature for us to rely on these types of tests. We're familiar with multiple choice items. We grew up on true and false tests in school. They're comfortable and seemingly easy.

Selecting the right assessment type

The reality is that these approaches to assessing a learner may not always be the best choice. How can you be sure that a student pilot can safely fly a plane using a 100-question multiple choice exam? How do you know if a customer service agent is truly able to handle a difficult customer from their answers to a series of true-false statements? How can you be certain that an analyst can correctly process the company's financials from a multiple choice test that focuses on their knowledge and not their abilities? Using the incorrect type of assessment in your eLearning course is akin to issuing 16-year old teenagers a driver's license after they pass just the written exam.

Figure 4 shows a simple process you can use to avoid such risks and select the right kind of assessment for your instructional needs.

Figure 4. The eLearning assessment process.

The first step in the process is to visit the instructional goals and objectives of your course. Second, Clark (2008) asserted that every instructional objective can be categorized into one of five types: fact, concept, process, procedure, or principle. Using the details shown in Figure 5, classify each of your instructional objectives as one of these five categories. Doing so will give you valuable clues to the proper way to assess your learner.

Figure 5. Categorizing instructional objectives (Adapted from Clark, 2008).

The third step is to select the most appropriate assessment for your objectives. The table in Figure 6 lists Clark's five objective categories on the left and the different types of assessment items offered by Adobe Captivate® 3. As you can see, certain types of assessment items are recommended for particular types of objectives. For example, if you are asking your learners to master facts or concepts about global warming, then creating essay, hotspot, matching, multiple choice, or short answer test items could provide an effective means to assess your learner. If you are teaching them how to perform procedures in software such as Microsoft Word, a common multiple choice or matching activity could work, but asking them to order the steps of the procedure using a sequencing test item or asking them to actually perform the task in a software simulation would be better. If you are teaching soft skills where answers are not always black and white, such as dealing with customer conflicts, providing a way for your learners to respond to problems in a written form or through structured scenarios would enable you to more effectively assess their abilities.

Figure 6. Types of assessment and objectives.

Note: True-False question types can be used for motivational purposes. For the purpose
of assessment they are not recommended as the learner has a 50/50 chance of being correct.

After you've identified the proper type of assessment, the next steps are to design and develop the test items. The specific steps to create each type of item in Adobe Captivate varies. For more information on how to design and develop the types you've identified, refer to the online help for creating question slides.

Assessment makeovers

Explore the assessment samples in Figures 7 through 9. These examples are based on the same content as the samples shown earlier but they have been revised to better assess the learner's mastery of the objectives. The sample in Figure 7 employs more instructionally aligned and robustly written multiple choice items that make use of the randomization and pooling features of Adobe Captivate 3. The sample in Figure 8 prompts the learner to set margins in a simulated version of Microsoft Word. The learner is provided parameters for the task with minimal guidance. All of the learner's interactions, both correct and incorrect, are sent to and tracked by a learning management system. Figure 9 represents a small portion of a scenario-based assessment item that immerses the learner in the role of hotel front desk representative. The learner is given a situation, choices, and feedback. Depending on their actions, learners would branch off to different parts of the scenario-based assessment and receive points based on the choices they make.

Figure 7. After: Sample of a test on global warming.

• Take the test: Global warming quiz created with Adobe Captivate

Figure 8. After: Sample of a test on setting margins in Microsoft Word.

• Take the test: Software simulations created with Adobe Captivate

Figure 9. After: Sample of a test in handling customer conflict.

• Take the test: Scenario simulation created with Adobe Captivate.

Needless to say, despite all of the dread and angst, I successfully passed my parallel parking test and drove home victoriously with my shiny new driver's license in hand. And despite the odds, I can still parallel park today (admittedly, not without bumping the curb a couple of times). By properly aligning the assessment items to the instructional objectives of your course, you too can more accurately measure not just your learner's knowledge but their ability to apply that knowledge and complete the tasks they need to perform to be successful.

Where to go from here

To learn more about assessments and how to select and create tests that accurately measure your learners' proficiency, consult the printed resources listed below, download the handout (PDF, 338K), and the Adobe Captivate Developer Center.

References

Clark, R.C. (2008).  Development Technical Training: A Structured Approach for Developing Classroom and Computer-basedInstructional Materials, 3rd Edition.  San Francisco: Pfieffer.

Shrock, S.A., & Coscarelli, W.C. (2007).  Criterion-referenced Test Development: Technical and Legal Guidelines for Corporate Training, 3rd Edition.  San Francisco: Pfieffer.

 

 

 

About the author