CTID Shuts Down

To everything (turn, turn, turn)

There is a season (turn, turn, turn)

And a time to every purpose, under heaven

A time to be born, a time to die

—Pete Seger

The Persian saying holds, “This too shall pass.” Accepting a CMA award (or a Grammy, I don’t recall which) soon after she had announced her retirement, Naomi Judd said, “It just hit me. That applies to good things, too.” So it comes to pass with another idea I had.

I started Central Texas Instructional Design (CTID) to share my experience and lessons learned in instructional design. It was part of a “free marketing” effort to help market myself by establishing a web presence and self-publishing credibility. But it turns out I have neither the time nor the energy to pursue the effort after taking a full-time position with the Fruit Company as World-Wide Manager for Training Assessment.

Given that the blog had actually attained a few followers, I probably should maintain the blog in event the Fruit Company and I should ever part ways, but....

Instead, I have migrated the postings from that blog into this one under the tag Central Texas Instructional Design. The posts are all back-dated to their original publication date.

 

Internal Validation on the Cheap: R(i)

This post originally appeared on the Central Texas Instructional Design blog on this date—or it would have if I had ever finished it.

ELAB 2019-03-01

Internal validation is a way to ensure multiple choice assessments are worth the time it takes to administer them. Without at least performing some kind of internal validation, an assessment isn’t anything more than a stressful method of tracking attendance.

My quick and dirty (cheap) method of internal validation uses the database described in the part 1 of this series, “Validating Multiple Choice Assessments on the Cheap.”

You also need a table that tracks the scores of each learner and ties to the previously described database. Here’s how I structure the record for this linked database:

1. Student ID*—a value that differentiates individuals but does not necessarily tie to any personally identifiable information
2. Assessment ID*—a value that distinguishes between the various assessments used in a curriculum and between versions of the same assessment
3. Score—what the learner scored on the assessment
4. Subgroup—the subgroup to which the learner belongs (more on this in a few minutes)

Fields marked with an asterisk tie this table to the previously described one.

The first step in the validation process is to divide the entire population into three roughly equal groups. These groups classify learners by how well they performed on the assessment. You can name the groups whatever you want. I usually number them to make sorting easier.

Formulas for this division exist, but unless you require a rigorous statistical methodology I prefer a simple method. Besides, smaller assessments don’t have a wide enough point spread to fine tune the size of the groups. So here’s what I do.

1. Calculate the mean score and standard deviation for the assessment. (Any spread sheet has built in functions for these operations.)
2. Place anyone whose score falls within ½-standard deviation of the mean in the middle group.
3. Place anyone whose score is more than ½-standard deviation above the mean in the upper group.
4. Place anyone whose score is more than ½-standard deviation below the mean in the lower group.

The size of the groups can differ significantly using this method. You can tweak the threshold scores a little, but small assessments and the tendency of scores to clump often make tweaking pointless.

With the learners assigned to groups, it’s time to use a database application such as Access or MySQL to help you analyze it. I usually generate a report defined something like this:

This is where I stopped. The report I created for the Fruit Company usually runs in excess of 15 pages, so I have no idea how I was going to simplify it.

The Difference Between a Performance Gap and a Knowledge Gap

This post originally appeared on the Central Texas Instructional Design blog on this date.

This post derives from conversations I recently had with one of my clients. Everyone who has designed instruction for corporations has had some version of these conversations. You know: “We need them to do X, and they’re not doing it. Train them to do it.”

This training request always begs for further analysis. Why are they not doing X? Is it that they don’t know how (a knowledge gap)? Or is it that they know how but are simply choosing to do something else (a performance gap)?

As an instructional designer, you should always pray for a knowledge gap.

Why? Because a knowledge gap is much easier to address than a performance gap, and proving you have addressed it is much more straight-forward.

Still, addressing a performance gap is an opportunity to prove your metal. Just be aware that you must fill many roles to bridge a performance gap. You must also function as:

• A business analyst
• An industrial psychologist
• A salesperson

You become a business analyst because your first task is to determine the underlying cause of the behavior. Since you have already determined it does not result from a lack of knowledge, many training managers are quick to say that it is not a training issue, just as many performance managers are look to training as a quick fix to every problem. 

As industrial psychologists, we know that we can work in the affective domain. We can use training to change emotions and motivations. But here were are as likely to undercut our own credibility as to make a real difference. Why? Because the real world takes precedence over our ivory tower training. We must ensure that systems in the real world reinforce what we are trying to change with the training. 

I have seen companies whose new hire training programs would not take responsibility for performance more than a few weeks out because it didn’t take long for the business to “corrupt” the newly hired employees. You have seen it. How many times have you heard, “I don’t care what you learned in training. I’m gonna show you how it’s really done.”

So our psychologist persona has to talk with our analyst persona. If we change their behavior, is there something in the business environment that will change it back over time. For example, if we train them to take as much time as needed with every customer but we pay them for each customer they talk to, the system overrides the training by positively reinforcing quickness over customer service.

Now comes our roles as salespeople. When we identify the conflicts in the systems, we have to sell the idea of fixing the systems. We often have to train our customers so that we can train their employees. Otherwise, the beatings will never stop.

 

Validating Multiple Choice Assessments on the Cheap

A question where the stem gives away the answer is an “antagonistic question.” No idea why. But without some form of internal validation, finding antagonistic questions, leading questions, and questions with more than one correct answer can be difficult to spot. Subject matter experts love their wording, even when it invalidates the question. Numbers never lie.

This post originally appeared on the Central Texas Instructional Design blog on this date.

As I mentioned last time, internal validation is a method of estimating the fairness and effectiveness of questions on a multiple choice assessment with data from the assessment itself. Note that it cannot determine the fairness and effectiveness of the assessment. That requires some form of external validation (Mehrens & Lehmann, 1973). You should not make retention or promotion decisions based solely on an assessment that has only been internally validated, but that assessment may still be of value in stack ranking learners or identifying areas where they can improve.

In this post, I give you a quick and dirty procedure for internal validation that you can perform with nothing more complicated than a PC database and spreadsheet. I used Access and Excel, but any database and spreadsheet would do.

Here’s how I structure the record:

• Student ID*—a value that differentiates individuals but does not necessarily tie to any personally identifiable information
• Assessment ID*—a value that distinguishes between the various assessments used in a curriculum and between versions of the same assessment
• Question ID*—a value that distinguishes between versions of the same question but may allow the question to be used on multiple assessments
Note: The Question ID should link to a separate table of questions that includes the text of the stem, correct answer, and distractors.
• Correct answer— the value of the correct option (may reside in an external table)
• Learner Selection—a value that identifies the option the learner chose

Fields with an asterisk are part of the key. This table links to another table that contained details about the question, including the text of the stem and options.

With this data, you can determine the statistical measurements that the Measurement and Evaluation Center of the University of Texas at Austin (2006) identifies as relevant for internal evaluation:

• Item difficulty—the percentage of learners who got the question correct
• Item discrimination—the relationship between how learners performed on the question and their overall score on the assessment
• Reliability coefficient—the margin of error in the overall score

You also have the information you need to evaluate the distractors, which may be the most useful result of this method. If you can determine why learners answer incorrectly, you can take steps—either in the learning environment or in the workplace—to correct this behavior.

In future posts, I’ll discuss how to calculate and interpret each of these measurements.

References

• Measurement and Evaluation Center (2006). Analyzing Multiple-choice Item Responses. Austin, Texas: The University of Texas. Retrieved November 16, 2008.
• Mehrens, W.A. & Lehmann, I.J. (1973). Measurement and valuation in Education and Psychology. New York: Holt, Rinehart, and Winston.

 

Seductive Distracters

This post originally appeared in the now defunct Central Texas Instructional Design blog on this date.

Back to distracters on multiple-choice assessments.

Distracters are simply incorrect options on multiple-choice assessments. To be useful, a distracter must be plausible and compelling—seductive, according to the University of St. Thomas Academic Support (n.d.). Distracters should be able to seduce learners who are uncertain of the correct answer into making an incorrect choice. At the same time, a good distracter must be thoroughly wrong, and the question of wrongness causes the most lively debates over whether a question is useful.

My rule of thumb is that if the experts argue over a distracter or question, learners will, too. I would not use any question that causes such arguments on an assessment, especially not a high-stakes assessment where a learner’s performance rating or job is on the line. There are plenty of opportunities to use these questions in the learning event. Arguable questions make excellent discussion points in face-to-face classes. You can even find creative ways to use them in online modules. Using them on an assessment only calls the validity of the assessment into question.

Assuming that a distracter is inarguably wrong, what makes it seductive? Let’s examine some example questions to identify their traits. The first example comes from the written portion of the test I took to obtain my Texas drivers license.

What does this sign mean?

• Edge of road
• Slow moving vehicle
• Stop for road-side barber shop

I still remember this question after all these years (I won’t say how many here) because it embarrassed me by making me laugh out loud while taking the test. The test writer probably intended to introduce a little levity with that last distracter. It worked, but a test is not the place for humor. Assuming that I didn’t know that the sign in question marked the edge of the pavement and was not familiar with the placard placed on slow moving vehicles, the humorous distracter improved my chances of guessing correctly by about 17%. It simply was not a plausible distracter.

You can find plausible distracters during the needs analysis or gap analysis. Corporate training usually addresses some performance gap or seeks to change a behavior. The best distracters come from what you are trying to teach people not to do. Here are a few examples of what I mean:

• If a number of people doing a job engage in behaviors they should avoid—such as interrupting a customer—those common misbehaviors are natural, plausible distracters on questions asking for the correct behavior.
• Similarly, if policies change, the old policy (which was once the correct answer) provides a plausible distracter.
• Applications can also provide plausible distracters. If an application provides a drop-down menu of choices to make based on situation, any of the choices that are not appropriate for the situation described in the stem make excellent distracters.
• Common sense also provides plausible distracters. Last month I mentioned an application that used color coding in a non-intuitive manner. In this case, choices listed in red were to be offered to customers when green or yellow choices were not appropriate, but employees never offered their customers red choices. If that client had not been willing or able to change their color coding, “Never mention this to a customer” would have been a compelling and plausible distracter to a question about the meaning of red choices in the application.

Here is an example of a question developed for one of my clients. The question passed all reviews but was not selected to be on an assessment. Some of the necessary context to answer this question, namely the product being trained, is absent, but you can still see what makes the distracters compelling.

What does it mean when the LED in the Wireless switch is flashing blue?

• The system has connected to a weak signal source.
• The system is communicating with a Bluetooth signal source.
• The system is communicating with a strong signal source.
• The system is searching for a signal.

Lets review each of the options as if they were all distracters.

• The first is plausible, if not compelling, because a weak signal source can be sporadic. The learner might interpret the blinking light as connecting and disconnecting to the source.
• The second is plausible because the learner might think that the blue LED indicates Bluetooth. Blinking also indicates traffic on some network adapters.
• The third is probably the least plausible of the options. It relies only on the assumption that the blinking light indicates traffic.
• This option is plausible because many network adapters have two LEDs. One that indicates connection when solid and one that indicates traffic. In this case, the assumption is that the blue LED is the one that indicates connection rather than traffic.

You probably noticed that all the examples are in the cognitive domain. They assess what a learner knows. Multiple-choice assessments are particularly suited to the cognitive domain, but they are not so applicable to other domains. For those domains, we need other types of assessment.

To sum up, I like to say that anyone can tell a really bad question. Only your learners can tell a good question, and then only if you have their performance data. I’ll talk about that soon.

References

• University of St. Thomas Academic Support. (n.d.). Multiple choice strategies for psychology tests. Retrieved May 15, 2008.

Punctuation Interlude

If you think quotation marks add emphasis, you really, really need to read today’s post.

This post originally appeared in the now-defunct Central Texas Instructional Design blog on this date.

Let’s take a break from multiple choice questions to revisit punctuation—quotation marks, in particular. When not setting off quoted text, what do they do?

One of my clients asked for my opinion on a marketing brochure. Marketing isn’t my specialty, but I agreed to look it over. One of the first things I noticed was an abundance of quotation marks bracketing single words and a couple of phrases. I asked, “When describing your new product, why is new in quotation marks? Why are so many other words in quotation marks?”

“I want to emphasize those words. The quotation marks will call the reader’s attention to those words.”

True enough, but what kind of attention do quotation marks draw? I told a story that one of my undergrad technical writing profs related in class. I wish I could take credit for it, but here is a short version of the story:

An editor and a writer were arguing over the use of quotation marks in a headline. The editor said they emphasize the meaning of the quoted word. The writer said they call the word into question. After much arguing, the writer said, “I can make my point if you give me control over tomorrow’s headline.” The overly-confident editor agreed. The next day, the banner headline read:

Editor Seen Leaving Motel with “Wife”

The next draft of the marketing brochure I saw had replaced the quotation marks with boldface type.

The Number of Distracters

A test-taker’s chance of guessing the right answer to a well-formed question is inversely proportional to the number of distracters.

This post appeared in the now-defunct Central Texas Instructional Design blog on this date.

Distracters are opportunities to choose incorrectly on a multiple choice assessment. The more distracters a question has, the less likely a correct answer results from a lucky guess. I mentioned last time that most of the companies I work with have standardized on a using four options (three distracters and one correct answer). Assessments in higher education frequently use an extra distracter to reduce the chance of guessing (CERNet, n.d.).

So, if having more options makes guessing harder, why standardize on four options?

• Four options is the point of diminishing returns.
• Writing good distracters is difficult.

Most of us are familiar with the point of diminishing returns from Economics 101. For a certain amount of work, we derive a certain benefit. At some point, we hit the point where there is not enough additional benefit to justify the additional work. The following chart and table show the returns on the work of writing additional distracters.

Number of Distracters	Chance of Guessing	Difference
1	50.000%
2	33.333%	16.667%
3	25.000%	8.333%
4	20.000%	5.000%
5	16.667%	3.333%
6	14.286%	2.381%
7	12.500%	1.786%

You can see that learners have a 50% chance of guessing correctly on a True/False question or a multiple choice question with only two options and one correct answer. Adding a third option reduces their chances of guessing by 16.7%, rounded. Going from four to five may still be worth the additional effort. But by the time you get to six options, the gain in accuracy is probably not worth the effort.

So, what is the correct number of options? It depends on the question and what the options are. When I took the written assessment for my driver’s license, one of the questions asked about the meaning of that little stripped sign you sometimes see at the roadside. One of the distracters was, “Stop for roadside barber shop.” My guess is that that distracter was never chosen, and its only reason for existence was to meet the magical number of required distracters. Since this distracter does not really distract, this was essentially a three-option question.

To sum up, there is no magic number of distracters. Remember that distracters should be “seductive alternatives” (University of St. Thomas Academic Support). They should “compellingly and confusingly” attract the test taker (Randall, 2003). If there are only two plausible alternatives, don’t waste your time trying to come up with two more implausible distracters that probably won’t have any effect on the outcome of the assessment.

Coming soon: Seductive Distracters

References

• Croation Academic and Research Network. (n.d.). 10 golden rules for writing multiple choice questions. Retrieved May 15, 2008, from Self-assessment and summative assessment in e-education.
• Randall, V. (2003, May 17). Recognizing distracters and foils. Retrieved May 15, 2008, from Passing the Bar.
• University of St. Thomas Academic Support. (n.d.). Multiple choice strategies for psychology tests. Retrieved May 15, 2008.