Internal Validation on the Cheap: R(i)

This post originally appeared on the Central Texas Instructional Design blog on this date—or it would have if I had ever finished it.

ELAB 2019-03-01

Internal validation is a way to ensure multiple choice assessments are worth the time it takes to administer them. Without at least performing some kind of internal validation, an assessment isn’t anything more than a stressful method of tracking attendance.

My quick and dirty (cheap) method of internal validation uses the database described in the part 1 of this series, “Validating Multiple Choice Assessments on the Cheap.”

You also need a table that tracks the scores of each learner and ties to the previously described database. Here’s how I structure the record for this linked database:

1. Student ID*—a value that differentiates individuals but does not necessarily tie to any personally identifiable information
2. Assessment ID*—a value that distinguishes between the various assessments used in a curriculum and between versions of the same assessment
3. Score—what the learner scored on the assessment
4. Subgroup—the subgroup to which the learner belongs (more on this in a few minutes)

Fields marked with an asterisk tie this table to the previously described one.

The first step in the validation process is to divide the entire population into three roughly equal groups. These groups classify learners by how well they performed on the assessment. You can name the groups whatever you want. I usually number them to make sorting easier.

Formulas for this division exist, but unless you require a rigorous statistical methodology I prefer a simple method. Besides, smaller assessments don’t have a wide enough point spread to fine tune the size of the groups. So here’s what I do.

1. Calculate the mean score and standard deviation for the assessment. (Any spread sheet has built in functions for these operations.)
2. Place anyone whose score falls within ½-standard deviation of the mean in the middle group.
3. Place anyone whose score is more than ½-standard deviation above the mean in the upper group.
4. Place anyone whose score is more than ½-standard deviation below the mean in the lower group.

The size of the groups can differ significantly using this method. You can tweak the threshold scores a little, but small assessments and the tendency of scores to clump often make tweaking pointless.

With the learners assigned to groups, it’s time to use a database application such as Access or MySQL to help you analyze it. I usually generate a report defined something like this:

This is where I stopped. The report I created for the Fruit Company usually runs in excess of 15 pages, so I have no idea how I was going to simplify it.