26 Item Analysis

Module 38.3 covered topics that rely on statistical analyses of data from educational and psychological measurements. These analyses are used to examine the relationships among scores on one or more test forms, in reliability, and scores based on ratings from two or more judges, in interrater reliability. Aside from coefficient alpha, all of the statistical analyses introduced so far focus on composite scores. Item analysis focuses instead on statistical analysis of the items themselves that make up these composites.

As previously noted, test items make up the most basic building blocks of an assessment instrument. Item analysis lets us investigate the quality of these individual building blocks, including in terms of how well they contribute to the whole and improve the validity of our measurement.

This module extends concepts developed early in the course to analysis of item performance within a CTT framework. The module begins with an overview of item analysis, including some general guidelines for preparing for an item analysis, entering data, and assigning score values to individual items. Some commonly used item statistics are then introduced and demonstrated. Finally, two additional item-level analyses are discussed, differential item functioning analysis and option analysis.

26.1 Objectives

Learning objectives

  1. Explain how item bias and measurement error negatively impact the quality of an item, and how item analysis, in general, can be used to address these issues.
  2. Describe general guidelines for collecting pilot data for item analysis, including how following these guidelines can improve item analysis results.
  3. Identify items that may have been keyed or scored incorrectly.
  4. Rescore variables to reverse their scoring or keyed direction.
  5. Use the appropriate terms to describe the process of item analysis with cognitive vs noncognitive constructs.
  6. Calculate and interpret item difficulties and compare items in terms of difficulty.
  7. Calculate and interpret item discrimination indices, and describe what they represent and how they are used in item analysis.
  8. Describe the relationship between item difficulty and item discrimination and identify the practical implications of this relationship.
  9. Calculate and interpret alpha-if-item-removed.
  10. Utilize item analysis to distinguish between items that function well in a set and items that do not.
  11. Remove items from an item set to achieve a target level of reliability.
  12. Evaluate selected-response options using option analysis.

In this module, we’ll run item and option analyses on PISA09 data using epmr, with results plotted, as usual, using ggplot2.

# R setup for this module
# Required packages are assumed to be installed,

library("epmr")
library("ggplot2")

# Functions we'll use in this module
# str() for checking the structure of an object
# rescore() for recoding variables
# colMeans() for getting means by column
# istudy() from epmr for running an item analysis
# ostudy() from epmr for running an option analysis