The effect of Sample Size on Differential Item Functioning and Differential Distractor Functioning in multiple choice items

Abstract

The current study investigated the effect of sample size on the number of items that show differential functioning (DIF) and the number of distractors that also show differential functioning (DDF) using the Mantel-Haenszel procedure. Data came from a national 8th grade mathematics exam that is composed of 40 multiple-choice items that was administered to 40,000 examinees. Eight samples with 250, 500 1250, 2500, 5000, 10000, 15000, and 20000 examinees were randomly selected. The findings of the current study indicated that increasing sample size increased the number of items detected with DIF and DDF. In addition, larger sample sizes are needed to detect items with nonuniform DIF and with negligible magnitude of DIF. Moreover, detecting DDF requires larger sample sizes as compared to the detection of DIF. Finally, sample size of 2,500 provided adequate number of items flagged with DIF (both types, and different magnitudes) and with DDF