The Probability of Obtaining Two Statistically Different Test Scores as a Test Index

Abstract

A new test index is defined as the probability of obtaining two randomly selected test scores (PDTS) as statistically different. After giving a concept definition of the test index, two simulation studies are presented. The first analyzes the influence of the distribution of test scores, test reliability, and sample size on PDTS within classical test theory. The second simulation manipulates the number of items and their locations over the latent continuum for a one-parameter logistic model. In addition, comparisons of both applications on the same data are made. The discussion focuses on advantages of applying PDTS for dimensional constructs in classical test theory and item response theory, as well as the disadvantages of making no assumptions about the test score distribution. The simulation studies provide the initial findings about the PDTS under several given empirical conditions. The discussion emphasizes why and when it is useful to consider PDTS in a test description.