Outliers in questionnaire data are unusual observations, which may bias statistical results, and outlier statistics may be used to detect such outliers. The authors investigated the effect outliers have on the specificity and the sensitivity of each of six different outlier statistics. The Mahalanobis distance and the item-pair based outlier statistics were found to have the best combination of specificity and sensitivity. Next, it was investigated how outliers influenced the bias in the percentile rank score, Cronbach’s alpha, and the validity coefficient. Outliers due to random responding and faking produced considerable bias, and outliers due to extreme responding produced little bias. Finally, the influence of removing discordant observations on bias was studied. Removing observations due to random responding identified by means of the Mahalanobis distance, the local outlier factor, and the item-pair based outlier statistic reduced bias.
AtkinsonA. C.RianiM.
(2000). Robust diagnostic regression analysis.
New York, NY:
Springer.
2.
AustinE. J.DearyI. J.GibsonG. J.McGregorM. J.DentJ. B.
(1998).
Individual response spread in self-reported scales: Personality correlations and consequences.
Personality and Individual Differences,
24,
421–438.
3.
BachmanJ. G.O’MalleyP. M.
(1984).
Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles.
Public Opinion Quarterly,
48,
491–509.
4.
BarnettV.LewisT.
(1994). Outliers in statistical data.
(3rd ed.).
New York, NY:
John Wiley.
5.
BarnetteJ. J.
(1999).
Nonattending respondent effects on internal consistency of self-administered surveys: A Monte Carlo simulation study.
Educational and Psychological Measurement,
59,
38–46.
6.
BaumeisterR. F.TiceD. M.
(1988).
Metatraits.
Journal of Personality,
56,
571–598.
ChatterjeeS.HadiA. S.
(1986).
Influential observations, high leverage points, and outliers in linear regression.
Statistical Science,
1,
379–416.
9.
ChristmannA.Van AelstS.
(2006).
Robust estimation of Cronbach’s alpha.
Journal of Multivariate Analysis,
97,
1660–1674.
10.
CohenJ.
(1988). Statistical power analysis for the behavioral sciences
(2nd ed.).
Hillsdale, NJ:
Lawrence Erlbaum.
11.
CostaP. T.McCraeR. R.
(1992). The NEO personality inventory and the NEO five factor inventory professional manual.
Odessa, FL:
Psychological Assessment Resources.
12.
CronbachL. J.
(1951).
Coefficient alpha and the internal structure of tests.
Psychometrika,
16,
297–334.
13.
DenolletJ.
(1994).
Health complaints and outcome assessment in coronary heart disease.
Psychosomatic Medicine,
56,
463–474.
14.
DenolletJ.
(2005).
DS14: Standard assessment of negative affectivity, social inhibition, and type D personality.
Psychosomatic Medicine,
67,
89–97.
15.
DevlinS. J.GnanadesikanR.KettenringJ. R.
(1975).
Robust estimation and outlier detection with correlational coefficients.
Biometrika,
62,
531–545.
16.
EmbretsonS. E.ReiseS. P.
(2000). Item response theory for psychologists.
Mahwah, NJ:
Lawrence Erlbaum.
17.
EmonsW. H. M.
(2008).
Person-fit analysis of polytomous items.
Applied Psychological Measurement,
32,
224–247.
18.
GuttmanL.
(1950).
The basis for scalogram analysis. In
StoufferS. A.GuttmanL.SuchmanE. A.LazarsfeldP. F.StarS. A.ClausenJ. A. (Eds.), Measurement and prediction (pp. 60–90).Princeton, NJ:
Princeton University Press.
19.
HampelF. R.RonchettiE. M.RousseeuwP. J.StahelW. A.
(1986). Robust statistics: The approach based on influence functions.
New York, NY:
John Wiley.
20.
HawkinsD. M.
(1980). Identification of outliers.
London, England:
Chapman and Hall.
21.
KaufmanL.RousseeuwP. J.
(2005). Finding groups in data: An introduction to cluster analysis.
Hoboken, NJ:
John Wiley.
22.
MahalanobisP. C.
(1936).
On the generalized distance in statistics.
Proceedings of the National Institute of Sciences of India,
249–55.
23.
MaxwellS. E.DelaneyH. D.
(1990). Designing experiments and analyzing data: A model comparison perspective.
Belmont, CA:
Wadsworth.
24.
McFarlandL. A.RyanA. M.
(2000).
Variance in faking across noncognitive measures.
Journal of Applied Psychology,
85,
812–821.
25.
MeijerR. R.
(1994).
The number of Guttman errors as a simple and powerful person-fit statistic.
Applied Psychological Measurement,
18,
311–314.
26.
MeijerR. R.SijtsmaK.
(2001).
Methodology review: Evaluating person fit.
Applied Psychological Measurement,
25,
107–135.
27.
MischelW.
(1968). Personality and assessment.
New York, NY:
John Wiley.
28.
MolenaarI. W.
(1991).
A weighted Loevinger H-coefficient extending Mokken scaling to multicategory items.
Kwantitatieve Methoden,
12
(37),
97–117.
29.
MolenaarI. W.
(1997).
Nonparametric models for polytomous responses. In
van der LindenW. J.HambletonR. K. (Eds.), Handbook of item response theory (pp. 369–380). New York, NY:
Springer.
30.
NunnallyJ. C.BernsteinI. H.
(1994). Psychometric theory
(3rd ed.).
New York, NY:
McGraw-Hill.
31.
RosnerB.
(1983).
Percentage points for a generalized ESD many-outlier procedure.
Technometrics,
25,
165–172.
32.
RousseeuwP. J.LeroyA. M.
(2003). Robust regression and outlier detection.
Hoboken, NJ:
John Wiley.
33.
SamejimaF.
(1997).
Graded response model. In
van der LindenW. J.HambletonR. K. (Eds.), Handbook of item response theory (pp. 85–100). New York, NY:
Springer.
34.
SelvinS.
(2004). Statistical analysis of epidemiological data.
New York, NY:
Oxford University Press.
35.
TukeyJ. W.
(1977). Exploratory data analysis.
Reading, MA:
Addison-Wesley.
36.
Van der LindenW. J.HambletonR. K.
(1997). Handbook of item response theory.
New York, NY:
Springer.
37.
Van GinkelJ. R.Van der ArkL. A.
(2005).
SPSS syntax for missing value imputation in test and questionnaire data.
Applied Psychological Measurement,
29,
152–153.
38.
YickJ. S.LeeA. H.
(1998).
Unmasking outliers in two-way contingency tables.
Computational Statistics & Data Analysis,
29,
69–79.
39.
ZickarM. J.DrasgowF.
(1996).
Detecting faking on a personality instrument using appropriateness measurement.
Applied Psychological Measurement,
20,
71–87.
40.
ZijlstraW. P.Van der ArkL. A.SijtsmaK.
(2007).
Outlier detection in test and questionnaire data.
Multivariate Behavioral Research,
42,
531–555.
41.
ZijlstraW. P.Van der ArkL. A.SijtsmaK. (2010). Discordancy tests for outlier detection in multi-item surveys. Manuscript submitted for publication.
