The Use of Random Coefficient Modeling for Understanding and Predicting Job Performance Ratings

Abstract

Earlier research using confirmatory factor analysis (CFA) suggests that most variance in job performance ratings is not attributable to ratee main effects. In this article, the authors point out several issues associated with CFA methodology and argue that random coefficient modeling (RCM) can be a useful alternative for estimating variances associated with ratee main effects, rater main effects, and the upper bound of Rater × Ratee interaction effects. Using an application of RCM on field data, the authors found that rater main effects variance was nearly two times as large as ratee main effects variance. They report meaningful contingencies of these findings by modeling rater familiarity with the ratee and the number of ratees rated by a rater. Finally, interactions revealed that Conscientiousness-related variables were positively related to job performance only when rater familiarity with the ratee was high or the number of ratees rated was high. The authors discuss how the RCM methodology can be used to assess the construct validity of job performance ratings and to test substantive hypotheses involving variance components, main effects, and interactions within nonindependent observations.