The authors assess the suitability of repeated-measures regression (RMR) to analyze multilevel data in four popular multilevel research designs by comparing results of RMR analyses to results of analyses using techniques known to produce correct results in these designs. The findings indicate that RMR may be suitable for only a small number of situations and that repeated-measures ANOVA, multivariate repeated-measures ANOVA, and multilevel modeling may be better suited to analyze multilevel data under most circumstances. The authors conclude by offering recommendations regarding the appropriateness of the different techniques given the different research designs.
Algina, J., & Keselman, H. J. (1997). Detecting repeated measures effects with univariate and multivariate statistics. Psychological Methods, 2, 208-218.
2.
Beersma, B., Hollenbeck, J. R., Humphrey, S. E., Moon, H., Conlon, D. E., & Ilgen, D. R. (2003). Cooperation, competition, and team performance: Toward a contingency approach. Academy of Management Journal, 46, 572-590.
3.
Bergh, D. D. (1995). Problems with repeated measures analysis: Demonstration with a study of the diversification and performance relationship. Academy of Management Journal, 38, 1692-1708.
4.
Berkhof, J., & Snidgers, T. A. B. (2001). Variance component testing in multi-level models. Journal of Educational and Behavioral Statistics, 26, 133-152.
5.
Boik, R. J. (1981). A priori tests in repeated measures designs: Effects of nonsphericity. Psychometrika, 46, 241-255.
6.
Box, G. E. P. (1954). Some theorems on quadratic forms applied in the analysis of variance: II. Effects of inequality of variance and correlation between errors in the two-way classification. Annals of Mathematical Statistics, 25, 484-498.
7.
Chan, D. (1998a). The conceptualization and analysis of change over time: An integrative approach incorporating longitudinal mean and covariance structures analysis (LMACS) and multiple indicator latent growth modeling (MLGM). Organizational Research Methods, 1, 421-483.
8.
Chan, D. (1998b). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
9.
Chan, D. (2002). Longitudinal modeling. In S. G. Roselberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 412-430). Malden, MA: Blackwell.
10.
Chan, D., Ramey, S.Ramey, C., & Schmitt, N. (2000). Modeling intraindividual changes in children’s social skills at home and at school: A multivariate latent growth approach to understanding between-settings differences in children’s social skill development. Multivariate Behavioral Research, 35, 365-396.
11.
Chan, D., & Schmitt, N. (2000). Interindividual differences in intraindivual changes in proactivity during organizational entry: A latent growth modeling approach to understanding newcomer adaptation. Journal of Applied Psychology, 85, 190-210.
12.
Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlational analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
13.
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
14.
Gully, S. M. (1994, April). Repeated measures regression analyses: A clarification with illustrative examples. Paper presented at the ninth annual conference of the Society for Industrial and Organizational Psychology, Nashville, TN.
15.
Gully, S. M. (1997, April). Effect size, theory, and tests of significance in repeated measures and multi-level analyses. Paper presented at the 12th annual conference of the Society for Industrial and Organizational Psychology, St. Louis, MO.
16.
Hofmann, D. A., Griffin, M. A., & Gavin, M. B. (2000). The application of hierarchical linear modeling to organizational research. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multi-level theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 467-511). San Francisco: Jossey-Bass.
17.
Hollenbeck, J. R., Ilgen, D. R., LePine, J. A., Colquitt, J. A., & Hedlund, J. (1998). Extending the multi-level theory of team decision making: Effects of feedback and experience in hierarchical teams. Academy of Management Journal, 41, 269-282.
18.
Hollenbeck, J. R., Ilgen, D. R., & Sego, D. J. (1994). Repeated measures regression and mediation tests: Enhancing the power of leadership research. Leadership Quarterly, 5, 3-23.
19.
Hollenbeck, J. R., Ilgen, D. R., Sego, D. J., Hedlund, J., Major, D. A., & Phillips, J. (1995). Multi-level theory of team decision making: Decision making in teams incorporating distributed expertise. Journal of Applied Psychology, 80, 292-316.
20.
Hox, J. (2002). Multilevel analysis techniques and applications. Hillsdale, NJ: Lawrence Erlbaum.
21.
Huynh, H., & Feldt, L. S. (1976). Estimation of the Box correction for degrees of freedom from sample data in randomized block and split-plot designs. Journal of Educational Statistics, 1, 69-82.
22.
Keselman, H. J., Algina, J., & Kowalchuk, R. K. (2001). The analysis of repeated measures designs: A review. British Journal of Mathematical and Statistical Psychology, 54, 1-20.
23.
Keselman, H. J., Algina, J., Kowalchuk, R. K., & Wolfinger, R. D. (1999). A comparison of recent approaches to the analysis of repeated measurements. British Journal of Mathematical and Statistical Psychology, 52, 63-78.
24.
Keselman, H. J., Rogan, J. C., Mendoza, J. L., & Breen, L. J. (1980). Testing the validity of conditions of repeated measures F tests. Psychological Bulletin, 87, 479-481.
25.
Klein, K. J., Bliese, P. D., Kozlowski, S. W. J., Dansereau, F., Gavin, M. D., Griffin, M. A., et al. (2000). Multilevel analytical techniques: Commonalities, differences, and continuing questions. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multi-level theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 512-553). San Francisco: Jossey-Bass.
26.
Klein, K. J., & Kozlowski, S. W. J. (Eds.). (2000). Multi-level theory, research, and methods in organizations: Foundations, extensions, and new directions. San Francisco: Jossey-Bass.
27.
Klein, K. J., Tosi, H., & Cannella, A., Jr. (1999). Multi-level theory building: Benefits, barriers, and new developments. Academy of Management Review, 24, 243-248.
28.
LePine, J. A., Colquitt, J. A., & Erez, A. (2000). Adaptability to changing task contexts: Effects of general cognitive ability, conscientiousness, and openness to experience. Personnel Psychology, 53, 563-593.
29.
LePine, J. A., & Van Dyne, L. (1998). Predicting voice behavior in work groups. Journal of Applied Psychology, 83, 853-868.
30.
Lorch, R. F., Jr., & Myers, J. L. (1990). Regression analyses of repeated measures data in cognitive research. Journal of Experimental Psychology, 16, 149-157.
31.
Marks, M. A., Zaccaro, S. J., & Mathieu, J. E. (2000). Performance implications of leader briefings and team-interaction training for team adaptation to novel environments. Journal of Applied Psychology, 85, 971-986.
32.
McCulloch, C. E., & Searle, S. R. (2001). Generalized, linear, and mixed models. New York: John Wiley.
33.
Padilla, M. A., & Algina, J. (2004). Type I error rates for a one factor within-subjects design with missing values. Journal of Modern Applied Statistical Methods, 3, 406-416.
34.
Pedhazur, E. J. (1982). Multiple regression in behavioral research (2nd ed.). Fort Worth, TX: Holt, Rinehart & Winston.
35.
Phillips, J. M. (2001). The role of decision influence and team performance in member self-efficacy, withdrawal, satisfaction with the leader, and willingness to return. Organizational Behavior and Human Decision Processes, 84, 122-147.
36.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models applications and data analysis methods. Thousand Oaks, CA: Sage.
37.
Rousseau, D. M. (1985). Issues of level in organizational research: Multi-level and cross level perspectives. In L. L. Cummings & B. M. Staw (Eds.), Research in organizational behavior (Vol. 7, pp. 1-38). Greenwich, CT: JAI.
38.
Ryan, A. M., McFarland, L., Baron, H., & Page, R. (1999). An international look at selection practices: Nation and culture as explanations for variability in practice. Personnel Psychology, 52, 359-391.
39.
Schmitt, N., Sacco, J. M., Ramey, S., Ramey, C., & Chan, D. (1999). Parental employment, school climate, and children’s academic and social development. Journal of Applied Psychology, 84, 737-753.
40.
Shah, B. V., Barnwell, B. G., & Bieler, G. S. (1997). SUDAAN user’s manual, release 7.5. Research Triangle Park, NC: Research Triangle Institute.
41.
Wolfinger, R. D., & Chang, M. (1995). Comparing the SAS GLM and MIXED procedures for repeated measures. SAS User’s Group International Conference Proceeding SUGI, 20, 1172-1182.
42.
Zeger, S., & Liang, K. (1986). Longitudinal data analysis for discrete and continuous outcomes, Biometrics, 42, 121-130.
43.
Zeger, S. L., Liang, K. Y., & Albert, P. S. (1988). Models for longitudinal data: A generalized estimating equation approach. Biometrics, 44, 1049-1060.
