When Do Interaction/Moderation Effects Stabilize in Linear Regression?

Abstract

Two-way interaction effects in linear regression occur when the relation between two variables changes depending on the level of a third. Despite their frequent use, interactions are notoriously difficult to estimate accurately and test for statistical significance because of small effect sizes and low reliability. In this study, we used Monte Carlo simulations to establish stability thresholds for two-way interactions between continuous variables across combinations of reliability (0.7–1.0), main effect size (0.1–0.5), collinearity (0.1–0.5), and interaction effect size (0.05–0.2). Stability was defined as the consistency of estimated effect sizes across repeated samples of the same size from the same population and operationalized using modified definitions of the corridor of stability and point of stability from Schönbrodt and Perugini. Results show that the stability of interaction estimates is primarily determined by sample size and predictor reliability. The case representing a realistic psychology field study, in which researchers have limited control over variables, stabilized at n = 3,800, requiring 72% statistical power. At n ≤ 100, 11% to 45% of the estimates were incorrectly signed (i.e., negative when the true effect was positive). Most psychology studies enroll far fewer than 500 participants, and our results indicate many published interactions may be unstable. Analyses involving highly reliable predictors, such as group assignment in experimental designs, may stabilize at lower sample sizes because they attenuate the expected effect size less than variables with more measurement error. Researchers are encouraged to avoid routine tests of two-way interactions unless sample size and reliability are adequate and hypotheses are specified a priori.

Interaction or moderation effects in linear regression occur when the relation between two variables changes depending on the level of a third variable. ¹ Interaction analyses are commonplace in the social sciences, in which researchers often seek to understand how the influence of one variable may change in different contexts or conditions. Interaction analyses are used to test a wide range of questions, including but not limited to gene–environment (Plomin et al., 1977), person–situation (Lewin et al., 1936), aptitude–treatment (Cronbach, 1957; Cronbach & Snow, 1981), and society–individual (Blumer, 1986) interactions, among many others. A significant interaction indicates that the relationship between the predictors and the outcome variable cannot be fully explained by the main effects alone because the predictors’ effects are interdependent in a nonadditive way (Jaccard & Turrisi, 2003). Interactions introduce unique complexities that can make them challenging to detect (MacKinnon, 2011), especially in nonexperimental or naturalistic investigations in which researchers have limited control over variables (i.e., field studies; McClelland & Judd, 1993). Some researchers use 80% power as a stopping rule for detecting interactions, but in practice, many reported interactions are underpowered. Power analyses are often omitted or conducted using conventional benchmarks that assume unrealistically large effect sizes for interactions. As a result, the stability of interaction effects, defined as the consistency of their estimated effect size across repeated samples, is frequently uncertain even when statistical significance is obtained.

In the context of null hypothesis significance testing, “power” refers to the probability of correctly detecting an effect if it exists. There are two primary issues with power in interaction effects: their typically small effect sizes (Aguinis et al., 2005; Vize, Sharpe, et al., 2023)and low reliability (Busemeyer & Jones, 1983). ² Beyond the normatively small effect sizes of interactions, they can be obscured by restricted ranges of predictors (McClelland & Judd, 1993), the size and collinearity of the main effects (Baranger et al., 2023; Dormann et al., 2013), and unequal cell sizes in categorical moderators (Frazier et al., 2004). The reliability of the interaction term is approximately the product of the reliabilities of its components, which can further attenuate effect size. As a consequence, large sample sizes tend to be necessary to detect interactions (Hyatt et al., 2022; Lakens, 2019; Vize, Baranger, et al., 2023). Even seasoned researchers easily fall into false dichotomies around statistical significance when interpreting interaction effects (McShane & Gal, 2017), yet many studies in the extant literature do not sufficiently power for interaction analyses.

Publication bias casts doubt on the understanding of published interactions (Sotola & Credé, 2023; Ferguson & Brannick, 2012). The number of significant interaction effects is likely overstated because of the selective reporting of exaggerated effect sizes (Simmons et al., 2011), failure to correct for multiple comparisons (see Pease & Lewis, 2015), and severe underpowering (see Jensen-Campbell et al., 2007; see Ode et al., 2008). Recent z-curve analyses of health-psychology journals confirm this pattern, in which researchers estimated replication rates below 50% and showed that for every significant interaction reported, nearly two nonsignificant results go unreported (Fremling et al., 2025). A meta-analysis found that only 22% of interaction analyses replicate in psychology (Open Science Collaboration, 2015), and similarly poor replicability exists in other fields (Credé & Sotola, 2024; Greenland, 1993). Some have even urged researchers to “mend it” or “end it” regarding the search for moderators (Murphy & Russell, 2017).

One factor to consider when designing interaction analyses is stability, defined as the consistency of estimated effect sizes across repeated samples of the same size from the same population. Larger samples yield tighter clusters of estimates and thus greater stability. Whereas standard measures of spread (e.g., standard errors, standard deviations, and confidence intervals [CIs]) describe uncertainty for a single analytic result, stability is about the analytic design of the study and reflects how reliably a given design reproduces estimates of effects.

“Power” and “stability” both describe the expected performance of a design rather than outcomes from a single data set, but they capture different features of that performance. Power pertains to the performance of decision rules in hypothesis testing: the probability of rejecting a null hypothesis at a chosen alpha when an interaction truly exists. Stability pertains to estimate reliability: the degree to which identically designed studies yield similar estimates of effects. Unlike power or confidence-interval coverage, stability does not depend on alpha or a specific test; it is purely about the spread of estimates generated by a study design across repeated simulations. Interactions are particularly prone to artifacts (Gelman, 2023; McClelland & Judd, 1993; Murphy & Russell, 2017; Rimpler et al., 2025), and conflating power or confidence-interval coverage with stability can lead researchers to draw theoretical conclusions from effects that are statistically significant yet unlikely to replicate.

These replication failures point to a deeper issue with conventional power analyses. Power is the probability of rejecting a null hypothesis at a chosen alpha when an interaction truly exists but does so based on assumed effect sizes. When those assumed effects are exaggerated because of publication bias, selective reporting, or unrealistic benchmarks, a study can be “powered” on paper yet still yield misleading results. In addition, finding that an effect differs from zero is not the same as estimating its magnitude accurately. For science, the critical question is not whether an effect exists but whether its size can be estimated with sufficient consistency and accuracy across samples. This is the question stability answers directly.

A well-known framework for quantifying stability was introduced by Schönbrodt and Perugini (2013) in the context of correlations. ³ Instead of focusing on the CIs around a single estimate, they used simulations to determine the sample size at which repeated estimates fell within a predefined range of the true value. They simulated a population correlation and drew 100,000 bootstrapped samples across a range of sample sizes (ns = 20–1,000). In each sample, they estimated the correlation between the two variables and traced the correlation values as a function of sample size to form trajectories. The number of trajectories falling within a fixed-width corridor around the population parameter, called the “corridor of stability” (COS), ⁴ was used to find the point of stability (POS), the smallest sample size at which a given percentage (80%, 90%, or 95%) of trajectories remained within the COS. For example, a correlation of r = .1 stabilized at n = 252 with 80% of estimates in the COS. At this sample size, there is approximately 38% power to detect the effect. ⁵ For $\mathrm{\rho }$ = .1, two of their three COS bounds permitted sign errors, in which effects could reverse direction and still be considered stable. Converting their COS bounds to percentages, we found that their narrowest COS allowed for these small correlation estimates to deviate by as much as 97% from the expected value; for their widest COS, estimates were permitted to deviate from the expected value by as much as 292% ⁶ for the smallest effect size they examined (r = .1) and still be regarded as stable. Although modest absolute deviations are informative for larger effects, they translate poorly to small effects. This is particularly problematic for interaction analyses, in which almost all empirically observed effect sizes in the psychological sciences have effect sizes of r ≤.1 (Murphy & Russell, 2017; Open Science Collaboration, 2015).

Although Schönbrodt and Perugini (2013) evaluated correlations between perfectly measured observed variables, research evaluating correlations between imperfectly measured latent variables has shown that such correlations require even larger sample sizes to stabilize because of unreliability. Kretzschmar and Gignac (2019) conducted Monte Carlo simulations similar to Schönbrodt and Perugini and used McDonald’s omega to introduce unreliability into latent-variable correlations. Although a latent-variable correlation of $\mathrm{\rho }=0.2$ stabilized at $n=220$ with perfect reliability, when McDonald’s $\mathrm{\omega }$ was simulated at .7, the requisite sample size for stability more than doubled to $n=490$ ; with $\mathrm{\omega }=0.4$ , this jumped to $n=1,340$ . Perfect reliability in psychological instruments is exceptionally rare, and most observed relations among psychological constructs are attenuated by the unreliability of their measures. The conclusions of Schönbrodt and Perugini regarding sample size are important for understanding stability but may not generalize to many applied scenarios. Interaction effects are particularly susceptible to low reliability because their reliability is approximately the product of the reliabilities of the main effects (Busemeyer & Jones, 1983). Coupled with their tendency toward small effect sizes (Vize, Sharpe, et al., 2023), interactions examined with linear regression possess unique complexities that may affect their stability, which warrants additional investigation.

In this study, we used Monte Carlo simulations to evaluate the stability of interaction effects under varying conditions. We aimed to find the sample sizes, main effect sizes, and predictor collinearity values that produce stable interactions. Our approach is different from Schönbrodt and Perugini (2013) in four notable ways. First, we considered more potential conditions by including three effect sizes: that of the interaction, main effects, and the collinearity of predictors. Second, our definition of stability accounts for effect size directly through the use of percentages instead of translating effect sizes using Fisher’s r to z and adding or subtracting predetermined widths (w = 0.1, 0.15, 0.2) before back-translating with Fisher’s z to r. Third, we tested a wider range of COS widths to allow us to consider conservative and less conservative thresholds for stability. Finally, we did not trace trajectories across sample sizes from the same population and instead resampled entirely at each new sample size because we had a uniform simulated population to draw from. Given that the mean effect size for interactions is below r = .1 in empirical research (Credé & Sotola, 2024; Freese & Peterson, 2017; Open Science Collaboration, 2015), the goal of this simulation study is to illustrate the conditions necessary for stable interactions. We answer the following research question:

• Research Question: How do main effect size, intercorrelation, and reliability affect the sample size required for stable interaction-effect estimates?

Method

In this preregistered study, we used Monte Carlo simulations to evaluate the conditions required for interaction effects to stabilize. Simulations were conducted in R (R Core Team, 2024), and the full reproducible code is available in the supplemental materials (https://osf.io/zmvsf/?view_only=46e3f25d45ea4c83a33ffaef111abef9). The R package InteractionPoweR (Baranger & Castillo, 2025) is available for researchers interested in conducting similar analyses. In this article, the regression model applied is

$$Y={\mathrm{\beta }}_0+{\mathrm{\beta }}_1{X}_1+{\mathrm{\beta }}_2{X}_2+{\mathrm{\beta }}_3(X_1{X}_2)+ϵ,$$ (1)

where $X_1$ and $X_2$ represent the main effects, $X_1{X}_2$ is the interaction term, $Y$ is the outcome variable, and $ϵ$ is normally distributed random error. Simulations were performed using the generate_interaction() function from the InteractionPoweR R package (see Baranger et al., 2023). In brief, variable correlations are input and then adjusted for reliability, and regression coefficients ( ${\mathrm{\beta }}_1$ , ${\mathrm{\beta }}_2$ , and ${\mathrm{\beta }}_3$ ) are computed. The variables $X_1$ and $X_2$ are simulated by drawing from a multivariate normal distribution, the interaction term $(X_1{X}_2)$ is calculated as the product of $X_1$ and $X_2,$ and $ϵ$ is then drawn from a normal distribution. All variables are standardized to have a standard deviation of 1 and a mean of 0, and thus, ${\mathrm{\beta }}_0=0.$

Simulation parameters

Reliability

Reliability reflects the degree to which predictor values are affected by random measurement error. We considered four levels of reliability for $X_1$ and $X_2$ , (1, 0.9, 0.8, 0.7). The reliability of the interaction term $rel(X_1{X}_2)$ is distinct from the predictors (Busemeyer & Jones, 1983) and is calculated as

$$rel(X_1{X}_2)=\frac{rel\left(X_1\right)rel\left(X_2\right)+cor{(X_1,X_2)}^2}{1+cor{(X_1,X_2)}^2}$$ (2)

Effect size

We evaluated two relevant effect sizes, those of the main effects ( ${\mathrm{\beta }}_1,{\mathrm{\beta }}_2$ ) and interaction ( ${\mathrm{\beta }}_3$ ). The main effects were simulated across a range of sizes $({\mathrm{\beta }}_1={\mathrm{\beta }}_2=0.1,0.15,0.2,0.3,0.5)$ . Note that our preregistration included only main effect sizes of $0.1,0.3,$ and $0.5$ . One main effect size $(0.2)$ was added after the preregistered analysis was conducted so the simulation parameters would more directly emulate additional effect sizes common in the literature.

Both ${\mathrm{\beta }}_1$ and ${\mathrm{\beta }}_2$ were simulated with equal effect sizes to minimize potential confounds and isolate the conditions under which the interaction stabilizes. Keeping ${\mathrm{\beta }}_1$ and β₂ equal allows the interaction to reflect a true bidirectional effect rather than a dominant main effect with a secondary modifier. In addition, the main effects were simulated with differing degrees of intercorrelation. Collinearity among the predictors is a common issue in regression analysis, and we considered several values for $cor\left(X_1,X_2\right)=,$ which are 0.1, 0.3, and 0.5. Note that under assumptions of multivariate normality, the interaction term $X_1{X}_2$ is independent of $X_1$ and $X_2$ ( $cor\left(X_1,X_1{X}_2\right)=cor\left(X_2,X_1{X}_2\right)=0$ ).

The expected interaction effect size ( ${\mathrm{\beta }}_3$ ) was calculated after attenuation from the population-level true value $\mathrm{\rho }$ using the procedure described in Baranger et al. (2023). ⁷ Multiple values were selected for $\mathrm{\rho }$ to reflect the effect sizes typically observed in the literature (Aguinis et al., 2005; Vize, Baranger, et al., 2023; Vize, Sharpe, et al., 2023), which were preregistered as $\mathrm{\rho }=0.1,0.15,$ and $0.2$ and later expanded to include $0.05$ to better illustrate the low end of common interaction effect sizes. Each combination of parameters attenuates $\mathrm{\rho }$ to a unique expected value $({\mathrm{\beta }}_3).$ For a complete table of parameter combinations, see the Supplemental Material available online.

Sample sizes $(n)$

Although the original preregistered approach was to examine sample sizes ranging from $n=5$ to $n=1,500$ in increments of $5,$ we identified a more efficient way to achieve our aims using a search algorithm. Data sets were generated at the median of a specified range, which then iteratively narrows until the outcome is identified. In our simulations, the range begins with $n=5$ up to a maximum of n = 350,000, and the changes reduced computational costs by several orders of magnitude. In addition, for illustrative purposes, we simulated data at standard sample-size benchmarks of interest $\left(n=100,200,300,400,500,1,000,1,500\right)$ using the same data-generation procedure. These additional sample sizes were generated to provide readers with an intuitive sense of the stability of interactions at common benchmarks for sample size. In light of the extent of these changes, for transparency, all original, preregistered analyses are reported in the Supplemental Material.

The simulation parameters produced 192 distinct combinations of variables, excluding sample size. At each examined sample size, for each combination of parameters, 10,000 data sets were generated.

Evaluating stability

Stability of the interaction is measured using a modified version of the COS and POS from Schönbrodt and Perugini (2013). Specifically, we used percentages to define how similar effect-size estimates must be to the expected effect to be considered stable. Statistical power was calculated at each examined sample size using the power_interaction_r2() analytic power function from the InteractionPoweR package in R (Baranger et al., 2023). The bias of the model was measured by evaluating how frequently the models recapture the true value $\mathrm{\rho }$ across simulations.

COS

The COS ⁸ is an interval in which estimates are considered stable if they fall within its bounds. Its width is determined using percentages, which we denote $P_{COS}$ . The COS is symmetric and centered on the expected effect after attenuation $({\mathrm{\beta }}_3=\mathrm{\rho }\times rel(X_1{X}_2))$ and is thus centered on a downwardly biased estimate, similar to CIs, which also do not account for bias.

To demonstrate how the $\mathrm{COS}$ is calculated, consider the following example: Suppose we select a $P_{COS}$ of $25\%$ , and the true effect size $(\mathrm{\rho })$ is 0.2. In a combination with $rel(X_1{X}_2)=0.5$ (selected for convenience), the expected effect size ${\mathrm{\beta }}_3$ is attenuated to 0.1. By applying the above formula, the $25\%\mathrm{COS}$ is $\left[0.1\times \left(1-0.25\right),0.1\times \left(1+0.25\right)\right]=\left[0.075,0.125\right].$ Under these stability requirements, estimates $(\widehat{{\mathrm{\beta }}_3})$ greater than $0.075$ and less than $0.125$ fall within the COS. If we were to create a $\mathrm{COS}$ for the same combination with a $P_{COS}$ of $50\%$ , the interval for stability would widen to $\left[0.05,0.15\right].$ Estimates falling within the smaller $P_{COS}$ will necessarily fall within a larger $P_{COS}$ , but the reverse does not hold.

POS

The POS is the sample size at which a certain percentage of the estimates (one of $80\%,90\%,$ or $95\%)$ of the 10,000 data sets generated for each sample size falls within the $\mathrm{COS}$ . We denote this percentage as $P_{POS}.$ As $P_{POS}$ increases, the sample-size requirement for stabilization becomes more stringent. Thus, stability in our simulations is operationalized similarly to Schönbrodt and Perugini (2013), diverging only in the definition of the $\mathrm{COS}$ to apply more directly to interaction effects.

The following example demonstrates how the COS and POS are calculated and applied in our study. Consider the case in which we select a $P_{COS}$ of $25\%,$ $P_{POS}$ of $80\%,$ and other parameters identical to the previous example $\left({\mathrm{\beta }}_3=0.1;COS=\left[0.075,0.125\right]\right).$ There are 10,000 estimates $\widehat{{\mathrm{\beta }}_3}$ at each sample size we consider. In this example, $n=600$ is identified as the $\mathrm{POS}$ because $80\%$ of the estimates of $\widehat{{\mathrm{\beta }}_3}$ are within $25\%$ of the expected effect. In our simulations, we searched for the sample size at which a $\mathrm{POS}$ is identified for each combination for each pair of $P_{COS}$ and $P_{POS}$ percentages.

We opted for percentage-based thresholds when quantifying stability to scale according to the magnitude of the effect. Schönbrodt and Perugini (2013) constructed a $\mathrm{COS}$ using Cohen’s benchmarks for width (w = 0.1, 0.15, 0.2) on correlations ranging from r = .1 to r = .7. This approach is suitable for larger effects, which almost never describe interactions effects in field studies in the psychological literature. For a demonstration of the application of their recommended widths to effects more typical of interactions, as percentages, see Table 1.

Table 1.

Percentage of Deviation Allowed From $\mathrm{\rho }$ Using the Corridor of Stability From Schönbrodt and Perugini (2013)

$\mathrm{\rho }$	w = 0.1	w = 0.15	w = 0.20
0.05	198%	295%	390%
0.10	98%	145%	192%
0.15	64%	95%	125%
0.20	47%	69%	91%
0.30	29%	43%	57%
0.40	20%	30%	38%
0.50	14%	21%	27%
0.60	10%	15%	19%
0.70	7%	10%	13%

Note: The italicized rows are the effect sizes employed in the present study. The bold rows represent effect sizes unique to the present study.

In six of 12 cases with the effect sizes we analyzed in our study, the $\mathrm{COS}$ bounds calculated using the fixed-width approach permit sign errors (i.e., a true positive effect falling within the COS despite being negatively signed). An interaction estimate deviating by 200% from the population-level effect with its sign inverted is best described as unstable (and inaccurate); it will tend to vary substantially across samples and prove difficult to reproduce. We thus sought to establish practical thresholds for stability for interaction effects. By opting for percentages, we offer a parsimonious alternative to calculate the $\mathrm{COS}$ with more informative bounds. In addition, our approach benefits from the width of the $\mathrm{COS}$ being a function of the effect size of the expected value, whereas Schönbrodt and Perugini’s (2013) $\mathrm{COS}$ width is independent of effect size. ⁹

Multiple values for $P_{COS}(25\%,50\%,75\%,100\%)$ were examined in this study to allow for the requirements for stability to range from conservative to less conservative. A $P_{COS}$ of $10\%$ was also preregistered but is not reported in this article because it was evaluated to be so conservative as to be not informative to readers. For this analysis, see the Supplemental Material.

True-value recapture rate

The true-value recapture rate is a measure of bias in the estimates of the interaction coefficients. It is the proportion of estimates falling within bounds centered on the true (unattenuated) interaction effect size $(\mathrm{\rho }),$ and is calculated similarly to the $COS$ :

$$TrueValueRecaptureRate=\frac{\begin{array}{l}Estimatesinside\left[\mathrm{\rho }\mp \mathrm{\rho }\times P_{COS}\right]\\ atsamplesizen\end{array}}{\begin{array}{l}Totalnumberofestimates\\ atsamplesizen\end{array}}.$$ (3)

Power

Power for the interaction term is calculated analytically using the InteractionPoweR (Baranger et al., 2023) package in R, which accounts for attenuation. This is different than the preregistered approach of recording the proportion of significant estimates during simulations. The change was made to improve computational efficiency.

Sign errors

Sign errors for the interaction term were calculated as the proportion of negatively signed interaction effect-size estimates to the total number of estimates because our simulated effect sizes ( $\mathrm{\rho }=0.05,0.1,0.15,0.2)$ are all positive.

Statistical analyses

All simulations and analyses were performed using the parameters and evaluation metrics described above. Information presented about the POS is calculated using data simulated with the search algorithm described above in Sample Sizes. When specific sample sizes were examined (see the Results section; $n=100,200,300,400,500,1,000,1,500)$ , the data were simulated separately from the search procedure. The data-generation process remained the same at any specific sample size (10,000 data sets were generated with the specified combination of parameters) regardless of whether the search algorithm was used to identify the sample size for simulation. For additional details on model performance and the properties of the simulated data, see the Supplemental Material.

Results

Stabilization across combinations

The subsequent analyses are reported with PCOS = 50% and $P_{POS}=80\%$ unless otherwise specified. Of the 192 combinations tested, 145 (76%) were stable at a sample size less than or equal to $1,500.$ Across all combinations with an effect size of $\mathrm{\rho }=0.05,$ the mean $\mathrm{POS}$ is $3,165(\mathrm{minimum}=1,445,\mathrm{maximum}=5,180,SD=928).$ When $\mathrm{\rho }=0.10,$ the mean $\mathrm{POS}$ is 789 $(\mathrm{minimum}=360,\mathrm{maximum}=1,300,SD=231)$ . As effect size increases, the number of estimates falling within the COS. increases; when $\mathrm{\rho }=0.15$ , there is an average $\mathrm{POS}$ of $348(\mathrm{minimum}=160,\mathrm{maximum}=580,SD=103).$ The largest simulated interaction effect ( $\mathrm{\rho }=0.20)$ has an average $\mathrm{POS}$ of $n=194\left(\mathrm{minimum}=90,\mathrm{maximum}=325,SD=58\right).$ An effect of $\mathrm{\rho }=0.2$ has $\mathrm{POS}$ values that are, on average, 16 times smaller than those from our smallest effect $\left(\mathrm{\rho }=0.05\right)$ under otherwise identical conditions. Larger interaction effects lead to more stable results in smaller samples; note that our two largest effect sizes are much larger than typical interaction effects.

As the reliability of the predictors increases, the POS decreases. The average $\mathrm{POS}$ for a reliability of $0.7$ is $1,534\left(\mathrm{minimum}=200,\mathrm{maximum}=5,180\right)$ . When reliability is $0.8,$ the mean POS is $1,207\left(\mathrm{minimum}=155,\mathrm{maximum}=4,105\right)$ . For a reliability of $0.9$ , the average $\mathrm{POS}$ is $967$ $\left(\mathrm{minimum}=115,\mathrm{maximum}=3,170\right)$ . In the case of perfect reliability, the mean $\mathrm{POS}$ is $788$ $\left(\mathrm{minimum}=90,\mathrm{maximum}=2,615\right)$ . Each 10% decrement in the reliability of the predictors results in an approximate increase of 20% to the sample size required for a stable prediction. If an interaction is estimated from predictors with reliabilities of $0.7$ , it requires a sample size 94% larger than the same interaction estimated with perfect reliability. Reliable predictors $\left(rel=0.9,1.0\right)$ lead to more stable estimates of the interaction in smaller samples.

Main effect size and collinearity also affect the $\mathrm{POS},$ although to a lesser degree than sample size, effect size, or reliability. Stability is evaluated at four predictor effect sizes. The smallest main effect size of ${\mathrm{\beta }}_i=0.1$ has a mean $\mathrm{POS}$ of $1,279\left(\mathrm{minimum}=155,\mathrm{maximum}=5,180\right).$ This pattern continues for ${\mathrm{\beta }}_i=0.2,0.3,0.5$ , which have average $\mathrm{POS}$ sample sizes of $1,216,$ $1,138,$ and $864$ , respectively. Collinearity among the main effects is considered at three levels. For the smallest level of collinearity $r_{X_1,X_2}=0.1,$ the mean $\mathrm{POS}$ is $1,151.$ At $r_{X_1,X_2}=0.3,$ the mean $\mathrm{POS}$ is $1,132.$ When the predictors are highly collinear ( $r_{X_1,X_2}=0.5)$ , the average $\mathrm{POS}$ is $1,089.$ Small, highly intercorrelated main effects result in larger sample sizes being required for stability.

Stability in the average study

A combination of parameters that approximates a field study in psychology ¹⁰ (Brysbaert, 2019; Nosek et al., 2022; Open Science Collaboration, 2015) has a point of stability of $n=3,800$ ¹¹ with $P_{COS}=50\%$ and $P_{POS}=80\%$ . At $n=100,$ the statistical power to detect the interaction effect is $6.88\%$ , and $33.57\%$ of the estimates are incorrectly signed! There is little change at $n=200$ $\left(\mathrm{power}=8.90\%,\mathrm{sign}\mathrm{error}\mathrm{rate}=28.36\%\right).$ . Even at $n=1,000,$ power is $25.57\%$ , and $9.40\%$ of the estimates are signed incorrectly. Power at the $\mathrm{POS}(n=3,800)$ is $72.17\%.$ With the most generous stability requirements $\left(P_{COS}=100\%,P_{POS}=80\%\right)$ , the $\mathrm{POS}$ is $n=940$ , and power at this $\mathrm{POS}$ is $24.40\%.$ By contrast, a rigorous stability requirement $\left(P_{COS}=25\%,P_{POS}=90\%\right)$ corresponds to an impractically large $\mathrm{POS}$ of $n=25,280$ and power $>99.99\%.$ When assuming the interaction effect is 10 times what is average in a psychological-science study ¹² with zero measurement error in the predictors, the sample size at which stabilization occurs is $n=150$ , which provides $70.60\%$ power. The sample sizes required to achieve stability are higher than those commonly found in published studies, even under ideal conditions with interaction effects of equal size to the main effects.

$f^2$ effect size

In a nonpreregistered analysis, to better illustrate the POS/COS trade-offs, we also calculated Cohen’s $f^2$ for each combination of simulation parameters. $f^2$ is a commonly used effect size in regression analysis that complements unstandardized coefficients. In a comparison of two nested models, a reduced model (e.g., $Y={\mathrm{\beta }}_0+{\mathrm{\beta }}_1{X}_1+{\mathrm{\beta }}_2{X}_2+{ϵ}_1$ ), and a full model (e.g., $Y={\mathrm{\beta }}_0+{\mathrm{\beta }}_1{X}_1+{\mathrm{\beta }}_2{X}_2+{\mathrm{\beta }}_3(X_1{X}_2)+{ϵ}_2$ ), the $f^2$ statistic ¹³ reflects how much of the residual variance not accounted for by the reduced model ( ${ϵ}_1)$ is captured by the full model. In this context, the predictor of interest is the interaction. We included $f^2$ primarily to aid in visualization (see Fig. 1). It provides a standardized metric that assists in illustrating several features from our simulations: sample size, effect size, and impact of the $\mathrm{COS}$ width and percentage on when the interaction can be regarded as stable. Cohen’s (1988) conventions suggest $f^2$ values of 0.02, 0.15, and 0.35 correspond to small, medium, and large effects, respectively. In line with our parameter selection, the $f^2$ values in this study encompass a range suited to interaction effects, although they are small in absolute terms. Figure 1 shows the relation between interaction effect size (measured using $f^2)$ and the $\mathrm{POS}$ sample size for all combinations. Each line represents a pair of $P_{COS}$ and $P_{POS}$ values.

Fig. 1.

Sample size and $f^2$ at the point of stability across different COS and POS percentages. Figure 1 traces the sample size required for stability as a function of the $f^2$ value for COS widths of 100%, 50%, and 25% with a POS threshold of 80%. All 192 combinations of simulation parameters are represented on each line. In the upper left, the numbers labeled “max” illustrate the sample size required for stability with the smallest $f^2$ in our simulations (0.0013). Points 1, 2, and 3 are for the same parameter combination and demonstrate how loosening stability requirements results in stability at smaller sample sizes. Points 2 and 4 lie along the same line and show how smaller effect sizes require larger sample sizes to stabilize, all other things being equal. COS = corridor of stability; POS = point of stability.

For example, consider Points 1 through 3 in Figure 1. ¹⁴ All illustrate the $\mathrm{POS}$ for an $f^2$ of $0.020$ and vary by $\mathrm{COS}$ width. Point 1 has a $\mathrm{POS}$ of $n=1,230$ , which is 4 times the $\mathrm{POS}$ for Point 2 ( $n=305)$ and 15 times that of Point 3 ( $n=80).$ Point 3 lies along the most generous $\mathrm{COS}$ and $\mathrm{POS}$ percentages we examined, and Point 2 represents the COS and $\mathrm{POS}$ percentages likely to be useful and practical to researchers. Point 3 is prohibitively strict and illustrates how the sample size required for stability increases as the $\mathrm{COS}$ narrows. In addition, Points 2 and 4 lie on the same line $\left(P_{COS}=50\%,P_{POS}=80\%\right)$ but with different effect sizes. Relative to Point 2, the $f^2$ for Point 3 is smaller $(0.005),$ and the $\mathrm{POS}$ is larger $(n=1,240).$ To stably estimate a small effect, sample-size requirements quickly balloon to be untenably large.

Power and stability

In a deviation from our preregistration, we chose to more specifically examine statistical power at the POS for all combinations considering various COS widths because we believe readers might find such an examination useful. With a $\mathrm{COS}$ width of $50\%$ and $P_{POS}$ of 80%, estimates of the interaction are stable in all 192 combinations when power is approximately 72%. To achieve stability with a $P_{POS}$ of $90\%$ holding the $\mathrm{COS}$ width at $50\%,$ the required power increases to $90\%,$ and with a $P_{POS}$ of $95\%,$ it is $97\%.$ With a narrower $P_{COS}$ of $25\%,$ holding $P_{POS}$ at $80\%,$ the average power for all 192 cases is $99.91\%\left(SD<0.01\%\right).$ The most lenient confidence requirements $(P_{COS}=100\%,P_{POS}=80\%)$ correspond with $24.52\%\left(SD<0.01\%\right)$ power on average at the $\mathrm{POS}.$ Stability and power are closely related, and greater power indicates more stable interaction estimates.

Sign errors and true-value-recapture rates

Sign errors are most frequent at small sample sizes across all combinations ( $m=47.56\%$ at $n=5;m=39.12\%$ at $n=10$ ). Overall, the average sign-error rate is $16.43\%$ at $n=100$ . Smaller effect sizes tend to have more sign errors; when $\mathrm{\rho }=0.05$ , the average sign-error rate is $33.00\%$ for $n=100,$ whereas a larger $\mathrm{\rho }$ of 0.20 has a sign-error rate of $4.30\%$ at the same sample size. The mean rate of sign errors in all simulations is $10.01\%$ at $n=200$ , $7.28\%$ at $n=300$ , $5.26\%$ at $n=400$ , and $4.53\%$ at $n=500$ . For larger sample sizes, such as $n=1,000,\mathrm{and}n=1,500$ , the average sign-error rate is $2.00\%$ and $1.04\%$ , respectively. Sign errors are strongly negatively related to power overall ( $r=-0.79$ ) and are most frequent for interactions with small effect sizes estimated at sample sizes under $n=100.$ When considering the ability of the models to recapture the true value $\mathrm{\rho }$ , we found that if the expected effect $({\mathrm{\beta }}_3)$ is similar in size and direction to $\mathrm{\rho },$ increasing the sample size improves accuracy. However, if ${\mathrm{\beta }}_3$ is biased because of measurement error, a larger sample does not correct the bias.

Case example

For the simulated data for the combination with median parameters, ¹⁵ see Figure 2. The data illustrated are not simulated using the search algorithm described in the method section because the plot would be incomplete. Instead, data are generated using the same procedure for each $n$ from $5$ to $1,500$ in increments of $5$ with $10,000$ data sets at each sample size.

Fig. 2.

Plot of the estimates for the interaction-term coefficient ( ${\mathrm{\beta }}_3$ ). Figure 2 demonstrates our simulations for the median combination (true interaction effect size = 0.15, main effect size = 0.3, main effect intercorrelation = 0.3, reliability = 0.8). There are 10,000 estimates of the interaction at each sample size. These estimates are split into 10 groups of even size (deciles), denoted by the horizontal, jagged lines. Darker shading indicates the estimates are tightly clustered, and lighter shading illustrates greater spread in the estimates. The horizontal dotted line is the expected effect ( ${\mathrm{\beta }}_3$ ), which is 0.11.

Stability is achieved at n = 425 with a COS of $[0.05,0.16]$ . Note that in Figure 2, the 80% point of stability is where the 90th and 10th deciles lines intersect $\mathrm{COS}$ .

Discussion

Despite their theoretical appeal, the results of this study suggest that estimating two-way interaction effects using linear regression presents major methodological and statistical challenges. An estimate of an effect is considered stable when it is likely to replicate in magnitude and direction across samples. Our simulations identified sample size, predictor reliability, and interaction effect size as the primary determinants of the stability of interaction estimates. When these parameters were evaluated at levels realistic for a field study, the sample size required for a stable estimate of the interaction coefficient was found to be 3,800.

An important takeaway from this study is that 72% power to detect the interaction is synonymous with estimates being reasonably stable ( $P_{COS}=50\%,P_{POS}=80\%$ ) because the stability is attained at 72% power with this COS width and POS percentage in all combinations. Thus, powering a study to at least 80% to detect an interaction will essentially guarantee stability at a level that will yield accurate results. Most studies fall well short of this mark (Open Science Collaboration, 2015). With a sample size of 500 or below, 49% of our parameter combinations were stable, ¹⁶ and it is effectively a coin flip of whether an interaction analysis in this sample-size range is replicable. The conditions that produce stable interaction estimates and the conditions of the many field studies diverge; without sufficient power from a robust sample size and reliable predictors, it is generally inadvisable to conduct interaction analyses because they are likely to be uninformative, and there is a large probability that the observed magnitude and sign of the interaction will not align with the true population effect.

Although there is no inherent flaw with interaction analyses, there are unique properties to interaction effects that explain the difficulties researchers encounter when testing for them. It has been repeatedly observed that interaction effects are small, with an absence of evidence from any field that there are many large interactions (Credé & Sotola, 2024; Freese & Peterson, 2017; Open Science Collaboration, 2015). Simultaneously, they are uniquely susceptible to attenuation because their reliability is approximately the product of the two main effects. Methodologically, the sample-size thresholds for stability we have outlined will likely be impractical for most researchers to meet because of constraints on resources and sociological pressures around publishing. The same pressures drive many interaction analyses. If the main effects in a study are not found to be significant, researchers may turn to testing interactions until a significant (and thus, publishable; Greenwald, 1975) result is found. In such cases, researchers are typically searching for interactions they are grossly underpowered to detect and that would yield highly unstable estimates. These conditions can encourage atheoretical serial testing of interactions. This will lead and has led to increased family-wise error rate and gross overestimations of population effect sizes (Sotola & Credé, 2023).

Power and stability are not the only issues with interactions. Even if a given sample size provides adequate power and stability for an interaction, there remain deeper conceptual and interpretive challenges. As Rohrer and Arslan (2021) emphasized, there are additional issues to consider: scale dependence, the distinction between moderation of slopes and moderation of correlations, and causal identification. Interactions can change in magnitude (or even reverse direction) depending on the measurement scale, leading to contradictory conclusions. Differences in correlations between groups may be mistaken for slope differences or vice versa, obscuring the true nature of the effect. Finally, significant interaction terms do not imply causal interactions unless both variables are appropriately manipulated or strong assumptions are met. Thus, even precise and stable estimates can be misleading without careful attention to these foundational issues.

Stability and power

Statistical power was identified as a strong proxy for stability, although the two constructs are theoretically and operationally distinct. For the 192 parameter combinations, stability (defined as $80\%$ of the estimates falling within a $\mathrm{COS}$ with width of $50\%$ ) was achieved with statistical power of approximately $72\%.$ Thus, if the probability of correctly detecting a true effect (power) is high, resampled estimates will tend to cluster tightly around the expected effect (stability). Although this operationalization of stability is less informative at larger effect sizes ( $r>.2)$ as a result of its percentage-based approach, it is optimal for application to interactions in which effect sizes almost universally fall below $r=.2.$ Furthermore, more stringent stability requirements demand greater power. If an interaction effect is to be replicable within $25\%$ of the expected effect, near $100\%$ power is required. Although the associated sample sizes for stabilization varies based on the size of the expected effect and other parameters, the power at the $\mathrm{POS}$ is consistent. There is 12% power to detect an interaction with $\mathrm{\rho }=0.10$ under conditions typical of psychology studies at $n=100$ (see Table 2; also see Aguinis et al., 2005; Vize, Sharpe, et al., 2023). Furthermore, when $n=100,$ approximately 85% of the estimated interactions vary by 100% or more from the expected value. The probability of detecting a true effect is affected by sampling variability, and stability is a measure of spread that can be largely attributed to sampling variability. Thus, if an interaction analysis is underpowered, it is also unstable.

Table 2.

Point of Stability (n) for Different $\mathrm{\rho }$ , COS, and POS Percentages and Reliabilities

			POS: 80%			POS: 90%			POS: 95%
		COS levels
$\mathrm{\rho }$	$rel$	${\mathrm{\beta }}_3$	25%	50%	100%	25%	50%	100%	25%	50%	100%
0.05	0.7	0.035	20,700	5,015	1,240	33,590	8,420	2,080	46,430	11,630	2,910
	0.8	0.040	15,455	3,800	940	25,280	6,300	1,585	36,390	9,060	2,290
	0.9	0.045	12,140	2,930	760	19,925	5,080	1,250	28,130	6,860	1,760
	1	0.050	9,590	2,435	610	15,900	4,075	980	21,825	5,630	1,415
0.10	0.7	0.070	5,080	1,240	325	8,195	2,060	520	11,725	2,930	730
	0.8	0.080	3,800	950	240	6,310	1,540	400	8,790	2,200	565
	0.9	0.090	2,955	750	190	5,055	1,220	310	6,985	1,810	430
	1	0.100	2,390	610	150	3,970	1,000	250	5,635	1,415	350
0.15	0.7	0.106	2,200	550	140	3,570	910	235	5,070	1,305	330
	0.8	0.120	1,650	425	110	2,770	690	175	3,890	960	250
	0.9	0.135	1,310	330	85	2,150	540	140	3,130	765	190
	1	0.150	1,040	270	70	1,720	440	110	2,450	625	155
0.20	0.7	0.141	1,200	310	80	2,060	500	130	2,885	715	180
	0.8	0.160	920	230	60	1,560	390	100	2,205	535	140
	0.9	0.180	720	190	50	1,170	305	80	1,695	420	110
	1	0.200	570	150	40	960	245	60	1,350	340	85

Note: Main effect size is 0.2, and the collinearity between the predictors is 0.1. We also calculated POS values for COSs with widths of 10% and 75%. For brevity, they are excluded from this table and are available in the Supplemental Material available online. Negative effect sizes were not evaluated in this study. In response to reviewer feedback, we evaluated one case from this table ( $\mathrm{\rho }=0.1,rel=0.8,\mathrm{POS}:80\%,\mathrm{COS}:50\%$ ) with the sign of the interaction effect inverted $(\mathrm{\rho }=-0.1)$ . We found the $\mathrm{POS}$ to be 983, a 3.2% increase from the positively signed equivalent, which has a POS of 950. POS = point of stability; COS = corridor of stability.

CIs are often misinterpreted as indicators of stability. A confidence interval reflects within‑studies uncertainty around the attenuated parameter targeted by the estimator under measurement error. It does not evaluate consistency across replications like the COS and POS. For example, at n = 100 with ρ = 0.1, the average 95% CI for the interaction term across 10,000 simulations was [–0.11, 0.27] around an attenuated β₃ ≈ 0.08, indicating substantial imprecision in a single study. By contrast, the COS and POS are design‑level metrics, and they quantify the proportion of replicated estimates that fall within a prespecified band around the attenuated population value. CIs summarize precision of one estimate, whereas the COS and POS summarize the ability of the design to produce stable results. Narrow CIs do not guarantee high stability if the corridor is tighter than the typical sampling variability; conversely, a design can meet a stability criterion even when single‑study CIs remain relatively wide.

Recapture rates and sign errors

There is a moderate to high risk of sign errors when the sample size is below $n=100,$ the effect size is small ( $\mathrm{\rho }=0.05,0.10$ ), and the predictors are less reliable ( $rel=0.7,0.8)$ . Main effects that are smaller ( ${\mathrm{\beta }}_1={\mathrm{\beta }}_2=0.1$ ) and highly correlated $(cor(X_1,X_2)=0.5)$ further increase the risk of sign errors. In the least reliable case, ¹⁷ defined as the combination of parameters with the highest observed $\mathrm{POS},$ the rate of incorrectly signed estimates is $35.72\%$ when $n=100$ and $10.95\%$ when $n=1,000$ . Unreasonably large sample sizes are necessary to reduce the sign-error rate to below $10\%$ under these conditions. By contrast, the most reliable case, ¹⁸ defined as the combination with the lowest observed $\mathrm{POS}$ , has zero sign errors by $n=400.$ The median case, ¹⁹ defined as the combination with median input parameters, has a sign-error rate of $10.65\%$ at $n=100$ and less than $0.01\%$ at $n=1,000.$ Of all the incorrectly signed estimates, only $2\%$ are statistically significant, on average. These results suggest incorrectly signed estimates are infrequent in the literature given that publication bias rewards significant results.

Likewise, the true population value ( $\mathrm{\rho }$ ) is recaptured more frequently for larger interaction effect sizes with more reliable predictors. As sample size increases, the precision of the estimates of attenuated effect ${\mathrm{\beta }}_3$ improves. The difference between ${\mathrm{\beta }}_3$ and $\mathrm{\rho }$ becomes more accentuated as the expected effect becomes smaller because of attenuation. As sample size increases, recapture rates tend to either zero (when the predictors are unreliable) or 100% (when the predictors are reliable). That is, when ${\mathrm{\beta }}_3$ is outside the $\mathrm{COS}$ around $\mathrm{\rho }$ , the recapture rate shrinks toward zero as sample size increases after an initial peak. In the aforementioned example ²⁰ representing a realistic field study, the true value is recaptured at a rate of $12.32\%$ when $n=100$ for a $\mathrm{COS}$ width of $50\%.$ This recapture rate suggests a relatively small amount of “true” signal is detected in most interaction estimates, again attributable to their small sample size and susceptibility to the reliability of their predictors. Functionally, this affirms what is already known about correlations: The reliability of the predictors can attenuate the observed effect size.

Alternatives

Our results accord with Murphy and Russell’s (2017) recommendation for researchers to end the search for moderators unless major improvements to their detection can be made. Estimating interaction effects necessitates at least four groups because they involve the examination of multiple levels of two independent variables simultaneously. Additive main effects are often mistaken to be interactions (Vize, Baranger, et al., 2023), and one cannot conclude an interaction exists simply by virtue of the two variables producing a larger effect jointly as opposed to separately. In linear regression, the interaction term models a situation in which variance is explicable beyond the sum of the constituent main effects. Thus, theoretical justification on why two main effects are insufficient for explaining the phenomena is critical when testing for interactions. A theoretical examination of all possible moderators should always be disclosed as exploratory. Given seven variables, there are 21 possible two-way interactions; one is likely to be significant at $\alpha =.05.$ At the very least, power should be calculated using thresholds consistent with the small anticipated effect sizes of interaction estimates, which are an order of magnitude smaller than the common benchmarks proposed by Cohen (1988). Cohen’s benchmarks are integrated into G*Power, ²¹ and there is a substantial risk of researchers erroneously underpowering their interaction analyses because of the gulf between these standard thresholds and actual interaction effect sizes. The problematic statistical and theoretical properties of interactions in the social sciences make their analysis with linear regression difficult for most researchers. In a field in which main effects are often insufficiently powered, a post hoc test of an interaction will almost never be sufficiently powered or stable unless using consortium-collected data (e.g., Adolescent Brain Cognitive Development Study).

Assuming a genuine interaction is hypothesized, we suggest that statistical approaches beyond linear regression may reduce the impact of measurement error and the signal-to-noise ratio inherent to small effects. Reliability affects both the point at which estimates for the interaction can be regarded as “stable” and on the observable effect size because of attenuation of $\mathrm{\rho }$ . Structural equation modeling (SEM) mitigates unreliability by separating true score variance from measurement error with explicit latent variables, which can reduce attenuation bias, improve stability, and increase statistical power for the modeled interaction. In genomics, methods have also been developed to infer the presence of interactions even while studies remain too underpowered to detect and characterize more than a handful of them (Zhu et al., 2023). Regardless of the method employed in testing for an interaction among two variables, we stress the importance of a priori hypotheses; if the main effects do not yield results viewed as publishable, an exploratory search for interactions significantly increases the risk of erroneous findings, especially given that the vast majority of studies in psychology are overwhelmingly underpowered for such searches.

Practical recommendations

Researchers planning interaction analyses with two continuous predictors can take several concrete steps to improve the replicability of their research. First and most important, developing an interaction hypothesis and powering for its analysis using realistic interaction effect sizes (r ≤ .10) rather than Cohen’s benchmarks will help avoid the pitfalls of underpowered, exploratory interaction testing. When feasible, SEM can mitigate some of the reliability-related challenges we have identified. Under an SEM framework, stability may be achieved at sample sizes comparable with our simulated cases that had perfect reliability (Hoyle, 2012), although many of the perfectly reliable cases still require enormous sample sizes for stability. Given the substantial sample-size requirements we identify, researchers unable to achieve at least 72% power for the interaction should consider whether their research questions can be addressed through alternative approaches.

Limitations

This study had several limitations. Any attempt to quantify the stability of an estimate in a regression framework across multiple trials requires an arbitrary selection of cut points to delineate “stable” and “unstable,” which applies to our percentage-based COS and POS, even though these had advantages over their existing definitions. Furthermore, the Monte Carlo simulations relied on synthetic data that may not reflect typical social-science data in which peculiarities such as nonnormality, omitted variables, or selection bias can appear. Measurement error similarly is modeled only through predictor reliability, which is but one potential source of noise in the data among many (Loken & Gelman, 2017; Schmidt et al., 2003). Potential concerns about the applicability of linear models to nonlinear patterns remain beyond the scope of this study. Large sample sizes required for stability may be impractical in empirical research, although alternative methods, such as SEM (Hoyle, 2012), could mitigate some of the identified issues with interaction analyses.

Conclusion

Our results strongly suggest that testing two-way interaction effects in psychology field studies using a linear regression framework is largely untenable. ²² Even under the most favorable conditions, achieving stability requires sample sizes in excess of what is common in empirical research. It follows from the results of this study and corroborating evidence from existing research (Murphy & Russell, 2017; Vize, Baranger, et al., 2023; Vize, Sharpe, et al., 2023) that published interaction effect using a linear-regression framework should be regarded with skepticism. Researchers should test interactions only if they have a specific interaction hypothesis that the relation between two variables changes depending on the level of a third and the study design is appropriate (sufficiently large N and reliable measures). Alternative methods to regression, such as SEM, may better address challenges related to reliability, power, and sampling variability in interaction analyses (Cole & Preacher 2014; Hoyle, 2012; Marsh et al., 2004). Neglecting these factors risks portraying negligible interactions as stable and reliable, leading to irreplicable studies that form the backbone of misguided theories built on the artifacts of misaligned incentives and null hypothesis testing.

Supplemental Material

sj-pdf-1-amp-10.1177_25152459251407860 – Supplemental material for When Do Interaction/Moderation Effects Stabilize in Linear Regression?

Supplemental material, sj-pdf-1-amp-10.1177_25152459251407860 for When Do Interaction/Moderation Effects Stabilize in Linear Regression? by Andrew Castillo, Joshua D. Miller, Colin Vize, David A. A. Baranger and Donald R. Lynam in Advances in Methods and Practices in Psychological Science