An excellent recent paper on between-Lab variability in effect sizes and the implications of that for power analysis is McShane & Böckenholt (2014). Among other cool things, the authors use data from Many Labs 1 to get an estimate of the average variance in effect sizes across labs. They come up with a mean estimate of around SD = 0.2 on the Cohen’s $$d$$ scale, which they say is probably more of a lower bound in general, since the Many Labs 1 experiments were more closely standardized across Labs than many other research programs will probably be. Either way, this is a great starting point for us.

Note that in the blog post we wrote the noncentrality parameter in terms of proportions of variance, so the next question is how to convert this SD estimate to the proportion $$L$$. Here’s my first crack at how we could do so.

We have a random variable $$d$$ which is the effect size that varies across Labs. $$\text{Var}(d) = \tau^2$$, where we just established above that a reasonable guess for $$\tau$$ is 0.2.

Now $$d = \beta/\sqrt{\Sigma}$$, where $$\beta$$ is the condition mean difference and $$\Sigma$$ is the (weighted) sum of all the variance components (equivalently, the average of $$\text{var}(y)$$ across both conditions) EXCEPT for the Lab-level variance components. In other words, we assume that $$d$$ is computed in each sample using only the variance components available in that particular sample. So, substituting in, we have $$\text{var}(\beta/\sqrt{\Sigma}) = \tau^2$$.

Assume that $$\Sigma$$ is a constant, that is, the denominator of $$d$$ is the same at each Lab except for sampling error. This is tantamount to assuming that the true values of the variance components are the same at each Lab, which is a standard assumption. Then we can solve the above for $$\text{var}(\beta) = \tau^2\Sigma$$.

Okay, now the proportion of variance that we seek is defined as $$L = \text{var}(\beta)/(\Sigma + \text{var}(\beta))$$, in other words, the proportion of all the variance (including between-Lab variance) that is due to between-Lab variance. Note that technically $$\text{var}(\beta)$$ only corresponds to the variance of the Lab * condition effect interaction, and that we should really also include the variance of the Lab main effects in the denominator of the proportion as well. For now I just assume it is 0. This is maybe okay since the Lab main effect variance does not directly affect the power calculations anyway, since it doesn’t appear in the noncentrality parameter, but rather only affects power indirectly by possibly altering the proportions $$E$$ and $$L$$. In any case, just be aware that this is an assumption.

Substitute into the $$L$$ equation the expression for $$\text{var}(\beta)$$ that we just derived, and factor out $$\Sigma$$ from the numerator and denominator, to obtain: $L = \frac{\tau^2}{1 + \tau^2}$ So if this is correct, then if $$\tau = 0.2$$ (which is the point estimate found in the McShane & Böckenholt paper), then $$L = 0.0385$$. And if $$\tau = 0.3$$ (consistent with McShane & Böckenholt’s conjecture that 0.2 is likely an underestimate), then $$L = 0.0826$$. Either of these would would be reasonable guesses for the between-Lab proportion of variance $$L$$, based on the analysis in McShane & Böckenholt (2014).