Tools for Those Who Summarize the Evidence Base
I'm looking for resources on using meta-regression to estimate the interactive effects of two moderators on effect sizes. For example, I have coded for scale specificity, which I expect to explain heterogeneity in effect sizes. However, I'd expect this to account for more heterogeneity in later years rather than early years, so I'd expect time and specificity to interact to predict effect sizes. I came across a PowerPoint presentation by David Wilson saying that this is possible but he doesn't go into the technical details. Is it as simple as creating an interaction term using meta-regression?
Also, I realize that hierarchical meta-analysis is probably more appropriate since studies are nested within years, but for now I'm just interested in the basic effects of the moderators for a conference presentation.
Yes, none of the textbooks currently out there explain how to do that. (Though they certainly imply it.)
If all you want is the estimated effect size values at different levels of the two (or more) moderators, the simplest thing is simply to use the meta-regression equation, inserting the values for which you want the estimates. In your example, your effect size (ES) is a function of year (IV1), scale specificity (IV2), and their interaction, you have a meta-regression equation predicting ES (EShat) in the form of:
ES(hat) = Constant + b1(IV1) + b2(IV2) + b3(IV1*IV2)
Where the b values are unstandardized regression weights (not the standardized beta weights) for each term in the model. So, with numbers replacing the example:
ES(hat) = 0.54 + 0.12(IV1) + 0.20(IV2) - 0.23(IV1*IV2)
Just insert the values for all of the combinations of IV1 and IV2 that you want to estimate.Typically people calculate 4 values to represent an interaction of 2 continuous variables, but of course it is up to you to decide what is optimal.
Presumably, if you are testing a meta-regression model like this one, you will zero-center your continuous predictors (including their interaction) prior to estimating the model in order to create more stable estimates (especially in relation to standardized coefficients). The constant in that model will differ, so what is entered in the equation above are zero-centered forms of the variables to be plotted.
If you want confidence intervals around particular estimates, then more work is involved, and entries on this same website describe how you do it. A year ago, a colleague and I published an article about what we call The Moving Constant Technique that shows how to do it and develops issues involved in this statistical practice. In particular, a preprint of the article (and some discussion) is here. The published article is:
We find the moving constant technique quite useful in a meta-analytic context because it is valuable to know whether the mean effect size differs from the value under the null hypothesis (e.g., a standardized mean difference statistically greater than zero), not just that effect sizes are bigger under one or another levels of a moderator. Social scientists doing meta-analysis have probably over-stressed the slope and neglected the constant. Public health and medicine meta-analysts have probably been more interested in the what values the moderators produced (though they all too often artificially dichotomize their continuous variables to do so). Convention can do better!
Hope that helps!
Thank you so much Blair! This is great information. I'm eager to jump in and read about the moving constant techique.
Thank you again.