Effect Size Adjustments for Mismatched Analyses

[Daniel Maggin]

Hi all. Hope you are all enjoying your new year! I have come across a problem that I thought might interest some folks out there so I thought I would pose it to the group to see if you all could help me out. So, here it goes...

I have a series of studies that conducted the analyses at the inappropriate unit of analysis. Specifically, these authors assigned schools or classrooms to the treatment condition but analyzed the data at the student level. Now, there is a procedure out there to correct for these mismatched analyses (Hedges, 2007) but I am having a tough time articulating (understanding?) the connection between the inflated statistical tests found within the primary studies and the importance of that to issues of weighting (i.e., the inverse variance weight) within meta-analysis. In particular, I am stumped because there is really no change in the actual standardized mean difference effect sizes themselves. 

So, I guess my questions are -- How does the mismatched analysis problem influence the inverse variance weight? and Why would this be such a problem for meta-analysis if you still have (essentially) the same estimate of overall treatment effect?

Any help or guidance out there would be great.

Dan

Hedges (2007). Effect sizes in cluster randomized control trials. http://www.northwestern.edu/ipr/publications/papers/2006/wp0613.pdf

[Blair T. Johnson]

Hmmm, yours is an interesting problem that probably occurs more often than analysts report. I browsed the Hedges report you linked, and it would rightly seem correcting the statistics would necessitate having the intraclass correlations. Do you have these?

 

It seems that you are intuiting the difference between the "wrong" analyses and the right ones--no differences in effect sizes but (probably) large differences in the inverse variances. But, again, you need the intraclass correlations to calculate the weights. In their absence, you need to do sensitivity analyses in which you examine a range of possible intraclass correlations and see what impact these have on statistical inference. Does that makes sense?

[ramesh athe]

Thanks