Tools for Those Who Summarize the Evidence Base
Hi Everyone! I'm currently planning a project that I'd like get underway but it's still unclear to me whether or not it's even possible. Enter this group! I want to look at researcher decisions when using structural equation modeling (SEM). Specifically, I want to examine which decisions contribute to changes in overall model fit. I could simply look at reported chi square values but I'd like to roll this up into a meta-analytic framework.
I am conceptualizing the overall model chi square as the effect size where instead of variable X relating with variable Y, it represents the "correlation" between the theoretical and observed model. A chi square of zero represents a perfect "correlation" between theory and data and large chi square represents a low correlation.
My question is if it's possible to calculate standardized effect sizes and variances from a chi square goodness of fit statistic from an SEmodel? I've read about Cramer's V for 2X2 contingency tables and that it *could* be applied to goodness of fit statistics, but I have not be able to find anything further about this. Could this be applied to chi-square goodness of fit for SEM?
I've also read about the possibility of using the log likelihood value in some meta-analyses but I don't think it was as an effect size. Would it be possible to use the log likelihood value as an effect size that's then converted into a standardized metric? If so, I believe the log likelihood function could be estimated from the chi-square value and used to create standardized effect sizes but I really do not know. So with that I have come to seek the experts' advice. Do you think this is even possible?
I also plan to post this on the MASEM page and get Dr. Cheung's thoughts.
It sounds interesting. But it appears that there are lots of issues that you need to address. The followings are some thoughts.
1. log likelihood is *not* an effect size. The difference between the log likelihood of the proposed model and the saturated model is the chi-square statistic reported in SEM.
2. Similar to t statistic and F statistic, chi-square statistic in *not* an effect size. It depends on the sample size.
3. Independent and dependent t statistics can be converted into d because it compares two means only (simple hypothesis). Similar to the F statistic, the chi-square statistic in SEM is testing a composite hypothesis.
4. There are a couple of factors affecting the chi-square statistic: (a) sample size; (b) model complexity (degrees of freedom); (c) degree of misspecification; and (d) violation of distribution.
If you want to examine what factors (not decisions) contribute to changes in overall model fit, you may do a computer simulation.
Hope it helps.
Hello Dr. Cheung,
Thank you for your response - it was very helpful.