Tools for Those Who Summarize the Evidence Base
I would like to open a discussion on the calculation and reporting of interrater reliabilty. First, what degree of importance is currently placed on this facet of meta-analysis in the peer-reviewed literature? Is there a gold standard for calculating, or do all commonly employed methods enjoy equal regard? How are these typically reported in the literature: to present a reliability score on every coded characteristic seems far too cumbersome, yet to generalize across codes risks obscuring important differences. What's the current standard? Is it where it should be?