Tools for Those Who Summarize the Evidence Base
Most of what scholars do hinges on having sufficient skill to create meaningful assessments of knowledge and the evidence base that underlies it. Meta-analysis is a skill set you can develop if you work at it with enough deliberative practice. How much deliberate practice does it take?
I few years ago, the editors of Dialogue, the official newsletter of the Society for Personality and Social Psychology, knew that I had created an interesting graph of how long it takes to gain mastery in a given domain and wondered if I would contribute an essay riffing on it. The resulting piece (with an improved graph, right) appeared in their Fall, 2009 issue. The meme of 10,000 hours being necessary to master a domain had been floating around culture--I still hear it--so the graph defined what this amount of time means in a more practical way, and it gave me a forum to recount some stories, make some jokes, and illustrate some points that I often share with my own students. (If you want the jokes, read the essay!)
Don't get me wrong: As I imply in the essay, it need not take 10,000 hours to master a particular domain, such as meta-analysis. I believe that knowledge and skills in other domains can help one master it faster. Further, mastery is something one must continue to stimulate. A musician must regularly try to play new, challenging pieces. A master meta-analyst still needs to keep up with new developments in the methods that other scholars produce. There are also sub-domains in meta-analysis. Classical, aggregate meta-analysis with one outcome per study is probably the most commonly mastered, to date. Bayesian approaches have been growing in prominence. New strategies now permit meta-analysis with two or more outcomes. New strategies to detect publication bias seem to appear every other month. Some strategies are less commonly mastered and therefore are less conventional. Thankfully, some members of this website have these skills and have reached out to assist other members solve advanced problems. It is my hope that this website to some extent helps the meta-analytic community do meta-analysis better in more different ways.
I've had so much fun sharing the essay with interested others that I offer it here as something that may amuse you, especially as 2012 ends and 2013 starts. It is available either in this link on ResearchGate (if you are a member) or access the one attached, below. This version has little edits from the original essay and the graph is now in color. Enjoy!
Happy New Year!