The Archives of General Psychiatry just published a study by John Ioannidis which basically says that for whatever reason, studies that say that brain volumes are different in different psychiatric disorders are more likely to be published. Ioannidis has shown similar meta-research type findings before in other fields already (see this article for example), and I highly doubt anyone is really surprised by this finding. Still, it’s cool to see this issue being acknowledged by a relatively high-profile journal like Archives.
Basically, Ioannidis takes a few meta-analyses that have examined “brain volume abnormalities” in different psychiatric disorders from 2006-2009, calculates the number of comparisons across studies that would be expected to have positive results based on statistical power, and then checks that against the actual number of positive findings reported in those studies. To make a long story short, whether he slices and dices it by disorder or by brain structure, studies were largely biased towards reporting statistically significant findings.
Now what would be really nice is if Archives (and other psychiatry journals) actually did something with this finding instead of continuing to publish articles that are basically just neurobabble.
John Ioannidis (2011). Excess Significance Bias in the Literature on Brain Volume Abnormalities Archives of General Psychiatry
Thanks for this interesting post. I'm no expert on this matter, but my guess is that it make an interesting plausible story when writing about brain size and different psychiatric disorders. I can image how hard it to would be to publish an article with no evidence whatsoever. I was watching
ReplyDeleteFalse Positives, False Negatives, and Small Effects: Genome, Exposome, and Nutrition and it seems that this is a widespread in life sciences in general. I'm collecting similar stories related to nutrition just to keep up with these changes.
That kind of bias sure doesn't help cognitive neuroscientists defend against being called "the new phrenology."
ReplyDelete@Sarkis
ReplyDeleteYep, publication bias is a pretty well-known problem in most academic fields. The bigger issue is when you have pretty much the entire academic structure built on top of this - funding, peer review, tenure, and ultimately the direction of discovery and science itself.
@Duval
I hadn't heard that phrase - "the new phrenology" - but I think it's pretty accurate. NIMH has started an entire initiative based on the premise of understanding the neurobiological bases of mental disorder (Research Domain Criteria or RDoc), and will be funding grants based on this. So you can see how this probelm will likely get worse in the foreseeable future.