Taking shit with a grain of salt: Or how to read health studies.

by Paleo Rob on 15/11/2010

The whole Twinkie Diet fiasco got me really annoyed. The fact that I had to explain why it is bullshit to as many people as I did, made me want to put my head through the wall.


I will have some salt with my shit.

It seems that every other day we get another ridiculous, outlandish study in the media about X causing Y or X found to cure Y. Ladies and gentlemen, these are called attention grabbing fillers. I have a friend who is an assistant editor to a local paper, and he loves these studies because they fill all the empty pockets of the newspaper, and require absolutely zero journalistic thought. (Why am I friends with this man? Cause other than that he has to be one of the nicest guys on the planet). How many times have you read in the paper or seen on TV something along the lines of “Eggs found to give you cancer”, “eating a bowl of cereal will help your hair grow back” or “Drinking pineapple juice before bed will cure your erectile dysfunction”. Too many to count right? Without going off topic too much here, it should be well known that news organisations these days are essential advertising vehicles.

Now of course some of these studies are actually correct and useful, but for the news media to give a shit, they need to be completely mind blowing, something that will make you stop and look at the screen and unfortunately, a lot of medical studies are just boring.

So before you chug down your pineapple juice or buy a bulk box of Twinkies, sit down for a sec and consider the following key questions. I will try to hold back on the cynicism.

Question 1: Was there a study done?

You won’t believe how many articles on the latest miracle cures are based on absolutely shit. A person with a doctorate says something outlandish, and because of his Phd postnomial it is taken as fact. Just because the doctor says it people, doesn’t make it right. How to tell? Articles like these have something along the lines of “however Dr X says”. Example: “… but Doctor ShizenFuller says that drinking the pineapple juice has made Mrs Shizenfuller a much happier lady, thus pineapple juice also helps with depression”.

Question 2: Was the study done on humans?

A lot of medical research is done on animals before it is done on humans. This generally is ok, because we have a lot in common genetically and physiologically with various critters. However there are plenty of times where something works for humans and doesn’t work for animals and vice versa. Give more credit to human studies than to animal one.

Question 3: Was there lots of data-points?

Any statistician will tell you, the more data points you have, the better. Most medical studies are just that, extrapolation of information based on data points. The more you have of them, the more confident you are that whatever conclusions they indicate are correct. Say you discover that if you sleep on the right side of your bed, you sleep for 1hr longer than if you sleep on the left side. Does that immediately mean everyone will have a similar reaction? Of course not, but what if you took a survey, of 5 people, and only 1 disagreed, that’s still 80% chance you are right, but still its only 5 people you surveyed. However if you survey 50, 500 or 5000 people and still got 80% of people agreeing with you, well then you are in the money. The more data points, the better.

Question 4: Was the study a double blind, placebo controlled, randomized study?

When considering new wonder drugs the gold standard of studies is the double-blind, randomized, placebo-controlled study. What does it all mean? Well double-blind and placebo controlled means that the neither the people in the experiment nor the experimenters themselves know who is taking the drug and who is taking the placebo. This is to eliminate the placebo effect in the people taking the drug (sometimes when people think something will do something it actually does, our brain is amazing) and it also takes out the bias of the experimenters (i.e subconsciously or not, an experimenter may give his great new heart medication to people who look healthier than others). Randomized means that to better represent the population as a whole, a wide variety of people are used. i.e different sex, age, weight, height or whatever. The more variation the better and the more people the more variation (i.e more datapoints!).

Question 5: Was the study paid for by people who don’t have vested interests?

This one is simple. Who paid for the study? Ok it may make me seem like a conspiracy theorist, but do you think doctors studying Pineapples on a grant paid for by the Pineapple Association of Australia will really create a study which shows pineapples in a bad light? Or would they find any remote correlation of health benefits from eating pineapples and then exaggerate them a little.

Question 6: Does the headline match the study?

Say that a study passes all the above tests, but all it finds is that after eating pizza people were more likely to smile. Not really that interesting or mind blowing right, but studies are expensive and you need some publicity. So you need to make your headline pop a little more. Let’s think about it, pizza makes people smile, smiling means you are happy, happy means you’re not depressed ergo “THIS JUST IN, PIZZA IS A CURE FOR DEPRESSION”. Ok it may not be out-landish as this but you get the point. Between the study and the headline itself, there is lots of room to adjust.

Take that salt, and shove it down your work colleagues throat.

So there you have it, 6 simple questions to ask yourself, every time you here some outrageous medical claim. Here I will even list them out for you again.

Q1: Was there a study done?

Q2: Was the study done on humans?

Q3: Was there ample data points?

Q4: Was the study a double blind, placebo controlled, randomized study?

Q5: Was the study paid for by people who don’t have vested interests?

Q6: Does the headline match the study?


If you answer no, 3 or more times, then you can pretty much call it as it is, a bunch of faecal matter.

{ 1 comment… read it below or add one }

Evolutionary Diet November 15, 2010 at 2:12 pm

“Was the study paid for by people who don’t have vested interests?”

Is it a coincidence that the California Walnut Commission pays for gazillions of studies promoting the health benefits of walnuts? Has there ever been a pro-walnut study that wasn’t funded by the California Walnut Commission?


Leave a Comment

Previous post:

Next post: