Personal scientists are well aware of the “replicability crisis” in the social sciences. Surprisingly few peer-reviewed and widely-discussed research results can be reproduced when another group tries to prove the exact same thing. But when attempting to compile a database of studies for her PhD, Anne Scheel found that the problem is even worse than that. For most studies it was nearly impossible to state formally what the researchers were trying to prove in the first place, much less describe whether they succeeded or not.

Just one example: Does screen time affect a person’s well-being? It depends on what you mean by “screen time” and what you mean by “well-being”. Most studies, she notes, don’t bother to define these terms in ways that other, independent researchers could use for replication. Among her suggestions, she concludes that these studies need to move toward more machine-generated hypothesis tests.

For the Personal Scientist, the takeaway is to trust scientific papers less and treat all claims with skepticism.

Prebiotics’ effect depends on the initial configuration

Even when you have a precise target for research, the final result can be less than actionable.

Duke University researchers tested 28 people with prebiotics over a 6 week period. The prebiotics, specific foods known to feed healthy gut microbes (as opposed to probiotics, which are live microbes taken in pill form) were given out at regular intervals and the people were tested to see the effects. The results? The prebiotics had an effect all right: overall “healthy” microbiome composition changed as expected when taking the prebiotics. But when the researchers looked more carefully at the results, they noticed that the changes depended more on the person’s initial microbiome than it did on the specific prebiotics they happened to take. Some people respond well to prebiotics and some people respond barely at all.

What should you take away from this study? That prebiotics affect the microbiome? Not if you’re one of the people below that yellow line.

About Personal Science

In a world of impossible complexity, we often have no choice but to rely on “experts” for medical and other advice. But if you want the truth, or if there’s a specific problem you must solve for health or other reasons, experts are no substitute for your own critical inquiry.

What are the skills necessary to navigate through this overabundance of conflicting information? We believe the techniques of science can help. Personal Science is the practice of applying the scientific method for personal reasons, instead of professional ones.

Share