Evidence Hierarchy and Reading the Literature

Lesson 5.1 · 12 min

The evidence hierarchy

Not all evidence carries the same weight. In rough order of typical strength: systematic reviews and meta-analyses that combine multiple studies sit near the top, followed by randomized controlled trials, followed by cohort and case-control studies, followed by case reports and case series, followed by in vitro and animal studies, followed by anecdote and expert opinion without supporting data.

This ordering describes typical reliability, not an absolute rule. A well-designed animal study can be more informative for a specific mechanistic question than a poorly designed trial in people. The hierarchy is a starting heuristic for how much weight to give a claim, not a substitute for reading the actual work.

A very common distortion in informal research communities is treating an in vitro or animal finding as though it were a demonstrated effect in people. The gap between a cultured cell result and a whole-organism outcome is large, and most such findings never translate.

In vitro:

Conducted outside a living organism, for example in cultured cells or a test tube.

In vivo:

Conducted within a living organism.

Lesson 5.2 · 13 min

The anatomy of a published study

The abstract is a compressed summary and is the section most often quoted out of context. The methods section describes exactly what was done and is where you check whether the study design actually supports the conclusion being drawn from it elsewhere.

The results section reports what was measured, usually with statistics describing how confident the authors are that an observed difference is not simply random variation. The discussion section is where authors interpret their own results, and it is also where the most speculative and least supported statements in a paper tend to live.

The limitations section, when present, is often the most honest part of the paper and the part least likely to be quoted by anyone selling something based on the study. Reading it is often the fastest way to calibrate how much weight the finding deserves.

Lesson 5.3 · 12 min

Claims versus the sources cited for them

A citation attached to a marketing claim does not mean the citation supports the claim as stated. Common distortions include citing an animal study for a claim about human effects, citing a study on one dose or route for a claim about an entirely different one, and citing a preliminary or retracted study without noting its status.

Before accepting a cited claim, find the actual paper, read the methods and results, and check whether the population, model, dose, and outcome measured actually match what is being claimed. This takes minutes and routinely reveals a mismatch.

Secondary summaries, including news coverage and vendor blog posts, compress and often distort findings. Treat them as a pointer to the primary source, never as a substitute for reading it.