Bias, Confounding, and Reproducibility

Lesson 3.1 · 12 min

Common sources of bias

Confirmation bias is the tendency to notice, remember, and record observations that match what you expected, while discounting or forgetting ones that do not. It requires no dishonesty at all, only a normal human mind and an unstructured record.

Selection bias enters when the sample studied is not representative of what the conclusion is claimed about, for example drawing conclusions about a general effect from a single self-selected case. Observer bias enters when the person measuring the outcome also knows which condition is being tested and that knowledge shapes the measurement.

Placebo and expectation effects are real and measurable even in contexts with no institutional oversight. A design that cannot separate an actual effect from an expectation effect cannot claim to have found the former.

Confirmation bias:

The tendency to notice and retain observations consistent with a prior expectation while discounting others.

Observer bias:

Distortion introduced when the person recording an outcome knows the condition being tested.

Lesson 3.2 · 12 min

Reproducibility as the standard

A single striking result, however carefully recorded, is an observation, not a finding. A finding is something that holds up when the same protocol is repeated, ideally by someone other than the original observer, under the same stated conditions.

Reproducibility failures are common and instructive. They can arise from an unrecorded variable that changed between attempts, from a protocol that was underspecified, or from the original result being a statistical fluctuation or an artifact of bias rather than a genuine effect.

Treating a single result as settled fact, and building further work on top of it, compounds the original uncertainty. Careful researchers explicitly flag single, unreplicated observations as preliminary.

Lesson 3.3 · 11 min

Practical bias reduction without institutional resources

You do not need a large team to reduce bias meaningfully. Deciding your measurement criteria and stopping rules in writing before you see any data, and then not revising them afterward, removes a large share of the opportunity for unconscious steering.

Where feasible, coding or labeling samples so that the person taking a measurement does not know which condition they are looking at removes observer bias directly. Where that is not feasible, at minimum record the raw measurement before forming any interpretation of it.

Keep a written log of everything, including the results that disappointed you. A record that only survives when it confirms the hypothesis is not a record, it is advocacy.