Recognize batch effects, confounding, and reproducibility failures

In BIO-5.1 you met the first way a result lies to you: chance. Test millions of hypotheses and pure luck manufactures false positives, so you correct for it with FDR and strict thresholds. This lesson is about the second, sneakier way. Not luck, but structure. A systematic, non-biological pattern in your data that looks exactly like signal, that survives every multiple-testing correction because it is real and reproducible, and that has nothing whatsoever to do with biology. These artifacts do not average out. They pass the statistics. And they have wrecked more published findings than chance ever did.

Batch effects: the artifact that looks like signal

A batch is a group of samples processed together: the same day, the same machine, the same reagent lot, the same technician, the same sequencing flow cell. A batch effect is a systematic technical difference between batches that shifts the measurements up or down. The reagents from March are not chemically identical to January's. The flow cell you ran on Tuesday is not the one from Thursday. Those differences are small per molecule, but they push on thousands of genes at once, in the same direction, for every sample in the batch.

Here is how it kills a study. You run an RNA-seq experiment (measuring how much each gene is transcribed, from S5 and BIO-4). You sequence your tumor samples in January and your healthy controls in March. Now you find hundreds of genes that differ between tumor and healthy. Real discovery? You cannot know. Disease status is perfectly aligned with run date, so every gene that "differs by disease" might just be reflecting January reagents versus March reagents. Disease and date are welded together, and no amount of statistical power can pry them apart after the fact.

Read the sanity check below and trace it by hand. It asks a blunt question: if I already know the batch label, how much of a gene's variation is left to explain? If the answer is "almost none", the batch is driving the numbers. This panel is illustrative, there is no Run button yet, so reason it through rather than expecting output on screen.

import statistics

def variance_explained_by_batch(expression, batch_labels):
    # expression: one measured value per sample
    # batch_labels: the technical batch each sample came from (e.g. run date)
    # Returns the fraction of total variance that batch alone accounts for
    # (this is the ANOVA quantity eta-squared).
    by_batch = {}
    for value, batch in zip(expression, batch_labels):
        by_batch.setdefault(batch, []).append(value)

    total_var = statistics.pvariance(expression)
    n = len(expression)
    within_var = 0.0
    for values in by_batch.values():
        if len(values) > 1:
            within_var = within_var + statistics.pvariance(values) * len(values) / n

    return 1.0 - (within_var / total_var)

# Two batches, perfectly split by run date. Biology contributes nothing here.
expr = [10.1, 9.9, 10.0, 20.2, 19.8, 20.1]
batch = ["jan", "jan", "jan", "mar", "mar", "mar"]
print(variance_explained_by_batch(expr, batch))   # near 1.0, a red flag

A value near 1.0 means the batch label predicts almost everything. That is not a p-value you report, it is an alarm you heed. The fix lives in the design, not the analysis: spread cases and controls evenly across every batch so that no technical variable lines up with your biological one. You cannot subtract a confound you deliberately confounded.

Confounding: the third variable

Batch is one instance of a bigger trap. A confounder is a variable linked to both your exposure and your outcome, forging a correlation that is real in the data but causally false. Batch is a technical confounder. Biology has its own: age, sex, ancestry, diet, time of sample collection. The famous one in genetics is population stratification. A genetic variant looks associated with a disease, but the variant is really just more common in one ancestry group, and that group happens to have more of the disease for entirely unrelated reasons. The association the model sees is genuine. The causal story you want to tell about it is fiction.

The defense is design, again. Randomize. Balance your groups across every nuisance variable you can think of. And where you cannot balance, at least measure the confounder so you can adjust for it in the model. You cannot correct for a variable you never recorded, which is why "what did you forget to write down" is a more dangerous question than "what was your p-value".

Reference bias: the map skews the territory

Recall the reference genome from S6 and read mapping from BIO-3.3. A reference is a single assembled consensus, and historically it leaned on a handful of individuals of limited ancestral diversity. Now recall how a mapper works: reads that match the reference align cleanly, reads carrying real differences align worse or not at all. Put those together and you get reference bias. The variants you systematically under-call are exactly the ones that differ most from the reference, especially large insertions and truly novel sequence, and that penalty falls hardest on genomes whose ancestry is furthest from whoever built the reference. The tool is not neutral. It sees the world it was shown. This is a live equity problem in medical genomics, and the field's answer is pangenome and graph references that hold many haplotypes at once instead of one.

Overfitting: learning the noise

Overfitting is a model that has memorized the noise in its training data instead of the signal, so it looks brilliant on the data it saw and falls apart on data it did not. Bioinformatics is a perfect breeding ground for it, because the shape of the data is upside down. You typically have far more features than samples: twenty thousand genes, or millions of variants, measured on maybe a few dozen patients (p far greater than n, in the usual shorthand). With more knobs than data points, you can always find some combination of features that perfectly separates your groups, purely by chance. That "gene signature" is not a discovery. It is a fingerprint of your specific noise.

The reproducibility crisis: track everything that touched the data

Step back and the pattern is bleak: a large share of published biological findings do not replicate when someone tries. Some of that is the pitfalls above. But a stubborn share is simpler and more embarrassing. The analysis itself cannot be re-run. The same input data, pushed through a different version of the same tool, produces different variant calls. Aligners change. Callers change. Reference builds change (GRCh37 versus GRCh38 will move coordinates and calls). Default parameters change between releases. If you cannot state exactly which tools, which versions, which parameters, which reference build, and which random seeds produced a figure, then nobody, including future you, can reproduce it or debug it.

Be honest about the ceiling, though. Provenance is necessary and not sufficient. You can reproduce a wrong answer perfectly forever. Pinning gives you the same result twice, which makes errors findable and comparisons fair, but it does not make the result correct. Reproducibility is a floor, not a proof.

Conservation and orthogonal evidence: cheap ways to catch yourself

Before you believe a striking result, run it past two gut-checks that no batch effect can fake.

The first is conservation. Recall multiple sequence alignment from BIO-3.3. If you predict that a mutation at some position matters, ask whether that position has stayed identical across species over hundreds of millions of years. A base held constant across deep evolutionary time is under purifying selection, so a change there is plausibly consequential. A position that drifts freely between species tolerates change, so a "damaging" prediction there earns a raised eyebrow. Conservation is powerful precisely because it comes from a completely different source than your assay, so your reagent lot cannot manufacture it. It is evidence, not a verdict: some real human-specific functional elements are not conserved, and some deeply conserved regions are conserved for reasons that have nothing to do with your hypothesis.

The second is orthogonal evidence. Does an independent method, ideally a different technology with different failure modes, agree? Confirm an RNA-seq expression change with qPCR. Confirm a variant call with Sanger sequencing. Take a GWAS hit more seriously if it also shows up as an eQTL and sits in the right regulatory chromatin (S8). The load-bearing word is orthogonal. The second method must not share the first method's batch, pipeline, or reference, or it merely reproduces the same artifact and dresses it up as confirmation. Two methods that fail the same way are not independent. They are one mistake, counted twice.