evidence
Can You Trust a Study You Found Online?
Sometimes, and you can usually tell within a couple of minutes. Four checks — what it was done in, how many, compared with what, and whether anyone else found the same — do most of the work.
Sometimes — and you can usually tell which within a couple of minutes by checking four things: what the study was done in, how many, compared with what, and whether anyone else has found the same. None of those needs a statistics background, and all four are answerable from the paper in front of you.
The instinct most people reach for instead is whether the study looks official: a database record, a named journal, a long author list. Those establish that a paper exists. But a carefully run human trial and a twelve-animal pilot look identical from the outside, and the difference lives in the methods.
Does being published mean it is reliable?
No. Publication means the paper passed review at one journal, and journals differ enormously in how demanding that is — from several specialists spending weeks trying to break the argument, to acceptance within days of submission. Both produce a citation that looks the same in a search result.
Peer review is also narrower than its reputation suggests. Reviewers read what the authors chose to write. They rarely see the raw data, almost never repeat the analysis, and cannot know how many other analyses were run before this one was submitted. They judge whether the account is coherent, not whether the result is true.
The published record also leans positive. A study that finds nothing is less likely to be written up, less likely to be submitted, and less likely to be accepted when it is — so what reaches print is a filtered sample of the work actually done. Add small studies, modest effects and flexible analytical choices, and the chance that a published positive finding is false rises steeply 1. Publication is a floor, just a lower one than the word implies.
What is the first thing to check?
The model system — whether the work was done in people, in animals, or in cells in a dish. It is the highest-value check available, it takes seconds, and it is nearly always stated in the first line or two of the abstract.
The vocabulary is standard once you know it. "In vitro" and "cell culture" mean cells in a dish; "murine", "rodent", "rats" and "mice" mean animals. Human work says so plainly and usually attaches a number of participants.
It matters because the step from animal to human fails far more often than people expect. When researchers followed up 76 highly cited animal studies, roughly 37% were replicated in subsequent human randomised trials, about 18% were contradicted by them, and around 45% were never tested in humans at all; only 8 of the 76 interventions went on to be approved for patient use 4. A systematic review comparing animal experiments against the human trials of the same six interventions found the animal data agreed for three and disagreed for the other three 2.
So a striking result in mice is a reason to run a human trial, not a preview of one. Cell work sits a step further back again: it shows that a molecule does something to cells under conditions chosen to make an effect visible, with no circulation, no metabolism, and no question of whether that amount could reach the tissue in a living body.
How many participants is enough?
It depends on how large an effect you are looking for, so there is no number worth memorising — but the intuition behind it transfers to everything you will read.
Large effects show up in small samples. If something halves a measurement, two dozen participants may be plenty to see it. The reverse does not hold: a small study that finds nothing has usually shown only that it was too small to tell. "No significant difference" in twenty people is close to uninformative, and it is routinely reported as though it had settled the question.
Harms follow harsher arithmetic. An adverse effect occurring in one person in 500 will, in a study of forty, almost certainly appear in nobody at all. That absence is not evidence the effect is rare; the sample was never large enough for the question to arise. Safety conclusions come from large trials and long-term monitoring, not from early studies.
One habit worth forming: when a headline percentage is quoted, find the number it was calculated from. The same figure from 12 participants and from 1,200 are not the same claim, and in the smaller one a single person moves it by several points.
What does "compared with what" mean?
It means the study included a control group — participants who went through the same process without the treatment — so that a difference can be attributed to the treatment rather than to everything else going on.
A great deal else does go on. Conditions improve on their own. People who volunteer often change other things at the same time. Repeated measurements drift towards the average for purely statistical reasons. And someone expecting to feel better frequently reports feeling better. A placebo control, ideally with neither participant nor assessor knowing who received which, separates the effect of a compound from the effect of expecting one.
So a single-arm study, where everyone gets the treatment and is measured before and after, cannot support a causal claim however clean the numbers look. It can describe what happened; it cannot say what caused it. "Open-label", "case series" and "uncontrolled pilot" all point at this design, and they are usually right there in the title.
Does one study settle anything?
No. A single study is a lead. A finding becomes something you can lean on when a different group, working independently, runs the experiment again and gets the same answer.
Replication is not a formality, and the record on it is sobering. When one team set out to reproduce the findings of 53 landmark preclinical cancer studies, the results held up in only 6 of them 3. These were not marginal papers — they were influential ones, and follow-up work had already been built on several of them.
Little of that gap is misconduct. Most of it is small samples, analytical choices made after seeing the data, results written up from the one experiment that worked, and methods described too thinly to repeat. So the practical question after any single paper is simply: has anyone else found this? Searching the intervention name by itself, rather than following the link you were sent, answers it quickly.
What about reviews and meta-analyses?
They are generally stronger than single studies, with one honest caveat: a review of weak studies is still weak. Pooling does not repair the underlying work. It averages it, and averaging can make shaky evidence look considerably more solid than it is.
The distinction to learn is between systematic and narrative. A systematic review sets out its search strategy in advance, states which studies qualified and why, and reports what it found including the inconvenient parts; a meta-analysis goes a step further and combines those results statistically. A narrative review is an expert summarising the literature as they see it, with no stated method for what was included.
The methods section separates them in about ten seconds: a systematic review names the databases searched, the dates covered and the inclusion criteria, and a narrative review does not, because there was no procedure to describe. Narrative reviews are often excellent reading. They simply cannot be audited, and you cannot see what was left out.
Who funded it, and does it matter?
It matters as a reason to want independent confirmation, not as grounds to dismiss a finding. That distinction is the whole of it, and it is the one most often collapsed.
Funding shapes research mostly upstream of the data: which questions get asked, which comparator a treatment is tested against, which outcome is designated the main one, and whether disappointing results are written up at all. Those choices are made long before any analysis. Outright falsification is rare, and it is not what the concern is really about.
But an interested funder does not make a result wrong. Most trials of medicines are paid for by the companies that make them, because nobody else will fund anything that expensive, and discarding that evidence would leave very little left to read. Treat the finding as provisional and look harder for an independent replication — which is what any unreplicated study deserves anyway. The declarations sit at the end of the paper, under funding and competing interests.
Is the abstract enough, or do I need the full paper?
The abstract is where a study sounds most confident, and it is frequently the only part anyone reads — which is exactly why it misleads so reliably.
It is a summary written by the authors to persuade you the work is worth your time, and compression drops the qualifications first. Effect sizes appear without their uncertainty. A secondary finding gets promoted because it was the interesting one. Careful hedging in the discussion flattens into plain assertion by the time it reaches the summary.
Everything the four checks need lives in the methods — what species or cell line, how many and in how many groups, randomised or not, compared with what, and for how long. Methods sections are dull on purpose and rarely run more than a few paragraphs. Where the full text is not available, the abstract usually still gives the model system and the sample size, which is two of the four.
What is the quickest way to check a study?
In order, stopping as soon as you have what you need:
- Find the model system. Humans, animals, or cells in a dish — usually in the first two lines, and often the only check you will need.
- Find the number, and whether it is the total or the count per group. Two arms of nine is a different study from one of 180.
- Find the comparison. Placebo, an active alternative, or nothing at all — and whether allocation was randomised and assessors blinded.
- Check what was measured and for how long. A marker shifting in a blood test is not an outcome anyone would notice.
- Check the date, then search the intervention name on its own to see whether anyone independent has found the same thing.
- Read the funding and competing-interests declarations at the end. They tell you how hard to look for confirmation, not whether to believe the result.
- Trace the claim that sent you here back to this paper, and check the paper actually says it. This step fails more often than any of the others.
Most of what gets sent around resolves at the first step. A great deal of confident writing about peptides rests on work done in cells or in a small number of rodents, and the paper generally says so in its second sentence. Reading that far is not scepticism — it is reading the study rather than the claim made about it.