What Is P-Hacking and How to Spot It

By Kevin Fernando · Sep 17, 2026 · 10 min read

P-hacking is repeatedly testing different hypotheses or analytical approaches on the same data until reaching statistical significance, then reporting only the successful tests. It occurs because each test carries a 5% false positive rate, and running many tests compounds this risk. A researcher testing 20 independent hypotheses will find roughly one false positive by chance alone, even if no true effect exists.

P-hacking is the practice of testing multiple hypotheses or analytical approaches on the same dataset until finding statistically significant results, then reporting only those that succeeded. It is also called data dredging or selective reporting. The term itself was coined in 2014 by researchers documenting a widespread problem in published science: the gap between what researchers actually tested and what they reported.

P-hacking distorts the published literature by making false findings appear real. When a researcher runs many statistical tests on the same data and reports only the ones that reached statistical significance, readers cannot tell whether the reported result is a genuine effect or a statistical accident. This matters because research synthesis relies on published findings, and if those findings are inflated by selective testing, the conclusions drawn from them are unreliable. Understanding how p-hacking works and learning to spot its markers is essential for anyone reading research or conducting their own studies.

What p-hacking actually is

P-hacking is the misreporting of true effect sizes in published studies that occurs when researchers try out several statistical analyses and data eligibility specifications and then selectively report only those that produce significant results. It is a form of selective reporting, also called data dredging or data snooping.

The core mechanism is simple: a researcher tests many hypotheses on the same dataset, obtains results for all of them, and reports only the ones that reached statistical significance. Readers see only the significant findings and assume they represent genuine effects. They do not see the dozens of tests that failed to reach significance. This creates a distorted picture of what the data actually show.

A common misunderstanding is that p-hacking requires intentional fraud. It does not. A researcher can engage in p-hacking while believing they are conducting honest science. They may genuinely explore their data, test multiple approaches, and report the ones that worked without consciously recognizing that they have inflated their false positive rate. The problem is structural, not necessarily moral: the incentive to publish significant results combined with flexibility in analytical choices creates conditions where false findings emerge by chance alone.

How researcher degrees of freedom create false positives

Researcher degrees of freedom are the analytical choices a researcher can make when working with data. Each choice expands the space of possible tests, and a larger search space makes it more likely that at least one test will produce a significant result by random chance alone.

Think of degrees of freedom as the number of different paths a researcher can take through the same dataset. A researcher might decide which variables to measure, how to code or transform them, which subgroups to analyze separately, which outliers to exclude, or when to stop collecting data. Each of these decisions is defensible on its own. But when a researcher makes many such decisions and then tests hypotheses based on the resulting dataset, they have effectively run many different analyses. The more analyses run, the higher the probability that at least one will produce a significant result purely by chance.

Why multiple testing inflates false positive rates

Each statistical test carries a built-in error rate. At a standard significance level of 5%, one in twenty randomly chosen hypotheses will appear statistically significant by chance alone. When a researcher runs many tests on the same dataset, these chances compound. If you run 20 independent tests at the 5% level, the probability of observing at least one false positive rises to approximately 0.64. You should expect roughly one false positive by chance alone, even if none of the tested relationships have any real effect. The more tests performed, the higher the odds that noise will be mistaken for signal.

Common researcher choices that expand the search space

Researchers face many legitimate decisions when analyzing data, but each choice creates an opportunity to p-hack. Should you exclude outliers or keep them? Should you use the raw data or a transformed version? Should you analyze all participants or only those who completed every step? Should you test for an overall effect or look separately at subgroups like men and women, age brackets, or income levels? A comprehensive review identifies 12 distinct p-hacking strategies. Each choice seems reasonable in isolation, but collectively they create a vast landscape of possible analyses. A researcher can navigate this landscape until finding the path that leads to statistical significance, then report only that path as though it were the plan all along.

A worked example of p-hacking in action

Suppose a researcher collects data on 100 subjects and measures 20 different variables for each one. They want to know whether any of these variables predict an outcome. If they test whether each of the 20 variables predicts the outcome, they are running 20 independent statistical tests on the same dataset.

Under standard significance thresholds, each test has a 5% false positive rate. With 20 independent tests at the 0.05 significance level, the probability of at least one false positive is approximately 0.64. In other words, even if none of the 20 variables actually predicts the outcome, the researcher should expect roughly one false positive by chance alone.

Now suppose the researcher obtains results for all 20 tests, finds that one of them is statistically significant, and publishes that finding. A reader sees a published paper reporting a significant relationship between a variable and an outcome. The reader does not know that 19 other tests were run and failed to reach significance. They do not know that the significant result was likely produced by random chance. The published finding appears to be a genuine discovery when it is actually a statistical accident.

This is p-hacking in its essence: running many tests, reporting only the significant ones, and leaving readers with the false impression that a real effect has been found.

Red flags that signal p-hacking may have occurred

Several reporting patterns and analytical choices suggest that a study's findings may be products of selective testing rather than pre-planned hypothesis testing. Learning to recognize these patterns helps readers distinguish credible findings from statistical artifacts.

Absence of a pre-registered analysis plan

A pre-registered analysis plan, filed before data collection or analysis begins, locks in the researcher's intended approach. It specifies which variables will be analyzed, how they will be measured, which statistical tests will be used, and what counts as a significant result. Without pre-registration, there is no independent record of what was planned versus what was discovered after looking at the data. A researcher who finds an unexpected result can claim it was always the hypothesis of interest, even if it emerged only after exploring the data. The absence of a registered protocol makes it impossible for readers to distinguish between confirmatory tests (planned in advance) and exploratory ones (suggested by the data itself).

Unexpectedly large number of tested relationships

A research paper that reports testing dozens of relationships raises a red flag. If the methods section mentions many outcome variables, many predictor variables, or many potential subgroups, the search space has expanded far beyond what a single study can reasonably explore. A legitimate study typically tests a focused set of hypotheses derived from theory or prior evidence. A paper that reports analyzing ten different outcomes, or testing effects separately in fifteen demographic subgroups, or examining correlations across a large matrix of variables suggests the researchers were searching for significance rather than testing predictions. The more relationships tested, the more likely at least one will appear significant by chance.

Selective reporting of results

Selective reporting occurs when researchers describe some findings in detail while burying others, or when they report only the analyses that reached significance while omitting those that did not. A paper might state that it examined five potential mediators but report results for only three, without explaining why the other two were excluded. It might present a main analysis and then note in passing that a robustness check "yielded similar results" without showing the actual numbers. It might report that a treatment worked for the overall sample but remain silent about subgroups where it did not. These omissions are often subtle and unintentional, but they create a misleading picture. Readers see only the findings that support the narrative, not the full landscape of what was actually tested.

Post-hoc explanations for unexpected subgroup effects

When an analysis reveals that a treatment worked for one subgroup but not another, researchers sometimes construct an explanation after the fact. They might argue that the effect makes theoretical sense for that particular group, even though no prior theory predicted it. They might frame the subgroup effect as a new insight rather than an accidental discovery. This is problematic because subgroup analyses multiply the number of tests performed. If you test a treatment in ten different demographic categories, you are running ten separate tests, each with a 5% chance of a false positive. Finding one subgroup where the effect is significant is unsurprising. Explaining why that particular subgroup should have been special, based on the data you just saw, is a form of p-hacking.

How to spot p-hacking when reading published research

When evaluating whether a study's reported findings are credible or likely products of selective testing, readers can check for several specific markers. These checks do not require statistical expertise; they require only careful reading of the methods section and supplementary materials.

Check for pre-registration or a registered protocol

Look for a statement in the paper indicating that the analysis was pre-registered, or search for a registered protocol in a public repository like the Open Science Framework or ClinicalTrials.gov. A pre-registration shows that the researchers specified their analysis plan before seeing the data. If the paper reports both confirmatory analyses (those specified in advance) and exploratory analyses (those suggested by the data), the distinction should be clearly marked. Be cautious if the paper makes no mention of pre-registration and offers no explanation for how the analysis was planned. This does not prove p-hacking occurred, but it removes an important safeguard against it.

Look for correction for multiple comparisons

When researchers test many hypotheses simultaneously, they should apply a statistical correction that adjusts the significance threshold to account for multiple testing. Common approaches include the Bonferroni correction, which divides the significance level by the number of tests, or the false discovery rate, which controls the expected proportion of false positives among all significant findings. Check whether the paper mentions any such correction. If the researchers tested many relationships but report results at the standard 5% significance level without adjustment, they have not accounted for the inflated false positive rate. The absence of a correction does not guarantee p-hacking, but it suggests the researchers were not taking the multiple testing problem seriously.

Examine the methods section for flexibility

Read the methods section carefully for signs of discretionary choices. Look for vague language like "we excluded participants who" without specifying the criterion in advance, or "we analyzed the data using" without explaining why that particular approach was chosen over alternatives. Notice whether the paper describes testing for outliers or data transformations, and whether these decisions were made before or after examining the results. Check whether subgroup analyses are mentioned, and if so, how many. A methods section that specifies exact inclusion criteria, pre-specifies all outcome measures, and explains the rationale for each analytical choice suggests a planned approach. A methods section that describes decisions in general terms or defers details to supplementary materials may be hiding flexibility in the analysis.

Why p-hacking matters for research synthesis

P-hacking distorts the published literature in a way that makes research synthesis unreliable. When many studies engage in selective reporting, the average effect size reported in the literature is inflated. A meta-analysis that combines published findings will overestimate the true effect because it is combining inflated estimates.

This matters for anyone synthesizing research to inform a decision. If you are reviewing published studies on a topic to understand what the evidence shows, you need to know whether the reported effects are real or artifacts of selective testing. A study that appears to show a strong effect may actually show no effect at all if that strong result was one of many tests and only the significant ones were reported. Identifying p-hacking when reading research is therefore essential to drawing accurate conclusions from the literature.

Related concepts that compound the problem

P-hacking is one of several mechanisms that distort the published literature. Understanding these related concepts helps explain why the problem is so widespread.

Publication bias is the tendency for journals to publish studies with significant results more often than studies with non-significant results. This creates a published literature that is skewed toward positive findings even if many studies found no effect. HARKing, which stands for Hypothesizing After Results are Known, is the practice of presenting a hypothesis that was generated by the data as though it had been specified in advance. The file drawer problem refers to the fact that studies with non-significant results are often never submitted for publication, so they remain in researchers' file drawers rather than appearing in the literature. Together, these mechanisms mean that the published literature contains a higher proportion of false positives than the true rate of false positives in all research conducted. P-hacking is one contributor to this distortion, and it works alongside publication bias, HARKing, and the file drawer problem to make the published literature less reliable than it appears.

Do this automatically

Manually reviewing a researcher's analytical choices across multiple datasets, checking each decision point for outliers excluded, variables transformed, and tests run, requires hours of careful audit work multiplied by the number of studies in a portfolio. Software can systematically flag the statistical patterns and researcher degrees of freedom that signal p-hacking may have occurred, enabling faster detection at scale. Academia research runs this kind of analysis end to end and cites every figure back to its original source, across as many subjects as you need, with Mimir attaching a citation to every figure. It will not write your paper for you. It gathers and cites the evidence; the argument stays yours, and deliberately so. Every plan includes it, including the free one.

Start your first report and run it on something you already know well.

This article is general information and research commentary. It is not professional advice, and it does not account for your particular circumstances.