Analytical Honesty

Also called Honest analysis, Analytic transparency. Also in Business

Quick answer

Analytical honesty is the practice of reporting every analysis choice, every result, and every limit of a study as it happened, including results that contradict the hypothesis. An analyst who practises it does not drop inconvenient data, pick the most favourable test after seeing the numbers, or present a pattern found afterwards as if it had been predicted.

~20 sec read

012 min

Why analytical honesty matters

A growth analyst opens an experiment report on a Friday. The main metric is flat. One segment, new users on mobile, shows a lift with a p-value of 0.03. The team ships the change and reports the segment result.

No number in that report is false. The problem is what the report leaves out: the analyst looked at many segments, and a result that looks significant is likely to appear in at least one of them by chance. A reader who sees only the winning segment cannot tell that it was one of twenty.

The cost is a decision that rests on weaker evidence than it appears to. The team may ship a change that does nothing, then build the next quarter's plan on a lift that never existed. Analytical honesty gives the reader what they need to judge the evidence: how many things were tried, in what order, and what the rest of them showed.

The arithmetic is simple. Suppose each look at the data has a 5% chance of a false positive, which is a result that looks like a real effect when there is none. If the looks are independent, the chance of at least one false positive grows quickly:

Looks at the dataChance of at least one false positive
15.0%
522.6%
1040.1%
2064.2%

Each figure is 1 minus 0.95 raised to the number of looks. A report that shows one of the twenty looks shows the reader a result that had a 64% chance of appearing even if nothing was going on.

022 min

What analytical honesty asks of an analyst

The practice is concrete. It asks for four disclosures, and each one answers a question a reader would otherwise have to guess.

  • Everything that was run. Not only the analysis that worked. If six outcome measures were tested, the report names all six.
  • Every exclusion. Which rows or participants were removed, by what rule, and whether the rule was set before or after the analyst saw the effect of removing them.
  • What was planned and what was found later. A hypothesis written before the data arrived is a prediction. A pattern noticed in the data afterwards is a postdiction. Both can be useful, but they carry different weight, so the report labels each one.
  • Results against the hypothesis, and the limits. A null result, a reversed effect, a small sample, a measure that only approximates what the team cares about. These go in the report, not in a file nobody opens.

The idea behind all four is the same. A reader can only judge a result if they can see how it was produced. Each undisclosed choice removes some of that view.

The choices that need disclosing

Methodologists call the choices open to an analyst researcher degrees of freedom. They include what to measure, which data to exclude, which statistical test to use, when to stop collecting data, and which comparisons to report. None of them is wrong in itself. The problem starts when they are made after seeing the results and then not mentioned.

Simmons, Nelson and Simonsohn (2011) showed that flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. A false positive is a finding that an effect exists when it does not. The authors added that in many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not.

031 min

Honest intent is not enough

It is tempting to think analytical honesty is only about not cheating. The research on analysis choices shows that it is harder than that.

Gelman and Loken (2013) argued that researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data. The authors call this the problem of forking paths. At each step an analyst picks one of several reasonable options. Had the data looked different, a different reasonable option would have been picked. The final analysis feels like the obvious one, but it was shaped by the data.

This matters because the analyst does not feel like they are searching for a result. They run one analysis, and it looks sensible. The reader cannot see that a different dataset would have led to a different sensible analysis, so the false-positive risk is higher than the single test suggests.

Two things follow. First, honesty has to be built into the process, not left to sincerity. Second, an analyst who really did run only one planned analysis can show that, but only if the plan was written down before the data arrived.

042 min

Many analysts, one dataset

The clearest demonstration is a study led by Silberzahn and published in 2018 in Advances in Methods and Practices in Psychological Science. Twenty-nine teams involving 61 analysts used the same dataset to answer the same question: whether soccer referees are more likely to give red cards to dark skin toned players than to light skin toned players.

In the study, twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a nonsignificant relationship. Estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. An odds ratio of 1 means no difference, so the teams disagreed on whether the effect was a little negative, small, or close to three times larger.

MeasureResult across the 29 teams
Teams finding a significant positive effect20 (69%)
Teams finding no significant effect9 (31%)
Lowest estimated odds ratio0.89
Median estimated odds ratio1.31
Highest estimated odds ratio2.93

The teams did not disagree because some were careless. The authors concluded that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. The same data produced a range of answers because every team made different, defensible choices, and the 29 analyses used 21 unique combinations of covariates. A covariate is another variable included in the model to account for its influence.

The lesson for a reader is not that analysis is hopeless. It is that a single published result is one draw from a range, and the report should say which analytic choices produced it.

052 min

How to practise analytical honesty

The most direct method is to write the plan before the data exists. This is called preregistration: the research question and the analysis plan are recorded in a dated place before anyone sees the outcome.

Nosek, Ebersole, DeHaven and Mellor (2018) wrote that mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. Postdiction means a claim built after seeing the data. Prediction means a claim stated before. The authors note that ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid the mistake. After a result is known, it feels as if it was always the obvious outcome.

A working routine for a team has five steps:

  1. Before looking at results, write down the hypothesis, the main metric, the exclusion rules, and the stopping rule.
  2. Run the planned analysis first and report it exactly as planned.
  3. Run any extra analyses, but label each one as exploratory.
  4. List every deviation from the plan and the reason for it.
  5. Report the negative and unclear results in the same place as the positive ones.

Exploratory work is not dishonest. Looking for patterns is how many good hypotheses start. It becomes dishonest only when the pattern is presented as if it had been the question all along.

A scenario for a product team

A staff engineer is asked whether a new onboarding flow raised activation. Before launch, they write down the metric (activation within 7 days), the sample size, and the end date. After the end date, the planned result is a small, non-significant lift. A segment cut shows a larger lift for one acquisition channel. The report gives the planned result first, then the channel finding under a heading that says it was not planned and needs a follow-up A/B test. The decision-maker sees both and knows how much weight each deserves.

061 min

Where analytical honesty falls short

Disclosure makes a result easier to judge. It does not make the result correct. Four limits are worth knowing.

  • An honest report of a weak design is still weak. If two things moved together, saying so openly does not show that one caused the other. See correlation and causation.
  • Honesty about the data does not fix missing data. If the analysis only includes the cases that survived some filter, a full report of what was analysed still leaves out what was filtered. See survivorship bias.
  • More disclosure is not always clearer. A report with 200 results and no ordering hides the important one. The aim is to show what was planned, what was exploratory, and what the main result rests on.
  • Uncertainty needs a number. Saying "this might be noise" is less useful than reporting the margin of error and the sample it comes from.

There is also a pressure problem. A team that is rewarded only for positive results will find it costly to report a null result. Practices such as preregistration help because they make the plan visible before the incentive to change it appears. They do not remove the incentive.

071 min

Analytical honesty vs. nearby ideas

Three ideas get mixed with analytical honesty. They differ in what they describe.

TermWhat it describesDeciding fact
Confirmation biasA tendency to favour evidence that supports what one already believesIt is a habit of the mind, usually unconscious. Analytical honesty is a set of practices that guard against it.
Intellectual honestyA broad attitude toward truth in any fieldIt covers all reasoning. Analytical honesty is limited to collecting, analysing and reporting data.
Fabrication or falsificationInventing or altering dataThese are deliberate misconduct. A report can fail analytical honesty with no false number in it, by leaving things out.

The line that matters in practice is the last one. Fraud is a false statement. A lapse in analytical honesty is usually an omission, and the person making it may not notice.

081 min

Where the idea comes from

No single paper defines analytical honesty, and the phrase has no one attributable origin. It names a practice that sits inside a wider idea, intellectual honesty. Intellectual honesty is described as a method in which relevant facts and information are not purposefully omitted, even when such things may contradict one's hypothesis.

The specific vocabulary for analysis choices came from methods research. Simmons, Nelson and Simonsohn described researcher degrees of freedom in 2011. Gelman and Loken extended the argument in 2013, and Nosek and colleagues set out preregistration as a response in 2018.

?7 questions

Questions people ask

What is analytical honesty?

Analytical honesty is reporting how a result was produced, not only the result. That includes every analysis run, every data exclusion, which choices were planned, and which findings went against the hypothesis.

Is analytical honesty the same as p-hacking?

No. P-hacking is trying analyses until one gives a significant result. Analytical honesty is the opposite practice: reporting all the analyses so a reader can see how much searching happened.

Can an analyst be dishonest without lying?

Yes. Leaving out the analyses that did not work, or changing the question after seeing the data, creates a misleading report with no false number in it.

How do you practise analytical honesty on a product team?

Write the metric, sample size, exclusion rules and stopping rule before the experiment starts. Report the planned result first, and label any later finding as exploratory.

Does preregistration guarantee honest analysis?

No. It makes the plan visible and dated, so deviations can be seen. A preregistered study can still use a weak design, so it supports honesty but does not replace good method.

Why do different analysts get different answers from the same data?

Each analyst makes different reasonable choices, such as which variables to include. In one study, 29 teams analysing one dataset produced odds ratios from 0.89 to 2.93.

Is exploratory analysis dishonest?

No. Exploring data is how many hypotheses begin. It is honest when the report labels the finding as exploratory and treats it as a lead to test, not a confirmed result.

▶3 videos

Watch

  • Understanding Analytical Honesty

  • Mengapa Scorpio Begitu Sulit Ditembus? Apa yang Membuat Mereka Begitu Misterius?

    Zodiak dan Astrologi

  • Thaqib Malaysia sebut indonesia tunduk dengan asing tidak seperti negaranya

    INformasi

§5 sources

Sources

Show all 5 sources
  1. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11). PubMed

  2. Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. Columbia University

  3. Silberzahn, R., Uhlmann, E. L., et al. (2018). Many analysts, one dataset: making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science, 1(3). White Rose repository

  4. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. PNAS, 115(11). PubMed Central

  5. Intellectual honesty. Wikipedia, a summary of the wider idea. Wikipedia

Keep reading

More from Research

All of Research
All of Research

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.