Accuracy

Also called Classification accuracy. Also in AI & Engineering, Business

Quick answer

Accuracy is how close a measurement or prediction is to the true value. A scale that reads 70.1 kg for a person who weighs 70.0 kg is accurate. In machine learning the word has a narrower meaning: the share of a model's predictions that are correct. Both meanings ask how far the result is from the truth.

~20 sec read

011 min

Why accuracy matters

People act on numbers as if the numbers were true. A product team reads "the new onboarding raised activation by 4%" and ships it. A support lead reads "the model is 97% accurate" and lets it route every ticket. If the measurement is off, or the 97% hides what the model gets wrong, the decision is wrong even though the number looks exact.

Accuracy is the property that tells you how far a number can be from reality. It is a different property from the number of digits. A reading of 72.13 kg looks more exact than a reading of 72 kg, and it can be further from the true weight.

Two costs follow. A measurement with low accuracy makes every later decision wrong by about the same amount. And one accuracy figure for a model can look high while the model fails on the cases that matter. Both are cheaper to catch before the decision than after it.

022 min

What accuracy actually measures

Start with the error. A measurement error is the measured value minus a reference value. The reference is the true value when it is known. When it is not known, the reference is the value of a calibrated standard, such as a certified 70 kg test mass.

The International Vocabulary of Metrology (VIM, JCGM 200:2012) defines measurement accuracy as the closeness of agreement between a measured quantity value and a true quantity value of a measurand. A measurand is the quantity being measured, such as a person's mass. The same document says a measurement is more accurate when it has a smaller measurement error. The VIM also notes that accuracy is not a quantity and has no numerical value. You report the error or the uncertainty, and you say a result is more or less accurate.

The two parts of an error

An error has two parts that behave differently when you repeat the measurement.

  • Systematic error stays the same, or changes in a predictable way, on every repeat. A scale that always reads 2 kg too high has a systematic error of 2 kg. An estimate of it is called bias.
  • Random error changes unpredictably from one repeat to the next. A scale that reads 69.0 kg, then 71.2 kg, then 70.1 kg for the same weight has random error.

Each part has a name for its absence. Trueness is how close the average of many repeated measurements is to the reference value. It is high when systematic error is small. Precision is how close repeated measurements are to each other. It is high when random error is small. The VIM defines precision as the closeness of agreement between measured values obtained by replicate measurements on the same or similar objects under specified conditions. A replicate measurement is a repeat of the same measurement.

ISO 5725-1 says accuracy consists of two parts, trueness and precision. The four combinations are all possible. A measurement can be precise and not accurate, accurate on average and not precise, both, or neither.

031 min

Where the two meanings of accuracy come from

Two bodies of standards shaped the modern meaning. The International Organization for Standardization (ISO) publishes the ISO 5725 series on accuracy of measurement methods. The Bureau International des Poids et Mesures (BIPM) publishes the VIM, the shared vocabulary of measurement science.

The meaning shifted with the publication of the ISO 5725 series of standards in 1994, and the 2008 edition of the VIM follows it. Before then, many textbooks used "accuracy" for the systematic part only and "precision" for the random part. That older usage did not disappear. Wikipedia's summary notes that accuracy still has two definitions in use, and that the more common one describes systematic error only.

The result is that the same word names two things. When you read a datasheet or a paper, check which one the author means.

041 min

A documented case: Millikan's electron charge

Robert Millikan's oil-drop experiment is the clearest documented case of a shared systematic error. Millikan measured the charge of a single electron by watching charged oil drops fall through air. The physicist Richard Feynman described what happened next in a 1974 Caltech commencement address.

Millikan measured the charge on an electron with falling oil drops and got a value that was slightly off. The cause was an incorrect value for the viscosity of air. The experiment needed that number to convert the drop's speed into a charge, so every result carried the same offset.

Feynman reported that later measurements of the charge came out a little bigger each time, until they settled on a higher number. He said researchers who got a number far above Millikan's value looked for a reason to doubt it.

Two lessons follow. Agreement between measurements shows precision, not accuracy: results that agree with each other can all be off by the same amount. And the reference value is part of the measurement. If the reference is wrong, a careful checker can make a wrong result look right.

051 min

A worked example: two scales and one reference mass

The numbers below are invented for illustration. A staff engineer compares two bathroom scales. They place a certified 70.0 kg reference mass on each scale three times and write down the readings.

Readings (kg)Mean (kg)Mean error (kg)Spread, max minus min (kg)
Scale A72.1, 72.0, 72.272.1+2.10.2
Scale B69.0, 71.2, 70.170.1+0.12.2

Scale A has a small spread and a large mean error. It is precise and has poor trueness. Scale B has the opposite: its mean is almost exact, but a single reading can be 1.2 kg off. Neither is accurate by the ISO 5725 definition, for different reasons.

The fix also differs. Scale A has a fixed offset, so calibration against the reference fixes it: subtract 2.1 kg from every reading. Calibration cannot remove random error. For Scale B, taking the mean of several readings reduces the random error, and the mean of three readings is already close to 70.0 kg.

A product analogy: an analytics script that drops 8% of events from visitors with ad blockers gives counts that are stable from day to day and too low every day. Trends are still readable. Absolute numbers are not.

062 min

Accuracy in classification and the accuracy paradox

In classification, accuracy has a narrower definition. Accuracy is the proportion of all classifications that were correct, whether positive or negative. A classification is one prediction of a label, such as "spam" or "not spam".

Accuracy = (TP + TN) / (TP + TN + FP + FN)

TP and TN are the true positives and true negatives, the predictions that were right. FP and FN are the false positives and false negatives, the predictions that were wrong. A confusion matrix is the table that counts all four.

For multilabel problems, scikit-learn's accuracy_score counts a prediction as correct only if the whole set of labels matches. A multilabel problem gives each item several labels at once. Getting four of five labels right still counts as wrong.

Why a high accuracy can mean nothing

The number counts right answers, but it does not ask which answers were right. Suppose a team screens 1,000 transactions, and 10 of them are fraud. This example is invented, and the counts are worked by hand.

FlagsTPFNFPTNAccuracyRecallPrecision
Model 1none010099099.0%0%not defined
Model 228822097097.8%80%28.6%

Model 1 never flags anything and scores 99.0%. It finds no fraud. Model 2 catches 8 of 10 frauds and raises 20 false alarms, and its accuracy is lower at 97.8%. Only Model 2 is useful for the job. The second row changes one thing from the first: the model now makes some positive predictions.

Google's Machine Learning Crash Course makes the same point: a model that predicts negative 100% of the time would score 99% on accuracy, despite being useless. This result is called the accuracy paradox. The fix is to report recall, the share of real positives the model finds, and precision, the share of flagged items that are real. The F1 score combines the two into one number.

071 min

How accuracy shows up in product, design, and engineering work

Accuracy questions come up in meetings where a number is about to become a decision. Three common ones, each with the question to ask.

  • A growth analyst reading a dashboard. Ask what the tracking misses. Events lost to ad blockers, consent banners or failed requests lower every count by a similar share. The trend is trustworthy and the absolute value is not.
  • A PM reading a vendor claim of "95% accurate". Ask what the test set was, how many items were in each class, and which kind of error costs more. A fraud filter that misses fraud and a spam filter that hides a real email have different costs.
  • An engineer comparing a sensor or a clock to a reference. Ask how it was calibrated, against what, and when. A device that was accurate at installation can drift.

In each case, the useful move is the same: find a reference value, measure the error against it, and report the size and direction of the error instead of one summary word.

081 min

When to report accuracy, and when to report something else

Accuracy needs a reference value. Without one, you can measure precision only, because you can compare readings with each other but not with the truth.

  1. Measurement or estimate, reference known.

    Report the mean error (bias) and the spread (precision) separately. One combined figure hides which problem you have, and the two problems have different fixes.

  2. Classification, balanced classes and equal error costs.

    Accuracy is a fair first summary. Google's guide calls it a coarse measure in this case.

  3. Classification, rare positives or unequal error costs.

    Do not lead with accuracy. Report recall and precision, or the F1 score, and show the confusion matrix.

  4. A sample survey.

    A margin of error covers random sampling error only. It says nothing about bias from who was asked, so a poll can have a small margin of error and still be inaccurate.

If the honest answer is "I have no reference value", say that, and call the result consistent rather than accurate.

091 min

Where accuracy goes wrong

Accuracy fails in a few specific ways. Each has a tell.

  • False precision. A result is reported with more digits than the method supports. The tell: an average shown to four decimals from an instrument that reads to the nearest unit.
  • A wrong reference. The thing you compare against is itself off. Millikan's viscosity value is the example above. In machine learning the equivalent is mislabelled test data.
  • A shared offset. Repeated agreement is read as proof of accuracy. The tell: independent methods give different answers, while each method agrees with itself.
  • Class imbalance. One class is rare, so predicting the common class scores high. The tell: accuracy is above the share of the most common class and recall is near zero.
  • Averages that hide groups. A model can score well overall and badly on a small group, because the small group barely moves the average. Report accuracy per group when the groups matter.
  • Scoring on training data. A model tested on examples it has already seen shows accuracy it will not have on new data.

101 min

Accuracy vs. precision and validity

Accuracy is confused with three neighbours. Precision is the one with a trap, because it has two meanings.

TermQuestion it answersFails when
Accuracy (measurement)Is the result close to the true value?There is systematic or random error
Precision (measurement)Do repeats agree with each other?Random error is large
Precision (classification)Of the items flagged, how many were real?The model raises many false alarms
ValidityDoes the method measure the thing you care about?The measure is a poor stand-in for the goal

The deciding fact: accuracy compares a result with the truth, and measurement precision compares results with each other. Classification precision is a third thing that shares the name. A survey question can be precise, accurate and still invalid, such as asking about intended purchases to learn about real purchases.

111 min

Where the definitions disagree

The main disagreement is about the word, not the evidence. The VIM notes that accuracy is sometimes understood as closeness of agreement between measured quantity values that are being attributed to the measurand. In that reading it means what ISO 5725 calls precision.

Google's guide warns that in machine learning, words like recall, precision, and accuracy have mathematical definitions that may differ from, or be more specific than, more commonly used meanings of the word.

No one disputes the underlying idea. The dispute is over which of three meanings a given author intends: closeness to the truth, closeness of the average to the truth, or the share of correct predictions. While the usage is unsettled, state your meaning in the first place the word appears. Report trueness and precision as two numbers, and report the classification counts, not only the rate.

122 min

Accuracy FAQ

What is accuracy? Accuracy is how close a measured or predicted value is to the true value. A smaller error means a more accurate result. In machine learning it also names a specific rate: correct predictions divided by all predictions.

What is the difference between accuracy and precision? Accuracy compares a result with the true value. Precision compares repeated results with each other. A scale that always reads 2 kg too high is precise and not accurate. In classification, precision has a different meaning: the share of flagged items that are real.

How do you calculate accuracy? For a classifier, divide the number of correct predictions by the total number of predictions. If a model gets 978 of 1,000 right, its accuracy is 97.8%. For a measurement, accuracy has no single formula. You report the error, which is the measured value minus the reference value.

What is a good example of accuracy? Millikan's oil-drop experiment is a documented one. His measurement of the electron charge was slightly off because he used a wrong value for the viscosity of air. Later measurements crept upward over time before they settled on a higher value.

Can a measurement be precise but not accurate? Yes. Repeated readings can agree closely and all be wrong by the same amount. This happens when a systematic error is present, such as a scale that was never zeroed. Calibration against a reference is the usual fix.

Why is 99% accuracy sometimes bad? It can be bad when one class is rare. If 1% of cases are positive, a model that predicts "negative" every time scores 99% and finds nothing. Check recall and precision, and look at the confusion matrix.

How do you improve accuracy? First find which part of the error is large. For systematic error, calibrate against a reference or fix the method. For random error, average more repeats or use a better instrument. For a classifier, improve the training data or change the decision threshold, then check recall and precision as well.

?7 questions

Questions people ask

What is accuracy?

Accuracy is how close a measured or predicted value is to the true value. A smaller error means a more accurate result. In machine learning it also names a specific rate: correct predictions divided by all predictions.

What is the difference between accuracy and precision?

Accuracy compares a result with the true value. Precision compares repeated results with each other. A scale that always reads 2 kg too high is precise and not accurate. In classification, precision has a different meaning: the share of flagged items that are real.

How do you calculate accuracy?

For a classifier, divide the number of correct predictions by the total number of predictions. If a model gets 978 of 1,000 right, its accuracy is 97.8%. For a measurement, accuracy has no single formula. You report the error, which is the measured value minus the reference value.

What is a good example of accuracy?

Millikan's oil-drop experiment is a documented one. His measurement of the electron charge was slightly off because he used a wrong value for the viscosity of air. Later measurements crept upward over time before they settled on a higher value.

Can a measurement be precise but not accurate?

Yes. Repeated readings can agree closely and all be wrong by the same amount. This happens when a systematic error is present, such as a scale that was never zeroed. Calibration against a reference is the usual fix.

Why is 99% accuracy sometimes bad?

It can be bad when one class is rare. If 1% of cases are positive, a model that predicts "negative" every time scores 99% and finds nothing. Check recall and precision, and look at the confusion matrix.

How do you improve accuracy?

First find which part of the error is large. For systematic error, calibrate against a reference or fix the method. For random error, average more repeats or use a better instrument. For a classifier, improve the training data or change the decision threshold, then check recall and precision as well.

▶3 videos

Watch

  • Understanding Accuracy

  • Accuracy and Precision

    Manocha Academy

  • Accuracy And Precision | Pharmaceutical Analysis | B Pharma First Semester

    Imperfect Pharmacy

§5 sources

Sources

Show all 5 sources
  1. Joint Committee for Guides in Metrology (2012). International vocabulary of metrology, basic and general concepts and associated terms (VIM), JCGM 200:2012, items 2.13 to 2.19.

  2. Feynman, R. P. (1974). Cargo Cult Science. Engineering and Science, Caltech.

  3. Google for Developers. Machine Learning Crash Course: Classification, accuracy, precision, and recall.

  4. scikit-learn documentation. sklearn.metrics.accuracy_score.

  5. Wikipedia. Accuracy and precision (a summary of ISO 5725-1 and the VIM definitions).

Keep reading

More from Research

All of Research
All of Research

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.