F1 Score

Also called F-measure, F-score, F1, Balanced F-score. Also in AI & Engineering

Quick answer

The F1 score is a single number from 0 to 1 that summarises how well a classifier finds positive cases. It is the harmonic mean of precision and recall, so it is high only when both are high. It counts true positives, false positives and false negatives, and it ignores true negatives.

~20 sec read

011 min

Why the F1 score matters

A model can look excellent on accuracy and still do nothing useful. Take a trust-and-safety PM who reviews a model that flags abusive messages. Abuse is 5% of all messages. A model that flags nothing is correct on the other 95%, so its accuracy is 0.95 while it catches no abuse at all.

The decision at stake is whether to ship, and which of two models to ship. Accuracy gives the do-nothing model a high score. The F1 score gives it 0, because it finds none of the positives. That one difference is why F1 is the usual main number for classification problems where the positive class is rare: fraud, abuse, defects, rare diseases, relevant search results.

The cost of reading the wrong number is a product that passes review and fails its users. F1 also has limits, covered below, so it is a better first number, not a final one.

022 min

What the F1 score actually measures

F1 is built from two rates, so they come first. Precision is the share of the items a model flagged as positive that really are positive. Recall is the share of the real positives that the model flagged. Both come from a confusion matrix, the table that counts each kind of outcome. TP means true positives (flagged and correct), FP means false positives (flagged but wrong) and FN means false negatives (real positives that were missed).

The two rates tend to move in opposite directions. A model that flags only the items it is very sure about gets high precision and low recall. A model that flags everything gets perfect recall and low precision. To compare models you need one number, and a plain average of the two rates would let a very good value hide a very bad one.

F1 uses the harmonic mean instead. The harmonic mean of two numbers is the reciprocal of the average of their reciprocals. In practice it stays close to the smaller of the two values.

F1 = 2 × precision × recall / (precision + recall) = 2 × TP / (2 × TP + FP + FN)

The second form is the one to compute from a confusion matrix. It shows what F1 counts: the true positives, twice, against every error on the positive class. True negatives appear nowhere in it.

The harmonic mean is strict. If precision is 1.0 and recall is 0.1, the arithmetic mean is 0.55, which sounds acceptable. F1 is 2 × 1.0 × 0.1 / 1.1, which is 0.18. The weak recall decides the result.

The scikit-learn library, which most Python teams use to compute the score, documents it in these words: "The F1 score can be interpreted as a harmonic mean of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0." That is also the range to expect: 0 when there are no true positives, 1 when there are no errors on the positive class.

031 min

Where the F1 score comes from

The measure comes from information retrieval, the field that studies how well a search system finds the documents a person needs. C. J. van Rijsbergen described a way to combine recall and precision into one measure in his 1979 book Information Retrieval.

The name and the use in machine learning evaluation owe a lot to Nancy Chinchor. In 1992 she described the scoring for the fourth Message Understanding Conference (MUC-4), a contest in which software read news reports and extracted facts such as who attacked what target. She wrote that ranking the systems was hard for a specific reason. Chinchor wrote in 1992 that it was difficult to rank the systems because recall and precision are often equally important yet negatively correlated.

Her answer was van Rijsbergen's F-measure. Her version let the two rates carry relative weights, which gave the measure the flexibility to be used for different applications. A parameter called beta sets the weight of recall relative to precision. The F1 score is the case where beta is 1 and the two rates count equally. The MUC-4 report states that the F-measure is higher when recall and precision are closer to each other, for the same sum.

The term F1 score is therefore a name for one setting of an older, more general measure. Many libraries still accept both names: scikit-learn calls it "balanced F-score or F-measure".

042 min

A worked example with four models

Take 1,000 messages, of which 50 are abusive. This is an illustrative setup, not real data. Four models label the same messages. The table gives each model's confusion matrix and the scores calculated from it.

ModelTPFPFNTNPrecisionRecallAccuracyF1
A: balanced4010109400.800.800.980.80
B: flags everything50950000.051.000.050.095
C: flags nothing0050950undefined0.000.950.00
D: very cautious200309501.000.400.970.57

Check model A by hand. Precision is 40 / 50 = 0.80 and recall is 40 / 50 = 0.80. F1 from the counts is 2 × 40 / (2 × 40 + 10 + 10) = 80 / 100 = 0.80. When the two rates are equal, F1 equals both.

Model B has perfect recall, and its arithmetic mean with precision is 0.525. Its F1 is 2 × 0.05 × 1.00 / 1.05, about 0.095. The perfect recall does not make up for the 950 clean messages it flagged.

Model C shows why accuracy fails here. It scores 0.95, higher than model D. Its F1 is 0, because 2 × 0 / (0 + 0 + 50) is 0. Its precision is 0 / 0, which has no value. Libraries must pick a convention: scikit-learn states that "F1 is by default calculated as 0.0 when there are no true positives, false negatives, or false positives" and lets the caller set the value through its zero_division parameter.

Model D is the most useful comparison. Its precision is a perfect 1.00 because it flags 20 messages and all 20 are abusive. It still misses 30 of the 50. F1 is 2 × 20 / (2 × 20 + 0 + 30) = 40 / 70, about 0.57. The simple average of its precision and recall would be 0.70, which overstates it.

051 min

Weighting recall or precision with F-beta

F1 treats a missed positive and a false alarm as equally costly. Often they are not. The general measure, the F-beta score, lets you set the weight.

F-beta = (1 + beta²) × precision × recall / (beta² × precision + recall)

A beta above 1 gives recall more weight. A beta below 1 gives precision more weight. The two common variants are F2, which favours recall, and F0.5, which favours precision.

Here is the effect on one model with precision 0.80 and recall 0.50. F1 is 2 × 0.80 × 0.50 / 1.30, which is 0.62. F2 is 5 × 0.80 × 0.50 / (4 × 0.80 + 0.50), which is 2.0 / 3.7, or 0.54. F0.5 is 1.25 × 0.80 × 0.50 / (0.25 × 0.80 + 0.50), which is 0.5 / 0.7, or 0.71. The same model looks weak or strong depending on which error the team fears more.

Choose beta before you look at results, and write down the reason. A screening step where a miss is expensive and a human reviews every flag suits a beta above 1. An automatic action that deletes or blocks suits a beta below 1.

062 min

A second case: the same model on rarer positives

F1 depends on how common the positive class is, even when the model does not change. Fix a model at recall 0.80 and a false positive rate of 0.02, meaning it wrongly flags 2% of the real negatives. Run it on two test sets that differ in one variable: the number of negatives.

Test setPositivesNegativesTPFNFPPrecisionF1
X1009008020180.820.81
Y1009,90080201980.290.42

On set X, F1 is 2 × 80 / (2 × 80 + 18 + 20) = 160 / 198 = 0.81. On set Y, F1 is 160 / (160 + 198 + 20) = 160 / 378 = 0.42. Recall is the same on both. The false alarm rate is the same on both. F1 falls by half because the false alarms now outnumber the true positives.

This is a property of precision, and F1 inherits it. A score measured on a balanced test set will not hold on production traffic where positives are rare. When you report F1, report the share of positives in the test set next to it. When you compare two models, use the same test set, such as a shared evaluation set, or the comparison means nothing.

072 min

F1 for more than two classes

F1 is defined for one positive class. For a problem with several classes, a library computes F1 once per class, treating that class as positive and the others as negative. Then it must combine the per-class scores into one. The three common choices give different answers.

  • Micro counts the true positives, false positives and false negatives across all classes first, then computes F1 once.
  • Macro computes F1 for each class and takes the unweighted mean. In scikit-learn's words, this "does not take label imbalance into account".
  • Weighted averages the per-class scores weighted by the number of real examples of each class.

Here is a three-class example with 120 items. Class A has 100 real items, class B has 10 and class C has 10. Per-class counts are below.

ClassReal itemsTPFPFNF1
A1009014100.88
B104460.44
C104460.44

Class A: 2 × 90 / (180 + 14 + 10) = 180 / 204 = 0.88. Class B: 2 × 4 / (8 + 4 + 6) = 8 / 18 = 0.44. Class C has the same counts as B.

Macro F1 is (0.88 + 0.44 + 0.44) / 3 = 0.59. Micro F1 pools TP = 98, FP = 22 and FN = 22, so it is 196 / (196 + 44) = 0.82. Weighted F1 is (100 × 0.88 + 10 × 0.44 + 10 × 0.44) / 120 = 0.81.

The gap from 0.59 to 0.82 matters. Micro F1 is mostly determined by the large class A, and for single-label problems it equals plain accuracy. Macro F1 shows that the two small classes are handled badly. If the small classes matter to users, report macro.

A further hazard is that "macro F1" itself has two definitions. Opitz and Burst (2019) report: "Yet, we find that there exist two different formulas to calculate this quantity." One averages the per-class F1 scores. The other computes F1 from the averaged precision and averaged recall. The two agree only in rare cases, so state which one you used.

082 min

How the F1 score shows up in product, design, and engineering work

Review queues. A trust-and-safety PM sizes a human review team from the flags a model produces. Precision tells them how many reviewers are wasted on clean items. Recall tells them how much abuse reaches users unseen. F1 summarises both for the launch review, but the staffing decision needs the two rates separately. Ask for all three numbers.

Search and recommendations. A search engineer evaluating whether results are relevant uses precision for the first page and recall for coverage. F1 over a labelled query set is a quick way to compare two ranking changes when no single click metric is trusted.

Entity extraction and routing. A support-tooling team that sorts tickets into queues evaluates the classifier per queue. The queue for billing disputes is small and important, so they report macro F1, not micro.

Model selection. An engineer tuning a model picks the decision threshold, the score above which an item counts as positive. Raising it trades recall for precision. Computing F1 at each threshold on a validation set and picking the peak is common. Do this on validation data, never on the final test set, or the reported score is optimistic. Using the test set to choose the threshold is a form of data leakage.

Monitoring. After launch, track F1 on a regularly labelled sample, not only the model's own confidence. If the share of positives in live traffic drifts, precision and F1 change even when the model is untouched, as the second case showed.

091 min

When the F1 score misleads

F1 fails in a few named ways. Each has a tell.

  • It ignores true negatives. Powers (2011) puts it directly: "Neither of these takes into account the number of True Negatives." Tell: a model can look good on F1 while mishandling the negative class.
  • It treats both errors as equal. A missed fraud case and a false alarm rarely cost the same. Tell: the team argues about which error matters. Use F-beta or report precision and recall.
  • It depends on which class is called positive. Swap the labels and the score changes, because the counts it uses change. Tell: two teams report different F1 for the same predictions.
  • It is undefined or zero at the extremes. With no predicted positives, precision has no value and a library returns a default.
  • It needs a fixed threshold. One F1 describes one cut-off. To judge a model across all cut-offs, use a curve such as the ROC curve.

The first failure is easy to produce on purpose. Chicco and Jurman (2020) give a test set where the confusion matrix is TP=90, FN=1, TN=0, FP=9. This classifier labels every healthy person in the set as sick, and still scores accuracy of 0.90 and an F1 of 180 / 190, about 0.95. A score of 0.95 suggests an excellent model. The model correctly identified none of the negatives.

101 min

Where the evidence is contested

Researchers disagree about whether F1 should be used at all. Powers (2011) argues that recall, precision and the F-measure are biased and should not be used without understanding the biases. His abstract says that these measures are "biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic". His proposed replacements are informedness and markedness, which correct for chance.

Chicco and Jurman (2020) go further for binary classification. They write: "We believe that the Matthews correlation coefficient should be preferred to accuracy and F 1 score in evaluating binary classification tasks by all scientific communities." The Matthews correlation coefficient (MCC) is a correlation between predictions and true labels that uses all four cells of the confusion matrix. Their argument rests on synthetic cases such as the one above.

The objection is strongest when the negative class matters. It is weaker for retrieval and extraction tasks, where the set of negatives is huge and unbounded, so true negatives carry little meaning. Information retrieval has used precision, recall and F-measure for decades for this reason.

Until the field settles, a practical rule works. Report F1 with precision and recall. Add MCC or the full confusion matrix when the negative class matters.

111 min

F1 score vs. accuracy and the ROC curve

The nearest neighbour is accuracy, and the deciding fact is which cells each one counts. Accuracy counts correct answers on both classes, so a large majority class dominates it. F1 counts only the positive class, so it is the better choice when positives are rare and the point is finding them.

F1 also differs from the ROC curve. F1 is one number at one threshold. The ROC curve plots the true positive rate against the false positive rate for every threshold, and it does use true negatives. Choose F1 to compare models at a decision you have already fixed. Choose the curve to compare how well models rank items before any threshold is picked.

122 min

F1 score questions

What is a good F1 score? There is no universal threshold, because F1 depends on how common the positive class is and on the task. A score of 0.80 can be excellent for a hard extraction task and poor for an easy one. Compare against a simple baseline on the same test set, and against the F1 of a model that predicts the majority pattern.

What is the difference between the F1 score and accuracy? Accuracy is the share of all predictions that are correct, including true negatives. F1 uses only true positives, false positives and false negatives. When positives are rare, accuracy stays high for a model that finds none, while F1 falls to 0.

How do you calculate the F1 score? Count true positives, false positives and false negatives, then compute 2 × TP / (2 × TP + FP + FN). With TP = 40, FP = 10 and FN = 10 the result is 80 / 100, which is 0.80. Equivalently, it is 2 × precision × recall / (precision + recall).

Why does the F1 score use the harmonic mean instead of the average? The harmonic mean stays close to the smaller of the two values. A model with precision 1.0 and recall 0.1 has an average of 0.55 but an F1 of 0.18. The harmonic mean prevents one strong rate from hiding a weak one.

When should you use F2 or F0.5 instead of F1? Use F2 when missing a positive costs more than a false alarm, for example a screening step before human review. Use F0.5 when a false alarm costs more, for example an automatic block. The number in the name is beta, the weight of recall relative to precision.

What is the difference between micro, macro and weighted F1? Micro pools the counts from all classes before computing F1. Macro computes F1 per class and takes the plain mean. Weighted takes the mean weighted by class size. Macro is the one that shows poor results on small classes.

Does the F1 score use true negatives? No. The formula contains true positives, false positives and false negatives only. This is why it suits problems where positives are rare, and why it can hide poor handling of the negative class.

Can the F1 score be undefined? Yes. If a model predicts no positives and there are no real positives, precision and recall are both 0 divided by 0. Libraries return a default, and scikit-learn returns 0.0 unless you set the zero_division parameter.

?8 questions

Questions people ask

What is a good F1 score?

There is no universal threshold, because F1 depends on how common the positive class is and on the task. A score of 0.80 can be excellent for a hard extraction task and poor for an easy one. Compare against a simple baseline on the same test set, and against the F1 of a model that predicts the majority pattern.

What is the difference between the F1 score and accuracy?

Accuracy is the share of all predictions that are correct, including true negatives. F1 uses only true positives, false positives and false negatives. When positives are rare, accuracy stays high for a model that finds none, while F1 falls to 0.

How do you calculate the F1 score?

Count true positives, false positives and false negatives, then compute 2 × TP / (2 × TP + FP + FN). With TP = 40, FP = 10 and FN = 10 the result is 80 / 100, which is 0.80. Equivalently, it is 2 × precision × recall / (precision + recall).

Why does the F1 score use the harmonic mean instead of the average?

The harmonic mean stays close to the smaller of the two values. A model with precision 1.0 and recall 0.1 has an average of 0.55 but an F1 of 0.18. The harmonic mean prevents one strong rate from hiding a weak one.

When should you use F2 or F0.5 instead of F1?

Use F2 when missing a positive costs more than a false alarm, for example a screening step before human review. Use F0.5 when a false alarm costs more, for example an automatic block. The number in the name is beta, the weight of recall relative to precision.

What is the difference between micro, macro and weighted F1?

Micro pools the counts from all classes before computing F1. Macro computes F1 per class and takes the plain mean. Weighted takes the mean weighted by class size. Macro is the one that shows poor results on small classes.

Does the F1 score use true negatives?

No. The formula contains true positives, false positives and false negatives only. This is why it suits problems where positives are rare, and why it can hide poor handling of the negative class.

Can the F1 score be undefined?

Yes. If a model predicts no positives and there are no real positives, precision and recall are both 0 divided by 0. Libraries return a default, and scikit-learn returns 0.0 unless you set the zero_division parameter.

§6 sources

Sources for the F1 score

Show all 6 sources
  1. Chinchor, N. (1992). MUC-4 evaluation metrics. In Proceedings of the Fourth Message Understanding Conference (MUC-4).

  2. van Rijsbergen, C. J. (1979). Information Retrieval, chapter 7, Evaluation. Butterworths. Author's online edition:

  3. Powers, D. M. W. (2011). Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. Journal of Machine Learning Technologies 2(1), 37-63.

  4. Chicco, D. and Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21, 6.

  5. Opitz, J. and Burst, S. (2019). Macro F1 and Macro F1. arXiv:1911.03347.

  6. scikit-learn developers. sklearn.metrics.f1_score.

Keep reading

More from Business

All of Business
All of Business

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.