011 min
ROC curve in short
- What it shows: how many real positives a classifier catches, and how many false alarms it raises, at each threshold you could choose.
- Origin: radar engineers in World War II, then signal detection theory, then medicine and machine learning.
- Confused with: AUC, which is one number taken from the curve, not the curve itself.
- Watch for: a test set with very few positives. The curve can look strong while most alerts are wrong.
021 min
Why the ROC curve matters
A classifier, such as a fraud model or a spam filter, usually does not output a yes or no. It outputs a score. Someone then picks a threshold: every case with a score at or above it is flagged as positive. That choice is a product decision, and it changes who is caught and who is bothered.
A single accuracy number hides this choice. It describes the model at one threshold, and often at a threshold nobody chose on purpose. If a PM approves a model because it reaches 95% accuracy, they may not have noticed that the model reaches it by never flagging anything on a dataset where 95% of the cases are negative.
The ROC curve removes the hidden choice. It shows the result for every threshold on one graph, so a team can ask a better question: which threshold gives us the catch rate we need at a false alarm rate we can afford?
032 min
How the ROC curve is built
Start with the four outcomes of a yes/no decision. A positive case flagged as positive is a true positive. A positive case missed is a false negative. A negative case flagged is a false positive. A negative case left alone is a true negative. A confusion matrix counts these four outcomes for one threshold.
Two ratios come from those counts. Fawcett (2006) describes a ROC graph as one in which the true positive rate is plotted on the vertical axis and the false positive rate on the horizontal axis.
- True positive rate (TPR): true positives divided by all real positives. This is the same quantity as recall, and it is also called sensitivity or hit rate.
- False positive rate (FPR): false positives divided by all real negatives. It is also called the false alarm rate.
To build the curve, take every score the model produced on a labelled test set. Use each one in turn as the threshold. For each threshold, compute one (FPR, TPR) pair and mark it as a point. Join the points from the strictest threshold to the loosest one.
The two ends are fixed. A threshold so high that nothing is flagged gives the point (0, 0): no false alarms and no catches. A threshold so low that everything is flagged gives (1, 1): every positive caught and every negative flagged. Moving along the curve from the first end to the second means loosening the threshold step by step.
Two shapes are worth knowing. The straight diagonal from (0, 0) to (1, 1) is what random guessing produces: at every threshold, the share of positives caught equals the share of negatives flagged. A perfect classifier reaches the top-left corner (0, 1), with every positive caught and no false alarms. Real models fall between the two, and a curve that bends further towards the top-left corner separates the classes better.
TPR = true positives / (true positives + false negatives) FPR = false positives / (false positives + true negatives)
041 min
Where the ROC curve comes from
The name comes from radar. Electrical engineers and radar engineers first developed the ROC curve during World War II, starting in 1941. A radar receiver picks up a faint signal, and an operator must decide if it is an aircraft or just noise. A lower bar for calling a signal an aircraft catches more real aircraft and also raises more false alarms. The curve describes that exact trade-off, and "receiver operating characteristic" refers to how a receiver operator performs.
After the war, psychologists and engineers developed signal detection theory, which uses the same two rates to describe how well any observer separates a signal from noise. Medicine adopted the graph to judge diagnostic tests. Machine learning adopted it later.
The term has not changed meaning on its way. A radar operator, a radiologist and a spam filter all produce a score, and all can be judged by the same graph.
052 min
A worked example with ten positives and ten negatives
The numbers below are illustrative. They are not from a real model. A model scores 20 labelled cases: 10 real positives and 10 real negatives. The scores run from 0 to 1.
- Positives scored: 0.95, 0.90, 0.85, 0.80, 0.70, 0.60, 0.55, 0.45, 0.35, 0.25.
- Negatives scored: 0.75, 0.65, 0.50, 0.40, 0.38, 0.30, 0.20, 0.15, 0.10, 0.05.
For each threshold, a case is flagged if its score is at or above it. Counting gives this table.
| Threshold | True positives | False positives | TPR | FPR |
|---|---|---|---|---|
| 1.00 | 0 | 0 | 0.0 | 0.0 |
| 0.80 | 4 | 0 | 0.4 | 0.0 |
| 0.70 | 5 | 1 | 0.5 | 0.1 |
| 0.60 | 6 | 2 | 0.6 | 0.2 |
| 0.50 | 7 | 3 | 0.7 | 0.3 |
| 0.40 | 8 | 4 | 0.8 | 0.4 |
| 0.30 | 9 | 6 | 0.9 | 0.6 |
| 0.20 | 10 | 7 | 1.0 | 0.7 |
| 0.00 | 10 | 10 | 1.0 | 1.0 |
Plot the last two columns and you have the ROC curve. At a threshold of 0.80 the model catches 40% of positives with no false alarms. At 0.50 it catches 70% and flags 30% of the negatives. At 0.20 it catches every positive, but 70% of the negatives are flagged too.
To check the shape, count pairs. There are 100 pairs made of one positive and one negative. In 81 of them the positive has the higher score. That share, 0.81, is the area under this curve, so the AUC is 0.81 for this example.
Read the table the way a product team would. Nothing in it says which row is correct. The model is the same in every row. Only the threshold changes, and the right row depends on what each kind of error costs.
062 min
Choosing a threshold from the curve
The curve does not choose a threshold. A team does, by pricing the two kinds of mistake. Continue with the same illustrative model, and suppose a subscription app uses it to flag accounts that may cancel. A missed account that cancels costs the business 5 units, for example the margin lost from one customer. A wasted retention offer to an account that was not leaving costs 1 unit.
Total cost at each threshold is 5 times the missed positives plus 1 times the false alarms.
| Threshold | Missed positives | False alarms | Total cost |
|---|---|---|---|
| 0.80 | 6 | 0 | 30 |
| 0.70 | 5 | 1 | 26 |
| 0.60 | 4 | 2 | 22 |
| 0.50 | 3 | 3 | 18 |
| 0.40 | 2 | 4 | 14 |
| 0.30 | 1 | 6 | 11 |
| 0.20 | 0 | 7 | 7 |
| 0.00 | 0 | 10 | 10 |
The lowest cost is at 0.20, so this team flags aggressively. If a false alarm cost 5 and a miss cost 1, the best row would move in the other direction. The graph stays the same. The cost ratio tells you where on it to stand.
Three steps turn this into a routine.
- Write down what a missed positive and a false alarm each cost, in the same unit.
- Estimate how common positives are in the live data, not only in the test set.
- Pick the threshold with the lowest expected cost, and re-check it whenever the costs or the positive rate change.
072 min
How the ROC curve shows up in practice
Comparing two models. A data scientist trains two classifiers and a PM asks which is better. If one curve lies above the other everywhere, the answer is simple: it is better at every threshold. If the curves cross, neither is better overall. One wins in the region where false alarms must be very low, and the other wins where catching nearly every positive matters most. A single number would have hidden that.
Summarising with AUC. Fawcett (2006) states the AUC property this way. The area under the curve equals the probability that the classifier ranks a randomly chosen positive case above a randomly chosen negative case. An AUC of 0.5 matches random guessing and 1.0 is perfect ranking. The number says nothing about any single threshold, which is why the curve is still needed for the final decision.
Tooling. Most machine learning libraries compute the curve from the true labels and the model scores. The curve can be drawn before any threshold is chosen, which is useful when a team wants to judge a model first and set the threshold later.
Medicine. A diagnostic test that returns a number, such as a blood marker level, is a classifier with a threshold. A clinician reads the same graph: how many sick patients does the test catch at each false alarm rate? The decision about the cutoff then depends on the cost of missing a disease and the cost of an unnecessary follow-up test.
A fixed rate target. Often the rule is set by operations, not by the model team. A fraud team may only have capacity to review 1 in 100 transactions. The curve lets them read off the true positive rate at the matching false positive rate, and see what the limit costs in missed fraud.
082 min
When the ROC curve misleads
The curve has one well-known weakness, and it is the same property that is usually praised. Fawcett (2006) notes that ROC curves do not change when the share of positives in the test set changes. That helps when the class mix differs between the test set and live use. It also means the graph cannot show you what the false alarms look like next to the true catches when positives are rare.
The reason is that FPR divides by the number of negatives. If negatives outnumber positives by 1,000 to 1, a very small FPR can still mean a large number of false alarms.
Use the illustrative model again at its 0.50 threshold, with TPR 0.7 and FPR 0.3.
| Test set | Positives | Negatives | True positives | False positives | Precision |
|---|---|---|---|---|---|
| Balanced | 10 | 10 | 7 | 3 | 70% |
| Rare positives | 10 | 1,000 | 7 | 300 | 2.3% |
The ROC point is identical in both rows: (0.3, 0.7). Yet in the second row about 98 of every 100 flagged cases are false alarms. A reviewer working that queue would stop trusting the model within an hour. Precision, the share of flagged cases that are truly positive, shows the difference at once. The ROC curve cannot.
Saito and Rehmsmeier (2015) found that ROC plots on imbalanced datasets can be deceptive about how reliable a classifier is. Their paper recommends the precision-recall plot for those cases.
Other limits are worth checking:
- A small test set. With 10 positives, one more case caught moves TPR by 10 points. The curve is a staircase, and its shape is uncertain.
- Costs are ignored by the graph. Equal-looking points can have very different business cost, as the cost table above showed.
- Different regions matter differently. If only the low-FPR region is usable in practice, a high overall area can come from a part of the curve you will never use.
- Score meaning. The curve depends only on the order of the scores. A model can have a good curve and still give probabilities that are far from the real rates.
091 min
ROC curve versus AUC, precision-recall curve and F1
AUC. The ROC curve is the whole graph. The AUC is one number, the area under it. Two curves with the same AUC can have different shapes, and the shape decides which one suits your cost ratio.
Precision-recall curve. It plots precision against recall (the same quantity as TPR). Because precision uses false positives as a share of flagged cases, not of all negatives, it changes when the class mix changes. Use it when positives are rare.
F1 score. The F1 score combines precision and recall at one threshold. The ROC curve covers all thresholds. A common pattern is to read the curve to choose a threshold, then report F1 at that threshold.
The axis that separates all of these is the question they answer. A ROC curve answers: how well does the model separate the two classes across thresholds? The others answer: how well does it perform at the threshold, or the class mix, we actually have?
101 min
Where to go when positives are rare
When the positive class is a small share of the data, see class imbalance. Report the ROC curve for model comparison, because it does not depend on the class mix. Add precision at the chosen threshold, because that figure depends on the class mix and tells the reviewers what their queue will contain.
A short checklist for a review meeting:
- State how many positives and negatives are in the test set.
- Show the curve, and mark the threshold that is in use.
- Report the true positive rate, the false positive rate and the precision at that threshold.
- Say what a missed positive and a false alarm each cost.
- Say how the positive rate in live data compares with the test set.
?7 questions
Questions people ask
What does a ROC curve show?
What does ROC stand for?
What is a good ROC curve?
How do I pick a threshold from a ROC curve?
Is the ROC curve the same as AUC?
Does class imbalance affect the ROC curve?
Can I use a ROC curve with more than two classes?
§3 sources
Sources
Show all 3 sourcesHide sources
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861-874. (open copy: )
Saito, T. and Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE.
Wikipedia. Receiver operating characteristic, history section.





