ROUGE

Also called ROUGE score, Recall-Oriented Understudy for Gisting Evaluation, ROUGE-N, ROUGE-L. Also in AI & Engineering

Quick answer

ROUGE is a family of automatic scores that rate a generated text summary by comparing it with one or more summaries written by people. It counts overlapping units such as single words, word sequences and word pairs. The main versions measure recall, meaning how much of the reference summary the generated summary also contains. Scores run from 0 to 1.

~20 sec read

011 min

ROUGE in brief

  • What it is: a score for how much a generated summary overlaps with human-written reference summaries.
  • Origin: Chin-Yew Lin, University of Southern California, 2004.
  • Confused with: BLEU, which counts the other way round and is used for translation.
  • Watch for: the score counts words, not meaning, and it depends on which variant and which references you use.

021 min

Why ROUGE matters when you ship a summarizer

A team builds a feature that summarizes support tickets. A new model version scores 2 points higher on ROUGE-L than the old one on the team's test set. This number is illustrative. A PM has to decide whether to tell customers that summaries improved. An engineer has to decide whether to keep the old version as a fallback.

ROUGE cannot settle either decision alone. It counts how many words and word sequences a summary shares with a summary that a person wrote. It does not read either summary for meaning. Two summaries can share most of their words and say opposite things. Two summaries can say the same thing in different words and share few.

A team that reads the score as a quality rating can ship a change that readers like less. It can also reject a change that readers would prefer, because the score did not move. The risk is highest for false statements. The main versions of ROUGE count how much of the reference the summary covers. They do not subtract anything for extra words, so a summary that adds one false sentence can keep a high score if the rest of it matches.

Knowing what ROUGE counts tells you what a difference in score can and cannot show.

032 min

What ROUGE actually measures

ROUGE counts overlapping units of text. A unit is a word, a run of consecutive words called an n-gram, or a pair of words. The text being scored is the candidate summary. The summaries that people wrote for the same document are the reference summaries. Lin's 2004 paper defines four families of measures: ROUGE-N, ROUGE-L, ROUGE-W and ROUGE-S.

ROUGE-N is the simplest. The N is the length of the n-gram. ROUGE-1 counts single words, and ROUGE-2 counts pairs of consecutive words. Lin defines ROUGE-N as an n-gram recall between a candidate summary and a set of reference summaries.

Recall is the share of the reference that the candidate found. For ROUGE-N the calculation is:

ROUGE-N = (reference n-grams that also appear in the candidate) / (all n-grams in the reference)

An n-gram that appears twice in the reference is matched at most twice. Lin explains why this makes ROUGE-N a recall measure: in ROUGE-N the denominator is the number of n-grams on the reference side, so the score is a recall measure. A candidate scores 1 when it contains every n-gram of the reference. It scores 0 when it shares none.

More than one reference

Two people rarely write the same summary, so ROUGE accepts several references. The package does not pool them. It scores the candidate against each reference and keeps the best result. It then applies a procedure called jackknifing. With M references, the package computes the best score over M sets of M-1 references and averages the M results. Lin gives the reason for this. The reference summaries are usually the only human summaries available, so the package needs a way to compare system and human performance. Scoring one reference against the rest lets it estimate average human performance.

Lin adds that more references widen what counts as a good summary. Every reference added to the pool expands the space of alternative summaries. He writes that by choosing which references to add, you can design an evaluation that focuses on different aspects of summarization.

043 min

A worked example of ROUGE

Lin's paper uses four short sentences to show how the measures disagree. The reference, S1, is "police killed the gunman". The three candidates are S2, "police kill the gunman", S3, "the gunman kill police", and S4, "the gunman police killed". S3 reverses who acts on whom, so its meaning is very different from S1. The paper says the same: S2 and S3 have very different meanings.

Against S1, S2 and S3 get the same ROUGE-2 score, because each has one bigram in common with it. A bigram is a pair of consecutive words, and here the shared one is "the gunman". The table below shows the other measures. The ROUGE-1 and ROUGE-2 columns were computed here from the definitions above, as recall. The ROUGE-L and ROUGE-S columns are the values in the paper, which uses F-measure with equal weight on recall and precision.

CandidateTextROUGE-1ROUGE-2ROUGE-LROUGE-S
S2police kill the gunman0.750.330.750.50
S3the gunman kill police0.750.330.500.17
S4the gunman police killed1.000.670.500.33

In the case of ROUGE-L, S2 has a score of 3/4 = 0.75 and S3 has a score of 2/4 = 0.5. ROUGE-L can separate the two because it keeps word order. ROUGE-S gives S2 a score of 0.5, S3 a score of 0.167, and S4 a score of 0.333.

The table shows that the choice of measure changes the ranking. ROUGE-1 puts S4 first, although its word order is the least like the reference, because it counts words and ignores order. ROUGE-S puts S2 first and S3 last, which matches what a reader would say. The paper reaches the same conclusion: S2 is better than S3 and S4, and S4 is better than S3.

What a length change does

A second example uses illustrative numbers. A reference summary has 100 words. Candidate A has 50 words, and 40 of them match the reference. Candidate B has 400 words, copied from the source document, and it contains all 100 reference words.

CandidateWordsMatchedRecallPrecisionF1
A, short50400.400.800.53
B, long4001001.000.250.40

Recall alone ranks B first, even though B is not a summary. It holds all of the source. Precision is the share of the candidate's words that match, and F1 is the combined score of the two. Both rank A first. Recall alone is only safe when every candidate has the same length, which is why summary-evaluation tasks fix a length limit.

052 min

The four families of ROUGE

Each family counts a different unit. The choice changes what the score rewards.

MeasureWhat it countsWhat it adds
ROUGE-NOverlapping n-grams of length NSimple and easy to explain. ROUGE-1 ignores order.
ROUGE-LThe longest common subsequenceKeeps word order without requiring words to sit side by side
ROUGE-WA weighted longest common subsequenceGives more credit to matches that are consecutive
ROUGE-SSkip-bigrams, which are word pairs with any gapCounts every in-order pair, so it is sensitive to order
ROUGE-SUSkip-bigrams plus single wordsGives some credit when no word pair matches

A subsequence is a set of words that appear in the same order in both texts, with other words allowed between them. An advantage of the longest common subsequence is that it needs no consecutive matches but still reflects word order. A second advantage is that it automatically includes the longest in-sequence common n-grams, so no n-gram length has to be chosen in advance.

The longest common subsequence has one weakness. It counts a single longest run and ignores shorter ones. ROUGE-W exists because two candidates can have the same ROUGE-L score even when only one of them has consecutive matches. In Lin's example, the reference is A B C D E F G and the two candidates are A B C D H I K and A H B K C I D. Both have a longest common subsequence of four words, but only the first has them side by side, and ROUGE-W scores them 0.571 and 0.286.

ROUGE-S uses a different idea. A skip-bigram is any pair of words in their sentence order, allowing for arbitrary gaps. Lin notes that unlimited gaps can count meaningless pairs such as "the the", so the package lets you cap the distance between the two words. With a cap of zero, ROUGE-S becomes a bigram overlap score. ROUGE-SU adds single words because ROUGE-S gives a score of zero to a candidate that has no word pair in common with the reference, even if it uses the same words in reverse order.

In the Document Understanding Conference setting, the weight in the F-measure is set so that only recall is considered. Many later tools report precision, recall and F1 for each family, so check which number a paper or tool calls its ROUGE score.

061 min

Where ROUGE comes from

Chin-Yew Lin of the Information Sciences Institute at the University of Southern California introduced ROUGE in 2004. The paper appeared at the workshop Text Summarization Branches Out in Barcelona. The name stands for Recall-Oriented Understudy for Gisting Evaluation. An understudy is a substitute, and a gist is the main content of a text. The score is meant to stand in for a person judging how much of the main content a summary kept.

The reason was cost. Lin wrote that even a simple manual evaluation of summaries at the scale of the Document Understanding Conference would require over 3,000 hours of human effort. That conference, called DUC, is a large yearly evaluation of summarization systems run with the US National Institute of Standards and Technology. A cost like that makes frequent evaluation impractical, and automatic evaluation promised a cheap substitute.

The idea came from machine translation. Lin and Hovy reported in 2003 that scores based on counting co-occurring words, similar to BLEU, could be applied to summaries. Their abstract says that automatic evaluation using unigram co-occurrences between summary pairs correlates surprisingly well with human evaluations. It also says that direct application of the BLEU procedure does not always give good results. ROUGE is the package built from that finding. Three of its measures were used in DUC 2004, a large-scale summarization evaluation sponsored by NIST.

072 min

How well ROUGE agrees with people

Lin tested the measures against human judgments from DUC 2001, 2002 and 2003. In those tasks, human judges gave each system summary a content coverage score by comparing it with a manual summary. Lin then computed the correlation between each system's average ROUGE score and its average human coverage score. As one example, he computed ROUGE scores for the 12 systems that took part in the DUC 2001 single document summarization evaluation. A correlation of 1 means the ROUGE ranking of systems matches the human ranking exactly.

He tested 17 variants, and the conclusions were specific to the task. In single document summarization, ROUGE-2, ROUGE-L, ROUGE-W and ROUGE-S worked well. For very short summaries of about 10 words, such as headlines, ROUGE-1, ROUGE-L, ROUGE-W, ROUGE-SU4 and ROUGE-SU9 did well, and ROUGE-N with N above 1 did significantly worse. For multi-document summaries the results were weaker: a correlation in the high 90 percent range was hard to achieve for multi-document summarization tasks.

Two practical findings follow. Exclusion of stopwords usually improved correlation, and correlations to human judgments were increased by using multiple references. Stopwords are very common words such as "the" and "of".

Lin explains part of the weaker multi-document result by sample size. In the single document summarization tasks there were over 100 samples, while there were only about 30 samples in the multi-document tasks.

The evidence has a limit. Each data point is a whole system, so these correlations show that ROUGE can rank 10 to 18 systems in a way that agrees with people. They do not show that the score of one summary is a reliable measure of that summary.

082 min

Using ROUGE in product and research work

A staff engineer comparing two summarization models on a fixed test set is the case ROUGE was built for. These steps keep the comparison honest.

  1. Fix the reference summaries before you run either model, and use the same ones for both.
  2. Record the variant, such as ROUGE-1, ROUGE-2 or ROUGE-L, and the text processing. Lin's results show that stopword removal and the number of references change the correlation with people.
  3. Compare average scores over the whole test set. A single summary's score tells you little.
  4. Read a sample of summaries from each model next to their scores. Choose summaries where the two models disagree most.
  5. Check facts by hand on that sample. ROUGE does not look at whether a statement is true.

A PM can ask three questions when a team reports a ROUGE number. Which ROUGE variant was it, and which references? How long were the summaries, and were the two models held to the same length? Did anyone read the outputs? If the answers are unclear, the number cannot be compared with any other number.

ROUGE needs reference summaries, so it fits tasks where a good output looks like a written reference. It fits a news summarizer with a standard dataset. It fits worse for summaries that should vary by reader, such as a summary for a manager and another for an engineer. Those need several references or a person's judgment.

092 min

Where the evidence is contested

ROUGE is widely used, and that has produced several objections.

The score has no natural ceiling. Natalie Schluter of the IT University of Copenhagen argued in 2017 that overall quality assurance is problematic: there is no natural upper bound on the quality of summarisation systems, and even humans are excluded from performing optimal summarisation. One reason is arithmetic. It is impossible to obtain 100% ROUGE-n scores unless the reference summaries contain precisely the same n-grams. When references differ, nobody can reach a perfect score, so a team cannot say how close a system is to the best possible result.

It was built for a different time. The 2004 evaluation covered the systems of that period. SummEval, a 2021 study by Alexander Fabbri and colleagues, reports that Peyrard showed that standard metrics agree when summaries are in the scoring range found in TAC summaries, but vastly differ in the higher-scoring range found in current models. The authors of SummEval also report that Rankel and colleagues found higher-order and less commonly reported ROUGE settings had a higher correlation with human judgments. In other words, the variant a field settles on may not be the one that agrees best with people.

It does not measure facts. SummEval reports that Maynez and colleagues found that currently available evaluation methods, such as ROUGE and BertScore, are not sufficient to study the problem of factual faithfulness. A summary that is hallucinated in one sentence and accurate in the rest can score well, because ROUGE counts overlap with a reference and does not check a claim against the source.

It measures wording overlap. SummEval describes ROUGE as a metric that measures lexical overlap between generated and target summaries. A summary that says the same thing in different words has little overlap, so it scores low.

The field's response is to pair ROUGE with human ratings. SummEval notes that the ROUGE metric is typically used together with crowd-sourced human annotations for model evaluation.

101 min

ROUGE vs nearby metrics

The nearest neighbour is BLEU, the score for machine translation that Lin used as his model. The deciding fact is the direction of the counting. BLEU is precision-based: it asks how much of the candidate appears in the references. ROUGE-N asks how much of the reference appears in the candidate, which is recall.

Precision and recall are two halves of one trade-off. Precision rewards a candidate for saying only things the reference says, and recall rewards it for covering everything the reference says. The F1 score combines the two, and Lin's ROUGE-L and ROUGE-S are defined as F-measures.

MetricCounting directionComparesTypical use
This pageROUGE-NSummarizationCounting direction: Reference n-grams found in the candidateCompares: Candidate and human summaries
BusinessBLEUMachine translationCounting direction: Candidate n-grams found in the referencesCompares: Candidate and human translations
BusinessF1 scoreClassificationCounting direction: Combines precision and recallCompares: Predicted and true labels
AI & EngineeringHuman evaluationFinal quality decisionsCounting direction: A person's rating of the outputCompares: The output itself

Human evaluation is the standard ROUGE tries to predict. Lin tested ROUGE by checking how well it ranked systems the way human judges did, so a human rating always outranks a ROUGE score when they disagree.

?8 questions

Questions people ask

What does ROUGE stand for?

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. Chin-Yew Lin introduced it in 2004. An understudy is a substitute, and a gist is the main content of a text.

What is the difference between ROUGE-1, ROUGE-2 and ROUGE-L?

ROUGE-1 counts overlapping single words, ROUGE-2 counts overlapping pairs of consecutive words, and ROUGE-L counts the longest run of words that appear in the same order in both texts. Only ROUGE-1 ignores word order completely.

Is a high ROUGE score good?

A high ROUGE score means the summary shares many words with the reference summaries. It does not show that the summary is true or readable. Compare scores only when the references, variant and summary length are the same.

What is a good ROUGE score?

There is no fixed good score. Schluter showed that a perfect score is impossible unless all references contain the same n-grams, so the ceiling depends on the dataset. Compare a system with other systems on the same test set.

How is ROUGE different from BLEU?

ROUGE-N divides matches by the number of n-grams in the reference, which is recall. BLEU divides by the number in the candidate, which is precision. BLEU was built for machine translation and ROUGE for summaries.

Does ROUGE measure whether a summary is accurate?

No. ROUGE counts overlap with a reference and does not check a statement against the source document. Maynez and colleagues found, as SummEval reports, that ROUGE is not sufficient to study factual faithfulness.

Why does ROUGE use more than one reference summary?

People write different good summaries of the same text. With several references, a candidate is not marked down for choosing a different but valid wording. Lin found that using multiple references increased correlation with human judgments.

Does ROUGE work for tasks other than summarization?

Lin's paper notes that ROUGE-L, W and S were also shown to be effective for evaluating machine translation. ROUGE does not fit open-ended tasks where many different answers are correct, because a reference gives little to match against.

§5 sources

Sources

Show all 5 sources
  1. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, Barcelona.

  2. Lin, C.-Y. and Hovy, E. (2003). Automatic Evaluation of Summaries Using N-gram Co-Occurrence Statistics. Proceedings of HLT-NAACL 2003.

  3. Schluter, N. (2017). The limits of automatic summarisation according to ROUGE. Proceedings of the 15th Conference of the European Chapter of the ACL, Short Papers.

  4. Fabbri, A. R., Kryscinski, W., McCann, B., Xiong, C., Socher, R., and Radev, D. (2021). SummEval: Re-evaluating Summarization Evaluation. Transactions of the ACL, 9.

  5. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the ACL.

Keep reading

More from Research

All of Research
All of Research

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.