Zero-Shot Learning

Also called ZSL, zero-data learning, dataless classification, zero-shot classification

Quick answer

Zero-shot learning is a machine learning setup in which a model handles a class or task it has no labelled examples of. Instead of examples, the model gets a description: a label name, a list of attributes, or an instruction. It compares each new input with that description, so teams can add new categories without first collecting training data.

~20 sec read

011 min

The problem zero-shot learning solves

A support team at a subscription app sorts every incoming ticket into one of three queues: billing, login and bugs. A classifier does the sorting. It learned the job from thousands of past tickets that agents had labelled by hand. This is supervised learning: the model learns from examples that already carry the right answer, so it can only ever reply with a label it saw during training.

Then the product team launches a family plan. Within a week, new tickets say things like my partner never got the invite. None of the three queues fits. The classifier must still choose one, so it sends each of these tickets to billing or login. There they wait for the wrong agents.

The usual fix is slow. Someone collects a few hundred family-plan tickets, labels them, and retrains the model. Until that work is done, the new queue cannot exist. The next launch starts the same cycle again.

Most real label sets change like this. New products, new kinds of fraud, rare diseases and new object types appear faster than people can label examples of them. Zero-shot learning removes the wait. The team writes down what the new queue means. The model uses that written meaning where it would normally need examples.

022 min

Where zero-shot learning came from

The idea that a label's meaning can replace examples of it is older than its name. Three research groups reached it in 2008 and 2009, working on text, brain scans and photos.

Dataless classification, 2008

Ming-Wei Chang, Lev Ratinov, Dan Roth and Vivek Srikumar at the University of Illinois started from a simple observation. A person can sort a forum post into a named category without training, because they know what the category name means. The team used Wikipedia as background knowledge. They turned both the category name and the document into sets of related concepts, then compared the two sets. They called the method dataless classification. With no labelled data from the task, it sorted posts from the 20 Newsgroups dataset with 85.29% accuracy.

The name zero-shot learning, 2009

Mark Palatucci, Dean Pomerleau, Geoffrey Hinton and Tom Mitchell used the term zero-shot learning in a 2009 paper at the NIPS conference. Their classifier did not learn a fixed list of answers. It learned to predict meaningful properties of the answer, then looked up which word in its knowledge base had those properties. Their test case came from neuroscience. From fMRI scans, which record activity in the brain, the classifier could often tell which word a person was thinking about, even for words it had no scans of. The name was a play on one-shot learning, an older term for learning a class from a single example.

Attributes, 2009

In the same year, Christoph Lampert, Hannes Nickisch and Stefan Harmeling built a dataset called Animals with Attributes. It holds more than 30,000 photos of 50 animal classes, each class described by 85 properties such as "striped" or "lives in water". A model trained to spot the properties could then recognise animals it had no photos of, from their property lists alone.

032 min

How zero-shot learning works

Every zero-shot system, from the 2009 animal model to the support team's classifier, has the same three parts. The seen classes are the ones with labelled examples: billing, login and bugs. The unseen classes have no examples at all: the new family-plan queue. The auxiliary information is a description that exists for both kinds, such as a label name, a sentence or a list of properties.

During training, the model learns how inputs relate to descriptions, using only the seen classes. At prediction time it applies that same relation to descriptions it never trained on. For the support team, the steps look like this:

  1. The support lead writes one short line for each queue, including the new one: an invite to the family plan did not arrive.
  2. A pretrained model turns the ticket and each line into an embedding: a list of numbers in which texts with similar meanings sit close together.
  3. The model measures the distance between the ticket and each line.
  4. The ticket goes to the queue whose line is nearest.

The support lead writes one line per queue, and the new ticket goes to the family plan queue because its meaning is closest to that line. Step 2 does the real work. The model never saw a family-plan ticket. It places my partner never got the invite near the family-plan line because it learned how words relate from a very large amount of other text. This is a form of transfer learning: knowledge from one set of data, reused for a problem the model was not trained on.

Three ways to build the shared space

  • Attributes. Each class is a list of yes-or-no properties. The model learns to detect the properties, then picks the class whose list fits best. The 2009 animal model worked this way.
  • Embeddings. Inputs and class descriptions go into one space of numbers, and the nearest description wins. Most current systems work this way.
  • Generated examples. A generative model, such as a GAN or a variational autoencoder, produces artificial training examples for the unseen classes from their descriptions. An ordinary classifier then trains on them, and the problem becomes normal supervised learning.

Zero-shot text classification as entailment

For text, a common method changes the question the model answers. It turns each label into a sentence, such as This text is about family plan invites. Then it asks whether the ticket implies that sentence. A model trained on natural language inference answers this. Natural language inference is the task of deciding whether one sentence follows from another. Wenpeng Yin, Jamaal Hay and Dan Roth framed zero-shot text classification as this kind of entailment test in 2019.

new ticketinvitequeuesfamily planbillingloginbugs
Look at where the ticket ends up: in the one queue that had no training examples, ahead of the three older queues.

042 min

A concrete example of zero-shot learning: CLIP

Text was the easier case, because the ticket and the queue line are both words. Photos are harder: an image is pixels and a label is a word, so the model needs a space that holds both.

In 2021, a team at OpenAI led by Alec Radford published CLIP (Contrastive Language-Image Pre-training). They collected 400 million images from the internet together with the text that came with them. CLIP has two parts: an image encoder and a text encoder. Each turns its input into an embedding. Training gave the model one job: for a batch of images and captions, find which caption belongs to which image. Contrastive training means the model pulls the embeddings of matching pairs together and pushes wrong pairs apart.

To use CLIP as a classifier, you write one caption for each class. Given a photo of a golden retriever, a researcher writes captions such as A photo of a dog. and A photo of a cat., and CLIP chooses the caption whose embedding sits nearest to the photo. No labelled dog photos are needed for this step.

The headline result came from ImageNet. ImageNet is a standard test of 1,000 object classes. On ImageNet, CLIP's best model reached 76.2% accuracy and matched the original ResNet-50 without using any of the 1.28 million labelled training photos. ResNet-50 was a widely used model trained on exactly those photos. CLIP saw none of them.

Why the wording of a zero-shot label changes the result

The class name on its own worked badly. In CLIP's training data, the text next to an image was usually a full sentence, rarely a single word. So the team put each label inside the template A photo of a {label}. This template alone raised CLIP's accuracy on ImageNet by 1.3%. Adding context helped more on narrow datasets: a type of pet for pet breeds, a type of food for dishes. Averaging the results of many templates raised zero-shot accuracy by almost 5 points on average across 36 datasets.

This matters for the support team. The line they wrote for the family-plan queue now decides which tickets the model sends there. Changing its wording is the cheapest way to change which tickets land there.

photocaptionphoto of a dogcaptionphoto of a cat
The answer is one of the written captions. Nothing in the picture was labelled dog before the captions were written.

052 min

Zero-shot learning in language models

CLIP and the support classifier both choose a class from a list. Language models used the term for something larger: the unseen thing became the whole task.

In 2016, a Google team led by Melvin Johnson trained one translation model on several language pairs at once. A token at the start of each sentence told the model which language to produce. The model could then translate between pairs it had never seen together in training. The team called this zero-shot translation.

In 2020, the GPT-3 paper by Tom Brown and colleagues at OpenAI used the word in a new sense. There, zero-shot means the model receives only a written instruction for the task, with no worked examples in the prompt. Few-shot means the prompt also includes a few examples. In both cases the model's weights, the numbers it learned in training, do not change.

Instruction tuning makes zero-shot prompts work better

A base large language model often does poorly with an instruction alone. In 2021, Jason Wei and colleagues at Google tested a fix. They took a 137-billion-parameter model and trained it further on more than 60 tasks, each written as a plain instruction. They called the result FLAN. On task types it had not been trained on, FLAN beat zero-shot GPT-3, a 175-billion-parameter model, on 20 of 25 tasks. Training on many tasks phrased as instructions taught the model to follow new instructions.

Comparing the two cases shows what stayed the same. In the classic setup, the unseen thing is one class inside a known task. In language models, it is the task itself. In both, zero-shot means no labelled examples for this specific job. It never means the model learned from nothing. The support team could use either route. They could keep their own classifier, or send each ticket to a language model with the list of queues as the instruction.

062 min

Where zero-shot learning breaks down

The support team's first week with the new queue shows the most common failure. Most family-plan tickets still land in billing.

Seen classes win in generalized zero-shot learning

Early research tested zero-shot models on unseen classes only. Real traffic mixes old and new classes, and this mixed case is called generalized zero-shot learning. In 2016, Wei-Lun Chao and colleagues showed that models which did well on unseen classes alone did badly once the seen classes were back in the list. Their test examples from unseen classes were almost always given a seen-class label. The cause is training: the model learned to give high scores to the seen classes, and it keeps doing so.

Calibrated stacking

Chao's team proposed a simple correction called calibrated stacking. Before picking the top class, subtract one fixed number from the score of every seen class. A larger number moves more inputs to the unseen classes, and a smaller one keeps more in the seen classes. The support team can do the same: lower the scores of billing, login and bugs by a small amount, then count how many family-plan tickets move to the right queue. Researchers now report the harmonic mean of seen-class and unseen-class accuracy, so a model cannot look good by ignoring one group.

Descriptions that do not separate the classes

Three other failures come from the shared space itself.

  • Domain shift. The link between inputs and descriptions was learned on seen classes, and it may not fit the unseen ones. A model trained on animals may place descriptions of furniture badly.
  • Hubness. In a space with hundreds of dimensions, a few points end up as the nearest neighbour of many inputs. The tell is one class that receives far more inputs than it should.
  • Semantic loss. Training on seen classes can teach the model to ignore properties that never mattered there but separate the unseen classes. Two queues described as "refund request" and "charge dispute" may look the same to it.

Inputs far from the pretraining data

Zero-shot models do well only where their pretraining data was rich. CLIP read printed text in images well, because the internet is full of it. But CLIP reached only 88% accuracy on the handwritten digits of the MNIST dataset. A basic logistic regression on raw pixels, one of the simplest models there is, did better. Test a zero-shot model on specialised images, such as medical scans or factory defects, before trusting it there.

072 min

Is zero-shot learning really zero-shot?

The MNIST result raises a harder question. If a zero-shot model only does well where its pretraining data was rich, what did it really learn without examples?

In 2017, Yongqin Xian, Christoph Lampert, Bernt Schiele and Zeynep Akata reviewed the field's benchmarks. They found that many published results could not be compared, and some were wrong. One reason was that image features had been pretrained on photos of the classes later used as unseen test classes. The test classes were not really unseen. The authors built a single benchmark with new data splits, and a new dataset, Animals with Attributes 2.

Zero-shot accuracy follows how often a concept appeared in pretraining

In 2024, Vishaal Udandarao and colleagues tested the same doubt on modern models. They studied 34 multimodal models and five large pretraining datasets. A model's zero-shot accuracy on a concept followed how often that concept appeared in its pretraining data. To raise accuracy by a fixed step, a model needed many times more examples of the concept. On rare concepts, every model they tested did poorly.

This does not make the method useless. It changes what the word promises. Zero-shot means no labelled examples for this task. It does not mean the model has never met the concept. For the support team, "invite" and "family" are common words, so the new queue works. A queue for a rare legal term or an internal product code name would be much less reliable. Test on your own rare classes before trusting a published zero-shot number.

082 min

Zero-shot learning vs. few-shot and transfer learning

If zero-shot learning still depends on earlier training, the next question is how it differs from methods that use a few examples. The deciding fact is how many labelled examples of the target class the model gets when it is put to work.

ApproachLabelled examples of the target classDo the model's weights change?
This pageZero-shot learningNoLabelled examples of the target class: None. Only a description
Not in the library yetOne-shot learningSometimesLabelled examples of the target class: Exactly one
AI & EngineeringFew-shot learningSometimes. In prompting, noLabelled examples of the target class: A handful, often 1 to 5
AI & EngineeringFine-tuningYesLabelled examples of the target class: Hundreds or more

Transfer learning is the wider family that contains all of these. It means reusing what a model learned on one task for another. Zero-shot learning is the case where the reuse happens through a description alone.

Zero-shot learning vs. zero-shot prompting. Zero-shot learning is the general research setup. Zero-shot prompting is one way to do it: give a language model an instruction with no examples.

When zero examples beat a few

It seems obvious that a few examples should beat none. CLIP's results did not always show that. On average across its test datasets, zero-shot CLIP matched a classifier trained on four labelled examples per class, using the same image features. The CLIP team gave a likely reason. A written caption states exactly what the class is. A few photos leave the model to guess which of their features matter.

The usual answer is to use both, in order. Start zero-shot on day one. Collect the inputs the model is unsure about, label them, and move to few-shot or fine-tuning once you have enough.

092 min

Using zero-shot learning in your work

That order, zero-shot first and labelled data later, is the main decision zero-shot learning gives a team. Teams use it to sort support tickets, emails and reviews, to tag products from their descriptions, to detect objects named in plain text, and to recommend new items that have no usage history yet. Each role makes a different decision with it.

  • Product manager. A new category can ship on launch day instead of after a labelling project. Decide in advance what accuracy is good enough, and when the category moves to a labelled model.
  • Designer. Label names and descriptions are now product text that changes results, as CLIP's 1.3% from one template showed. Write each description so it is clearly different from its neighbours. Show low-confidence results for human review instead of hiding them.
  • Engineer. Build the test set the way traffic really arrives: old and new classes mixed. Tune the seen-class correction on that set. Watch for one class that suddenly receives far more inputs than expected.
  • Founder. Zero-shot lowers the cost of testing a new category to almost nothing. The lasting advantage comes later, from the labelled data you collect while it runs.

What zero-shot learning costs

There is no training bill for a new class, but each prediction can cost more. The entailment method runs the model once per label, so a ticket checked against 50 queues needs 50 runs. The embedding method computes each label's embedding once, stores it, and compares the ticket against all of them in one step. With many labels or a lot of traffic, choose the embedding method.

Use zero-shot learning when new classes appear faster than you can label them, and the classes are described in common words. Plan to replace it with a labelled model when a class matters enough that its errors cost more than the labelling.

?4 questions

Questions people ask

Can a zero-shot classifier give one input more than one label?

Yes. Score each label on its own instead of choosing the single best one, and keep every label above a threshold. The entailment method suits this, because it already answers yes or no per label.

Do I need a GPU to run zero-shot classification?

Not always. Small entailment and embedding models run on an ordinary CPU when traffic is low. Large image-and-text models, or high traffic, usually need a GPU or a hosted service that runs one.

Is zero-shot learning a kind of unsupervised learning?

No. Unsupervised learning finds structure in data with no labels at all. Zero-shot models are trained on labelled or paired data, then applied to classes that had no labelled examples.

Which datasets are used to test zero-shot learning?

Image research mostly uses Animals with Attributes 2, CUB, a set of bird species photos, and SUN, a set of scene photos, with the shared splits from Xian and colleagues. Text research often uses the benchmark from Yin and colleagues.

§11 sources

Sources for zero-shot learning

Show all 11 sources
  1. Chang, M.-W., Ratinov, L., Roth, D. & Srikumar, V. (2008). Importance of Semantic Representation: Dataless Classification. AAAI 2008.

  2. Palatucci, M., Pomerleau, D., Hinton, G. & Mitchell, T. (2009). Zero-Shot Learning with Semantic Output Codes. NIPS 2009.

  3. Lampert, C. H., Nickisch, H. & Harmeling, S. (2009). Learning to Detect Unseen Object Classes by Between-Class Attribute Transfer. CVPR 2009. ISTA research repository record.

  4. Johnson, M. et al. (2016). Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. arXiv:1611.04558.

  5. Chao, W.-L., Changpinyo, S., Gong, B. & Sha, F. (2016). An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild. ECCV 2016. arXiv:1605.04253.

  6. Xian, Y., Lampert, C. H., Schiele, B. & Akata, Z. (2017). Zero-Shot Learning: A Comprehensive Evaluation of the Good, the Bad and the Ugly. arXiv:1707.00600.

  7. Brown, T. et al. (2020). Language Models are Few-Shot Learners (GPT-3). NeurIPS 2020. arXiv:2005.14165.

  8. Yin, W., Hay, J. & Roth, D. (2019). Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. EMNLP 2019. arXiv:1909.00161.

  9. Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML 2021. arXiv:2103.00020.

  10. Wei, J. et al. (2021). Finetuned Language Models Are Zero-Shot Learners (FLAN). arXiv:2109.01652.

  11. Udandarao, V. et al. (2024). No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance. arXiv:2404.04125.

Keep reading

More from AI & Engineering

All of AI & Engineering
All of AI & Engineering

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.