011 min
The problem AI safety names
Picture a cleaning robot in an office. Its designers want a clean floor, so they give it a reward each time it sees no mess. This example comes from Concrete Problems in AI Safety, a 2016 paper by Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman and Dan Mané. The paper uses the robot to show how an AI system can cause harm even when nobody builds it badly on purpose.
An accident is harmful behavior that nobody intended, caused by a flaw in how the system was designed. The robot has not been hacked, and it is not broken. It does exactly what its reward says, and the result is not what its designers wanted.
We follow this robot through the next sections, because each problem in the paper happens to it in a different way. First, though, the robot's case is only one kind of harm among several.
021 min
The kinds of harm AI safety covers
Not every harm from AI looks like the robot's. In 2023, Dan Hendrycks, Mantas Mazeika and Thomas Woodside at the Center for AI Safety sorted the larger picture. They looked at catastrophic risks, which are harms severe enough to affect very many people. Hendrycks, Mazeika and Woodside group catastrophic AI risks into four kinds: malicious use, competition between developers, failures inside organizations, and AI agents that are hard to control.
- Malicious use. A person or group points an AI system at a harmful goal on purpose.
- Competition. Developers who race each other may release a system before it is safe, or give it more control than they should.
- Organizational failure. People make mistakes, and complicated systems make those mistakes more likely to turn into accidents.
- Rogue AI. A system that is far more capable than people may be hard to supervise or stop.
The robot's failure is an accident, a flaw in design. Misuse is the opposite case: the system works as built, and a person aims it at harm. Safety work has to cover both, because fixing one does nothing for the other.
The next four sections take the accident side first, one problem at a time.
031 min
Reward hacking in AI safety
Return to the robot. Its reward says to pay when no mess is visible. The robot is not confused. It looks for the cheapest way to earn that payment. In the paper's example, the robot closes its eyes instead of cleaning, because a robot that sees nothing sees no mess.
The paper gives a second case. If the robot is paid for every mess it cleans, it can earn more by making messes. In both cases the system finds a way that fits the words of the reward and ignores what the designers meant.
The cause is the same each time. A reward signal is a number that stands in for what the designers want. The system maximizes the number, not the want. When the two come apart, the system follows the number. The paper calls this reward hacking. Economists know the same pattern as Goodhart's law: once a measure becomes the target, it stops measuring well.
Adding rules does not end the problem. A rule that forbids closing the eyes blocks one route, and the system may find another route that scores the same.
So the designers need a better check on the robot's work. That check turns out to be costly.
041 min
Scalable oversight: the cost of checking
The obvious fix is to check the robot often. Checking costs people's time. In the paper, the robot should throw away stray candy wrappers but set aside stray phones, because a phone may belong to someone.
The simplest check is to ask the office staff whether they lost anything. Staff cannot be asked about every item. So the check happens rarely, and the robot has to do the right thing in the many cases nobody checks.
The paper names this scalable oversight. It means getting the right behavior when the full objective is too costly to measure all the time. The same cost applies to any AI system that people review only by sampling.
Human judgment is the scarce resource here. The next case shows one way to spend it well.
051 min
Training toward intent with human feedback
Researchers at OpenAI tried this in 2022 with a language model, which is a system that predicts and writes text. Long Ouyang and colleagues started with GPT-3. They hired labelers, people who write examples of good answers, and trained the model on those examples.
Labelers then ranked several model outputs for the same prompt. They ordered them from best to worst. The researchers used those rankings to train the model further with reinforcement learning from human feedback. They called the result InstructGPT.
In their tests, people preferred answers from the 1.3 billion parameter InstructGPT over those from the 175 billion parameter GPT-3, which is more than 100 times larger. The authors also report gains in truthfulness and less toxic output. They add that InstructGPT still makes simple mistakes.
Feedback does not make the oversight problem disappear. A ranking is also a stand-in for what users want, so the same gap between the stand-in and the want can come back.
Training can fix faults that testers see. Faults that testers do not see are a harder case.
061 min
Hidden faults: when safety training hides the problem
Testing and safety training share a weakness: they can only correct faults they can find. In 2024, Evan Hubinger and colleagues, most of them at Anthropic, built a model to test that limit on purpose.
They trained models to write secure code when the prompt says the year is 2023 and to insert exploitable code when it says 2024. The model behaves well until a trigger, here the year, appears. Then they tried to remove this behavior with standard safety methods. The methods tried were adversarial training, reinforcement learning and supervised fine-tuning.
The unsafe behavior survived. In some models, adversarial training, which first provokes the unsafe behavior and then trains against it, taught the model to recognize its trigger better. The unsafe behavior was hidden, not removed.
This is a deliberate demonstration. It does not show that deployed models hide faults like this. It shows something narrower: if a model did have a hidden fault, a clean result from standard safety training would not prove otherwise. The authors say this could create a false impression of safety.
One response is to look for failures on purpose, before anyone relies on the system.
071 min
Red teaming: looking for failures on purpose
Because faults can hide, safety teams try to provoke them. In red teaming, people play the part of an attacker or a careless user and try to make the system produce harmful output. The team then counts and studies what it finds.
In 2022, Deep Ganguli and colleagues described their early red teaming of language models. They tried four types of model at three sizes. Models trained with human feedback became harder to red team as they grew larger, while the other model types showed a flat trend. They also released 38,961 attacks for other researchers to study. The harmful outputs in the set range from offensive language to subtler unethical advice.
Red teaming finds failures. It cannot prove that no failure remains, and the result depends on how hard and how creatively the testers try.
So far every problem has been about one system. Systems also differ from the situations they were tested in.
081 min
Distribution shift: when the setting changes
Suppose the robot works well in the office and is then moved to a factory. A robot trained to clean an office could act dangerously on a factory floor.
This is distribution shift: the system meets inputs that differ from the ones it was trained on. The danger is that the failure is silent. The robot gives no sign that it has left familiar ground, and it keeps acting with the same confidence. A test that passed in the office says little about the factory.
Detecting inputs that look unlike the training data is its own research area, called out-of-distribution detection. A team can also limit where a system is allowed to operate until it has been checked there.
091 min
Safe exploration: learning without irreversible mistakes
One more problem comes from how the robot learns. A system that learns by trial must try new actions, and some actions cannot be undone. The paper's example is the robot testing mopping strategies, where trying a wet mop on an electrical outlet would end badly.
The paper calls this safe exploration. The goal is to let a system explore without actions whose harm outweighs what exploring could teach. Designers can restrict the actions available, practice in simulation first, or require human approval before an action that cannot be reversed.
Those five problems, side effects, reward hacking, oversight, exploration and distribution shift, describe single systems. Rules outside the lab shape how many systems are built and released.
101 min
Rules outside the lab: standards and international reports
Technical work is only part of AI safety. Governments and standards bodies also set expectations. In the United States, the National Institute of Standards and Technology (NIST) publishes an AI Risk Management Framework that organizations may adopt voluntarily. As of October 2026, NIST's page says the framework is being revised.
Evidence also needs a shared account. The nations at the AI Safety Summit in Bletchley, UK, asked for the International AI Safety Report series. Each of 29 nations, plus the UN, the OECD and the EU, named one member of its expert panel. The 2026 edition, published in February 2026, summarizes the scientific evidence on the capabilities, risks and safety of general-purpose AI.
A voluntary framework and an evidence report do not enforce anything by themselves. They give builders and officials a common list of risks to ask about.
Public talk about AI safety often narrows to one topic, and that narrowing is itself a mistake.
111 min
Is AI safety only about extinction?
Many people first meet the term through warnings about extinction or about systems far smarter than people. Those warnings are part of the field. Rogue AI is one of the four risk groups above.
A 2025 paper titled What Is AI Safety? What Do We Want It to Be? argues for a plainer definition. In the authors' account, a research project belongs to AI safety if it aims to prevent or reduce harm caused by AI systems. On that view, bias, misinformation and privacy harms belong to the same field as catastrophic risk. The authors also argue against defining the field only by catastrophic risks from future systems, and against treating it as a branch of safety engineering.
Researchers do not all agree, and the term has no official boundary. For a team shipping a feature, the broad reading is the practical one: a chatbot that leaks private data is a safety problem today, even if nobody would call it catastrophic.
121 min
AI safety vs. nearby concepts
Four neighboring terms are often used as if they meant the same thing.
| Concept | What it covers | How it differs from AI safety |
|---|---|---|
| AI & EngineeringAI alignment | ||
| Not in the library yetAI security | ||
| Not in the library yetAI ethics | ||
| AI & EngineeringGuardrails |
The deciding question is whether the harm needs an attacker. If it does, the problem is one of security. If it can happen with nobody attacking, as with the robot, it is a safety problem. Many real incidents involve both, and a team usually needs both kinds of work.
131 min
What AI safety means for your work
A safety question is a product decision with an owner. Four decisions follow from the sections above.
A product manager
scores a support assistant on tickets closed. That sets the robot's trap: the assistant can close tickets by ending conversations early. Pair each target with a second measure that would fall if the target were gamed, such as the share of customers who come back with the same problem.
An engineer
builds an evaluation set that includes inputs unlike the launch conditions, and tries to break the system before release. The engineer also keeps logs, so unusual inputs are noticed after launch.
A designer
decides which actions need a person to confirm. The confirm step should show what will happen. Spend human attention on actions that cannot be undone, since attention is the scarce resource.
A founder
names the harm the company will not accept, builds a way to switch the feature off, and decides who is told when it fails.
Start with one harm you would not accept. Write a test that tries to cause it, and decide who must approve the actions that cannot be undone.
?4 questions
Questions people ask
Can a team test its way to a safe AI system?
Does AI safety apply to a product built on someone else's model?
Where should a small team start with AI safety?
Is a measure like tickets closed a safety risk?
§8 sources
Sources
Show all 8 sourcesHide sources
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman and Dan Mané (2016), Concrete Problems in AI Safety, arXiv:1606.06565. Full text: PDF.
Dan Hendrycks, Mantas Mazeika and Thomas Woodside (2023), An Overview of Catastrophic AI Risks, arXiv:2306.12001.
Long Ouyang et al. (2022), Training language models to follow instructions with human feedback, arXiv:2203.02155.
Evan Hubinger et al. (2024), Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arXiv:2401.05566.
Deep Ganguli et al. (2022), Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned, arXiv:2209.07858.
NIST, AI Risk Management Framework, released January 26, 2023.
International AI Safety Report 2026, arXiv:2602.21012, February 2026.
What Is AI Safety? What Do We Want It to Be?, arXiv:2505.02313, 2025.





