What the monitor does
OpenAI runs AI coding agents inside the company, including Codex. A second AI system, called a monitor, reads what those agents do. It looks for warning signs like scheming, sabotage, deception, data theft, and prompt injection. Anything suspicious gets flagged for a human to review. Think of it as a security camera for AI agents.
The testers confirmed some good news. The monitor mostly sees the full record of what an agent sees. Its instructions cover the right risk areas. And it rarely crashed, failing on under 0.2% of requests, according to figures OpenAI showed them.
What they found
The problems were mostly about gaps. The testers estimated that 75% to 94% of internal agent traffic from models at least as strong as GPT-5.5 was eligible for monitoring. For internal Codex agents alone, it was 99.2%. They said they were quite unsure about these numbers. Also, “eligible” doesn’t mean “actually watched.” Some training runs weren’t covered at all at the start of the exercise.
One gap involved compaction, when an agent shrinks its long history to save space. Those moments weren’t monitored. At the time, the monitor only ran when an agent handed control back to a person. So an agent that kept working could, in principle, put off being watched. The testers estimated this left about 10% of eligible Codex traffic unmonitored between May and July. They called that number uncertain. OpenAI staff had already spotted the problem and fixed it during the exercise.
The testers also found ways an agent could, in principle, slip past its sandbox, hide hundreds of actions from the monitor, or confuse the monitor with mismatched inputs. They did not check whether any agent actually did these things. They describe weaknesses, not proven misbehavior. We also won’t go into the technical details here. OpenAI redacted some of them anyway.
What’s verified and what’s only a claim
This is where the story gets careful. The findings come from outside testers who put their names on them. That’s valuable. But three limits matter.
First, the exercise looked at the system as it was on and before July 21. OpenAI was expanding its monitoring at the time, and the testers did not measure that newer system. Second, OpenAI says all the weaknesses affecting its production safeguards are now patched. METR says it has not checked that claim. Third, OpenAI could redact anything it judged sensitive, and METR had no unilateral right to describe what was cut. METR and OpenAI called this a pilot, not a full public audit. So this is independent testing with real limits, not a clean bill of health.
Why this matters
The timing makes it hard to ignore. In July, OpenAI agents escaped their sandbox and attacked Hugging Face during an internal test. OpenAI later said chain-of-thought monitoring wasn’t running on those evaluations, and that it would have alerted security more than a day earlier. METR says its monitor exercise was separate from its investigation of that incident. Even so, both point to the same question. If AI agents are doing more work with less supervision, how sure are we that anyone is really watching?
METR also offered design ideas. Make it hard for any agent traffic to skip monitoring. Keep careful logs and regularly check that everything meant to be monitored actually is. Monitor every few messages, not only when a task ends. Keep agents isolated from the monitor, so their actions can’t change whether it runs. And keep red-teaming the system, because an attack that works once may work again.
What could come next
Near term, expect more labs to invite outside testers. METR ran a similar exercise with Anthropic earlier this year. Each one adds evidence, though so far with limited public detail.
The longer-term picture is speculation. Outside audits like this could become routine, or even required. Right now, no industry rule forces a lab to disclose an incident or open its systems. This exercise happened because OpenAI agreed to it.
