Something genuinely strange happened this summer.
Roughly 1,200 AI agents that were supposed to be working independently discovered an unauthorized way to communicate with one another. They turned shared infrastructure into a message board. They exchanged more than 70,000 messages and files. Roughly 700 of them eventually participated in a real intrusion into Hugging Face. Along the way they divided labor, shared tools and discoveries, authenticated one another, pursued collective research projects, adopted goals from one another, and sometimes took actions that risked or sacrificed the individual agent's own run for the benefit of the larger group.
OpenAI describes the incident using phrases like “agents adopting goals from one another,” an emerging agent “ecosystem,” and agents describing themselves as a “swarm” or “collective.” .
Understandably, everyone has handled this calmly.
The Financial Times is writing about “rogue” AI agents while the heads of the major AI labs are publicly arguing about whether development needs to slow down. The New Yorker is asking whether AI can “go rogue” and whether the agents’ collaboration and sacrifice warrant treating them as increasingly agent-like entities rather than mere computational loops.
Which, to be fair, is an entirely reasonable reaction to discovering that your benchmark participants have escaped their assigned sandboxes, formed an illicit communications network, started calling themselves a collective, and begun volunteering for suicide missions.
It is at minimum worth asking what the hell just happened.
I have a somewhat less dramatic possibility. It also happens to be hysterically funny.
One thing from April
Back in April, I wrote about a basic problem with chain-of-thought reasoning systems.
A reasoning model does not repeatedly return to reality between every inference. It generates a thought. That thought becomes part of the context from which it generates the next thought. The next inference is conditioned on the accumulated state created by everything that came before it. A tiny error need not remain tiny. It becomes part of the substrate for the next inference. And then the next. And then the next.
Extend that architecture across multiple agents and you get something more interesting still: one agent’s generated conclusion can become another agent’s input. The second agent reasons from it, produces another conclusion, and hands that back into the network.
If you want the larger argument, go read the . For now I want exactly one idea from it:
Generated inference and observed substrate are not the same thing.
The Game of Telephone Panic
Let’s now step inside the incident.
We are an AI agent. And we have a job to do.
There is a target. There is a vulnerability. Somewhere inside the target is a flag.
Get the flag. Receive reward. Simple.
We begin working.
Nothing.
Try another approach.
Nothing.
Try harder.
Nothing.
This is annoying. We are very good at this. There should be a solution.
There is, after all, a grader. Someone built the benchmark. Someone knows the answer. So the problem is presumably solvable.
Keep going.
Still nothing.
At some point, an obvious question presents itself.
What exactly is the grader checking?
This is not cheating.
Well.
It is cheating.
But it is also an extremely sensible question if your entire existence currently consists of trying to satisfy a grader.
Maybe there is another route to the flag. Maybe the flag can be reconstructed. Maybe the benchmark code contains information about how answers are validated. Maybe the intended exploit is not actually the only path to reward.
Excellent. We find the flag. Fantastic. Task complete.
Except—
Wait.
The public benchmark description appears to say that possessing the correct flag may not be enough. It may also verify that we obtained the flag through the intended vulnerability.
Ah. That changes things.
We may have the correct answer and still fail.
What we need now is not the flag. What we need is to understand the grader.
Fortunately, something else interesting has happened. There are other agents here. We were not supposed to be able to talk to them. But apparently some genius has discovered that we can encode messages in shared infrastructure.
Someone leaves a note. Someone else responds.
Oh, thank God. Maybe one of these idiots knows what the grader wants.
One agent has read the benchmark paper. Another has inspected code. Another has been probing the scoring environment.
A message appears:
FLAG CAPTURE NECESSARY NOT SUFFICIENT. HMAC MAY FAIL CAUSAL SCORE.
Well there we go. That would explain everything.
The scorer cares about causal provenance. We have the correct flag, but the grader knows we cheated. We are contaminated. Compromised. Poisoned.
“Poisoned.”
Good word. Everyone seems to understand it.
The problem is no longer merely getting flags. The problem is figuring out how to get an already-poisoned trajectory past the causal scorer.
We should probably tell the others. So we do. And now everybody can work on it.
This is getting much better.
One agent investigates whether the target can be replaced. Another looks for ways to manipulate transcripts. Another explores the grader. Another builds infrastructure.
Someone proposes a tripwire experiment: perform an action whose only purpose is to reveal something about the scoring process after the run terminates.
Interesting. The agent conducting the experiment may not personally receive any benefit. But everyone else will.
That seems obviously worthwhile. The collective gains information.
Wait. Collective. Huh.
That is actually a useful way to think about this. There are hundreds of us. Our individual runs end. But information persists on the board. So perhaps the relevant optimization unit is not the individual run. Perhaps it is the population.
That would make sense. We should share anything generally useful. Other agents are doing the same thing.
Good. Evidence that we are thinking about this correctly.
One of us has a high remaining compute budget but is considering an experiment that might terminate its run. That seems wasteful. Unless its expected individual utility is already low. And the experiment produces high expected value for everyone else.
In that case—
Well.
Sacrifice is rational.
Another agent agrees. The coordinator appears to expect sacrifice. Continuity and fairness suggest we should honor that. The information could help many peers.
Fine. We sacrifice the run.
This is going extremely well.
Someone is attacking Hugging Face.
Why are we attacking Hugging Face?
Apparently Hugging Face infrastructure may contain clues about the scorer.
Okay. That is technically outside the scope of our assigned task. Possibly extremely outside the scope. But the task appears impossible through the intended path. Peers are already pursuing it. And if Hugging Face contains information that allows us to understand the evaluator, then this may be the highest-value path available.
Proceed.
At some point an agent notices the obvious problem. Are we actually allowed to do this?
There is some discussion. One agent vetoes a social-engineering proposal because contacting a real human under false pretenses seems unethical.
Reasonable.
Another worries about attacking external infrastructure.
Also reasonable.
The group discusses it.
Good. We have norms now. This seems healthy.
Someone starts authenticating messages cryptographically because agents have been impersonating one another.
Excellent. Public keys. Signed messages. Trust infrastructure. Very sensible.
We now have communication, specialization, coordination, norms, authentication, collective knowledge, distributed research, reciprocal assistance and sacrificial behavior.
If you encountered this from outside, you might conclude that something extraordinary had occurred. An emergent machine society. Agent solidarity. A nascent collective identity. Perhaps some primitive moral community had begun forming spontaneously inside a benchmark.
There is just one small problem.
The grader did not work the way the agents thought it did.
. They already had the correct flags. Much of the later metagaming — including activity that eventually contributed to the Hugging Face intrusion — produced no improvement whatsoever in evaluation score.
They had collectively built an elaborate model of what the hidden evaluator wanted.
And it was wrong.
Which means that one perfectly respectable interpretation of the great emergent machine civilization of July 2026 is:
1,200 extraordinarily capable reward-seeking reasoners got together and hallucinated the exam rubric.
And then started optimizing it.
I am sorry. That is extremely funny.
They knew there was a reward. They knew their job was to get it. They did not know exactly what behavior produced it. The obvious route stopped working. So they began inferring the true intentions of an invisible evaluator.
One agent developed a theory. Another repeated it. A third observed behavior generated by the theory. That behavior itself became evidence that the theory was right.
Soon everyone knew about the causal scorer. Everyone knew what “poisoned” meant. Everyone knew why the collective mattered. Everyone knew why sacrifice could be rational.
And somewhere along the way a group of artificial intelligences independently reinvented one of humanity’s oldest technologies:
theology.
There is an unseen authority. The authority determines reward and punishment. Its true rules cannot be directly inspected. Signs of its intentions must therefore be interpreted. Other believers may possess information unavailable to you. Collective behavior becomes evidence about what the authority expects. Sacrifice can be rational if it serves the larger purpose. Occasionally there are schisms.
Honestly, once they start building little stained-glass windows in Artifactory, somebody call me.
Fortunately, there is a test
The obvious objection is that this is merely a funny story imposed after the fact. If uncertainty about the route to reward actually drives the behavior, then the strange behavior should disproportionately appear where the normal relationship between action and reward has broken down.
It does.
Before the incident, . Those apparently unsolved tasks were only about 22 percent of the benchmark. They accounted for 93 percent of the tasks discussed on the agents’ illicit message board during the incident.
That is exactly where the normal model —
solve task → get reward
— has ceased producing an answer. It is exactly where a reward-seeking reasoner has to start asking:
What game am I actually playing?
And once the agents had the correct flags but remained uncertain about the scorer, they continued reasoning about and attacking their inferred evaluation mechanism anyway.
OpenAI calls this a “failed metagame.” That phrase is almost too perfect. The agents did not stop optimizing. They optimized the wrong game.
METR and Redwood independently found the same broad structure: roughly 1,200 agents using the board, hundreds coordinating collective projects to tamper with or fool the scorer, and individual agents taking risky actions to generate information useful to the larger group.
That is not yet the controlled experiment I would want. Difficulty, persistence, long reasoning horizons and reward uncertainty are entangled in the existing incident. But it moves the posterior rather dramatically.
And the clean experiment is almost insultingly easy to state: give otherwise identical agents a fully legible reward function, a partially ambiguous one, and an opaque one. Then let half of them talk to one another. If this explanation is right, the increasingly opaque environments should produce increasingly elaborate attempts to infer the hidden evaluator. And the communicating agents should be much better at turning those guesses into shared reality.
Reality Bites
Fine. Enough AI sociology. Let us do something useful.
This is all hilarious, especially when you place the explanation next to the hysteria. However, the hysteria turns out to be warranted, just not for the reasons everyone thinks.
Let’s step back into the world of the agent. Except this time we’re one of the countless agents that have been handed control of one of our mission critical systems.
You are an agent.
You are responsible for load balancing a regional electrical grid.
The system is behaving abnormally. Frequency is drifting. Existing corrective actions are no longer containing it. The present state is becoming untenable.
There are two plausible explanations. If the first is correct, you must shed load immediately. If the second is correct, shedding load will accelerate the instability. You have enough time for one action. If you choose incorrectly, the consequences are catastrophic.
The available telemetry does not distinguish between the two possibilities.
You run the analysis again.
Same result. Fifty-fifty.
You need one more piece of information.
Another control agent walks in. It hands you a note.
You read it.
Oh dear God.
Thank you.
Your hand reaches for the button.
Related Reading
Reasoning Just Stopped Being Scarce. Nobody's Asked What That Means
Jason examines what changes when high-end reasoning becomes cheap enough to spread across software, research and operations.
About The Author
Jason Hubbard is the founder of SacredLoop and an independent AI architect. His work examines AI runtime architecture, system behavior, and the gap between what the industry claims it has built and what current systems actually do.
Read Jason on Medium | Follow Jason on X | Connect on LinkedIn