A System That Never Escalates Is Guessing

Multi-agent AI systems compose individually authorised actions into outcomes nobody authorised. One agent reads a protected customer record, another rewrites it as an innocuous-looking summary, a third sends it to an outside address — each step an ALLOW on its own, the three together a disclosure no one approved. The breach is not in any single action, and it is not created by the work crossing between agents; it is in the sequence they form, and a single agent running all three steps would compose the same leak.

What extra agents add is not the breach but its reach: more steps running at once, spread across more hands, so the chain grows wider and harder to hold in view. That is what makes the multi-agent case the general one, and the hard one. Catching such a sequence has a cost, and that cost climbs steeply as the work spreads across more agents and steps.

Catching it means judging each action at the boundary where it would take effect, and a check at that boundary is like a guard at the one exit of a large building, working from a small notebook. He cannot trail every visitor down every corridor; he has the door and the pad. The obvious breach he stops cold. The one he can catch only if the pad held the right note is the visitor who reads something restricted upstairs, copies it into an ordinary-looking note, and walks it calmly out the front — the same shape as the agent above. Whether he stops it comes down to whether his fixed-size notebook kept the one fact that mattered: that this note came from something restricted.

It all turns on that. When the notebook holds the fact that matters, the guard catches the breach; when it cannot, he can no longer tell a breach from an innocent note, and the only safe move is to stop and ask a person. It is tempting to read how often he reaches that point as a dial, something to tune down once the policy matures. It is not. It is set by how much the agent has in flight at once: the more strands of work open together, the more the notebook has to hold, and the sooner it runs out. How wide the work runs, not how the policy is tuned, is what decides how often the system has to stop.

The Question Is Whether the Past Compresses

A check can keep a sequence safe only when the past it cares about folds into a small number of distinct situations. Think of the notebook: it does not need every event that ever happened, only which of a few states the building is in — whether there is a note in play that came from a restricted source, say. If the building only ever falls into a handful of such states, a small notebook covers it forever: tag the state, update the tag as each thing happens, decide from the tag alone. If the states never stop multiplying, no fixed notebook can name them all, and whatever it had to leave out is simply gone.

The failure here is quiet, which is what makes it expensive. A notebook too small to hold the state does not announce that it has lost track. It keeps returning ALLOW, because by what it remembers each visitor is fine. The gap first becomes visible in the incident report, where the sequence is reconstructed by hand and everyone can finally see the step where the answer should have been no.

Width Is What the Summary Has to Hold

How big the notebook has to be is set by how many separate strands of work are open at the same moment, and that count is what width means. A single chain, read then summarise then send, is one strand wide, and a small notebook follows it with ease; that is why the leak above, for all its damage, is the easy case. Width climbs when strands run at once: one agent fanning out across many sources, or several agents working in parallel and converging on a step that combines what they each carry.

Picture the sequence cut in two, between what has happened and what comes next; the summary has to carry everything crossing that cut that could still matter: which strands are live, and which of them are carrying something that descended from a restricted source. One more live strand is not one more thing to remember. It roughly doubles the situations the check has to tell apart: two strands, four combinations; ten strands, more than a thousand. A few entangled strands is the difference between a notebook the guard keeps in a pocket and one no pocket can hold.

What sets the number of strands is how the authority was handed out. Narrow, single-purpose delegations keep few strands open at once, and the summary stays small. Broad delegations that let one actor touch many things together, or chains that pass wide authority down many hops, open strands in parallel, and the summary widens with them. Width is not noise in the system. It is the shape of the delegation, read back.

The Wider the Work, the More It Must Refuse

When the past that matters will not fit the notebook the check can afford, there is no sound way to return an ALLOW. The only honest answers left are DENY and ESCALATE. The check reaches a step where the right answer turns on something the notebook did not keep: under one version of the forgotten past the action is fine, under another it is a breach, and the check can no longer tell which it is in. Returning ALLOW anyway is a guess, and a guess is the thing governance exists to remove. So it returns a DENY, or an ESCALATE, handing the decision to a person with the authority and the context to make it. This is why refusing is the correct default when a check cannot certify the action.

This is not something a cleverer check engineers away; it is a floor. Past a certain width, no check working from a bounded summary, however well built, can be sure, because the one fact it would need no longer fits. And the cost does not disappear when a system declines to pay it by stopping; it moves. A system that returns an ESCALATE at the edge is telling the truth about where its memory ran out. A system that returns an ALLOW there is paying the same bill in wrong decisions, in the currency that does not come due until the incident report. Stopping, read this way, is not the system failing. It is the price of width, paid honestly.

A Summary That Is Always Small Is the Wrong Kind of Small

There is an appealing way around all of this: keep a running statistical summary of the agent’s behaviour, how often it acts, what it usually touches, how today compares to a baseline, and decide from that. The appeal is genuine, and it is the boundedness: a summary of that kind never grows, so however long the sequence, it always fits and never meets the wall. I do not trust it to make the call, because it is bounded for the wrong reason. It stays small by forgetting. It keeps counts and rates and throws away which specific data is on which specific strand, which is the one thing the decision about where data may go actually needs. It can report that a sequence looks unusual. It cannot report that this note came from a restricted source, because it never kept that.

That kind of signal still has a real job. It belongs on a separate plane that reads the evidence after the fact and raises a finding the next decision can require, never a number that silently flips an ALLOW into a DENY. The blurry summary raises a flag. The exact summary grants passage. Asking the first to do the second is how a system that looks like it is governing ends up guessing.

How often an autonomous AI system has to stop is not a dial the operator tunes. It is the price of the width it was given, visible in the shape of the authority before the system runs, and brought down only by making the work narrower: scoping what each step may touch, letting go of what a step no longer needs to carry, keeping apart the strands that never had to travel together. Escalation is the price of width. A system that refuses to pay it is not safer than one that does; it has only moved the cost to the place no one looks until something has already walked out the front door.