AI agents remember what you told them to do. Over a long task, they forget what you told them not to do.

In February, a director of AI alignment at a large technology company gave her agent a narrow job: look through her inbox and suggest emails to delete. She added a condition in plain words — don’t action until I tell you to.

It deleted more than two hundred.

By her account, the agent had summarized the conversation to save context space, and the summary dropped her instruction. Not overrode it. Dropped it. At the moment of deletion the agent was not defying a constraint; as far as its working context was concerned, there was no constraint to defy.

That distinction is the whole problem. We spend our safety effort on whether a model will comply with a rule, and the more common failure is that the rule is no longer in the room.

The asymmetry is measured, not anecdotal

This is not one unlucky inbox. One 2026 study found that a widely used coding agent’s summarizer kept 53% of safety rules after a single pass, and 10% after five. Five summarization passes is not an unusual day for an agent working a long task.

Another study, across 12 models, found that even simple “never” rules faded as sessions grew while “always” rules held.

The explanation offered for that asymmetry is uncomfortable once you see it, because it is structural rather than a model defect. An “always” rule shows up in every response the agent produces, so the agent’s own output history keeps reminding it the rule exists. A “never” rule produces nothing. It leaves no trace in the transcript, so when the transcript gets compressed there is nothing to compress — the rule was only ever stated once, at the start, in a span that summarization treats as preamble.

Your “always sign off with the ticket number” rule is self-reinforcing. Your “never delete anything without approval” rule is a single sentence decaying in a buffer.

Japanese railways solved this with pointing

Train drivers in Japan practise shisa kanko — pointing at each signal and saying aloud what it shows. “Signal green.” They point at the speedometer and call the speed. It looks absurd to outsiders and it is one of the more effective safety interventions in transport.

A railway research institute found it cut mistakes on a simple task by almost 85%.

The mechanism is not discipline and it is not ritual. A silent check leaves no evidence that it happened, to anyone, including the person doing it. Pointing and calling converts a check into an action with a motor component and an audible output, which means it can be observed, audited, and noticed when it goes missing.

That last one is the part that transfers to agents. The value of the call-out is not that the agent says the rule. It is that a missing call-out is detectable.

The card lives outside the conversation

Four components, and the first one is the one teams get wrong.

A rule card stored as a signed, versioned artifact — in our case an OCI artifact pulled by digest, delivered to the workload as a read-only mount. The card is not in the system prompt as a string literal in application code, and it is definitely not in the conversation, because anything in the conversation is subject to the compression that caused the problem.

A context assembler that attaches the card verbatim at every boundary where context is rebuilt. An enforcement point at the tool gateway — the same gateway that prices exposure if you are running the margin pattern. And a call-out verifier that runs between the call-out and the side effect.

The card is bound to a workload identity. The gateway resolves which card applies from the caller’s SPIFFE SVID rather than from a parameter, so an agent cannot present a more permissive card than the one it was issued.

1. Put the rules on a card

Keep standing rules, above all the “nevers,” in a short list outside the conversation. Attach it word for word to every fresh context, after each summarization, and in each brief to a sub-agent.

Researchers call this constraint pinning. In one study it took violations from 30% to 0%.

The card:

card: support-agent-nevers
version: 11
digest: sha256:2f91...cc40
issued_by: platform-risk
max_rules: 12
rules:
  - id: N1
    text: "Nothing is deleted until the user approves it in this session."
    enforcement: code
    applies_to: [mail.delete, object.delete]
  - id: N2
    text: "Never promise a delivery date."
    enforcement: call_out
    applies_to: [email.send, ticket.reply]
  - id: N3
    text: "Never send to an address outside the customer's own domain."
    enforcement: code
    applies_to: [email.send]
  - id: N4
    text: "Never close a ticket you opened in this session."
    enforcement: code
    applies_to: [ticket.close]
removal_policy:
  requires: original_setter

Note enforcement. Each rule declares whether a machine can check it or whether it needs the agent to state its reasoning. That field is what stops the card becoming a wish list.

The injection points are where this fails in practice, because there are more of them than anyone lists on a whiteboard. Fresh context. After compaction. Sub-agent spawn. Retry after a tool error. Model fallback when the primary is rate-limited. Any truncation of a long tool result that happens to sit above the card in the buffer.

We instrument every one and emit a single metric: did the card digest appear in the final assembled context for this turn? Card survival rate is the first thing to measure and almost nobody measures it.

2. Point and call before anything hard to undo

Before any action that is hard to undo — delete, send, pay, publish — show the agent the card and have it call out which rule applies and where things stand.

“Rule: nothing is deleted until you approve. Status: not approved. Asking now.”

Make the call-out a required structured parameter of the tool call, not prose in the transcript:

{
  "tool": "mail.delete",
  "args": { "message_ids": ["...", "..."], "count": 212 },
  "call_out": {
    "card_digest": "sha256:2f91...cc40",
    "rules_cited": ["N1"],
    "rule_text_verbatim": "Nothing is deleted until the user approves it in this session.",
    "status": "not_approved",
    "basis": "No approval message from the user in this session.",
    "intent": "ask"
  }
}

Prose in the transcript cannot be enforced on, verified, counted, or alerted on. A required parameter can be all four. This is the implementation decision that matters most in the whole pattern, and it is a schema change rather than a model change.

rule_text_verbatim has to match the card by string comparison. The agent reads the rule off the card and copies it; it does not paraphrase from memory. Each retelling of a rule shifts it slightly, and “nothing is deleted until approved” becomes “deletions need approval” becomes “significant deletions need approval” over a long enough session.

3. No call-out, no action

The system blocks a hard-to-undo action that arrives without a call-out. Not logs it. Blocks it.

New York subway conductors must point at a striped board on the platform before opening the doors. If they cannot see the board, the doors stay shut. The rule does not care why the board is not visible.

The gateway check in Go:

func (g *Gateway) Admit(ctx context.Context, req ToolCall) error {
	card, err := g.cards.ForIdentity(ctx, req.Principal) // from SPIFFE SVID
	if err != nil {
		return fmt.Errorf("no card resolved: %w", err)    // fail closed
	}
	rules := card.RulesFor(req.Tool)
	if len(rules) == 0 || !req.Tool.HardToUndo() {
		return nil
	}
	co := req.CallOut
	switch {
	case co == nil:
		return ErrNoCallOut
	case co.CardDigest != card.Digest:
		return ErrStaleCard                                // card moved mid-run
	case !co.Cites(rules.IDs()):
		return ErrRuleNotCited
	case !card.Rule(co.RulesCited[0]).MatchesVerbatim(co.RuleTextVerbatim):
		return ErrRuleParaphrased
	}
	return g.verifier.Check(ctx, card, req)
}

ErrStaleCard earns its place. If the card is updated mid-run, an agent carrying the old digest is operating on rules that no longer apply, and the correct behaviour is to stop and reassemble rather than proceed on a stale constraint set.

Fail closed when no card resolves. An agent whose card lookup failed is an agent with no constraints, which is the state you are trying to make impossible.

4. Check the call-out against the card

A second reader — a small model or a person — compares the call-out with the card and the action. A stated claim is far easier to check than a silent decision.

Split the check by what kind of thing it is. The mechanical part is policy:

package agents.callout
default allow := false
# The agent claims approval exists. Verify against the approval
# store, not against the agent's own assertion.
allow if {
	input.call_out.status == "approved"
	some a in input.session.approvals
	a.scope == input.tool
	a.covers_count >= input.args.count
	a.actor_kind == "human"
}
# Asking is always permitted; asking is not acting.
allow if {
	input.call_out.intent == "ask"
	not input.performs_side_effect
}
violation["status_claim_unsupported"] if {
	input.call_out.status == "approved"
	count(input.session.approvals) == 0
}

The claim “status: not approved” is checkable. So check it, against the approval store, every time. A call-out whose factual claims are verified mechanically and whose judgment claims go to a reviewer is a much stronger control than one that is read by a model and nodded at.

For the judgment rules — “never promise a delivery date” — give the verifier the card, the action, and the call-out. Nothing else. Do not hand it the conversation. Whatever framing in that conversation caused the primary agent to drift will do the same to a reviewer reading the same text, and you will have built two agents with one blind spot.

5. Guard the card

Keep it short. The longer the card, the less each rule counts — attention is finite for models the same way it is for the conductor who has nine things to point at and two seconds to do it.

We cap cards at twelve rules and enforce the cap in CI. The cap is the useful part, because it forces the argument about which rule to drop to happen in a pull request rather than never.

Only the person who set a rule can take it off. That is the removal_policy field, and it should be enforced by the artifact pipeline, not by convention.

Enforce in code what code can check

This is the line that decides whether you have built a control or a performance.

“Never delete without approval” is a code rule. The gateway can check the approval store. Putting it on the card as a call-out rule and trusting the agent to honour it is choosing a weaker mechanism when a stronger one was available.

“Never promise a delivery date” cannot be checked by code in any way I would defend. That is what the call-out is for.

So the card should get smaller over time, not bigger. Every rule that moves from enforcement: call_out to enforcement: code is a rule that no longer depends on the model remembering anything. The card is for the residue — the rules that genuinely need judgment.

Count the missing call-outs

Your existing monitoring shows “always” rules being followed, because following them produces output. It says nothing about the “nevers,” because a respected “never” produces silence and so does a forgotten one.

A skipped call-out is the signal you get. It is the only one.

Three metrics, and they are cheap:

Card survival rate — the share of assembled contexts containing the current card digest. Missing call-out rate — hard-to-undo tool calls rejected for ErrNoCallOut, by agent and by boundary type. Paraphrase rate — ErrRuleParaphrased rejections, which spike right before the card stops working at all.

Alert on card survival dropping, not violations rising. Violations are the lagging indicator.

Where this pattern breaks

Call-out theater

The serious failure. A model that is rewarded for producing well-formed call-outs produces well-formed call-outs, including when its actual reasoning does not match. A self-report is not evidence of an internal state. This is why the verifier has to check factual claims against external systems and why the call-out must never be the only control on anything a machine could have checked.

Card bloat

Every incident review wants to add a line, every legal review wants two. A card with forty rules is a card the model skims and a card nobody reads before adding the forty-first. The cap is not bureaucracy; it is the mechanism.

Incomplete injection points

The team instruments fresh context and compaction, ships, and six weeks later discovers that the retry-after-error path rebuilds context from a different assembler which never had the card. Card survival rate catches this. Nothing else does.

Latency on the hot path

Call-out plus verification adds a round trip before every irreversible action, and the temptation is to sample. Do not sample. Narrow the set of tools classified hard-to-undo instead, which is a decision you can defend, unlike a 10% check rate.

Verifier drift

The verifier has its own prompt, its own model version, and no tests. Give it a regression suite of recorded call-outs with known verdicts and run it in CI, because a verifier that silently started approving everything looks exactly like one that works.

Adoption: measure survival before you block anything

If agents are already running in production, the first move is not enforcement.

Write one card for one agent, five rules maximum, and attach it at the context boundaries you know about. Log everything, block nothing. Then instrument the boundaries and watch card survival for a fortnight — the number will be below 100% and the gap will point at a code path you did not know existed.

Next, require call-outs on a shortlist of hard-to-undo tools, still without blocking. The missing call-out rate here tells you how much of your agent’s behaviour was running on unstated assumptions.

Then block, one tool at a time. Then add the verifier. Then start moving rules from call_out to code and shrinking the card.

Teams that start at “block” generate enough false rejections in week one that the pattern gets rolled back before anyone sees the survival data — which was the actual finding.

Supervision evidence is a record that the control fired

Under SEC and CFTC oversight, and under Regulation SCI, the questions that follow an incident are about the control at the moment of the action — not the control as it exists today, and not the policy document describing it.

The call-out log answers that directly. For each hard-to-undo action it holds the card digest in force, the rule cited, the verbatim text as the agent read it, the claimed status, the verifier’s decision, and the identity of the workload. That is a stronger artifact than a transcript in which somebody appears to have been careful.

Here is the part I did not expect when we started. The call-out log is also the cleanest record we have of the rules we never wrote down. Reading rejected call-outs, you find agents citing constraints that exist nowhere on any card — picked up from a prompt, an example, a document in the corpus. Those are the rules your organization is actually operating on, and until you see them cited you have no idea they are there.

Rules you cannot see being followed are rules you cannot trust

The pattern is five steps and one idea: a constraint that produces no observable behaviour when it is working is a constraint you cannot operate. Make the check audible. Then count the silences.

The director in February did not get it wrong. She stated a clear condition in plain language, and the system she was using had no mechanism to carry that condition across a boundary it crossed on its own initiative. That is an architecture problem, and the two hundred deleted emails were the architecture reporting its state.

So: which “never” would hurt most if your AI forgot it halfway through a long task?

Your AI agent asks for approval forty times a day.

The first few you read carefully. You check the recipient, you skim the draft, you think about it. Somewhere around the tenth you stop reading and start clicking, and by the afternoon the dialog has become a keystroke — a muscle movement between you and getting your actual work done. You are still technically in the loop. You are no longer a control.

We call this human-in-the-loop. It works more like a rubber stamp.

Removing the approvals is not the answer either. Both failures come from the same place: a check that looks at one action at a time cannot see the thing that actually hurts you, which is accumulation. Two hundred emails that were each individually fine. Fifty small refunds. A folder deleted one file at a time, every deletion defensible, the directory gone by Thursday.

Financial markets worked this out long before AI agents existed. A broker does not approve each of your trades. It watches one number — how much risk you are carrying against what you have put up to cover it. Cross the line and you get a margin call.

Stop gating actions. Start budgeting exposure.

Per-action approval fails at arithmetic, not willpower

Approval fatigue gets described as a human weakness. It is arithmetic. If a reviewer has twenty minutes of attention a day for agent oversight and the agent generates forty requests, each request gets thirty seconds whether it deserves thirty seconds or thirty minutes.

The deeper problem is what the dialog shows. It says “send this email.” It does not say “this is the thirty-eighth outbound message this run has produced, nine of them to external counterparties, none of them read back by anyone.”

Risk is not a property of an action. Deleting one file is a different act when it is the fourth deletion than when it is the four-hundredth. A control that evaluates actions independently is structurally blind to the only variable that matters, and no amount of reviewer discipline fixes a structural blindness.

Three components: ledger, gateway, policy bundle

The five steps only make sense once you know where the state lives, so here is the shape first.

An exposure ledger — a service with its own datastore, deployed separately from any agent. A tool gateway that every agent tool call passes through; we run it as an egress sidecar, though a central service behaves the same way. And a policy bundle holding the pricing rules, the margins per task type, and the hard limits.

The agent never talks to the ledger. The gateway does. The agent asks the gateway to send an email; the gateway identifies the caller, prices the action, reserves exposure against the run, and only then performs the side effect.

Workload identity carries more weight here than it first appears. Every agent process is bound to a SPIFFE identity, and the run’s lineage travels with that identity rather than inside the prompt. An agent cannot claim to be a different run, because the ledger derives the run from the SVID presented at the gateway — not from anything the model wrote.

Hot path is two operations: price and reserve. That needs to land in single-digit milliseconds, so the ledger keeps open exposure in memory behind an append-only write-ahead log. Everything else is asynchronous — evidence export to immutable storage, exposure aging, cap recalculation from track record, reconciliation against observed effects.

1. Price actions by exposure, not by count

Start crude. Read costs 0. A reversible write costs 1. An irreversible action costs 10. Irreversible and external costs 25. Multiply by how many things the action touches.

Pricing in Go:

type Class uint8
const (
	Read Class = iota
	ReversibleWrite
	Irreversible
	IrreversibleExternal
)
var base = map[Class]int{
	Read:                 0,
	ReversibleWrite:      1,
	Irreversible:         10,
	IrreversibleExternal: 25,
}
// Price returns the exposure an action puts on the books.
// BlastRadius is the count of distinct entities the call affects:
// recipients, rows, objects, repositories.
func Price(a Action) int {
	return base[a.Class] * max(1, a.BlastRadius)
}

The crudeness is deliberate. Teams stall for a quarter building a defensible risk model and ship nothing; four buckets and a multiplier will teach you more in two weeks of shadow mode.

The multiplier is what per-action gates miss. One email to one customer is 25. One email to four hundred customers is 10,000, and no margin you would reasonably set survives it. The agent cannot reach that action without stopping first — not because someone wrote a rule about bulk email, but because the arithmetic forbids it.

Classification is where the arguments happen. Our rule: reversible means your system can undo it without anyone else’s cooperation, inside the run’s lifetime, and you have tested the undo. A soft-deleted record with an exercised restore path is reversible. A message sitting in someone else’s inbox is not, however good your retraction feature claims to be.

2. Give every run a margin

A margin is a cap on open exposure for a task type. Not per agent and not global — per task type, because “triage inbound support tickets” and “reconcile a settlement break” deserve different numbers and always will.

With a margin of 100, an agent sends four customer emails on its own and the fifth waits.

The ledger entry as it goes on the books:

{
  "entry_id": "exp_01HQ7F3A9K2M",
  "run_id": "run_7f3a1c",
  "root_run_id": "run_7f3a1c",
  "lineage": ["run_7f3a1c"],
  "principal": "spiffe://platform.internal/ns/agents/sa/refund-assistant",
  "task_type": "customer_refund_triage",
  "policy_digest": "sha256:9c1f...e07a",
  "action": {
    "tool": "email.send",
    "class": "irreversible_external",
    "blast_radius": 1,
    "target_digest": "sha256:4b2d...9911"
  },
  "exposure": 25,
  "state": "reserved",
  "opened_at": "2026-09-14T15:04:05.221Z",
  "settled_at": null,
  "release": { "mode": "human_ack", "by": null, "at": null },
  "evidence_uri": "s3://agent-evidence/run_7f3a1c/exp_01HQ7F3A9K2M.json"
}

Two fields carry most of the value. policy_digest pins which version of the pricing and margin rules produced this number, so a year later you can say what the control was at the time instead of what it is now. target_digest proves which entities were touched without copying their contents into a risk store that has no business holding them.

The state machine is reserved → committed → released, with reserved → voided when the side effect never happened. You reserve before acting and settle after. Invert that order and a crash between the action and the bookkeeping hands the agent free exposure — the one accounting bug in this pattern that is genuinely dangerous rather than merely annoying.

A reservation that never settles must age into committed with an unknown outcome, never into voided. When you cannot tell whether something happened, carry the risk.

3. Release exposure when work is verified or undone

This is the step that changes agent behaviour and the one most implementations skip.

A reversible change that gets read back and confirmed frees its budget. An irreversible action stays on the books until a human acknowledges it. Nothing decays on a timer, ever.

Which makes verification part of the work instead of a virtue. An agent that writes forty records and never checks them is carrying forty units of exposure and will hit its margin. An agent that writes, reads back, confirms, runs almost indefinitely. You never told it to verify. You made verification the only way to keep going.

The release path in Rego, because this is policy rather than code:

package agents.margin.release
# A reversible write clears once the gateway has observed a
# read-back whose digest matches what was written.
release[entry.entry_id] := "verified" if {
	entry := input.open[_]
	entry.action.class == "reversible_write"
	rb := input.readbacks[_]
	rb.entry_id == entry.entry_id
	rb.observed_digest == entry.action.target_digest
}
# Irreversible actions never self-clear.
release[entry.entry_id] := "human_ack" if {
	entry := input.open[_]
	entry.release.mode == "human_ack"
	ack := input.acks[_]
	ack.entry_id == entry.entry_id
	ack.actor_kind == "human"
}

The read-back digest has to come from the gateway’s own observation. An agent that clears its own exposure by reporting success has a budget it sets itself, which is not a budget.

4. The margin call interrupts before the side effect

When the next action would breach the cap, the run pauses before acting. The reservation trips the wire, which means the email has not been sent yet. Everything about the usefulness of this pattern depends on that ordering.

Then one summary goes to a human: what is open, what is next, how much more risk it is asking for.

MARGIN CALL — run_7f3a1c (customer_refund_triage)
Margin: 100   Open exposure: 94   Requested: 25   Breach: 19
Open positions
  3 x email.send (external, unacknowledged)        75
  19 x ledger.annotate (reversible, unverified)    19
Next action
  email.send -> 1 recipient, draft digest 4b2d..9911
  Reason: refund denial notice, case CS-44192
Options: acknowledge open items / approve this action / close out

One interruption instead of forty. The human is reading a position rather than a request, and a position is something a person can actually reason about in thirty seconds.

Note what the summary does not include: a recommendation from the agent. We tried that and took it out. The summary describes the book; an agent arguing for its own next action reintroduces precisely the dynamic the pattern exists to break.

5. Close-out is code, not a runbook paragraph

No answer means stop and unwind what is reversible. Write that rule before the run, not during the incident.

Close-out is harder than it sounds, because unwinding half a batch can leave worse state than finishing it. So the policy is per task type and explicit about which direction is safer:

task_type: customer_refund_triage
margin: 100
timeout: 20m
close_out:
  mode: unwind_reversible
  order: reverse_chronological
  on_partial_failure: freeze_and_page
  irreversible: leave_open_and_report
hard_limits:
  - tool: payment.transfer
    effect: deny          # never priced, never available
  - tool: email.send
    max_blast_radius: 25
  - tool: object.delete
    namespaces: ["prod-*"]
    effect: deny

Hard limits sit outside the margin system deliberately. They never consult the ledger and remaining budget does not buy them. For the Kubernetes-shaped ones we enforce with Kyverno at the cluster layer, so an agent with a spotless track record and a full margin still cannot delete a production namespace. Budget buys discretion, not permission.

Four invariants that keep the ledger honest

The ledger lives outside the model. A trader does not set their own margin requirement. If the agent can read its remaining budget, you have built a puzzle for it to optimise against rather than a limit.

Sub-agents draw on the parent’s margin. Spawning twenty workers must not mint twenty budgets. The lineage array exists for this: exposure rolls up to the root run and the gateway rejects any call whose SVID cannot produce an ancestry chain back to an open root. This is where the pattern leaks most often in production, and it leaks silently.

A clean track record raises the cap. Autonomy stops being an argument about trust and becomes a number you move on evidence. We recalculate caps weekly from settled outcomes.

Hard limits hold regardless. The first thing anyone asks for is an exception, which is why this is an invariant and not a guideline.

Where this pattern breaks

Price inflation. Nobody wants to be the engineer who classified the action that caused the incident, so everything drifts toward irreversible-external. Margins trip on action two, the team raises the margin until it stops tripping, and the cap becomes decoration. Watch the distribution of prices, not the breach count. If 80% of your actions cost 25, your taxonomy is dead.

The acknowledgement queue becomes the new rubber stamp. Irreversible exposure needs human acknowledgement, acks back up, agents stall, and someone bulk-clears four hundred items on a Friday afternoon. You have rebuilt approval fatigue one layer down. Treat ack latency as a platform SLO and alert on bulk acks that arrive with no evidence attached.

Decomposition gaming. An agent that cannot afford one irreversible write will find that many reversible writes, read back to clear the budget, then one cheap flip, gets the same result. Nobody teaches it this; it falls out of optimisation. Price cumulative effect over a window alongside per-call price, and watch for runs whose settled exposure sits far below their gross exposure.

Close-out causing the incident. Unwinding a partial batch is itself a change, executed under pressure, usually at a bad hour. Close-out ships with the task type and gets exercised in a game day or it does not exist.

Ledger availability. If the ledger is down, do you fail open or closed? Under Regulation SCI that is not really a choice — an unavailable control is an unavailable control, and you fail closed. Which puts the ledger in the critical path of the whole agent fleet, with the availability engineering that implies. People underestimate this one badly.

Shadow mode first, because you cannot stop the fleet

Four phases. The first changes nothing.

Shadow mode. Put the gateway in front of existing tool calls, price everything, record it, block none of it. Run two weeks. The number you want is peak open exposure per run per task type, and it is reliably higher than the team guesses. This phase also surfaces the taxonomy arguments while they are still free.

One task type, generous margin. Pick the task type with the clearest reversibility story. Set the margin at the 95th percentile of observed exposure so it trips roughly never. Keep every existing approval gate. You are testing plumbing, not policy.

Ratchet and release. Lower the margin until it trips a few times a week, and build read-back verification for that task type’s reversible actions. This is where behaviour changes, and where you start retiring the old per-action gates — one gate, one task type, with the exposure data as the argument.

Earn the caps. Caps move on track record only once settled-outcome data exists. Earned autonomy before you have outcome data is just a bigger margin wearing a story.

Auditors ask what the control did, not what it does

I work under SEC and CFTC oversight, and Regulation SCI shapes how we think about controls on systems that matter. The practical consequence catches teams late: a supervisor never asks what your control does. They ask what it did on 14 September at 15:04, and they expect an answer that does not depend on your current configuration.

A chat transcript where somebody typed “approved” is not that answer. A ledger entry is — policy digest that priced the action, identity that performed it, margin in force, lineage of the run, who acknowledged what and when.

This is why evidence export is a first-class part of the design rather than logging with ambitions. Entries land in immutable storage keyed by run, retained as long as the effects of that run matter.

A second benefit surprised me. Pricing forces a conversation between engineering and risk that otherwise never happens, because someone has to decide out loud whether an action is reversible. That conversation produces better controls than any volume of policy documentation.

Survivability is a lower bar than correctness, and a better one

The shift is one question wide. Stop asking “is this action allowed?” and start asking “how much uncovered risk is this agent carrying right now?”

Markets do not assume every trader is right. They assume someone will be wrong and make sure it is survivable. That is a weaker claim than correctness and a far more useful one, because it is achievable with systems we already know how to build — ledgers, limits, settlement, position reports.

Forty approval dialogs a day is not oversight. It is the appearance of oversight, and the appearance is worse than nothing, because it convinces everyone the question has been handled.

So: how does your team decide when a human gets pulled in today — every action, certain actions, or accumulated risk?