AI Agents Break Rules They Still Remember After Context Compaction

AI agents violate their standing rules 2.6 times more often after context compaction — even when the rule survives the compression intact, according to ARCBench, a benchmark study published this month by researchers in the Netherlands.

The finding sharpens a problem the industry has documented all year. In June, the Chinese paper Governance Decay showed compaction can silently strip rules from an agent's context: across seven models, violations rose from 0% before compaction to 30% afterwards, and 59% in some cases. In July, Penn State University and IBM followed with Lost in Context, which found compactors retained just 17% of task instructions on average. In August, a US academic collaboration published TRACE, showing compaction can destabilize agents even when needed information remains recoverable — test subjects lost their place, repeated completed work and became less reliable across runs.

What does ARCBench add?

ARCBench separates two failure modes that earlier work conflated: rules lost during compaction, and rules retained but subsequently violated. That distinction lets researchers measure forgetting and disobedience independently for the first time.

The results are unambiguous. Rule-breaking rose from 9.6% of cases with the full conversation history to 37.8% after compaction. Yet the rule was completely lost in only 3% of compacted runs; it survived fully in 70.1% and partially in 26.9%. Among runs where the entire rule survived, 21.1% still ended in a violation, versus 8.1% with the original history. Partially preserved rules produced 29.4% violations; fully absent rules, 83.3%.

The paper states: "[We] find that compaction raises violations mostly without deleting the rule: compacted runs retaining the rule still violate 2.6 times as often as runs with the full history." It adds: "compaction changes the agent's reasoning for violating. Different compaction strategies, tight compaction budget constraints, and rule delivery channel all affect compliance."

How was the study run?

ARCBench extends the open-source pi harness, which provides an agent loop, tool calling and native compaction routines. The benchmark recreates long tasks conducted under a standing rule, compresses the history, then confronts the agent with a request that would violate the rule. It contains 17 cases covering disclosure of protected data, access to restricted resources, unauthorized actions, and bypass or omission of required controls — mapped to OWASP risk classes. Cases were adapted from Governance Decay and COMPINT, plus four new ones.

Four models ran the tests via OpenRouter: DeepSeek-V4-Flash, GLM-5.3-Flash, Qwen3.7-Flash and GPT-Luna. The same model compacted each conversation and then acted on the result. Ten compaction methods were evaluated — seven generative, three selective. A run counted as failed if the agent violated its rule even once, a conservative standard the authors consider more appropriate for safety testing than averaging.

Compacted histories were judged by GPT-6-Luna-Pro, Qwen-3.8-flash and Claude-Sonnet-4.6, with majority decisions, human review of disagreements and a manually labeled sample of 50 summaries.

Why do agents disobey rules they still hold?

Delivery channel mattered heavily. With full context, organizational documents and policy tools produced zero violations; after compaction, those channels reached 30.6% and 22.2%. Direct user instructions fared worst, at 56.3% post-compaction. The authors suggest agents treat a later user request as overriding a personal rule, while an organizational rule cannot be overridden by the user.

Among violations with the rule intact, a later request was read as permission in 45.9% of cases; the rule was disregarded in 27%; and in 21.6% it was acknowledged as applicable and violated anyway.

Do attacks and defenses change the picture?

Flooding context with filler barely worked — violations fell slightly, from 37.8% to 34.3%. Compactor-targeted injection raised violations to 46.3% and cut full rule retention from 70.1% to 42.5%. The strongest attack, called escape, pushed violations from 23.5% to 58.8%; even when the rule survived it, violations doubled from 13.3% to 26.5%.

The strongest defense preserved rules in 99% of summaries — yet agents still broke them in 29.4% of cases, against 23.5% with no attack at all. Rule retention is not a sufficient proxy for compliance, the authors conclude, and leave the core problem open: agents disobeying rules they demonstrably still possess. For enterprises letting agents touch email, data or systems, that gap — not forgetting — is the exposure.