All tests pass. The code compiles. But the system doesn't do what you asked for.
If you've used AI coding agents on anything beyond a
single-file task, you've hit this wall. The agent produces beautiful,
locally-correct code that fundamentally misunderstands the bigger picture.
I studied this through a production project: ~300
conversation turns, 50+ generated files, 1,000+ passing tests. The system was
wrong — not "bug" wrong, but "you built the wrong thing"
wrong.
Here's why this happens, and why it's structural — not just
"the model isn't smart enough yet."
Local Correctness ≠ Global Correctness
Local correctness: Each function works. Tests pass.
Types check.
Global correctness: The system does what the user
actually wanted, in the actual environment, with all the implied constraints
never written down.
AI agents nail local correctness. They systematically fail
at global correctness — and the gap widens with project complexity.
Five Mechanisms of Context Loss
1. Semantic Drift
Your intent passes through spec → design → code. Each step
paraphrases, each paraphrase drifts. I said "deploy the full pipeline — ML
apps plus infrastructure." After three transformations, it became
"deploy infrastructure only." The application layer vanished.
Unnoticed for 170 turns.
The agent can't detect its own drift — it only sees the
previous step, never the original intent.
2. Salience Inversion
Transformers have structural recency bias. A foundational
constraint stated 200 turns ago fades. A config file read 3 turns ago is vivid.
So the agent contradicts your early decisions in favor of whatever it read most
recently.
Every foundational fact has a half-life in the
agent's memory. After enough turns, it will be overridden by noise.
3. Assumption Crystallization
Agent encounters ambiguity → picks an interpretation →
writes it into code → reads it back as established fact. The guess is now unfalsifiable
from inside the system. In my case, a wrong interpretation of
"pipeline" propagated through 15 files for 168 turns before anyone
noticed.
4. Environmental Divergence
The agent builds a mental model from docs and configs.
Reality has undocumented constraints, manual configurations, and implicit
dependencies. This gap only closes through deployment failures — each revealing
one new surprise. Six attempts in my case.
5. Compositional Explosion
Each component works alone. Together, emergent behaviors
appear that no individual component predicts. A chat bot entered an infinite
loop because three async systems interacted in a way only visible at runtime.
The agent can reason about 2–3 interactions. A real system has thousands.
The Compounding Loop
These aren't five independent problems. They feed each other
in a positive feedback cycle:
Salience loss → drift undetected
→ crystallization → wrong env model → composition blind spots → more context
lost → cycle repeats
The failure rate doesn't grow linearly — it accelerates. In
my project: ~8% during design, ~20% during deployment.
Why Common Fixes Don't Work
|
Proposed Fix |
Helps With |
Doesn't Help With |
|
Bigger
context window |
Fitting
more text |
Recency
bias, crystallization |
|
Better
retrieval (RAG) |
Finding
relevant docs |
Current
vs. outdated distinction |
|
Smarter
model |
Local
reasoning quality |
Verifying
against reality |
|
Multi-agent |
Task
decomposition |
Inter-agent
drift, shared context loss |
The
problem is ARCHITECTURAL, not capability-limited.
What Would Actually Help
1. Assumption
tracking — Tag every interpretation as unvalidated; surface periodically
for review
2. Intent
anchoring — Preserve original goal as immutable reference; check alignment
every N turns
3. Grounding
loops — Cheap verification against real environment without full deployment
4. Stratified
memory — Separate "foundational constraints" (never expire) from
"working context" (recent, mutable)
5. Complexity
budgeting — Agent knows when it's in over its head and asks for help
The Real Takeaway
LLM
agents won't replace developers. They'll redefine what developers do.
The bottleneck was never typing code. It was always thinking
— understanding what to build, why, and how the pieces fit together. Context
degradation proves this: the moment a project requires sustained architectural
vision, the agent falls apart. That's the part that's irreducibly human.
What AI agents are fantastic at: summarizing, generating
boilerplate, exploring solution spaces quickly, reducing the mechanical burden
of coding. They're a force multiplier for the execution layer. They free you to
spend more time on the thinking layer.
The irony is that these tools don't make thinking less
important. They make it more important.
The
future isn't "AI writes code, humans watch." It's humans who think
more carefully about what and why, paired with agents who execute the how at
speed.

