I’ve been building an API smoke test pipeline — a blocking gate that hits a handful of test-environment endpoints before anything moves to pre-prod. In test, we use a mock server to isolate the environment, so we can validate all the infra, networking, and pods without depending on anything external. The mock data structures had some mistakes which was causing errors in the pods. I figured an AI agent could chase the logs and then apply the fixes faster than I could.
Twenty-six prompts later, it had read the pod logs, the mock api data responses, the Kubernetes manifests, the Helm values, and the CI pipeline definition. It worked out how the data needed to be reshaped to stop the services erroring, fixed it, and the pipeline went green. Genuinely useful work. Total damage: 1.8 million input tokens — about $5-6 in API costs just to re-read the same context over and over.
That’s the number that made me stop and do the maths. Five bucks, for work that was worth doing. The problem isn’t that the agent wasted tokens. It’s that even when it’s doing something real, the architecture makes every step expensive. And that’s about to matter a lot more, because the pricing model recently changed.
What Even Is a Token?
Quick primer, before we get into things.
LLMs don’t read words. They read tokens — chunks of text from a vocabulary that varies by model but typically sits somewhere between 30,000 and 200,000 entries. A token might be a whole word (“apple”), a partial word (“appl”), or a single character in edge cases. Spaces, brackets, punctuation — all tokens.
“Hello, world!” is about 5 tokens. This paragraph so far? Roughly 50.
Every model charges by the token — input tokens (what you send) and output tokens (what it generates). And here’s where it gets interesting: the ratio between input and output is rarely what you’d expect.
The Autoregressive Loop
Here’s the architectural fact that drives everything else.
LLMs generate text one token at a time. Each token requires a full forward pass through the network. If the output is 1,000 tokens, the network runs 1,000 times. And every single one of those 1,000 passes re-reads the entire context — your system prompt, the conversation history, the file contents, everything.
There’s a clever optimization called KV caching that saves intermediate calculations so you don’t recompute the whole context from scratch every time. Without it, generating a 1,000-token response from a 10,000-token context would be ~200x more expensive than the naive number suggests.
But the KV cache lives in GPU memory. Expensive. Finite. In most API services, the cache is discarded after each request — you pay the prefill cost again on every new message. Even in persistent sessions, GPU memory pressure from other requests can evict the cache, forcing a rebuild. Either way, that rebuild is a “pre-filling” step that costs as much as the original context length.
This matters for one reason: the longer the conversation gets, the more each additional token costs.
The Simple Chatbot Math
Let’s start with the cheap case — a single-turn chatbot exchange:
- System prompt: 1,000 tokens
- User query: 100 tokens
- Model’s thought process (CoT): 10,000 tokens
- Final response: 1,000 tokens
Total: ~12,000 tokens. At ~$2-3 per million input and ~$15 per million output, that’s a fraction of a cent. Basically free.
This is why chatbots feel magical. For a single Q&A, the economics are invisible. You’d have to ask thousands of questions to notice the cost.
The Agentic Coding Math
Now let’s look at what happens when your AI coding agent gets to working solving a problem for you.
Here’s a typical trajectory:
- System prompt + user request (“find the bug in this code”) → 2K tokens
- Agent thinks, writes a plan → adds 5K output tokens
- Agent calls
read_fileonmain.py→ 10K tokens dumped back into context (file contents) - Agent thinks, decides to check another file →
read_fileonconfig.py→ another 10K - Agent makes a code patch → adds 2K output tokens
- Agent verifies → runs tests → test output comes back → 5K
- Tests fail → agent debugs → more thinking → more file reads
By step 5, you’re at ~30K tokens of accumulated context. By step 7, you’re at 55K-60K. And critically — every output token at step 7 is priced against the full 60K context.
That smoke test example from the opening? Twenty-six prompts, five or six file reads, a few back-and-forth loops. The session logs show it consumed 1,812,400 input tokens and 38,900 output tokens. At current pricing, that’s about $5-6 in input alone. For a mock data reshape.
Not $5-6 total. $5-6 just to read the files and think about it. The actual codegen is extra.
Why the Rug Got Pulled
This math was always true. But for the first year or two, it was subsidized. Flat-rate pricing ($10/month for Copilot, $20/month for ChatGPT Plus) made the input-to-output cost invisible.
That was never sustainable.
GitHub switched Copilot to AI credit pricing — you now pay per “agentic” operation, not per seat. Anthropic tightened usage caps for premium users earlier this year (widely reported, March 2026). OpenAI’s Pro tier ($200/month) has hard usage limits that kick in faster than most power users expect.
The pattern is consistent: the flat-rate era is ending. Companies that were eating the token cost to win adoption are now passing it through. And the users who feel it first are the power users — the ones running agentic loops, not single-turn completions.
What the Ecosystem Is Doing About It
The token tax is real, but the industry isn’t just sitting there taking the hit. A few things are already working:
Prompt caching is the biggest win available today. OpenAI, Anthropic, and Google all support it now. The idea is simple: if your system prompt and conversation prefix stay the same between calls, the provider caches the KV tensors and charges you 50-90% less for cache hits. A PwC evaluation across 500+ agent sessions found prompt caching cuts API costs by 41-80% and improves time-to-first-token by 13-31%. The catch: you have to structure your prompts carefully, static content first, dynamic content at the end, or you’ll bust the cache.
Context compaction is what agentic coding tools do when sessions get long. Claude Code, Codex CLI, and OpenCode all compress older conversation history when you approach the context limit. Factory’s evaluation across 36,000 real engineering messages showed anchored iterative summarisation (merging new summaries into persistent state rather than regenerating from scratch) preserves 95%+ task accuracy while cutting token usage by 50-80%. The trade-off: every compaction is a hard semantic break that invalidates cached prefixes, so you’re trading one cost for another.
Speculative decoding speeds up output generation by using a smaller “draft” model to propose multiple tokens at once, then verifying them in parallel with the big model. vLLM with EAGLE achieves up to 2.8x latency reduction. This doesn’t reduce input costs, but it gets you through the output phase faster, which matters when you’re paying per second of GPU time on self-hosted setups.
Quantised KV caches compress the Key/Value tensors from FP16 to FP8, cutting memory usage roughly in half. More tokens fit in GPU memory, which means more concurrent users and longer contexts before eviction. vLLM supports this out of the box.
Smarter model routing is the pragmatic fix. Not every task needs Opus or GPT-5. Use Haiku or Flash for file reads and simple edits, reserve the big models for complex reasoning. A bug fix that takes Opus 3 turns might take DeepSeek 5 turns, but at $0.10/$0.20 versus $5/$25, the cheaper model wins on total cost even with more turns.
None of these solve the fundamental architecture problem. The autoregressive loop will always re-read context. But stacked together, they turn a surprise $200 bill into something you can actually justify.
What This Means For Developers
The cheap things stay cheap. Code completion — finish my for-loop, write this docstring, suggest the next line — costs almost nothing. Single-turn Q&A is a rounding error. These will keep getting cheaper, maybe free.
The expensive things — agent loops that autonomously read your codebase, plan, execute, verify — will stay expensive because the architecture demands it. Not because someone’s price-gouging. Because every single step re-reads the entire conversation. That’s not a billing problem. That’s the architecture.
This doesn’t mean agentic coding is bad or overpriced. For complex, multi-file refactors, $5-6 in tokens is still cheaper than an hour of your time — and it works while you sleep. But it means you need to think about when you use it.
Letting an agent run wild on every prompt is like taking a helicopter to the corner shop. It’ll get you there, but maybe save it for the trip that actually needs one.
The cheap things — completion, summarization, single-turn Q&A — will keep getting cheaper, maybe free. The expensive things — agent loops that read your codebase, plan, execute, verify — will stay expensive because the architecture demands it. Not because someone’s price-gouging. Because every single step re-reads the entire conversation. That’s not a billing problem. That’s the architecture.