This is not a serious proposal for Alan's stack. I take the problem Alan described in public, make assumptions around it, and walk towards an answer with cost, latency and money in view. Everything runs on AWS, by choice. Alan's real platform is its own, and I make no claim about it or about its bill.
Every amount below prices my exercise. Prices are AWS list prices in USD before tax, for the EU (Paris) Region, with models read through Amazon Bedrock's EU profile, from the offer files in the sources. List prices overstate what a company of this size pays, and private discounts are not public. Each figure is tagged public, inferred or assumed, and each comes from one script that applies every lever only to what the previous lever left.
The problem
In October 2026, Alan wrote up Hopper, its internal agent. It was built for code, so "every session ran the agent inside its own development sandbox". That made much less sense for a question about a Notion page, "which still waited for a sandbox and a full agent runtime to start". So:
In October, we moved the agent out of the sandbox. The agent loop now runs in our backend, and a sandbox starts only when a task needs a filesystem, a shell or a browser.
Alan reports that the median time before the agent starts responding has halved, and calls that early. The move is about latency. My question starts where it stops: once the loop is out of the sandbox, where does the money go, and how far can it fall without making the agent slower?
What I assume
Only the first two lines rest on public counts: "3k sessions are created each day", 80% of staff use the agent, and the press room gives 790+ staff. Everything about tokens is mine, so read the result as an order of magnitude.
The scale behind the exercise
| Quantity | Low | Base | High | Tag |
|---|---|---|---|---|
| Sessions a month | 63,000 | 63,000 | 90,000 | Inferred, 21 or 30 days |
| Users | 632 | 632 | 632 | Inferred |
| Sessions that need a sandbox | 20% | 35% | 50% | Assumed |
| Chat session: turns × context | 3 × 15,000 | 5 × 20,000 | 8 × 30,000 | Assumed |
| Build session: turns × context | 20 × 40,000 | 30 × 60,000 | 40 × 80,000 | Assumed |
| Sandbox size and life | 2 vCPU, 8 GB, 10 min | 2 vCPU, 8 GB, 20 min | 4 vCPU, 16 GB, 40 min | Assumed |
| Files kept in S3, 1 MB each, for 24 months | 500 GB | 1,000 GB | 5,000 GB | Assumed |
Where it runs matters here. Amazon EKS, AWS Fargate, Aurora, ElastiCache, S3, S3 Vectors, OpenSearch Serverless and G6 GPU nodes all have Paris price rows. Claude does not run in Paris alone: the Opus 5.5 card offers in-Region inference in London only, and from Paris the EU profile routes to Frankfurt, Stockholm, Milan, Spain, Ireland and Paris. Haiku 5.5 has no in-Region endpoint at all. For health data that routing is a compliance question, and I do not answer it here.
Version 0: the naive build
The first version is deliberately careless, the one a team ships to prove the idea. The loop runs on EKS, out of the sandbox as Alan's now is. Every turn calls Claude Opus 5.5 through the EU profile, with no cache, because it is the best model and the demo must work. A knowledge base answers document questions from a classic vector index. Sandboxes clone the repository through a NAT gateway. Everything else sits in S3 Standard and Aurora.
Requestsweb and Slack
Agent loopon EKS
Opus 5.5EU profile, no cache
Knowledge baseOpenSearch Serverless
SandboxFargate, clone via NAT
StateAurora and S3 Standard
Next turnresults return
Answerone stream back
Version 0, USD a month at list, rounded
| Line | Low | Base | High |
|---|---|---|---|
| Opus 5.5 tokens ($4.40 in, $22 out, per million) | 57,700 | 199,800 | 702,900 |
| Sandboxes on Fargate | 200 | 800 | 6,700 |
| Warm pool in working hours | 200 | 700 | 3,800 |
| Agent nodes on EKS | 1,200 | 1,700 | 3,400 |
| Aurora Serverless v2 | 1,000 | 1,400 | 2,300 |
| OpenSearch Serverless, classic minimum | 400 | 400 | 400 |
| NAT gateways and repository clones | 400 | 1,200 | 6,900 |
| S3 Standard | under 100 | under 100 | 100 |
| Total | 61,200 | 206,200 | 726,400 |
| Per user | 97 | 326 | 1,149 |
The model is 94% to 97% of every column. The spread is about twelve to one, because tokens are unknown. The sandboxes that started the story, warm pool included, are under 1% of the base.
Optimisation 1: prompt caching
The problem. Each of the 866,250 turns in the base re-sends a long prefix at $4.40 per million, and nothing remembers it.
The change. Put cache checkpoints on the stable prefix. I assume 80% of input is read at $0.22, 10% written at $5.50 and 10% sent fresh. A write costs 25% more than plain input, so a prefix that churns costs money. A session idle for over five minutes loses the cache, and the 1-hour write costs $8.80.
Latency. AWS's lens says caching reduces inference latency, without a number. I assume time to first token falls by a third on long prompts.
Dollars. It saves about $141,600 (about $100,200 at a 60% hit rate, $162,300 at 90%). That leaves about $64,600, and it is 69% of the start in one change. Nothing else on this page comes close.
- Added
Optimisation 2: a session store on ElastiCache
The problem. Every turn reads session state from Aurora, and that read is the last thing a person should wait for.
The change. Two cache.m7g.large Valkey nodes hold the live session. I assume Aurora loses 20% of its capacity and 30% of its I/O.
Latency. I assume a few milliseconds saved per turn, small against a model call but paid on every turn.
Dollars. This buys latency and fewer database reads. It saves no token, since the model still receives the same prompt. The nodes cost about $213 and Aurora gives back about $337, a net $124. That leaves about $64,400. If Aurora does not shrink, this lever costs money.
- Added
Optimisation 3: a semantic cache, which is the one that saves tokens
The problem. Many people ask the same question about the same document, and each one pays the model. A session store cannot help: it remembers a conversation, not an answer.
The change. Vector search on ElastiCache for Valkey can look up a similar past question on the same nodes. It only suits first questions of chat sessions, never code. Answers must be keyed by permission scope, or one person gets an answer built for another, which also lowers the hit rate. I assume 8% of chat sessions are answered this way.
Latency. A hit answers in about 100 ms instead of seconds, and a miss adds about 20 ms (both assumed).
Dollars. About $490, because the chat lane is the small lane once caching is on. That leaves about $63,900.
- Added
Optimisation 4: a capped answer with maxTokens
The problem. Output is the dear token, at $22 per million, and nothing bounds it.
The change. Cap the answer. A cap only bites on answers that would run past it, so I assume 12% less output. The risk is truncation: a cut answer, or a cut tool call, is sent again with its whole context. I assume 2% of turns.
Latency. Long answers end sooner, and each truncated turn adds a full extra turn (assumed).
Dollars. It saves about $848 gross, and each 1% of truncated turns gives back about $158, so it stops paying at about 5.4%. Net about $532. That leaves about $63,400. The better argument is the quota: Bedrock deducts input plus max_tokens at the start of a request, so a tighter cap throttles later.
Optimisation 5: Guardrails on Knowledge Base questions
The problem. Some questions should never reach the knowledge base or the model at all. Stopping them costs something too.
The change. A denied-topics policy, at $0.15 per 1,000 text units, on the question alone. A blocked prompt pays the guardrail and no model. I assume half of chat turns reach the knowledge base and 4% of those are blocked.
Latency. About 200 ms more on each guarded turn, and a blocked question answers at once (both assumed).
Dollars. The guardrail costs about $14 and avoids about $110 of tokens, a net $96. That leaves about $63,300. It pays above a block rate of about 0.5%. Scope it to the question: reading the whole 20,000-token prompt is 80 text units a turn and about $1,130 a month per policy, about ten times the tokens it avoids.
- Added
- Replaced
Optimisation 6: an agent router, Haiku 5.5 instead of Opus 5.5
The problem. Opus 5.5 reads a Notion page and renames a variable at the same price as a hard refactor.
The change. A small classifier sends what it can to Haiku 5.5, at $0.11 in and $0.55 out, a fortieth of Opus. Bedrock's own Intelligent Prompt Routing is listed as unsupported on both model cards, so I write the router. I assume 70% of chat and 25% of build turns move, at 15% more tokens because a weaker model takes more steps. Keep a session on one model, since a cache belongs to its model. Haiku's rates rise fivefold above 100,000 tokens in a prompt, so a long session must compact first.
Latency. About 300 ms for the classifier call, then a faster model on the routed turns (assumed).
Dollars. About $16,100, after a router that costs about $30 a month. That leaves about $47,200, and it is why choosing the model needs an evaluation per route.
- Added
- Replaced
Optimisation 7: distilled models for small questions
The problem. Small one-shot questions could run on something smaller still.
The change, and why I do not make it. AWS's page is blunt: "Distillation is not currently available for Anthropic models on Amazon Bedrock. There is no confirmed timeline for when Anthropic distillation will be restored." What remains is Nova and Llama teachers, in N. Virginia and Oregon. No EU Region is listed, so the nearest is across the Atlantic, which costs latency (I assume about 80 ms a call) and possibly compliance on health data.
Dollars. I priced it anyway, on a distilled Nova Lite at $0.06 and $0.24, against Haiku 5.5 for a 3,000-token question. It saves $0.000243 a question. A job of 15,000 pairs costs about $248, so it pays from about 178,000 such questions a month if retrained twice a year, about 349,000 if quarterly, and about 1 million if monthly. Haiku 5.5 is already that cheap. In my base, this lever is $0, and $47,200 stands.
Optimisation 8: a local model for code, on GPU nodes
The problem. What is left on Opus is mostly build work, and part of it is mechanical.
The change, and what happened to the first choice. I set out with Devstral Small 2. Its model card says Apache 2.0, and the licence holds. Its status does not, and Mistral's own pages disagree. The model page lists it as deprecated, dated February 27, 2026, with no retirement date. The overview gives the same deprecation date and a retirement on March 31, 2026. Both name Mistral Medium 3.5 as the replacement, and the overview lists that one as deprecated too, from October 31, 2026, with retirement on April 30, 2027. So I do not treat it as a safe replacement, and I keep this as the lesson: an open licence protects the weights, not the upkeep.
I price Mistral Small 4 instead, which Mistral's page lists as generally available under Apache 2.0, covering coding among other things. The price moves with the swap: the page gives 119 billion parameters against Devstral's 24, with 6.5 billion active. Its FP8 checkpoint holds about 121 GB of weights, which needs a g6.48xlarge (8 NVIDIA L4, 192 GB, $16.95 an hour) for one replica. The card's server example uses two GPUs, so eight on one replica is my sizing. It takes 20% of the build turns still on Opus, on EKS nodes that scale down, and I assume it carries 85% of them and 720 turns a node-hour. How good it is at code is the test to run first.
Latency. About 20 seconds a turn on my assumed throughput, likely slower than Opus, and a load of some minutes each morning (both assumed). The upside is that data stays in Paris instead of the EU profile.
Dollars. One node always on plus a second in working hours, the sizing I used for Devstral at about $5,528, would cost about $15,967 here against about $6,939 of Opus, so that does not pay. One node in working hours only, with overflow and nights left on Opus, costs about $3,577 and replaces about $5,898 of Opus net of 10% redone, so it saves about $2,353. It runs at 55.8% of paid capacity and pays from about 51,000 turns a month. That leaves about $44,800. In the low scenario it does not pay, and I leave it out. Against Haiku it never pays: the same turns would cost Haiku about $186. It only earns its place on work Haiku cannot do.
- Added
Optimisation 9: S3 Vectors against a classic vector index
The problem. OpenSearch Serverless bills about $410 a month at its classic minimum before a single query.
The change. Pick the store for the query volume. S3 Vectors has a Paris endpoint, a knowledge base can use it wherever both services exist, and it bills storage and queries. My assumed index holds 1 million vectors of 5 KB.
Latency. AWS states subsecond for infrequent queries and as low as 100 ms for frequent ones. At about 0.035 queries a second I read mine as infrequent, so slower than a warm index. The pages I read give no OpenSearch figure. Against a model turn of many seconds, that is acceptable (assumed).
Dollars. About $410 falls to about $2, a saving of about $409. Parity sits near 29.6 million queries a month against my 92,000. That leaves about $44,400.
- Replaced
Optimisation 10: the repository off the NAT gateway
The problem. This is the sandbox problem again, seen in the bill: each sandbox pulls about 1 GB of repository through a NAT gateway, at $0.05 per GB, while a person waits.
The change. Put the repository in the image, or fetch a snapshot from S3 through a gateway endpoint, which has no charge. I assume 90% of that traffic goes.
Latency. The sandbox starts tens of seconds sooner (assumed). For the people on the build path, this is the change they feel.
Dollars. About $992. That leaves about $43,400.
- Added
- Replaced
Optimisation 11: S3 tiers for sites and artifacts
The problem. Everything sits in S3 Standard forever, and few files are read after a week.
The change, step by step. I priced each class on 1 TB of 1 MB files kept 24 months, with 1% read back each month.
1 TB of artifacts, USD a month in a steady state, with the first-time transition of what already exists
| Option | A month | First transition | The catch |
|---|---|---|---|
| S3 Standard | 24.00 | ||
| Intelligent-Tiering | 8.97 | $2.50 monitoring. The fee pays from 229 KB to the first tier, 132 KB to the second. It takes 90 days to reach the cheap tier, and files under 128 KB never move | |
| Standard, then Glacier Instant at 60 days | 7.72 | 18.33 | 90-day minimum, $0.03 per GB read |
| Standard, then Glacier Flexible at 60 days | 7.44 | 29.15 | Restore in minutes to hours, 40 KB of metadata per file |
| Standard, then Deep Archive at 60 days | 6.59 | 55.00 | 180-day minimum, restore within 12 hours |
Latency. Glacier Instant reads in milliseconds. The two archives break any link a person clicks, so I keep them out. Sites stay in Standard: they are small, and their small files could not transition by default anyway.
Dollars. One lifecycle rule: Standard for 60 days, then Glacier Instant. It saves about $16 a month. Intelligent-Tiering does not pay here over that rule. Deep Archive saves $1.13 more a month and costs a 12-hour wait, so it does not pay either. A file deleted at day 75 still bills to its 90-day minimum in Glacier Instant, so if artifacts live only three months, leave them in Standard. The total stays about $43,400. This lever is under a thousandth of the bill.
- Replaced
Optimisation 12: a Compute Savings Plan
The problem. The steady compute is billed on demand.
The change. A one-year, no-upfront plan on the agent nodes ($0.3808 to $0.2888 an hour) and an assumed 70% of Fargate. The GPU node runs in working hours only, so it stays out. The commitment comes after the reductions, so it does not reserve waste.
Latency. None.
Dollars. About $630, which leaves about $42,800. It is thin for a reason: no Savings Plan covers Bedrock tokens, and tokens are most of what remains. The eligible lines are about 14.7% of the bill.
Optimisation 13: the Bedrock Reserved tier
The problem. Tokens need a commitment too, and I wanted to sign it last, because committing first would reserve the waste. Before the levers, this bill would have reserved $199,800 a month of tokens.
The change, and why I do not make it. It is not on offer. Both model cards mark Reserved, Priority, Flex and batch as unsupported for Opus 5.5 and Haiku 5.5. The rows exist for older models such as Sonnet 4.6, and the Bedrock article shows what they buy: a three-month term costs nine tenths of Standard tokens at full use, so it needs about 90% use just to break even. It buys capacity, and barely a discount.
Latency. None here. Elsewhere, it would buy steadier capacity.
Dollars. $0. $42,800 stands.
Optimisation 14: negotiation
The problem. A company at this scale does not pay list.
The change. A private discount, and I assume 8%. Nothing about any real agreement is public. Whether third-party models on Bedrock count towards one is not known: Claude is sold through AWS Marketplace, and the charges appear under the model provider. So I apply it only to what AWS sells itself.
Latency. None.
Dollars. About $620, which leaves about $42,200. If models counted, it would be about $3,422 and the end about $39,400.
Where it lands
Requestscaches first
Routerone Haiku call
Opus 5.5the hard work, cached
Haiku 5.5questions, easy steps
LocalMistral Small 4 on GPU
Next turnresults return
Answerone stream back
Start and end, USD a month, rounded
| Low | Base | High | |
|---|---|---|---|
| Version 0 | 61,200 | 206,200 | 726,400 |
| After every lever | 13,800 | 42,200 | 143,900 |
| Cut | 77.5% | 79.6% | 80.2% |
| Per user | 22 | 67 | 228 |
The model still takes about 83% of what is left, and the Opus line alone is about $34,300. Caching and the router did almost all of the work. The other twelve levers together save about $6,300, and two of them save nothing. Those two are the most useful results here, because both looked obvious until I read the model cards.
The next move sits outside this page's router. Sonnet 5.5 costs half of Opus 5.5 on every rate. If the remaining Opus work could move to it, about $17,100 of that line would go, and whether it can is a question for an evaluation.
The funnel
1 · $206,200Version 0
2 · $64,600Prompt caching
3 · $63,300Caches, cap and guardrail
4 · $47,200Router to Haiku 5.5
5 · $44,800Small 4 on a GPU node
6 · $44,400S3 Vectors
7 · $43,400NAT and S3 tiers
8 · $42,800Compute Savings Plan
9 · $42,800Bedrock Reserved tier
10 · $42,200Negotiated discount
- Version 0Base · Opus 5.5 on every turn, no cache$206,200 a month
- Prompt cachingOptimisation · 80% read, 10% written, 10% fresh$64,600 a month leftsaves $141,60068.7% of the base
- Caches, cap and guardrailOptimisation · Four small levers, about $1,300$63,300 a month leftsaves $1,3000.6% of the base
- Router to Haiku 5.5Optimisation · 70% of chat, 25% of build turns$47,200 a month leftsaves $16,1007.8% of the base
- Small 4 on a GPU nodeOptimisation · Distilled models add $0: no EU Region$44,800 a month leftsaves $2,4001.2% of the base
- S3 VectorsOptimisation · $410 floor down to about $2$44,400 a month leftsaves $4000.2% of the base
- NAT and S3 tiersOptimisation · $992 of clones, $16 of storage$43,400 a month leftsaves $1,0000.5% of the base
- Compute Savings PlanCommitment · Nodes and Fargate; no tokens$42,800 a month leftsaves $6000.3% of the base
- Bedrock Reserved tierCommitment · Not offered on Opus or Haiku 5.5$42,800 a month leftsaves $00% of the base
- Negotiated discountNegotiation · 8% assumed, on AWS-sold lines only$42,200 a month leftsaves $6000.3% of the base
Your own numbers will differ, and yours can be measured. The calculator prices your stack line by line, and a FinOps audit starts from your real billing data, in the same order: reduce first, then commit, then negotiate.
What I cannot see from outside
- Tokens per session, and the model mix. Everything here depends on them. None of it is public.
- Whether caching is already on. If it is, my first lever is taken and my start is far too high.
- Every share in the levers. The hit rates, block rates, routed shares and redo rates are mine. An evaluation would replace each one.
- Whether the EU profile fits a health-data path. I price it. I do not say it clears anything.
- GPU throughput. I measured nothing. If Mistral Small 4 serves fewer turns than I assume, on one replica across eight GPUs, it stops paying.
- What counts towards a private discount. Especially third-party models.
- Everything else in the company's cloud. This is one internal platform in an exercise, not a bill.