This is not a serious proposal for BlaBlaCar's stack. I take the problem BlaBlaCar described in public, make assumptions around it, and walk towards an answer with cost, latency and money in view. Everything runs on AWS, by choice. BlaBlaCar has written that it runs on Google Cloud, and I make no claim about its platform today or about its bill.
Every amount below prices my exercise. Prices are AWS list prices in USD before tax, for the EU (Paris) Region, from the offer files in the sources and the site's own price data. List prices overstate what a company of this size pays, and private discounts are not public. Each figure is tagged public, inferred or assumed, and each comes from one script that applies every lever only to what the previous lever left.
The problem
The problem is in one sentence of BlaBlaCar's engineering post of September 2022: going to the cloud "is mostly associated with a 'pay what you request model' and not a 'pay what you use model'". The same post gives the number most companies keep to themselves:
At the beginning of the project, we were only using 25% of the requested CPU !
A marketplace feels this harder than most: "our members travel more during weekends than on Tuesday morning". The post reached 53%, and its first principle was "reliability before costs". That work was on Google Cloud, in 2022, and it was not a cost project. My question moves it: if that platform had been built on EKS by a team in a hurry, what would it cost, and which levers would actually matter?
What I assume
No public source gives today's fleet. About 3,000 pods in 2017 and a hundred database clusters around 2019 are the only anchors, so the range is wide on purpose. Low pairs the small fleet with the kindest lever assumptions, and high the large fleet with the harshest.
The scale behind the exercise
| Quantity | Low | Base | High | Tag |
|---|---|---|---|---|
| Stateless pods | 3,000 | 5,000 | 8,000 | Assumed |
| Requested vCPU per pod | 1 | 1.5 | 2 | Assumed |
| Share of requests used | 25% | 25% | 25% | Public, 2022 |
| Peak of use over the mean | 1.3 | 1.4 | 1.6 | Assumed |
| Database clusters | 100 | 100 | 100 | Public, around 2019 |
| Nodes and vCPU per cluster | 3 × 8 | 3 × 16 | 3 × 24 | Assumed |
| Disk per node | 500 GiB | 1,000 GiB | 2,000 GiB | Assumed |
| Internet egress a month | 150 TB | 300 TB | 600 TB | Assumed |
| S3 | 1 PB | 2 PB | 4 PB | Assumed |
I price CPU and not memory. On m7i and m7g a vCPU comes with 4 GiB, and if pods bind on memory first, the savings below are too high.
Version 0: the naive build
The first version is what a team ships when the migration has a deadline. EKS in Paris, three clusters, On-Demand m7i.2xlarge nodes running day and night. Requests are copied from the old platform, generously, because nobody wants to be the one who caused the outage. Each service gets its own Application Load Balancer. Every call to S3 and every image pull crosses a NAT gateway. Databases run as StatefulSets on r7i.4xlarge, and the internet takes everything else directly.
Membersweb and apps
ALB per service300 of them
Podsx86, On Demand, always on
DatastoresStatefulSets on r7i
S3 and imagesthrough NAT gateways
Retriesand calls between Pods
Responseone stream back
Version 0, USD a month at list, rounded
| Line | Low | Base | High |
|---|---|---|---|
| Stateless nodes, x86 On Demand | 171,700 | 459,900 | 1,056,600 |
| Datastore nodes | 136,100 | 272,300 | 408,400 |
| Node root volumes | 4,600 | 12,400 | 28,600 |
| Database disks and snapshots | 19,200 | 38,400 | 76,900 |
| S3 Standard | 22,600 | 44,600 | 88,600 |
| Internet egress | 11,300 | 18,800 | 33,800 |
| Load balancers | 10,900 | 10,900 | 10,900 |
| NAT gateways, traffic between zones | 4,300 | 8,300 | 16,300 |
| EKS, three clusters | 200 | 200 | 200 |
| Total | 381,000 | 865,800 | 1,720,200 |
Stateless nodes are 53% of the base and datastores 31%. In the base, 1,339 nodes are running for 7,500 requested vCPU, of which 1,875 are used. Everything else on this page matters less than that gap.
Optimisation 1: requests set from use
The problem. The scheduler places Pods by what they request, so a request set out of caution is a bill. Pods use a quarter of it.
The change. Set requests from measured use, with the Vertical Pod Autoscaler in recommendation mode and BlaBlaCar's own Horizontal Pod Autoscaler target as the model. I size requests so that the peak of use reaches 60% of them, between the post's published 53% and its 70% target. The peak matters for honesty: the mean belongs to the night lever below, and counting it here would count it twice.
Latency. A tight request is throttled sooner on a spike. At 60% at the peak I assume no change at p50, and the Saturday is the test, as the post says.
Dollars. 7,500 requested vCPU become 4,375, and 10,714 vCPU of nodes become 6,250. It saves about $197,000, the largest lever on the page, and it takes engineering time rather than a purchase. Measured last, it is worth about $67,000, since the later levers shrink the same dollars. That leaves about $669,000.
- Replaced
Optimisation 2: bin-packing with Karpenter
The problem. Nodes are 70% full on requests, because a node group scales in fixed shapes and rarely removes a half-empty node.
The change. Karpenter chooses instance shapes and consolidates. I assume 85% full. The post's "up to 95%" is a ceiling, and it was reached with a scheduler profile. The trap is that consolidation moves Pods, and a PodDisruptionBudget or a slow start then decides whether anybody notices. I also keep it away from the datastores.
Latency. Each move is a Pod restart. I assume it stays invisible with enough replicas and a budget in place, and it will not with one replica.
Dollars. 6,250 vCPU become 5,147. About $49,000. That leaves about $620,000.
- Added
- Replaced
Optimisation 3: scaling down at night
The problem. Nodes are still sized for the peak, and the post reports use falling to about 20% of requests at night.
The change. Let the Horizontal Pod Autoscaler and Karpenter follow the traffic, so the fleet averages the mean of use, plus 10% of headroom, instead of the peak. This is the gain the first lever left out. The trap is the morning: scale-up must beat the commute, and holidays add "hard-to-anticipate usage peaks", so the floor has to be generous.
Latency. A cold ramp costs seconds on the first requests of a spike (assumed), and a warm pool on the morning peak removes most of it.
Dollars. 5,147 vCPU become 4,044 on average. About $49,000. That leaves about $572,000.
- Replaced
Optimisation 4: Graviton
The problem. The nodes are x86, and the same size costs 19% less on Graviton: $0.0476 against $0.0588 per vCPU-hour.
The change. Build multi-architecture images and move 80% of stateless to m7g. Native dependencies and third-party sidecars are what stop the rest, which is why the Graviton article starts with what does not fit. I assume equal throughput per vCPU, and it needs a load test per service.
Latency. Per AWS's argument it may improve, and I assume no change until measured.
Dollars. About $26,000, which leaves about $545,000. This figure flatters the lever, because it comes before Spot. On Spot the gap shrinks to 7%, $0.0224 against $0.0241. Measured last, Graviton is worth about $13,000. If a team could do only one of the two, Spot is the one.
- Replaced
Optimisation 5: Spot where a Pod tolerates it
The problem. Most stateless Pods can survive a lost node, and they are paying for a promise they do not need.
The change. Spot for interruptible Pods, 60% of stateless, on a hard requirement rather than a preference: the post found that pods which tolerated preemption kept landing on on-demand nodes because a preferred rule gives no control. In Paris, the Spot Instance Advisor showed m7g.2xlarge at 53% off and m7i.2xlarge at 59% off, both with interruption at "15-20%" a month, on the day I read it, as a reference for the mix. That is one day's reading, and it is a poor bucket, so Karpenter must spread across many shapes, and I hold 5% more capacity for churn.
Latency. A node lost at a two-minute notice means retries, and p99 suffers on the day (assumed). The Pods that stay On Demand are the ones where that matters.
Dollars. About $46,000. That leaves about $499,000.
- Added
- Replaced
Optimisation 6: shared load balancers
The problem. 300 services, 300 Application Load Balancers, each billing its hours and three public IPv4 addresses before a single request.
The change. Put several applications behind one load balancer: two per cluster, six in all, 50 services on each, under the default quota of 100 rules. The capacity units follow the traffic, so they stay. The trap is the blast radius: one bad rule now touches 50 services.
Latency. None per request (assumed). A shared listener takes slightly longer to evaluate its rules.
Dollars. About $9,000, 1% of the start. That leaves about $491,000.
- Replaced
Optimisation 7: NAT Gateway and VPC endpoints
The problem. Calls to S3 and image pulls cross NAT gateways, at $0.05 per GB.
The change. Gateway endpoints for S3 and DynamoDB, which have no charge, and interface endpoints for ECR. I assume 60% and 25% of the 80 TB. Interface endpoints bill by the hour, so they pay only with volume: here 18 of them cost about $145 and $200 of data, against about $1,000 saved.
Latency. Image pulls are a little faster, and scale-up with them (assumed).
Dollars. About $3,000. At these volumes it is hygiene. That leaves about $488,000.
- Added
- Replaced
Optimisation 8: datastores, and why I keep them on nodes
The problem. After stateless, the largest line is a hundred clusters on r7i.4xlarge, and databases do not follow the night.
The change. Right-size by 20% on average across the clusters, from measured use, and move to r7g, 19% lower. The post left the Vertical Pod Autoscaler on stateful sets as a "maybe", and a database under pressure is an outage, so this is slow work done by cluster.
I also priced the two obvious moves, and neither pays in dollars.
One cluster of 1 TB, three ways, base
| Option | Against nodes on r7g | The catch |
|---|---|---|
RDS for MariaDB, Multi-AZ, db.r8g.4xlarge |
41% more, about $84,000 a month on 100 clusters | Buys operations, not dollars. Only some of the eight engines exist there |
| DynamoDB, on demand | Cheaper below about 447 writes a second, with five reads per write | Above that it costs more, and it means rewriting the access patterns |
RDS may still be right: a team that stops running a hundred clusters by hand gets something a price list does not show. It just is not a FinOps saving.
Latency. Right-sizing has a price under load, and that is why 20% is the assumption and not more.
Dollars. About $96,000. That leaves about $392,000.
- Replaced
Optimisation 9: CloudFront, on a flat-rate plan or not at all
The problem. 300 TB a month leaves straight through the load balancers.
The change. CloudFront in front, with a catch in the sums. Pay-as-you-go costs about $20,000, more than the $19,000 of direct egress, because Europe's tiers are close and every request bills. So the flat-rate plan is the lever: the $6,000 Premium level covers 350 TB and 3.5 billion requests. AWS says "you will not incur overage charges", and that a first spike "up to 3x your monthly allowance won't affect your service that month", which suits a holiday. Sustained excess is different, and "traffic delivery might be adjusted".
The real catch is the domain. Each plan covers one distribution and one apex domain. I assume one. With a domain per country, the plans multiply, and this stops paying at about 18 plans at the $1,000 default level, or 3 at the $6,000 level.
Latency. Better for anything cached, and TLS ends nearer the traveller (assumed). The plan includes the web ACL it requires.
Dollars. About $13,000. That leaves about $379,000.
- Added
- Replaced
Optimisation 10: S3 tiers
The problem. Two petabytes in S3 Standard, with photos, logs and backups treated alike.
The change. Expire what nobody needs, 20% in my assumption, then move what cools: 40% to Glacier Instant Retrieval. The trap is its 90-day minimum: a file deleted early still bills to day 90, and very small objects do not pay for their transition.
Latency. Milliseconds in Glacier Instant, so a member opening an old photo sees nothing.
Dollars. About $22,000, and I do not price the first transition, which depends on an object count I cannot see. That leaves about $356,000.
- Added
Optimisation 11: Cost Explorer and allocation
The problem. The post speaks of being "haunted by orphaned cloud resources". Nothing here says who owns what.
The change. Tag and categorize first, so that each team sees its namespace and each volume has an owner. Cost Explorer itself saves nothing: it shows. What it finds, I assume, is 15% of disks and 30% of snapshots that nothing needs. It needs a retention policy, not a sweep, because the wrong snapshot is unrecoverable.
Latency. None.
Dollars. About $7,000. That leaves about $349,000. In truth this lever is how the others stay won, and that part has no line here.
Optimisation 12: Savings Plans on the floor
The problem. What runs all month is on On-Demand rates.
The change. Commit last, because a plan bought before the first lever would have reserved the waste. A one-year Compute Savings Plan covers 1,132 vCPU, the bottom of the night among what is not on Spot, at 24.2% off m7g. A three-year plan covers the datastores, at 47.8% off r7g, because they do not move. It is the same order the Graviton article gives: move, then commit. That rate is a three-year Compute Savings Plan, which follows a family change. The trap is the term: a dollar commitment for three years, while the next right-sizing shrinks what it covers. An EC2 Instance plan would be $0.455, deeper, and tied to one family.
Latency. None.
Dollars. About $94,000, of which $84,000 is the datastores and only $10,000 the nodes. The elastic part has already gone to Spot. That leaves about $255,000.
Optimisation 13: negotiation
The problem. A company at this scale does not pay list.
The change. A private discount, and I assume 5%, on everything but Spot. Nothing about any real agreement is public.
Latency. None.
Dollars. About $11,000, which leaves about $244,000.
Where it lands
CloudFrontflat-rate plan
Shared ALBstwo per cluster
Spot PodsGraviton, Karpenter
Floor PodsSavings Plan
Datastoresr7g, three-year plan
S3 and imagesVPC endpoints
Callsbetween Pods
Responseone stream back
Start and end, USD a month, rounded
| Low | Base | High | |
|---|---|---|---|
| Version 0 | 381,000 | 865,800 | 1,720,200 |
| After every lever | 93,600 | 244,100 | 661,400 |
| Cut | 75.4% | 71.8% | 61.6% |
That 71.8% measures how careless version 0 is, on purpose. It says nothing of what BlaBlaCar could save today, since its post reached 53% in 2022, on another cloud.
The order of the money is the finding. Requests, packing and the night together are about $294,000, and they cost engineering time alone. Datastores and commitments add $190,000. Load balancers, NAT, CloudFront and allocation, the usual list of quick wins, add $32,000 between them, less than 4% of the start. The largest line that remains is the datastores, at about $92,000, followed by nodes at $49,000 and Spot at $42,000.
The funnel
1 · $865,800Version 0
Optimisation11 steps, saves $516,800
Commitment1 step, saves $94,300
Negotiation1 step, saves $10,600
14 · $244,100Assumed negotiated discount
- Version 0Base · Naive EKS, x86 On Demand, always on$865,800 a month
- Requests set from useOptimisation · 60% of the peak, not 25% of a guess$669,000 a month leftsaves $196,80022.7% of the base
- Karpenter bin-packingOptimisation · Nodes 85% full, not 70%$620,400 a month leftsaves $48,6005.6% of the base
- Scale down at nightOptimisation · Nodes follow the mean, not the peak$571,800 a month leftsaves $48,6005.6% of the base
- Graviton nodesOptimisation · 80% of stateless, 19% lower$545,300 a month leftsaves $26,5003.1% of the base
- Spot where Pods tolerate itOptimisation · 60% of stateless, a day's reading$499,500 a month leftsaves $45,8005.3% of the base
- Shared load balancersOptimisation · 300 become 6$490,600 a month leftsaves $8,9001% of the base
- NAT off for AWS trafficOptimisation · 85% of the bytes through endpoints$487,500 a month leftsaves $3,1000.4% of the base
- Datastores right-sizedOptimisation · 20% smaller, r7g; RDS costs 41% more$391,600 a month leftsaves $95,90011.1% of the base
- Flat-rate CloudFront planOptimisation · Pay-as-you-go would cost more$378,800 a month leftsaves $12,8001.5% of the base
- S3 tiers and expiryOptimisation · 20% expired, 40% in Glacier IR$356,400 a month leftsaves $22,4002.6% of the base
- Allocation finds the orphansOptimisation · 15% of disks, 30% of snapshots$349,000 a month leftsaves $7,4000.9% of the base
- Savings Plans on the floorCommitment · 1 year on nodes, 3 on datastores$254,700 a month leftsaves $94,30010.9% of the base
- Assumed negotiated discountNegotiation · 5% assumed, not on Spot$244,100 a month leftsaves $10,6001.2% of the base
Your own numbers will differ, and yours can be measured. The calculator prices your stack line by line, and a FinOps audit starts from your real billing data, in the same order: reduce first, then commit, then negotiate.
What I cannot see from outside
- The size of the fleet today. Every total scales with it, and my range is a factor of four.
- What happened after 2022. If use over request is near 70%, the first lever is taken, and the page is about purchasing.
- Memory. The post reports CPU only. If pods bind on memory, the savings are too high.
- Traffic. Egress, requests, NAT volume and cross-zone bytes are all mine.
- Apex domains. The flat-rate lever assumes one.
- Database load. It decides whether DynamoDB pays anywhere.
- Spot in Paris. One day of the Advisor is not a record, and "15-20%" is a poor bucket.
- Everything else in the company's cloud. Data, observability and managed services are not here, and on some companies they are larger than compute.