We Deleted Autoscaling & Finally Scaled

Our platform runs chemistry. A researcher describes a molecule they want to make, and we go find the synthesis routes, read the patent landscape, search the literature, design the experiment, and lay out a multi-stage project to get there. Under the hood that means a deep research run, a literature phase, a project architect pass — each one an LLM-driven pipeline that reads hundreds of papers, embeds them, reasons over them, and writes a structured result.

A single one of those tasks takes ten to twenty minutes and wants at least 16 GBs of RAM.

That one sentence is the whole story of our infrastructure. Everything we got wrong, we got wrong by not taking it seriously enough.

Act I: one big server

We started where everyone starts. One EC2 instance running the API, a Celery worker for background work, Redis as the broker. It was simple, we could reason about it, and for a long time it was correct.

Then researchers started using it like researchers use things — which is to say, all at once and without warning. A few concurrent deep research runs plus a couple of experiment literature searches, and the box was gone. Not slow: stuck. The embedding work is CPU-bound, the pipelines are memory-hungry, and Celery pre-fork workers will happily accept more tasks than the machine can survive. Meanwhile the API — the part that just serves CRUD, the part where someone is actively clicking and waiting — was stuck behind the same starved CPU.

The obvious diagnosis was "we need more servers." That diagnosis cost us two architectures.

Act II: autoscaling, and the five-minute problem

So we did the textbook thing: a load balancer, an autoscaling group, scale out under pressure.

It didn't work, and the reason it didn't work is the most useful thing we learned.

Autoscaling is a control loop, and control loops need time. Something has to notice load is high, decide to add capacity, provision an instance, boot it, start the app, warm it, and register it as healthy. For us that round trip was five to ten minutes.

Now look at the shape of our load. It is not a ramp. Nobody's traffic gently climbs toward lunchtime. One researcher kicks off a multi-stage project, and within seconds we owe the system four heavy tasks. Two colleagues do the same thing on the same afternoon. Load goes from nearly nothing to fully saturated in the time it takes to click a button, stays there for ten minutes, then vanishes.

Against that shape, a five-minute reaction time isn't slow — it's useless. By the time the new instance was serving traffic, the burst that triggered it was half over. In the meantime, the existing box was already slowed, which meant we'd managed to build a system that reliably added capacity after it stopped being needed, while the user watched a loading spinner.

Worse, scaling the whole application to absorb heavy background work meant paying for API capacity we didn't need in order to get worker capacity we did. The two resources had nothing to do with each other, and we'd bolted them together.

Act III: containers, then Kubernetes

We assumed the problem was our crude autoscaling, so we went up the ladder. Containers first — Docker, a registry, ECS. That was real progress and we kept it; a single image that can boot as any role is genuinely good engineering and it's load-bearing for us to this day.

Then we trialled Kubernetes, because that's what you do.

And we hit exactly the same wall, wearing much nicer clothes.

Here's the thing nobody tells you clearly enough. Kubernetes' Horizontal Pod Autoscaler scales replicas of a long-lived service based on aggregate metrics. That is a beautiful abstraction when your workload is a stream of small, uniform, interchangeable requests, and average CPU across the fleet is a meaningful signal. Add pods, spread the load, remove pods.

Our workload is none of those things. Our unit of work is:

• Expensive — 2 vCPU and 16 GB, for one task
• Long — minutes, not milliseconds
• Indivisible — you cannot serve one deep research run 30% faster by adding a second pod; it is one pipeline with one sequential reasoning chain
• Bursty and random — no ramp, no pattern, no warning

Average CPU across a fleet tells you almost nothing when a single unit of work can saturate a node by itself. Furthermore, a pod still has to be scheduled onto a node that exists, which meant we were back to users waiting for availability — or paying to keep warm nodes idling for requests that might not come today.

Autoscaling, at every level of sophistication we tried, required us to have a strategy: a prediction about the shape of future load, expressed as thresholds and cooldowns and buffers. We didn't have a prediction. We had researchers.

The reframe

The most critical pivot wasn't a new tool. It was a question:

Are we scaling requests, or are we scaling jobs?

Every tool we'd reached for was built to scale requests. More concurrent HTTP traffic, so run more copies of the server behind a balancer. We'd been trying to express "I need one 16 GB machine for ten minutes, right now, and then never again" in a vocabulary that only knows how to say "run N copies of my service."

We didn't have a traffic problem. Our API traffic was trivial — a few hundred CRUD calls, some polling, nothing a modest instance couldn't handle for years.

We had a job problem. And there is an entire class of infrastructure built for exactly that, which the web-app world mostly ignores because it grew up in scientific computing and data pipelines.

Act IV: AWS Batch on Fargate

The final architecture is almost boring, which is the highest compliment I can pay it.

One modest EC2 instance runs the API and a warm Celery worker. It handles everything cheap and interactive: CRUD, auth, the lightweight background tasks, the quick asks, the scheduled watchdogs. It is never asked to do heavy lifting again, so it is never starved, so the UI is always responsive. This box does not autoscale. It doesn't need to.

AWS Batch on Fargate runs everything heavy. When a user starts a deep research run, we don't queue it for a worker — we submit a job. Batch launches a container with exactly 2 vCPU and 16 GB, dedicated to that one task. The task runs. The container exits. We pay for the minutes it ran.

Two pieces of jargon, since they're the crux:

• Fargate is serverless containers. You ask for a container with a given CPU and memory; AWS finds it somewhere. There is no cluster, no node pool, no capacity to provision, nothing to keep warm, nothing to patch.
• AWS Batch is a job scheduler on top of that. You define a job once, then submit executions of it. Its primitive is the job — one submission, one run, a lifecycle, a retry policy, a terminate call.

That last sentence is the entire argument of this post. Batch's primitive is a job, and our unit of work is a job. Everything that had been friction dissolved, not because Batch is clever, but because we stopped translating.

No autoscaling strategy, because there's no fleet to size. Fifty tasks arriving at once is fifty containers — Batch doesn't care whether we submit one job or fifty, and no amount of burstiness makes it "lag," because nothing is shared. Perfect resource isolation: one task's compute cannot touch the API resources or another user's run. This uses the same container image code that the EC2 worker runs, booted into a different role, so there are no two code bases to maintain.

We've verified 50+ concurrent heavy task runs, and 100+ is comfortable. The relevant comparison isn't that number in isolation — it's that a fraction of this load used to hang a dedicated EC2 instance, or send users to wait for a ten-minute scale-up process.

The constraint that made it legal

Fargate containers are not instant. Cold start is around two minutes.

For most web architecture, two minutes is a problem, and this is where I expect readers to close the tab. So, to be precise about why it's fine for us: our users don't wait.

Multi-stage project creation, experiment design, deep research — these are understood by the people using them to be long. A researcher starts one the way you'd start a load of laundry: they kick it off, go do something else, and come back later to a finished result and a notification. Nobody is watching a loading state. Against a ten-minute task that the user has already walked away from, two minutes of startup is invisible.

This is the load-bearing condition for the whole design, and it's the first thing to check before copying it. If your heavy work has a latency budget measured in seconds, none of this applies to you. Our advantage wasn't that we found better infrastructure — it's that our UX was honest about being asynchronous, which bought us the freedom to choose it.

We did keep one escape hatch. A couple of our background tasks belong to a heavy family but finish in well under a minute, and for those, two minutes of container startup costs more than the work itself. They stay on the warm Celery worker. The routing rule is "what is this task," not "which server do we use now" — which is a healthier way to think about it anyway.

The parts nobody blogs about

Elastic compute was the easy half. Two unglamorous things did more for reliability than Batch did.

Admission control came first. Before any of this, we capped concurrency per user: one running heavy task per category, a small queue behind it, and a clear error when you exceed it. This is not a scaling feature, it's a bounding feature, and it's the reason elastic compute didn't turn into an elastic bill. Infinite capacity with no admission control isn't an architecture, it's a billing incident waiting for a trigger.

Ephemeral compute needs a reaper. A traditional Celery worker acknowledges tasks late, so if it dies mid-task the broker hands that task to someone else — a safety net you inherit for free and stop thinking about. Batch has no equivalent. If a container dies, nothing anywhere knows it was supposed to be doing something. The task's record sits there marked "running" forever, holding that user's concurrency slot hostage.

So we built heartbeats: running tasks touch a timestamp, and a scheduled sweep fails anything that's gone quiet for five minutes, freeing the slot budget for this work. Moving from a broker-backed worker to ephemeral jobs means re-implementing the durability guarantees the broker was quietly providing — and that recovery code is the least-exercised code you own, because by definition it only runs when something else has already gone wrong. Give it its own tests and its own alerting.

So should you do this?

Not necessarily. The point isn't that Batch beats Kubernetes — it's that they answer different questions, and the industry has gotten into the habit of reaching for the same answer regardless of the question.

Use this if your heavy work looks like ours:

	Job runner (Batch/Fargate)	Autoscaled cluster (K8s/ECS)
Unit of work	Long, expensive, indivisible	Short, cheap, uniform
Load shape	Bursty, random, unpredictable	Steady, or predictably cyclical
Latency budget	Minutes — users work async	Milliseconds — users are wait
What scales	Number of jobs in flight	Requests per second
Idle cost	Zero	Warm capacity
You must predict load	No	Yes

Kubernetes is genuinely the right tool when you're running many long-lived services that need service discovery, rolling deploys, sophisticated networking, and a platform team to own it. If that's you, it earns its complexity — and "it's complicated" was never our objection. Our objection was that its core scaling primitive didn't match our core unit of work, and no amount of configuration fixes a mismatch at that level.

What we'd actually recommend is smaller than a tool choice: separate your interactive path from your heavy path before you do anything else. Almost all of our pain came from one machine serving both, so every scaling decision had to satisfy two workloads with nothing in common. Once we split them, the right answer for each became obvious, and it turned out to be a small fixed server for one and ephemeral jobs for the other. No cluster in sight.

The most popular solution is popular because it fits the most common problem. Check whether you have that problem first.