We Deleted Autoscaling & Finally Scaled

Our platform runs chemistry. A researcher describes a molecule they want to make. We then find the synthesis routes, read the patent landscape, search the literature, design the experiment and lay out a multi-stage project to get there. Under the hood, that means a deep research run, a literature phase and a project architect pass. Each one is an LLM-driven pipeline that reads hundreds of papers, embeds them, reasons over them and writes a structured result.

A single one of those tasks takes ten to twenty minutes and needs at least 16 GB of RAM.

That sentence is the whole story of our infrastructure. Everything we got wrong, we got wrong because we didn't take it seriously enough.

Act I: One big server

We started where everyone starts: one EC2 instance running the API, a Celery worker for background work and Redis as the broker. It was simple. We could reason about it. For a long time it was the right call.

Then researchers started using it the way researchers use things, which means all at once and without warning. A few concurrent deep research runs plus a couple of literature searches were enough to take the box down. It didn't get slow. It got stuck. The embedding work is CPU-bound, the pipelines are memory-hungry and Celery's pre-fork workers will happily accept more tasks than the machine can survive. Meanwhile the API, where someone is actively clicking and waiting, was stuck behind the same starved CPU.

The obvious diagnosis was "we need more servers." That diagnosis cost us two architectures.

Act II: Autoscaling and the five-minute problem

So we did the textbook thing. We added a load balancer and an autoscaling group so we could scale out under pressure.

It didn't work. Why it didn't work turned out to be the most useful thing we learned.

Autoscaling is a control loop. Control loops need time. Something has to notice that load is high, decide to add capacity, provision an instance, boot it, start the app, warm it up and register it as healthy. For us that round trip took five to ten minutes.

Now look at the shape of our load. It isn't a ramp. Nobody's traffic gently climbs toward lunchtime. One researcher kicks off a multi-stage project. Within seconds we owe the system four heavy tasks. Two colleagues do the same thing that afternoon. Load goes from almost nothing to fully saturated in the time it takes to click a button. It stays there for ten minutes, then disappears.

Against that shape, a five-minute reaction time is useless. By the time the new instance was serving traffic, the burst that triggered it was half over. The existing box had already slowed down in the meantime. We had built a system that reliably added capacity right after it stopped being needed, while the user stared at a loading spinner.

It got worse. Scaling the whole application to absorb heavy background work meant paying for API capacity we didn't need just to get the worker capacity we did. The two resources had nothing to do with each other, yet we had bolted them together.

Act III: Containers, then Kubernetes

We assumed the problem was our crude autoscaling, so we moved up the ladder. Containers came first: Docker, a registry, ECS. That was real progress and we kept it. A single image that can boot into any role is good engineering. We still rely on it today.

Then we trialled Kubernetes, because that's what you do.

We hit exactly the same wall. It was just wearing nicer clothes.

Something that doesn't get said clearly enough is that Kubernetes' Horizontal Pod Autoscaler scales replicas of a long-lived service based on aggregate metrics. That's a great abstraction when your workload is a stream of small, uniform, interchangeable requests and average CPU across the fleet actually means something. Add pods, spread the load, remove pods.

Our workload is none of those things. Our unit of work is:

• Expensive: 2 vCPU and 16 GB for one task
• Long: minutes, not milliseconds
• Indivisible: a second pod won't make one deep research run 30% faster, because it's one pipeline with one sequential reasoning chain
• Bursty and random: no ramp, no pattern, no warning

Average CPU across a fleet tells you very little when a single unit of work can saturate a node by itself. A pod also still has to land on a node that exists. That put us right back to users waiting for capacity, or paying to keep warm nodes idle for requests that might never come.

At every level of sophistication we tried, autoscaling asked us for a strategy. It wanted a prediction about future load, expressed as thresholds, cooldowns and buffers. We didn't have a prediction. We had researchers.

The reframe

The pivot that mattered most wasn't a new tool. It was a question:

Are we scaling requests, or are we scaling jobs?

Every tool we had reached for was built to scale requests. More concurrent HTTP traffic means more copies of the server behind a balancer. We had been trying to say "I need one 16 GB machine for ten minutes, right now, then never again" in a vocabulary that only knows how to say "run N copies of my service."

We didn't have a traffic problem. Our API traffic was tiny: a few hundred CRUD calls and some polling. A modest instance could handle that for years.

We had a job problem. There's a whole class of infrastructure built for exactly that. The web-app world mostly ignores it because it grew up in scientific computing and data pipelines.

Act IV: AWS Batch on Fargate

The final architecture is almost boring. I mean that as a compliment.

One modest EC2 instance runs the API and a warm Celery worker. It handles everything cheap and interactive: CRUD, auth, lightweight background tasks, quick requests and scheduled watchdogs. It never does heavy lifting anymore, so it never gets starved, so the UI stays responsive. This box doesn't autoscale. It doesn't need to.

AWS Batch on Fargate runs everything heavy. When a user starts a deep research run, we don't hand it to a worker. We submit a job. Batch launches a container with exactly 2 vCPU and 16 GB dedicated to that one task. The task runs, the container exits and we pay for the minutes it used.

Two bits of jargon are worth explaining, since the whole design depends on them.

• Fargate is serverless containers. You ask for a container with a certain amount of CPU and memory, then AWS finds room for it somewhere. There's no cluster, no node pool, no capacity to provision, nothing to keep warm and nothing to patch.
• AWS Batch is a job scheduler on top of that. You define a job once, then submit runs of it. Its basic unit is the job: one submission, one run, with its own lifecycle, retry policy and terminate call.

That last point is really the argument of this whole post. Batch's basic unit is a job. Our unit of work is a job. The friction went away, not because Batch is clever, but because we stopped translating between two different models.

There's no autoscaling strategy because there's no fleet to size. Fifty tasks arriving at once means fifty containers. Batch doesn't care whether we submit one job or fifty. Burstiness can't make it lag, because nothing is shared. Resource isolation is complete, so one task's compute can't touch the API or another user's run. It uses the same container image as the EC2 worker, just booted into a different role, so we maintain one codebase instead of two.

We've verified more than 50 concurrent heavy runs. Going past 100 is comfortable. The number on its own isn't the point, though. What matters is that a fraction of this load used to hang a dedicated EC2 instance or leave users waiting ten minutes for a scale-up.

The constraint that made it work

Fargate containers aren't instant. Cold start takes about two minutes.

For most web architectures, two minutes is a real problem. I expect this is where some readers close the tab. So let me be precise about why it's fine for us: our users don't wait.

Our users understand that multi-stage project creation, experiment design and deep research take a while. A researcher starts one the way you'd start a load of laundry. They kick it off, go do something else, then come back later to a finished result and a notification. Nobody is staring at a loading screen. Next to a ten-minute task the user has already walked away from, two minutes of startup is invisible.

The whole design rests on this condition, so check it before copying us. If your heavy work has a latency budget measured in seconds, none of this applies to you. We didn't get here by finding better infrastructure. We got here because our UX was honest about being asynchronous, which gave us the freedom to choose it.

We did keep one escape hatch. A couple of background tasks belong to the heavy family but finish in well under a minute. For those, two minutes of container startup costs more than the work itself, so they stay on the warm Celery worker. The routing rule is "what kind of task is this," not "which server should we use right now." That's a healthier way to think about it anyway.

The parts nobody blogs about

Elastic compute was the easy half. Two unglamorous things did more for reliability than Batch did.

Admission control came first. Before any of this, we capped concurrency per user: one running heavy task per category, a small queue behind it and a clear error when you go over. This is about bounding, not scaling. It's the reason elastic compute didn't turn into an elastic bill. Unlimited capacity with no admission control is just a billing incident waiting for a trigger.

Ephemeral compute also needs a reaper. A traditional Celery worker acknowledges tasks late. If it dies mid-task, the broker hands that task to someone else. You get that safety net for free and eventually forget it exists. Batch has nothing like it. If a container dies, nothing anywhere knows it was supposed to be doing something. The task's record sits there marked "running" forever, holding that user's concurrency slot hostage.

So we built heartbeats. Running tasks update a timestamp. A scheduled sweep fails anything that has gone quiet for five minutes, which frees up the slot. Moving from a broker-backed worker to ephemeral jobs means rebuilding the durability guarantees the broker was quietly giving you. That recovery code is the least-exercised code you own, because it only runs when something else has already gone wrong. Give it its own tests and its own alerts.

So should you do this?

Not necessarily. The point isn't that Batch beats Kubernetes. They answer different questions, but the industry has gotten into the habit of reaching for the same answer no matter what the question is.

This approach fits if your heavy work looks like ours:

	Job runner (Batch/Fargate)	Autoscaled cluster (K8s/ECS)
Unit of work	Long, expensive, indivisible	Short, cheap, uniform
Load shape	Bursty, random, unpredictable	Steady or predictably cyclical
Latency budget	Minutes, users work async	Milliseconds, users are waiting
What scales	Number of jobs in flight	Requests per second
Idle cost	Zero	Warm capacity
You must predict load	No	Yes

Kubernetes is the right tool when you run many long-lived services that need service discovery, rolling deploys, sophisticated networking and a platform team to own it all. If that's you, it earns its complexity. Complexity was never our objection anyway. Our objection was that its core scaling primitive didn't match our core unit of work. No amount of configuration fixes a mismatch at that level.

What we'd actually recommend is smaller than a tool choice: separate your interactive path from your heavy path before you do anything else. Almost all of our pain came from one machine serving both, which meant every scaling decision had to satisfy two workloads with nothing in common. Once we split them, the right answer for each became obvious. It turned out to be a small fixed server for one and ephemeral jobs for the other, with no cluster in sight.

The most popular solution is popular because it fits the most common problem. Make sure you actually have that problem first.