Scaling a Node.js Backend from 10K to 1M Users

Most teams optimise the wrong layer.

Something gets slow, someone profiles the application code, a few functions get tightened, and the graph barely moves. So they buy a bigger instance. That works briefly and then stops working, because the bottleneck was never CPU — it was a missing index, or a connection pool, or an unbounded queue quietly filling.

I have taken a Node.js backend to 600,000 concurrent users, and cut another team's cloud bill by 40% without touching a single feature. Neither involved clever code. Both involved finding the layer that was actually saturated.

This is what breaks, in the order it breaks.

The layers, and why order matters

A request passes through roughly five layers, and each has a different failure signature:

1. The event loop — Node's single thread
2. The database — connections, indexes, query shape
3. Caching — what you avoid doing at all
4. External calls — everything you do not control
5. The process/instance boundary — how many of you there are

The critical insight is that fixing a layer that is not your bottleneck produces no improvement at all — not a small one, none. That is why the "we optimised and nothing changed" experience is so common. Diagnose before you fix.

Stage 1: up to ~10K users — the event loop

At this scale you are almost certainly fine, with two exceptions worth checking because they are cheap now and painful later.

Blocking the event loop. Node handles concurrency on one thread. Any synchronous work — a big JSON parse, crypto, image manipulation, a tight loop over a large array — stops everything. Not slows: stops. One 200ms synchronous operation under 100 concurrent requests is a 20-second tail.

Symptom: latency that degrades across all endpoints simultaneously, including trivial ones. If your health check gets slow when a report runs, this is you.

Memory leaks that only appear over days. The classic is a setInterval that is never cleared, holding a closure over something large. It leaks quietly, looks fine in testing, and takes the process down four days later — which is why it gets misdiagnosed as "the server needed a restart." The full pattern is in Avoiding Memory Leaks in Node.js: Why setInterval Can Be Dangerous.

The broader set of practices at this stage is in Node.js Performance Optimization: Best Practices.

Stage 2: 10K–100K — the database, always

This is where nearly everyone's first real wall is, and it is almost never the application.

Missing indexes. A query fine at 10,000 rows is catastrophic at 5 million. Nothing changed in your code; the data grew. Find them by logging slow queries — not by guessing.

N+1 queries. Fetch 50 records, then issue one query per record for a related field. 51 round trips where 2 would do. ORMs make this effortless to write and invisible to notice.

Connection pool exhaustion. The subtle one. Your pool holds, say, 10 connections. A slow query holds one for 2 seconds. At 50 requests/second you are out of connections and every request queues — including the fast ones. The symptom looks like "the whole API is slow" and the cause is one slow endpoint.

This is precisely the failure the shock-absorber pattern prevents: Handling High Concurrency in Node.js.

Fix order at this stage: indexes first (hours, enormous payoff), then N+1 (days), then pooling and timeouts. Do not add caching yet — caching an unindexed query hides the problem and you will meet it again with a cold cache, at the worst possible moment.

Stage 3: 100K–1M — caching, properly

Now caching earns its place. The mistake is treating it as one thing. It is four, and they are not interchangeable:

1. In-process — nanoseconds, per-instance, small and hot. Config, feature flags, hot lookups. Careful: it multiplies by instance count and is a common hidden memory leak.
2. Distributed (Redis) — sub-millisecond, shared across instances. Sessions, computed results, rate-limit counters.
3. Database-level — materialised views, denormalised counters. Best for expensive aggregations that tolerate slight staleness.
4. CDN/edge — anything public and identical for many users. Cheapest possible request: the one that never reaches you.

The discipline is deciding, per data type, how stale is acceptable — and being honest that the answer is rarely "zero". Most teams cache too little because they have not had that conversation with the product side. This is the strategy that carried 600K concurrent users.

Invalidation is the hard half. Write it down before you build: what event invalidates what key. Undocumented invalidation is how you serve a customer someone else's data — the worst bug in this entire article.

Stage 4: everything you do not control

At scale, your reliability is largely a function of your dependencies. Every external call needs four things, and most codebases have none of them:

• A timeout. No exceptions. A call without a timeout is an unbounded resource hold, and defaults are usually far too generous.
• A retry policy with backoff and a cap. Naive retries turn a degraded dependency into an outage — you add load to something already struggling.
• A circuit breaker. After N failures, stop calling and fail fast. Recover gracefully rather than queueing requests against a dead service.
• A fallback. Cached value, degraded response, queued-for-later. Decide what the product does when the dependency is gone.

Anything slow belongs on a queue rather than in the request path — the same conclusion I reach from a different direction in Stop Polling, Start Reacting.

The cost dimension

Scaling and cost are the same conversation. The 40% reduction I wrote about in Is Your Code Burning Money? came from no feature changes at all — it came from the same diagnosis discipline:

• Instances sized for a peak that happens twice a year
• Queries scanning far more data than they return
• Cache hit rates nobody had ever looked at
• Logs at debug level in production, at volume
• Data transfer between zones that did not need to cross zones

The pattern: teams scale up to hide inefficiency, then pay for that inefficiency monthly, forever. Find the bottleneck and the bill usually falls as a side effect.

A note on security under load

Two things degrade badly at scale. Auth token handling — where you store a JWT becomes a real question when you are issuing millions (JWT Token: Where to Store JWT in Browser?). And rate limiting, which is a capacity control, not just an abuse control. An unauthenticated endpoint with no limit is a free denial-of-service against yourself.

The diagnosis checklist

When something is slow, in this order:

1. Is it all endpoints or one? All → event loop or connection pool. One → that endpoint's query.
2. Is it constant or under load? Constant → bad query or missing index. Under load → pooling, back-pressure, or a saturated dependency.
3. Does it get worse over hours/days without deploys? → leak.
4. Is CPU high? Usually not the problem in Node. Low CPU with high latency means you are waiting on something — that is where to look.
5. What is your cache hit rate? If you cannot answer, that is the next thing to instrument.

Most teams skip straight to step 5's fix without doing steps 1–4, which is exactly how you end up optimising a layer that was never the bottleneck.

The short version

• Diagnose the layer before optimising anything.
• Under 10K users: watch the event loop and long-lived timers.
• 10K–100K: it is the database. Indexes, N+1, connection pools — in that order.
• 100K–1M: four distinct caching layers; invalidation is the hard half.
• Always: timeout, retry cap, circuit breaker, fallback on every external call.
• Cost usually falls out of the same work.

None of this is exotic. It is unglamorous, sequential, and it works — and it is almost always cheaper than the bigger instance you were about to buy.