Answer Box
A shift from round-robin to least-connections load balancing with slow-start reduced 99th-percentile latency from 420 ms to 110 ms and cut error rates from 2.3% to near zero during traffic spikes, according to a real-world case study. The algorithm routes each new request to the backend with the fewest active connections, naturally absorbing bursts and preventing any single node from being overwhelmed.
The Problem with Round-Robin
A simple round-robin load balancer distributes requests sequentially across backends, ignoring each server's actual load. During a traffic spike, one node can become overloaded while others sit idle, causing 502 errors and frustrated users. The author initially considered adding more servers, but realized the real issue was distribution logic, not capacity.
Round-robin works fine under steady, homogeneous traffic, but fails when backends have different sizes, varying request costs, or occasional garbage collection pauses. Adding a new server causes an immediate shift in traffic patterns, potentially overwhelming a cold host.
How Least-Connections Works
The least-connections algorithm maintains only a counter of active connections per backend. When a new request arrives, the load balancer selects the backend with the lowest count. This creates natural fairness: a node handling long-running requests accumulates more connections and receives fewer new ones until it catches up.
To protect newly added servers, a slow-start penalty is applied for a configurable window (e.g., 30 seconds). During that period, the effective load is artificially inflated (by 1 million connections in the example) so the fresh node receives minimal traffic until it warms up. Static weights can also be factored in for heterogeneous instances.
Implementation Details
A minimal Go implementation requires about 60 lines of code. The `Next()` function iterates over backends, computing an effective load as `current connections + slow-start penalty`, optionally adjusted by weight. Two helper functions—`ConnStarted` and `ConnFinished`—update the connection counter atomically.
Key pitfalls include forgetting to decrement connections on error paths (use `defer` or `finally` to guarantee cleanup) and setting the slow-start penalty too low. A penalty of ~1 million connections effectively makes a new node invisible until the slow-start window expires.
Measurable Impact
In production, the switch transformed a fragile, over-provisioned cluster into a self-balancing system. During flash sales, 99th-percentile latency dropped from 420 ms to 110 ms, and the error rate fell from 2.3% to virtually zero. The algorithm works with any HTTP service, gRPC server, or TCP pool.
The author recommends testing the approach by replacing a round-robin proxy's selection logic with the least-connections snippet and running a load test with tools like `hey` or `wrk`. Observing how distribution changes when a backend is paused or added demonstrates the algorithm's effectiveness.