Applying Little’s Law to Connection Pool Sizing

This guide is part of Connection Pool Sizing Formulas. Little’s Law is the only sizing tool that produces a number from measurement rather than from intuition, and it takes two values that most services already emit. It is also easy to apply wrongly in two specific ways, each of which changes the answer by roughly an order of magnitude — which is why services that “used the formula” so often end up badly sized anyway.

The failure it prevents looks like this: a pool sized at 50 that queues continuously at 300 requests per second, alongside a database sitting at 20% CPU. Nothing in the pool metrics explains it, because the pool is doing exactly what it was configured to do.

HikariPool-1 - Connection is not available, request timed out after 3000ms.
  (total=50, active=50, idle=0, waiting=61)
# meanwhile:
postgres=# SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
 count
-------
     4

Fifty connections checked out and four actually executing is the signature of a hold-time measurement that counted the wrong interval. This guide fixes that.

Rapid Incident Diagnosis

Three readings, in order, establish whether Little’s Law will help and what inputs to use.

  1. Pool checkout rate. Not request rate — the number of borrows per second, which is requests multiplied by queries per request. Most pools expose this as a counter; if not, the database’s own statement rate for this service is a close proxy.
  2. Hold time distribution. The interval between checkout and check-in. HikariCP publishes it as hikaricp.connections.usage; SQLAlchemy needs the checkout/checkin event pair; Go requires instrumenting around db.Conn(). This is the number that goes wrong.
  3. Backend state on the database. SELECT state, count(*) FROM pg_stat_activity WHERE application_name = 'your-service' GROUP BY state. If checked-out connections greatly exceed active backends, the connections are held rather than working, and no sizing formula applies until that is fixed.

The decision rule: if active backends roughly match checked-out connections, Little’s Law describes your system and the derivation below applies. If they diverge sharply, hold time is being spent outside the database and the correct action is to reduce it, not to compute a larger pool.

Hold time is not query duration A checked-out connection covers query execution plus result processing plus any application work inside the checkout window, so hold time can be many times query duration. One checkout window, 62 ms connection checked out — this is W query 6 ms result-set materialisation 16 ms object mapping and business logic 34 ms commit 6 ms using query duration: W = 12 ms at 300 borrows/s → a floor of 4 connections using hold time: W = 62 ms at 300 borrows/s → a floor of 19 connections A five-fold error, from measuring the wrong interval And the amber segment is also the thing to shorten — it is the only part holding a connection for no database reason
Query duration and hold time differ by everything the application does between checkout and release. Using the first as `W` understates the requirement by whatever that gap is.

The Formula and Its Two Inputs

L = λ × W

L  concurrent connections in use  (the floor for the pool)
λ  borrows per second             (not requests per second)
W  mean hold time in seconds      (not query duration)

Both notes in parentheses are where the derivation goes wrong.

λ is borrows, not requests. A request that issues four queries produces four borrows on a per-statement-checkout framework and one on a per-request one. Using request rate on a per-statement framework understates λ by the queries-per-request factor. The reliable source is the pool’s own checkout counter; the database’s statement rate for this service is a good second choice.

W is hold time, not query duration. The gap between them is everything the application does while holding the connection — result-set materialisation, object mapping, and any business logic inside the transaction. On a per-request-checkout framework that gap is most of the request. Measuring it requires instrumenting the checkout and check-in events rather than reading query latency from the database.

With both measured correctly, add headroom for variance. A 20–30% allowance covers ordinary jitter; anything more suggests the distribution is heavy-tailed enough that the acquisition timeout, rather than the pool size, should absorb the tail.

pool_floor = ceil(λ × W × 1.25)

Worked Example

A checkout service on a per-request-checkout framework, measured over a busy hour:

Measurement Value Source
Requests per second, fleet-wide 420 Load balancer
Queries per request, mean 3.2 Pool checkout counter ÷ request counter
λ — borrows per second 1,344 The product
Hold time, mean 11 ms Checkout-to-check-in interval
Hold time, p99 180 ms Same histogram
Query duration, mean 3 ms Database statement statistics

L = 1344 × 0.011 = 14.8 concurrent connections. With 25% headroom, an aggregate floor of 19.

Had query duration been used instead of hold time, the result would have been 1344 × 0.003 = 4.0, and with headroom, 5 — a pool that would queue continuously while appearing to follow the formula. That single substitution is the most common way this derivation fails.

The p99 of 180 ms is worth a moment. Sizing for it would give 1344 × 0.18 = 242 connections, which is absurd — it assumes every borrow simultaneously takes the worst-case time. The correct treatment is to size for the mean and let the acquisition timeout absorb the occasions when several slow borrows coincide, with a circuit breaker behind it for the case where they persist.

Converting the Floor to a Configuration Value

The 19 above is an aggregate figure across the whole fleet. What gets configured is per process:

per_process = ceil(aggregate_floor ÷ (max_replicas_at_surge × workers_per_replica))

With 8 replicas at maximum, 25% surge, and 2 workers each, the divisor is 10 × 2 = 20, giving a per-process floor of 1. That is uncomfortably small, and the discomfort is informative: twenty processes sharing an aggregate requirement of 19 connections means each has essentially no headroom for a local burst.

The options at that point are to reduce the process count — fewer, larger processes with more threads — or to accept that individual processes will queue briefly and rely on the acquisition timeout. What is not an option is configuring each process for the aggregate figure, which multiplies the requirement twentyfold.

From aggregate floor to configured value The aggregate connection floor is divided by the number of processes at maximum scale, and the result is what goes in the configuration file. aggregate floor 19 λ × W × 1.25 ÷ processes at maximum 10 replicas × 2 workers includes the rollout surge per-process floor 1 too small for comfort — that is the finding Configuring each process with the aggregate figure multiplies the requirement by the process count Nineteen per process across twenty processes is 380 backends for a workload that needs 19 — the most expensive arithmetic error available
The formula gives an aggregate. Putting that number in each process's configuration multiplies the requirement by the process count, which is how a correct derivation still produces a badly oversized fleet.

Validation and Verification

Three checks confirm the derived value is right, and all three are available within minutes of deploying it.

Pending waiters return to zero. The definitive check. If hikaricp.connections.pending, SQLAlchemy’s queue depth, or Go’s WaitCount rate stays at zero through a peak, the floor was adequate. Any sustained value means λ or W was underestimated.

Acquisition wait p99 stays well inside the timeout. A p99 above roughly 10% of the acquisition timeout means the pool is close enough to its limit that a modest traffic increase will start queueing.

Active backends approximately match checked-out connections. From pg_stat_activity. If checked-out connections substantially exceed active backends, hold time is being spent outside the database and the formula’s input was measured on a system that should be changed rather than sized around.

-- Run during peak: checked-out connections should be doing database work.
SELECT state, count(*)
FROM pg_stat_activity
WHERE application_name = 'checkout-service' AND backend_type = 'client backend'
GROUP BY state;
-- active: 14, idle: 5   → the derivation holds
-- active: 3, idle in transaction: 16   → hold time is not database time
Pending waiters before and after Before the change, pending waiters are non-zero through every peak. After the derived value is deployed they return to zero and stay there, which is the confirmation that the floor was adequate. 0 pending derived value deployed queueing through every peak zero, including at peak Pending waiters, not utilisation, is the metric that confirms the floor — a healthy pool at peak is fully utilised and has no waiters.
The confirmation is the pending-waiter series flattening at zero through a peak, which utilisation alone would never show.

Frequently Asked Questions

What if hold time is not instrumented and cannot easily be?
Use request duration as an upper bound on a per-request-checkout framework, and query duration as a lower bound elsewhere. Sizing between the two, closer to the upper bound, is safer than sizing at the lower one — an oversized pool wastes budget, an undersized one queues.
Does the law still hold when the system is saturated?
Little’s Law describes a stable system, which a saturated one is not — the queue grows without bound and the law’s assumptions break. That is why it gives a floor rather than a prediction: below λ × W the system cannot be stable, and above it stability depends on the database keeping up.
Should λ be peak or average?
Peak, measured over a window comparable to the hold time. Averaging over an hour smooths away the minute in which the queue actually formed, and it is that minute the pool has to survive.
How does this interact with the acquisition timeout?
The floor sizes for the mean; the timeout absorbs the variance. A pool at exactly the floor with a very short timeout will shed load during normal jitter — pair the derived floor with headroom and a timeout comfortably above the observed acquisition p99.
Why not just use the p99 hold time and be safe?
Because it assumes every concurrent borrow is simultaneously at the worst case, which is not how a distribution works. On a heavy-tailed hold-time distribution it produces a pool many times larger than the database can usefully serve, pushing the service past the knee described in Connection Pool Sizing Formulas.
Does the law apply to a proxy’s server pool as well?
Yes, and with the same inputs read one layer down. The proxy’s borrow rate and the duration a server connection is assigned give the proxy’s own floor, which is what default_pool_size should be derived from rather than guessed.
What changes when several services share one database?
Nothing about the derivation — each service computes its own floor independently. What changes is that the sum of those floors must fit inside the database’s capacity, which is the budget constraint rather than the demand one.