Autoscaling Services on Connection Pool Saturation Signals

This guide is part of Detecting Connection Pool Saturation. CPU-based autoscaling misses the most common way a database-backed service runs out of capacity: request threads parked waiting for a pooled connection use no CPU at all. So teams add pool wait as a scaling signal, and it works — until the day the database itself is the bottleneck. Then every new pod brings a new pool, the database receives more concurrent sessions than it can schedule, queries slow down, connections are held longer, pool wait rises further, and the autoscaler adds more pods. The scale-out that was meant to relieve saturation becomes the thing that turns a slowdown into an outage:

14:02  hpa/orders-api  New size: 12; reason: external metric pool_wait_p95(orders) above target
14:05  hpa/orders-api  New size: 18; reason: external metric pool_wait_p95(orders) above target
14:07  hpa/orders-api  New size: 24; reason: external metric pool_wait_p95(orders) above target
14:08  postgres        FATAL:  sorry, too many clients already

The objective is an autoscaling policy that adds replicas when pool saturation is caused by too few application instances, refuses to when it is caused by the database, and can never scale past the connection budget.

Rapid incident diagnosis

Establish which side is saturated. Pool wait means demand for connections exceeds supply in the pod. Whether more pods help depends on whether the database has spare capacity to serve the extra connections. Look at both sides for the same window:

# Application side: are callers waiting for connections?
histogram_quantile(0.95, sum by (le) (rate(hikaricp_connections_acquire_seconds_bucket{service="orders-api"}[2m])))

# Database side: how busy is PostgreSQL relative to its cores?
sum(pg_stat_activity_count{datname="orders", state="active"}) / 16

The second query uses the PostgreSQL exporter’s session counts divided by the database’s vCPU count.

  • Pool wait high, active sessions well below the core count, database CPU moderate: the application tier is the constraint. More pods — each with its own pool — will serve more requests.
  • Pool wait high, active sessions at or above the core count, database CPU high or lock waits rising: the database is the constraint. More pods add sessions that will queue inside PostgreSQL instead of in the pools.
  • Pool wait high, active sessions low, many sessions idle in transaction: connections are held, not used. Neither scaling nor pool size helps; the transaction scope must change.

Check whether the autoscaler amplified an incident. Replay the HPA events against database metrics:

kubectl get events -n orders --field-selector involvedObject.name=orders-api,reason=SuccessfulRescale \
  --sort-by=.lastTimestamp -o custom-columns=TIME:.lastTimestamp,MSG:.message

If replica increases coincide with rising database CPU and falling throughput per pod, the scaling signal did not distinguish the two causes.

Deciding whether pool wait should trigger a scale-out High pool wait with database headroom means the application tier is short of instances and scaling out helps. High pool wait with the database at its core count means the database is the constraint and scaling out adds queueing. High pool wait with idle in transaction sessions means connections are held and scaling does not help. Pool wait p95 above target Active sessions below cores DB CPU moderate Active sessions at cores DB CPU high, lock waits Many idle in transaction connections held Scale out pods add capacity Do not scale fix queries or DB size Do not scale fix transaction scope
Pool wait is a symptom with two causes. Scaling out treats one of them and worsens the other, so the policy must look at the database before adding pods.

Mathematical sizing or parameter formula

Hard ceiling from the budget. No policy should be able to exceed the database connection budget, whatever the signal says:

maxReplicas = floor( service_connection_budget ÷ (pool_max × processes_per_pod) )  −  surge_pods

With a budget of 300 connections for this service, a pool of 12 and one process per pod, floor(300 ÷ 12) = 25; subtract 2 for rolling-update surge, giving maxReplicas: 23.

Scale-out target from pool demand. When the application tier is the constraint, the replicas needed follow from the average number of goroutines or threads waiting plus those holding connections:

desired_replicas = ceil( current_replicas × (in_use + waiting) ÷ (pool_max × target_utilisation) )

At 10 replicas, with an average of 12 connections in use and 5 callers waiting per pod, a pool of 12 and a 75 % target: ceil(10 × 17 ÷ 9) = 19. The HPA computes the same proportion when given in_use + waiting per pod as an AverageValue metric with a target of 9.

Database headroom gate. Allow scale-out only while the database has room to run more concurrent work:

headroom = db_vcpus × concurrency_factor − active_sessions
scale_out_allowed = headroom > pool_max × replicas_to_add × expected_active_fraction

With 16 vCPUs, a concurrency factor of 2, 26 active sessions, and 9 new replicas whose connections are 30 % active on average: headroom is 32 − 26 = 6, and the new pods would add about 12 × 9 × 0.3 = 32 active sessions. Scale-out is not allowed; the database is near its limit, and the right response is on the database side.

Replica count against the connection budget cap Replicas requested by pool wait rise from 10 to 30, but the connection budget caps replicas at 23. The headroom gate blocks scale-out beyond 12 replicas while the database is at its concurrency limit. requested by pool wait alone allowed with cap and headroom gate maxReplicas 23 from connection budget gate holds at 12: database at its core limit 14:00 14:10
Replicas are capped by the connection budget regardless of the scaling signal. Within the cap, the headroom gate decides whether the signal is allowed to act.

Exact remediation & configuration

Export a per-pod demand metric. For HikariCP with Micrometer, hikaricp_connections_active and hikaricp_connections_pending are already per pod. Record their sum as the scaling metric, and a database headroom gauge next to it:

groups:
  - name: autoscaling-inputs
    rules:
      - record: orders:pool_demand:per_pod
        expr: hikaricp_connections_active{service="orders-api"} + hikaricp_connections_pending{service="orders-api"}
      - record: orders:db_headroom:sessions
        expr: 16 * 2 - sum(pg_stat_activity_count{datname="orders", state="active"})

Gate the scaling metric on headroom. Expose a single query to the autoscaler that returns the demand when headroom exists and the current utilisation target when it does not, which makes the HPA hold its size:

avg(orders:pool_demand:per_pod)
  and on() (orders:db_headroom:sessions > 8)
or
  (9 * count(orders:pool_demand:per_pod) / count(orders:pool_demand:per_pod))

When headroom falls to 8 active sessions or fewer, the query returns exactly the target value of 9, and the HPA’s desired replica count equals the current count.

KEDA ScaledObject with a budget cap and conservative behaviour.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-api
  namespace: orders
spec:
  scaleTargetRef:
    name: orders-api
  minReplicaCount: 6
  maxReplicaCount: 23                # floor(300 ÷ 12) − 2 surge
  cooldownPeriod: 300
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 60
          policies:
            - { type: Pods, value: 3, periodSeconds: 60 }   # at most 3 pods (36 connections) per minute
        scaleDown:
          stabilizationWindowSeconds: 600
          policies:
            - { type: Percent, value: 10, periodSeconds: 120 }
  triggers:
    - type: prometheus
      metricType: AverageValue
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: |
          avg(orders:pool_demand:per_pod) and on() (orders:db_headroom:sessions > 8)
          or (9 * count(orders:pool_demand:per_pod) / count(orders:pool_demand:per_pod))
        threshold: "9"
    - type: cpu
      metricType: Utilization
      metadata:
        value: "70"

The scale-up policy limits how fast connections are added — three pods per minute is 36 connections — so the database has time to absorb them and the headroom gauge has time to reflect the change before the next step. The CPU trigger remains for CPU-bound load; the HPA uses whichever trigger asks for more replicas, subject to the same cap.

Page on the gate, not only on the scaler. When the headroom gate is holding scale-out while pool wait stays high, the database is the constraint and a human should know:

- alert: OrdersScaleOutBlockedByDatabase
  expr: avg(orders:pool_demand:per_pod) > 9 and on() orders:db_headroom:sessions <= 8
  for: 5m
  labels: { severity: critical }
  annotations:
    summary: "orders-api pools saturated and database has no headroom; autoscaling is holding"
Autoscaling pipeline with a database headroom gate Pod pool metrics and PostgreSQL session metrics flow into Prometheus recording rules. A gated query returns pool demand only while database headroom exists. KEDA reads it and drives the HPA, which scales the deployment within the maxReplicas cap. Pods active + pending PostgreSQL active sessions Recording rules demand per pod DB headroom Gated query demand only if headroom > 8 KEDA + HPA max 23 3 pods per min
The gate sits between the metric and the autoscaler. KEDA and the HPA see a value that only exceeds the target while the database can take more sessions.

Validation & verification

Application-tier saturation scales out. In staging, give the database generous capacity and load the service until pools wait. Replicas must increase in steps of at most three per minute, pool wait must fall, and database active sessions must stay below the core-based limit.

Database saturation holds. Repeat the test with the database deliberately constrained — a smaller instance, or a CPU-heavy query mix. When active sessions reach the limit, the replica count must stop increasing, the OrdersScaleOutBlockedByDatabase alert must fire, and too many clients must never appear in the database log.

Budget cap holds under any signal. Temporarily set the trigger threshold very low so the autoscaler always wants more replicas; the deployment must stop at maxReplicaCount, and the connection count from the service must stay within the budget:

SELECT count(*) FROM pg_stat_activity WHERE application_name = 'orders-api';

It must not exceed 23 × 12 + surge, and the free-slot margin on the server must remain positive.

No oscillation. Over an hour of variable load, the HPA’s replica history should not alternate up and down within the stabilisation windows. Frequent reversals mean the scale-down window is too short for how long new pods take to affect pool demand.

Vertical scaling and pool resizing as alternatives

Horizontal autoscaling is one response to pool saturation. Two others are sometimes better, and a policy that considers all three avoids reaching for pods when the problem is elsewhere.

Resize the pool within the pod. If each pod has spare CPU and memory while its pool is saturated, a larger pool uses capacity that already exists. This is common for I/O-heavy services where request threads mostly wait on the database. A larger pool per pod and fewer pods consume the same database budget as a smaller pool and more pods, but save the per-pod overhead and reduce cross-pod imbalance. The limitation is that pool size is usually fixed at startup, so resizing means a rollout.

Scale the database. When the headroom gate blocks scale-out, the database is the constraint. For read-heavy saturation, adding read replicas and routing reads to them raises capacity without touching the writer’s connection budget. For write-heavy saturation, a larger instance class raises both CPU and — on managed services where it derives from memory — max_connections. Neither is instant, which is why the gate’s alert should reach a human.

Reduce hold time. The fastest capacity gain is often making each request hold its connection for less time. Pool demand is arrival rate times hold time; halving hold time halves demand at any traffic level. Traces and use_time histograms show where connections are held longest.

Situation Best response Autoscaler role
Pool saturated, pod CPU high, DB headroom Add pods Scale out
Pool saturated, pod CPU low, DB headroom Larger pool per pod Hold; change configuration
Pool saturated, DB at core limit, read-heavy Add replicas, route reads Hold
Pool saturated, DB at core limit, write-heavy Larger instance or query work Hold
Pool saturated, connections held idle Shorten transactions Hold

The autoscaler should own only the first row. Encoding the others as alerts with clear runbooks keeps the automated response narrow and predictable, and makes each human decision explicit.

Frequently Asked Questions

Why not scale on pool utilisation instead of demand?
Utilisation caps at 100 %, so it cannot express how far demand exceeds supply; a pool at 100 % with one waiter and one with fifty look identical. Active plus pending connections keeps growing with demand, which gives the HPA a proportional signal.
Should the connection pool grow instead of adding pods?
Only if the pod has spare CPU and the database has headroom. A larger pool in fewer pods and more pods with smaller pools consume the same database budget; the choice depends on where CPU and memory are cheaper and how quickly each can change. Pods can be added without restarts; pool sizes usually cannot.
Does a proxy change this?
With PgBouncer in transaction mode, the database sees a fixed backend pool regardless of replica count, so the budget cap relaxes. Saturation moves into PgBouncer’s client queue, and the headroom gate should use PgBouncer’s waiting-client metric alongside database CPU. See PgBouncer Metrics Monitoring.
How does this relate to alerting on saturation?
The alerts in Alerting on Connection Pool Saturation tell humans; the gated metric here tells the autoscaler. Keep them consistent — the same demand and headroom definitions — so the page and the scaling decision never disagree about what is happening.
Can the Vertical Pod Autoscaler help?
VPA adjusts CPU and memory requests, which helps when pods are CPU-bound, but it does not change pool size and restarts pods to apply changes. It is complementary to the approach here, not a replacement for the connection budget cap.