Diagnosing Saturation Cascades Across Services

This guide is part of Detecting Connection Pool Saturation. A single service saturating its pool is a contained incident. A cascade is what happens when the response to that saturation — retries, timeouts that are longer than the caller’s patience, a shared database — pushes the pressure outward until several services are failing and none of them looks like the origin:

14:02  checkout-api    p99 2.4 s, pool active 16/16, waiting 40
14:03  orders-api      p99 3.1 s, pool active 24/24, waiting 61
14:04  orders-db       FATAL: sorry, too many clients already
14:05  billing-worker  ConnectionTimeoutError (shares orders-db)
14:07  checkout-api    circuit breaker open, 503s to the edge

By 14:07 four systems are unhealthy and the on-call engineer has four dashboards, each showing a full pool. The objective is to find the origin quickly, and then to install the bulkheads that keep the next occurrence contained.

Rapid incident diagnosis

Order the events by time, not by severity. The origin is the first thing that changed, which is rarely the loudest alert:

# per service, the first minute in which waiting rose above zero
min_over_time(timestamp(hikaricp_connections_pending > 0)[30m:1m])

Sorting that across services gives the sequence. The service whose waiters rose first, with no upstream cause, is the origin.

Separate shared-database cascades from retry cascades. They have different signatures and different fixes:

Check the retry multiplier. A caller that retries three times turns one failing request into four, and the callee sees the multiplication:

sum(rate(http_client_requests_seconds_count{service="checkout-api", peer="orders-api"}[1m]))
/ sum(rate(http_server_requests_seconds_count{service="checkout-api"}[1m]))

A ratio that rises from 1.0 to 3.5 during the incident is retry amplification, and it is usually the reason a small disturbance became an outage.

Check the database’s attribution. When services share a database, the connection count by application name says who took the budget:

SELECT application_name, count(*), count(*) FILTER (WHERE state='active') AS active
FROM pg_stat_activity GROUP BY 1 ORDER BY 2 DESC;

Look for the slow dependency. A pool full of connections whose sessions are idle means the service is holding connections while waiting for something else — the classic shape of a cascade arriving from downstream.

Mathematical sizing or parameter formula

Retry amplification. With r retries and a failure rate f, the load a callee sees is:

effective_load = base_load × (1 + r × f)

At 1,000 requests per second with three retries and a 40 % failure rate, the callee sees 1000 × (1 + 3 × 0.4) = 2,200 requests per second — while it is already struggling. Retry budgets cap this:

retry_budget = max_retries_per_second ÷ base_requests_per_second     (typically 0.1)
effective_load ≤ base_load × (1 + retry_budget)

A 10 % budget limits the amplification to 1.1× regardless of the failure rate.

Timeout ordering across hops. A caller must give up before its callee does, or its connections are held for work that is already lost:

timeout(caller) < timeout(callee) − network_margin

Inverted ordering is what makes a slow callee saturate its caller’s pool: the caller waits longer than the callee’s own deadline, holding a connection the whole time.

Shared-database bulkheads. Per-role connection limits partition the budget so one service cannot take another’s share:

Σ role_limits ≤ usable_connections
role_limit(service) = its claim, including surge

Time to cascade. The window between the origin saturating and a neighbour failing is roughly the neighbour’s remaining headroom divided by the rate at which the origin consumes the shared resource:

seconds_to_spread ≈ neighbour_headroom ÷ origin_consumption_rate

With 40 free connections and an origin opening 10 per second, a neighbour has four seconds — far too short for a human response, which is why the bulkheads have to be in place beforehand.

Exact remediation & configuration

Cap retries with a budget, not a count. A per-attempt count multiplies load; a budget bounds it:

// Resilience4j: at most 10 % of calls may be retries, with a circuit breaker in front
RetryConfig retry = RetryConfig.custom()
    .maxAttempts(2)                                   // one retry
    .waitDuration(Duration.ofMillis(100))
    .retryOnException(e -> e instanceof IOException)
    .build();

CircuitBreakerConfig breaker = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)
    .slowCallRateThreshold(50)
    .slowCallDurationThreshold(Duration.ofSeconds(1))
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .permittedNumberOfCallsInHalfOpenState(5)
    .build();

The circuit breaker is what stops retries entirely once the callee is clearly unhealthy, which is the difference between a bounded multiplier and none.

Order timeouts from the edge inward:

edge gateway        5 s
checkout-api → orders-api   2 s
orders-api → database        1.5 s (pool wait 0.5 s + statement 1 s)

Each hop must be able to give up before the one above it does.

Partition the shared database. Per-role limits turn a shared budget into private ones:

ALTER ROLE orders_app     CONNECTION LIMIT 150;
ALTER ROLE billing_worker CONNECTION LIMIT 40;
ALTER ROLE reporting      CONNECTION LIMIT 20;

A runaway reporting now exhausts 20 connections and fails alone.

Give heavy work its own pool and its own path. Reporting queries against a replica, with their own role and their own pool, remove the most common origin entirely:

app:
  datasource:
    reporting:
      jdbc-url: jdbc:postgresql://orders-replica.internal:5432/orders
      maximum-pool-size: 4
      connection-timeout: 10000

Shed load at the edge rather than queueing it. A bounded queue with a fast rejection keeps a saturated service from becoming a slow one:

server:
  tomcat:
    threads:
      max: 200
    accept-count: 100          # beyond this, connections are refused rather than queued

Make failures fast and typed. A pool timeout should produce a 503 with Retry-After, not a hung request — which is what lets the caller’s budget work.

Building bulkheads before the next cascade

Cascades are prevented by structure, and the structure is worth writing down.

Map the shared resources. For each database, list every service that connects, its role, its claim and its limit. For each service, list its callees and their timeouts. Most cascades run along an edge that nobody had drawn.

Give every shared database per-role limits. The totals should sum to the usable connections with headroom, exactly as in the budget method. Without them, the database’s capacity is allocated on a first-come basis during an incident — which favours whichever service is misbehaving.

Isolate by pool, then by replica, then by instance. Escalating separation costs more and isolates more:

Rehearse the cascade. A game day that saturates one service’s pool on purpose and watches what happens to its neighbours is the only way to know whether the bulkheads work. Run it in staging with production-like limits, and record the blast radius.

Alert on the origin’s signature, not only on the symptom. An alert on a single service’s waiting queue fires four times during a cascade; an alert on the database’s free connections fires once, at the moment that matters:

- alert: DatabaseConnectionBudgetLow
  expr: (pg_settings_max_connections - pg_stat_activity_count) < 25
  for: 2m
  labels: { severity: critical }
  annotations:
    summary: "{{ $labels.instance }}: fewer than 25 free connections — a cascade is likely"

Validation & verification

The origin is identifiable in under a minute. During the rehearsal, time how long it takes to name the first service to saturate using the dashboards as they are. If it takes longer, the timestamp query above belongs on a panel.

Retry budgets hold. Saturate a callee in staging and measure the caller’s request rate to it; the multiplier must stay within the budget rather than tripling.

Role limits contain the origin. With limits in place, saturate one service and confirm the others’ connection counts are unchanged and their error rates stay flat.

Timeouts are ordered. Log the effective timeout at each hop and confirm they decrease inward. A single inverted pair is enough to propagate a stall.

The database alert fires first. In the rehearsal, the free-connections alert should fire before the per-service saturation alerts, because it is the shared resource that binds.

Frequently Asked Questions

Related

• Detecting Connection Pool Saturation — the parent guide to saturation signals.
• Implementing Circuit Breakers for Connection Acquisition Failures — the breaker configuration referenced above.
• Propagating Request Deadlines into Connection Acquisition — making timeout ordering automatic.
• Calculating a Connection Budget Across Microservices — the per-role limits that partition a shared database.