Implementing Circuit Breakers for Connection Acquisition Failures

This guide is part of Connection Acquisition Timeout Strategies. A timeout bounds how long one caller waits. It does nothing about the hundred callers behind them who will each also wait the full timeout, all queued for a pool that is not going to free up. A circuit breaker is what stops that queue from forming.

The failure this prevents has a distinctive shape. The database slows down; borrows start taking longer; the pool fills; new requests queue; each waits the full acquisition timeout before failing; request threads are consumed by waiting rather than by working; the service becomes unresponsive even for endpoints that never touch the database. The database recovering does not immediately fix it, because by then there is a backlog of queued callers and retrying clients. That is the cascade, and the acquisition timeout is what paces it rather than what prevents it.

Where the breaker belongs

The breaker must sit around the borrow, not around the query and not around the whole request. That placement is what makes it work:

request handler
  └── circuit breaker            <- here
        └── pool.getConnection()  <- the call that queues
              └── execute query

Around the query only, and you never see the acquisition failures — the breaker’s own call has already blocked in the pool. Around the whole request, and the breaker trips on every kind of failure, including ones a database has nothing to do with, which makes its state meaningless as a signal.

There is a second, more important reason for this placement. When the breaker is open it must fail without attempting to borrow. If it borrows and then fails, it has still occupied a connection slot and still added to the queue — the exact behaviour it exists to prevent.

Java with Resilience4j

Resilience4j’s CircuitBreaker wraps a supplier, which maps naturally onto the borrow:

CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .slidingWindowType(SlidingWindowType.TIME_BASED)
    .slidingWindowSize(30)                  // seconds of history
    .minimumNumberOfCalls(20)               // do not judge on a handful
    .failureRateThreshold(50)               // open at 50% failures
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .slowCallRateThreshold(60)              // slow borrows also count
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .permittedNumberOfCallsInHalfOpenState(3)
    .recordExceptions(SQLTransientConnectionException.class)
    .build();

CircuitBreaker breaker = CircuitBreaker.of("orders-pool", config);

public <T> T withConnection(SqlFunction<T> work) throws Exception {
    return breaker.executeCallable(() -> {
        try (Connection c = dataSource.getConnection()) {
            return work.apply(c);
        }
    });
}

Three parts of that configuration carry the design.

recordExceptions(SQLTransientConnectionException.class) is the important narrowing. That is the exception HikariCP throws when acquisition times out. Recording every SQLException would trip the breaker on a constraint violation, which is a bug in a query and has nothing to do with pool health — and opening the breaker would then hide the real error from everyone.

slowCallDurationThreshold matters as much as the failure rate. Acquisition failures are a lagging signal: borrows get slow long before they start timing out. Counting slow borrows as partial failures opens the breaker while the pool is degrading rather than after it has failed.

waitDurationInOpenState is the recovery pacing. Ten seconds is a reasonable default for a database: long enough for a saturated pool to drain, short enough that recovery is not needlessly delayed. Combined with permittedNumberOfCallsInHalfOpenState, it means recovery is probed by three requests rather than by the full arriving load — which is what prevents the breaker from re-opening immediately on the first attempt to close it.

Spring Boot wires the same thing declaratively:

resilience4j:
  circuitbreaker:
    instances:
      ordersPool:
        sliding-window-type: TIME_BASED
        sliding-window-size: 30
        minimum-number-of-calls: 20
        failure-rate-threshold: 50
        slow-call-duration-threshold: 2s
        slow-call-rate-threshold: 60
        wait-duration-in-open-state: 10s
        permitted-number-of-calls-in-half-open-state: 3
        record-exceptions:
          - java.sql.SQLTransientConnectionException

Go and Node

In Go the breaker wraps the operation that obtains a connection — in practice, the query call, since database/sql hides the borrow inside it. Use the context deadline to bound the wait and treat context.DeadlineExceeded as the failure signal:

func (r *Repo) query(ctx context.Context, q string, args ...any) (*sql.Rows, error) {
    return r.breaker.Execute(func() (*sql.Rows, error) {
        ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
        defer cancel()
        rows, err := r.db.QueryContext(ctx, q, args...)
        if errors.Is(err, context.DeadlineExceeded) {
            // could be pool wait or slow query; both justify opening
            return nil, err
        }
        return rows, err
    })
}

Go’s pool does not distinguish “waited for a connection” from “query was slow” in the error it returns, so the breaker treats them together. That is acceptable — both are reasons to shed load — but it means you should read db.Stats().WaitCount alongside the breaker state to know which one actually happened.

In Node with node-postgres, the borrow is explicit and the breaker can be placed precisely:

import CircuitBreaker from 'opossum';

const acquire = () => pool.connect();   // rejects after connectionTimeoutMillis

const breaker = new CircuitBreaker(acquire, {
  timeout: 3_000,             // must exceed connectionTimeoutMillis
  errorThresholdPercentage: 50,
  volumeThreshold: 20,
  resetTimeout: 10_000,
});

breaker.fallback(() => { throw new ServiceUnavailable('database busy'); });

export async function withClient(fn) {
  const client = await breaker.fire();
  try { return await fn(client); }
  finally { client.release(); }
}

The finally { client.release() } is not optional. A breaker that leaks connections on its own error paths creates exactly the exhaustion it was added to prevent.

Choosing thresholds

Time-based windows are the right choice over count-based for this application. A count-based window of 100 calls behaves completely differently at 10 requests per second than at 1,000, and the whole point is to react to a condition that develops in wall-clock time.

Set the slow-call threshold from measurement, not intuition. Take your normal p99 borrow duration from pool metrics and double it. If p99 acquisition is 5 ms, a 2-second threshold is enormously permissive; if it is 800 ms, your pool has other problems.

Common failure patterns and remediation

What to do when the breaker is open

Opening the breaker is not the whole design — deciding what happens next is. Three responses are worth having explicitly:

Serve stale. For read paths with a cache, return the cached value with a header or field noting staleness. This is the highest-value fallback and the reason to keep a cache in front of read-heavy endpoints even when the database is normally fast enough.

Shed with a retryable status. Return 503 with a Retry-After header so clients back off in a coordinated way rather than retrying immediately and adding to the load. A breaker that sheds into a client that retries instantly has moved the queue rather than removed it.

Degrade the response. Return the parts of a page or payload that do not require the database. A product page without live stock counts is worth more than an error.

Emit the state transitions as events and put them on the same dashboard as pool utilisation. A breaker opening is the highest-signal alert a pooled service can produce: it means saturation was detected and load was shed, which is both an incident and evidence that the protection worked. Wire it alongside the metrics in Detecting Connection Pool Saturation.

Operational boundary

A circuit breaker limits damage. It does not fix a pool that is too small, queries that are too slow, or connections that leak. If your breaker opens regularly, that is not a well-tuned breaker — it is an unaddressed capacity problem with a good alarm on it. Work back to the underlying cause using pool metrics and server-side state, and treat the breaker as the thing that bought you time to do so.

A breaker also cannot help with a slow database that never quite times out. If borrows take 1.9 seconds against a 2-second slow-call threshold, everything is technically succeeding and your latency is ruined. That is what the slow-call rate exists for, and it needs its threshold set from measured p99, not from a round number.

Frequently Asked Questions

Related

• Connection Acquisition Timeout Strategies — the parent guide on bounding the wait.
• Detecting Connection Pool Saturation — the metrics that should fire before the breaker does.
• Connection Pool Sizing Formulas — addressing the capacity problem the breaker only masks.
• Configuring the HikariCP Leak Detection Threshold — finding leaks that exhaust a correctly sized pool.
• Node.js Async Connection Limits — where the borrow is explicit and easiest to wrap.