Measuring Connection Acquisition Latency Percentiles

This guide is part of Java Connection Pool Benchmarks. It covers how to instrument and interpret the time a thread spends waiting to borrow a connection from the pool — the single most actionable latency signal for a saturated or undersized pool. The objective is precise: capture p50, p95, and p99 of getConnection() wait time, distinguish a healthy pool (sub-millisecond acquisition) from a saturated one (p99 climbing toward the connection timeout), and avoid the classic mistake of averaging away the tail that actually breaks request SLAs.

Connection acquisition latency is invisible to most application dashboards because it hides inside the data-access layer. A request that takes 800ms looks like a slow query, but the query may have run in 4ms after the thread waited 780ms for a free connection. When that happens you will eventually see HikariCP log Connection is not available, request timed out after 30000ms — the terminal symptom of acquisition latency exceeding connectionTimeout. Long before that hard failure, rising p99 acquisition latency is the leading indicator of pool exhaustion. This guide instruments it with Micrometer, reads the hikaricp_connections_acquire_seconds histogram, and separates JMH microbenchmark numbers from production telemetry.

Rapid incident diagnosis

When request latency rises but query execution time is flat, suspect acquisition latency. Triage in this order.

First, check whether the pool is exhausted. HikariCP exposes hikaricp_connections_pending (threads currently blocked in getConnection()) and hikaricp_connections_active (connections in use). When pending > 0 sustained and active == maximumPoolSize, threads are queuing for connections and acquisition latency is the bottleneck — not the database.

Second, read the acquire-time percentiles directly rather than the mean. A mean of 2ms can hide a p99 of 400ms when 1% of borrows block behind a full pool. The tail is what violates SLAs, so always pull p95/p99 from the histogram, never the average.

Third, distinguish acquisition latency from connection creation latency. hikaricp_connections_creation_seconds measures how long opening a brand-new physical connection takes (TCP + TLS + auth). A slow creation time points at the network or database, not pool sizing; a slow acquire time with fast creation points at an undersized pool or a connection leak.

The wider context of how pools are benchmarked across implementations is covered in the parent topic, Java Connection Pool Benchmarks, and exporting these metrics to a dashboard is detailed in Exposing HikariCP Metrics with Micrometer and Prometheus.

Capturing percentiles with Micrometer

HikariCP publishes a Micrometer Timer named hikaricp.connections.acquire (exported to Prometheus as hikaricp_connections_acquire_seconds). To get true percentiles you must enable client-side distribution statistics — by default Micrometer exports only count and sum, from which p99 cannot be derived.

// Enable histogram buckets + client-side percentiles for the acquire timer
@Bean
MeterFilter hikariAcquireHistogram() {
    return new MeterFilter() {
        @Override
        public DistributionStatisticConfig configure(Meter.Id id,
                DistributionStatisticConfig config) {
            if (id.getName().startsWith("hikaricp.connections.acquire")) {
                return DistributionStatisticConfig.builder()
                    .percentilesHistogram(true)                 // export buckets for server-side quantiles
                    .percentiles(0.5, 0.95, 0.99)               // client-side pre-computed quantiles
                    .serviceLevelObjectives(                    // explicit SLO boundaries
                        Duration.ofMillis(1).toNanos(),
                        Duration.ofMillis(10).toNanos(),
                        Duration.ofMillis(100).toNanos())
                    .build()
                    .merge(config);
            }
            return config;
        }
    };
}

percentilesHistogram(true) emits cumulative histogram buckets so Prometheus can compute quantiles across instances with histogram_quantile(). percentiles(0.5, 0.95, 0.99) adds per-instance pre-computed values, useful for a single JVM but not aggregatable across pods — never average pre-computed percentiles from multiple instances. For a fleet, rely on the histogram buckets and aggregate server-side.

Bind the HikariCP datasource to the registry so the metrics are published:

HikariDataSource ds = new HikariDataSource(config);
ds.setMetricRegistry(meterRegistry);   // Micrometer-backed registry

In Prometheus, the p99 across the whole fleet over a 5-minute window is:

histogram_quantile(0.99,
  sum(rate(hikaricp_connections_acquire_seconds_bucket[5m])) by (le, pool))

Mathematical sizing and the timeout relationship

Acquisition latency is governed by queuing theory. When borrow demand exceeds the number of connections, threads queue, and wait time follows the pool’s utilization. A useful operational bound comes from Little’s Law applied to the pool: the average number of in-use connections equals the borrow arrival rate multiplied by the average hold time.

active_connections = borrow_rate (req/s) × hold_time (s)

Worked example: a service handles 2,000 req/s, each request holds a connection for 5ms (query + result processing). Required concurrent connections:

2000 req/s × 0.005 s = 10 connections in use on average

Average use is 10, but tail behavior demands headroom — set maximumPoolSize above the average to absorb bursts and variance in hold time. If you size at exactly 10, any burst pushes borrows into the wait queue and p99 acquisition latency spikes immediately. A pool sized at 15–20 keeps p99 acquisition near zero for this load. The HikariCP guidance of a small, fixed pool holds: a right-sized pool of 20 beats an oversized pool of 100, because oversizing pushes contention to the database instead of the pool. See Optimizing HikariCP maximumPoolSize for High Concurrency for the full sizing derivation.

The hard ceiling on acquisition latency is connectionTimeout. Any borrow that cannot be satisfied within that window fails with the timeout exception. So the practical target is: keep p99 acquisition latency at least an order of magnitude below connectionTimeout (e.g. p99 < 50ms when connectionTimeout = 30000ms). When p99 approaches the timeout, exhaustion failures are imminent.

Exact remediation and configuration

When p99 acquisition latency is high, the fix is one of: increase maximumPoolSize (if the database can absorb more connections), reduce hold time (shorten transactions, move work outside the connection scope), or shed load. Apply pool sizing changes carefully — oversizing moves the bottleneck to the database.

# application.yml — HikariCP sized for the worked example (2000 req/s, 5ms hold)
spring:
  datasource:
    hikari:
      maximum-pool-size: 20          # > average in-use (10) for burst headroom
      minimum-idle: 20               # keep pool full; avoid cold-acquire latency
      connection-timeout: 10000      # fail fast; tighten from default 30000
      max-lifetime: 1800000          # recycle below DB/proxy idle cutoff
      metric-registry-enabled: true  # ensure Micrometer timers are published

minimum-idle set equal to maximum-pool-size keeps connections warm, eliminating the creation-latency spike that otherwise shows up in the acquire tail when the pool grows on demand. Lowering connection-timeout to 10s makes saturation fail fast and visible rather than silently inflating request latency. Apply changes via rolling restart; HikariCP does not support live resizing of maximumPoolSize without recreating the datasource.

Validation and verification

After tuning, confirm the tail has flattened and the pool is no longer queuing.

Query the live percentiles and pending count from the Prometheus endpoint or JMX:

# p50, p95, p99 acquisition latency, fleet-wide
histogram_quantile(0.50, sum(rate(hikaricp_connections_acquire_seconds_bucket[5m])) by (le))
histogram_quantile(0.95, sum(rate(hikaricp_connections_acquire_seconds_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(hikaricp_connections_acquire_seconds_bucket[5m])) by (le))

# threads currently blocked waiting for a connection
max(hikaricp_connections_pending)

For a controlled measurement before deploying, run a JMH microbenchmark of getConnection()/close() against the configured pool. JMH gives clean, isolated borrow-latency numbers without application noise, but remember it measures the uncontended or fixed-contention borrow cost — it cannot reproduce production hold-time variance. Treat JMH as the floor (best-case borrow cost) and production telemetry as the truth (real tail under real load).

@Benchmark
@BenchmarkMode(Mode.SampleTime)            // SampleTime captures the latency distribution, not just mean
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public void borrowAndReturn(Blackhole bh) throws SQLException {
    try (Connection c = dataSource.getConnection()) {
        bh.consume(c);
    }
}

Mode.SampleTime is essential — it records a histogram of individual call times so JMH reports p99/p99.9, which Mode.AverageTime discards. Compare the JMH p99 (microseconds, uncontended) against production p99 (often milliseconds, contended); the gap is your queuing penalty.

Frequently Asked Questions

Related

• Java Connection Pool Benchmarks — the parent topic on benchmarking pool implementations and methodology.
• Benchmarking Connection Pool Algorithms for Read-Heavy Workloads — sibling guide comparing borrow algorithms under read-heavy load.
• Optimizing HikariCP maximumPoolSize for High Concurrency — sizing derivation that drives acquisition latency.
• Exposing HikariCP Metrics with Micrometer and Prometheus — wiring the acquire histogram into dashboards and alerts.