Building a Grafana Dashboard for Connection Pool Metrics

This guide is part of Prometheus and Grafana Pool Metrics. Collecting pool metrics is the easy half. The hard half is arranging them so that someone paged at three in the morning reaches the right conclusion quickly, rather than looking at nine graphs and guessing.

A pool dashboard has exactly one job: distinguish between the three states a pool can be in. Not busy, busy and coping, and saturated. Every panel should serve that distinction, and panels that do not should be removed.

The four panels that matter

Panel Question it answers Why it is here
Utilisation How full is the pool? Context — but never an alert on its own
Pending waiters Are callers queuing? The unambiguous saturation signal
Acquisition latency What is the queue costing? Converts saturation into user-visible impact
Borrow hold time Why are connections not coming back? Separates a small pool from a slow database

The ordering is deliberate: it is the order an investigation actually runs in. Utilisation tells you the pool is full; pending waiters tells you that is a problem; latency tells you how bad; hold time tells you what to change.

The single most important design decision is that utilisation must not be the alert. A pool at 100% utilisation with no waiters is a correctly sized pool doing its job efficiently. Paging on it trains people to ignore the dashboard, which is worse than having no dashboard.

Panel layout follows the investigation Four panels in a two by two grid. The top row establishes whether there is a problem: utilisation for context and pending waiters for the verdict. The bottom row explains it: acquisition latency for impact and borrow hold time for cause. Top row says whether there is a problem; bottom row says what to change 1. utilisation active / max, per pool context only — never the alert 2. pending waiters callers queued for a connection this is what you alert on 3. acquisition latency p50, p95, p99 of the wait how much of user latency is queueing 4. borrow hold time how long each borrow keeps a connection small pool or slow database? Panels five onward belong on a second row, collapsed by default. Four is what fits on one screen.
Four panels in investigation order. Anything else goes in a collapsed row below.

The PromQL

These use Micrometer’s HikariCP metric names. The Go and Node equivalents follow the same shapes with different series names.

Panel 1 — utilisation. A ratio, so it is comparable across pools of different sizes:

sum by (pool) (hikaricp_connections_active)
  /
sum by (pool) (hikaricp_connections_max)

Set the panel unit to percentunit, the range to 0–1, and thresholds at 0.8 amber and 0.95 red. Those colours are informational, not alerting.

Panel 2 — pending waiters. The saturation verdict:

sum by (pool) (hikaricp_connections_pending)

Use a bar or a stepped line rather than a smoothed one, because the value is a small integer and smoothing hides the transition from 0 to 1 — which is the entire signal.

Panel 3 — acquisition latency. A summary or histogram, depending on your Micrometer configuration:

histogram_quantile(0.99,
  sum by (pool, le) (rate(hikaricp_connections_acquire_seconds_bucket[5m])))

Plot p50 alongside p99. A p50 near zero with a high p99 means a small fraction of requests are queuing — usually a periodic batch competing with request traffic. Both rising together means the whole workload is queuing.

Panel 4 — borrow hold time. The diagnostic panel:

histogram_quantile(0.99,
  sum by (pool, le) (rate(hikaricp_connections_usage_seconds_bucket[5m])))

This is the panel that decides what to change, and it is the one most dashboards omit. Read it against panel 2: waiters rising while hold time is flat means the pool is too small for the arrival rate. Waiters rising because hold time rose means the database slowed down, and enlarging the pool sends more concurrent work at an already-struggling database.

Wiring the metrics up

Micrometer publishes HikariCP metrics automatically in Spring Boot once the actuator and a registry are present:

management:
  endpoints.web.exposure.include: prometheus,health
  metrics.tags.application: ${spring.application.name}
  metrics.distribution:
    percentiles-histogram:
      hikaricp.connections.acquire: true
      hikaricp.connections.usage: true
    slo:
      hikaricp.connections.acquire: 1ms,5ms,10ms,50ms,100ms,500ms,1s,5s

percentiles-histogram: true is required for histogram_quantile to work — without it Micrometer publishes client-side percentiles that cannot be aggregated across instances, and a p99 averaged across pods is not a p99 of anything.

For Go, export DBStats on a collection interval:

prometheus.MustRegister(collectors.NewDBStatsCollector(db, "orders"))

That gives go_sql_stats_connections_waited_for_total and go_sql_stats_connections_blocked_seconds_total, which are counters rather than gauges. The equivalent of panel 2 is therefore a rate:

sum by (db_name) (rate(go_sql_stats_connections_waited_for_total[5m]))

For Node, expose pool.waitingCount as a gauge on your existing metrics endpoint. The counters are described in tuning node-postgres pool timeout settings.

Labels and variables

Every series needs at least three labels: application, pool, and instance. Without pool you cannot tell a primary pool from a replica pool; without instance you cannot tell whether saturation is fleet-wide or concentrated on one pod.

Add Grafana variables so one dashboard serves every service:

$application  = label_values(hikaricp_connections_max, application)
$pool         = label_values(hikaricp_connections_max{application="$application"}, pool)

Then use sum by (instance) inside a panel repeated over $pool, so a service with a primary and a replica pool gets one set of panels each rather than a single graph that adds them together. Summing a primary pool and a replica pool produces a number that describes nothing.

One instance saturating while others are fine is a distinct and common failure — usually an uneven load balancer, or one pod with a leaked connection. That pattern is only visible if instance is preserved rather than aggregated away, which is why the panels above use sum by (pool) for the fleet view and repeat by instance below.

Aggregation hides per-instance saturation A fleet-aggregated utilisation graph sits at a comfortable average while one instance in the fleet is fully saturated with queued callers. Only a per-instance breakdown reveals it. A comfortable average can contain a fully saturated pod fleet average — 55% looks entirely healthy no alert fires per instance 100% one pod saturated with queued callers a leak, or uneven load balancing Keep the instance label and add a per-instance panel below the fleet view. Alerting on max over instances rather than on the average is what catches this case.
Aggregating across instances hides single-pod saturation, which is one of the most common pool failures.

Alert rules to attach

Two rules cover the pool. Both are deliberately about waiters, not utilisation:

groups:
  - name: connection-pool
    rules:
      - alert: PoolSaturated
        expr: max by (application, pool) (hikaricp_connections_pending) > 0
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.application }}/{{ $labels.pool }} has queued callers"

      - alert: PoolAcquisitionSlow
        expr: |
          histogram_quantile(0.99,
            sum by (application, pool, le)
              (rate(hikaricp_connections_acquire_seconds_bucket[5m]))) > 1
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.application }}/{{ $labels.pool }} p99 acquisition above 1s"

The for: 5m matters. Pending waiters spike briefly under any burst, and a rule without a duration will page on normal traffic. Five minutes of sustained queueing is a real condition; five seconds is a burst the pool absorbed.

max by rather than avg by is what makes the single-saturated-instance case alertable. The threshold selection and severity split are developed further in Detecting Connection Pool Saturation.

What each candidate alert actually fires on Alerting on utilisation fires whenever the pool is efficient. Alerting on pending waiters fires only when callers are queued. Alerting on acquisition latency fires when the queue is costing user-visible time. Only two of these three are worth waking someone for utilisation > 90% fires on efficient operation fires during every burst trains people to ignore alerts use as a panel, not an alert pending waiters > 0 fires only when callers queue needs a for: 5m duration warning severity the primary saturation alert p99 acquisition > 1s fires when queueing costs latency already user-visible critical severity the paging alert Use max by rather than avg by on all of these, so a single saturated instance is not averaged away. A pool at full utilisation with no waiters is a correctly sized pool, not an incident.
Utilisation belongs on a panel. Waiters and acquisition latency belong on alerts.

Common failure patterns and remediation

Symptom Underlying cause Remediation
histogram_quantile returns no data percentiles-histogram not enabled Enable it in Micrometer; client-side percentiles cannot be aggregated
p99 across pods looks wrong Averaging pre-computed percentiles Use histogram buckets and compute the quantile in PromQL
Dashboard shows one flat line for everything Pools summed without a pool label Add the label; repeat panels by $pool
Alerts fire constantly Alerting on utilisation rather than waiters Alert on pending waiters with a for duration
Saturation invisible in the fleet view Aggregation hid a single instance Alert on max by and add a per-instance panel
Waiters panel looks empty Smoothed line hiding integer transitions Use a stepped line or bar
Cannot tell whether to grow the pool No borrow hold time panel Add it; it is what distinguishes the two causes
Metrics missing after a deploy Pool renamed, so series changed Pin poolName in configuration rather than letting it default

What to add on a second row

Four panels fit on one screen and answer the primary question. A collapsed second row is the right home for everything else:

  • Connection creation rate. A high rate means connections are churning — usually minimumIdle below maximumPoolSize, or an age cap set aggressively.
  • Timeout rate. How often acquisition actually failed, as opposed to merely being slow.
  • Total connections against the database ceiling. The fleet-wide sum compared against max_connections, which is a capacity-planning signal rather than an incident one.
  • Leak detection warning rate. If leak detection is enabled, the count of warnings without a matching return — see configuring the HikariCP leak detection threshold.
  • Database active backends. From the exporter on the database side, to confirm whether the database is the constraint.

That last one is worth linking rather than embedding: a dashboard that mixes application-side and database-side metrics without clearly separating them leads people to compare numbers that are not comparable. Keep them adjacent and labelled.

Operational boundary

A dashboard describes the pool. It cannot tell you whether the database has capacity for a larger pool, whether a query plan regressed, or whether a connection was leaked in a code path that has since been deployed away. When the four panels point at the database rather than the pool, the next stop is the server side — active backends, wait events, and lock contention, as covered in interpreting pg_stat_activity wait events for pool issues.

Resist the urge to add panels. A dashboard with twenty graphs is not more informative than one with four; it is less, because nobody reads twenty graphs at three in the morning.

Frequently Asked Questions

Should I alert on pool utilisation?
No. A pool at 100% with no queued callers is working correctly. Alert on pending waiters and acquisition latency instead.
Why does my p99 look wrong across instances?
You are almost certainly averaging pre-computed percentiles. Enable histogram buckets and compute the quantile in PromQL from the buckets.
What is the single most useful panel?
Pending waiters. It is the only metric that unambiguously distinguishes a busy pool from a saturated one.
Why include borrow hold time?
Because queue depth alone does not say whether to grow the pool. Rising hold time means the database slowed; flat hold time with rising waiters means the pool is too small.
How do I handle multiple pools per service?
Keep the pool label, repeat the panels by a $pool variable, and never sum a primary pool with a replica pool.
What for duration should alerts use?
Five minutes for the saturation warning. Shorter durations page on normal traffic bursts that the pool absorbed correctly.
Should the database metrics be on the same dashboard?
Adjacent and clearly separated, or linked. Mixing them without separation encourages comparisons between numbers measured on different sides of the connection.