Exporting Go database/sql Pool Metrics to Prometheus

This guide is part of Prometheus and Grafana Pool Metrics. Go’s database/sql package keeps precise statistics about its connection pool in sql.DBStats — how many connections are open and in use, how many times a caller had to wait, how long those waits took in total, and why connections were closed. None of it is exported by default. Teams discover the pool only when context deadline exceeded errors appear on queries that are individually fast:

{"level":"error","ts":"2026-09-15T11:02:44Z","msg":"list orders failed",
 "error":"context deadline exceeded","route":"/v1/orders","elapsed_ms":2001}

The deadline expired while the goroutine was waiting inside database/sql for a free connection. The objective is to export the pool statistics — for database/sql and for pgxpool — derive the rates that matter, and alert before callers time out.

Rapid incident diagnosis

Read the stats directly, before any exporter exists. A temporary debug endpoint returns the live struct:

http.HandleFunc("/debug/dbstats", func(w http.ResponseWriter, r *http.Request) {
    json.NewEncoder(w).Encode(db.Stats())
})

curl -s localhost:8080/debug/dbstats | jq
# { "MaxOpenConnections": 20, "OpenConnections": 20, "InUse": 20, "Idle": 0,
#   "WaitCount": 184213, "WaitDuration": 912455000000,
#   "MaxIdleClosed": 1402211, "MaxIdleTimeClosed": 0, "MaxLifetimeClosed": 3120 }

Interpret the fields.

• InUse == MaxOpenConnections with Idle == 0: every connection is lent out. Callers arriving now wait.
• WaitCount and WaitDuration are cumulative since the *sql.DB was opened. Two samples a minute apart give the wait rate and the mean wait: in the example above, if a second sample shows WaitCount up by 6,000 and WaitDuration up by 30 seconds, callers waited 6,000 times with a mean of 5 ms — but some waited much longer.
• MaxIdleClosed growing quickly means connections are returned to a full idle set and closed, then reopened for the next request: SetMaxIdleConns is too low for the traffic. That churn costs a connection handshake per request and is visible as latency, not as waiting.
• MaxLifetimeClosed and MaxIdleTimeClosed show recycling by SetConnMaxLifetime and SetConnMaxIdleTime, which should be slow and steady.

Separate pool wait from slow queries. If WaitCount is flat but requests time out, the time is spent in queries or in the network, not waiting for a connection. Check pg_stat_activity for long-running statements from the service.

Mathematical sizing or parameter formula

The cumulative counters turn into three operational rates.

Wait rate — how often callers wait, per second:

rate(go_sql_wait_count_total{db_name="orders"}[5m])

Mean wait per waiting acquisition:

rate(go_sql_wait_duration_seconds_total{db_name="orders"}[5m])
  / rate(go_sql_wait_count_total{db_name="orders"}[5m])

Fraction of time the pool is exhausted, approximated by how much waiting time accrues per second of wall time — a value of 3 means that, on average, three goroutines were waiting at any moment:

rate(go_sql_wait_duration_seconds_total{db_name="orders"}[5m])

That last expression is the average wait queue length, directly from Little’s Law: queue length equals arrival rate of waiters times their mean wait. Worked example. Over five minutes, WaitCount rises by 36,000 (120 waits per second) and WaitDuration by 1,800 seconds (6 seconds per second). The mean wait is 6 ÷ 120 = 50 ms, and on average 6 goroutines are queued for a connection. With a 2-second request deadline, a mean of 50 ms is tolerable — but a queue of 6 against a pool of 20 means demand is running about 30 % above capacity during those minutes.

Sizing then follows from demand: if in-use connections sit at 20 and the queue averages 6, a first estimate for SetMaxOpenConns is 20 + 6 = 26, checked against the database’s connection budget across all replicas. The Go-specific tuning rules are in Configuring SetMaxOpenConns and SetMaxIdleConns.

Churn rate — connections closed because the idle set was full:

rate(go_sql_max_idle_closed_total{db_name="orders"}[5m])

Any sustained value above a few per second means SetMaxIdleConns is below steady-state concurrency. Go’s default MaxIdleConns is 2, which on a busy service produces hundreds of closes per second.

Exact remediation & configuration

Register the built-in collector for database/sql. The Prometheus Go client ships a collector for sql.DBStats; register one per *sql.DB, with a db_name label that identifies the pool:

import (
    "database/sql"
    "log"
    "net/http"
    "os"
    "time"

    _ "github.com/jackc/pgx/v5/stdlib"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/collectors"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))
    if err != nil { log.Fatal(err) }
    db.SetMaxOpenConns(26)
    db.SetMaxIdleConns(26)                 // equal to max open: no churn under steady load
    db.SetConnMaxLifetime(30 * time.Minute)
    db.SetConnMaxIdleTime(5 * time.Minute)

    reg := prometheus.NewRegistry()
    reg.MustRegister(
        collectors.NewGoCollector(),
        collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
        collectors.NewDBStatsCollector(db, "orders"),     // db_name="orders"
    )
    http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{}))
    log.Fatal(http.ListenAndServe(":9102", nil))
}

For pgxpool, write a small collector. pgxpool.Pool.Stat() exposes richer counters than sql.DBStats, including the total number of acquisitions, which lets you compute the share that had to wait:

type pgxPoolCollector struct {
    pool *pgxpool.Pool
    name string
    total, idle, acquired, maxConns     *prometheus.Desc
    acquireCount, emptyAcquire, canceled *prometheus.Desc
    acquireSeconds, newConns             *prometheus.Desc
}

func newPgxPoolCollector(pool *pgxpool.Pool, name string) *pgxPoolCollector {
    l := prometheus.Labels{"pool": name}
    d := func(n, h string) *prometheus.Desc { return prometheus.NewDesc("pgxpool_"+n, h, nil, l) }
    return &pgxPoolCollector{pool: pool, name: name,
        total: d("total_conns", "Open connections."), idle: d("idle_conns", "Idle connections."),
        acquired: d("acquired_conns", "Connections in use."), maxConns: d("max_conns", "Pool maximum."),
        acquireCount: d("acquire_total", "Successful acquisitions."),
        emptyAcquire: d("empty_acquire_total", "Acquisitions that waited because the pool was empty."),
        canceled: d("canceled_acquire_total", "Acquisitions cancelled by context."),
        acquireSeconds: d("acquire_duration_seconds_total", "Total time spent acquiring."),
        newConns: d("new_conns_total", "Connections created."),
    }
}

func (c *pgxPoolCollector) Describe(ch chan<- *prometheus.Desc) { prometheus.DescribeByCollect(c, ch) }

func (c *pgxPoolCollector) Collect(ch chan<- prometheus.Metric) {
    s := c.pool.Stat()
    g, k := prometheus.GaugeValue, prometheus.CounterValue
    ch <- prometheus.MustNewConstMetric(c.total, g, float64(s.TotalConns()))
    ch <- prometheus.MustNewConstMetric(c.idle, g, float64(s.IdleConns()))
    ch <- prometheus.MustNewConstMetric(c.acquired, g, float64(s.AcquiredConns()))
    ch <- prometheus.MustNewConstMetric(c.maxConns, g, float64(s.MaxConns()))
    ch <- prometheus.MustNewConstMetric(c.acquireCount, k, float64(s.AcquireCount()))
    ch <- prometheus.MustNewConstMetric(c.emptyAcquire, k, float64(s.EmptyAcquireCount()))
    ch <- prometheus.MustNewConstMetric(c.canceled, k, float64(s.CanceledAcquireCount()))
    ch <- prometheus.MustNewConstMetric(c.acquireSeconds, k, s.AcquireDuration().Seconds())
    ch <- prometheus.MustNewConstMetric(c.newConns, k, float64(s.NewConnsCount()))
}

Alert on waiting, not on utilisation. A pool at 100 % in-use is working hard; a pool where callers queue is about to fail them:

groups:
  - name: go-sql-pool
    rules:
      - alert: GoSQLPoolQueueing
        expr: rate(go_sql_wait_duration_seconds_total[5m]) > 1
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.db_name }}: on average more than one goroutine waiting for a connection"
      - alert: GoSQLPoolChurn
        expr: rate(go_sql_max_idle_closed_total[5m]) > 5
        for: 15m
        labels: { severity: info }
        annotations:
          summary: "{{ $labels.db_name }}: connections closed on return; SetMaxIdleConns too low"
      - alert: PgxPoolCanceledAcquires
        expr: rate(pgxpool_canceled_acquire_total[5m]) > 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.pool }}: requests cancelled while waiting for a connection"

CanceledAcquireCount in pgxpool counts exactly the failure in the opening log: a context that expired during acquisition. It is the most direct page-worthy signal a Go pool offers.

Validation & verification

Metrics are present and labelled.

curl -s localhost:9102/metrics | grep -E '^go_sql_(in_use|idle|wait_count|wait_duration|max_idle_closed)'

Each series must carry db_name="orders"; a service with two pools must show two distinct db_name values.

The queue signal responds. In staging, lower SetMaxOpenConns to 2 and run a load test. rate(go_sql_wait_duration_seconds_total[1m]) must climb well above 1 within a minute, and the GoSQLPoolQueueing alert must enter pending. Restore the setting and confirm it drops back to near zero.

Churn disappears. After raising SetMaxIdleConns, the churn rate should fall to near zero under steady load, and the database’s own session creation rate — pg_stat_database.sessions growth on PostgreSQL 14 and later — should fall with it.

Cancelled acquisitions match errors. For pgxpool, compare increase(pgxpool_canceled_acquire_total[10m]) with the count of context deadline exceeded errors from acquisition in logs over the same window. They should agree closely; a large difference means some deadlines expire during queries rather than during acquisition.

Dashboard panels for a Go service’s pool

A dashboard for a Go pool needs to answer the same four questions as any pool dashboard — is anyone waiting, how long, how close to the ceiling, and are connections churning — using the series this page exports.

Two practical details make the panels reliable. First, use a $db template variable built from label_values(go_sql_max_open_connections, db_name) so every pool in the service appears in a selector — services with separate reader and writer pools show both. Second, graph per instance, not only summed: a pool problem often affects one pod that received a burst of long requests, and a fleet sum hides it.

For pgxpool, the richer counters allow an exact share of acquisitions that waited, which is a better headline number than queue length:

rate(pgxpool_empty_acquire_total{pool="$db"}[5m])
  / rate(pgxpool_acquire_total{pool="$db"}[5m])

A value of 0.02 means 2 % of acquisitions found no idle connection. Values near zero are healthy; a value that climbs with traffic shows the pool approaching its ceiling long before cancellations begin.

Add one annotation source: deployments. Most step changes in pool behaviour follow a release that changed query patterns, pool settings or replica counts, and seeing the deploy marker on the queue-length panel usually explains the step immediately. The full dashboard layout conventions are covered in Building a Grafana Dashboard for Connection Pool Metrics.

Frequently Asked Questions

Related

• Prometheus and Grafana Pool Metrics — the parent guide to the pull pipeline and dashboards.
• Go database/sql Pool Internals — how the connection request queue that these counters measure works.
• Understanding Connection Acquisition Timeouts in Go — contexts, deadlines and cancellation during acquisition.
• Building a Grafana Dashboard for Connection Pool Metrics — panels for the rates derived above.