Alerting on PgBouncer Client and Server Limits

This guide is part of PgBouncer Metrics Monitoring. PgBouncer enforces several limits, and each one fails differently. max_client_conn refuses new clients outright; default_pool_size makes them wait; max_db_connections and max_user_connections cap backends across pools. Teams usually alert on waiting clients and discover the others during an incident, when the log says something they have never seen:

WARNING C-0x55c9: (nodb)/(nouser)@10.4.2.31:52134 pooler error: no more connections allowed (max_client_conn)
WARNING C-0x55d1: orders/orders_app@10.4.2.44:41002 pooler error: server login has been failing
LOG C-0x55e2: orders/orders_app@10.4.2.12:33120 closing because: query_wait_timeout (age=15s)

The first is a hard refusal that looks like a network failure to the application. The objective is an alert for each limit, with thresholds that fire while there is still room to act.

Rapid incident diagnosis

Read every limit and its current usage in one pass. The console’s views cover them:

SHOW CONFIG;      -- max_client_conn, default_pool_size, max_db_connections, max_user_connections
SHOW LISTS;       -- databases, users, pools, free_clients, used_clients, free_servers, used_servers
SHOW POOLS;       -- per pool: cl_active, cl_waiting, sv_active, sv_idle, sv_used, maxwait
SHOW DATABASES;   -- per database: pool_size, max_connections, current_connections

SHOW LISTS is the one that answers the client-limit question directly: used_clients approaching the sum of free and used is the approach to max_client_conn.

Match the symptom to the limit:

Check whether the database is the real limit. PgBouncer cannot open more backends than PostgreSQL allows:

SELECT current_setting('max_connections')::int AS limit,
       count(*) AS in_use,
       current_setting('max_connections')::int - count(*) AS free
FROM pg_stat_activity;

Identify which pool. With several databases and users, the limits are per pool, and an aggregate view hides which one is short. Always group by database and user.

Mathematical sizing or parameter formula

Client limit headroom. Clients are cheap, so the limit should sit well above the fleet’s maximum:

peak_clients = Σ over services ( replicas × workers × pool_max )
max_client_conn ≥ peak_clients × 1.5 + admin_and_tools

With 40 processes holding 10 connections each, peak clients is 400, and max_client_conn of 1,000 leaves room for a deploy overlap and ad-hoc sessions. Alert at 70 % of the limit:

warn_at = 0.7 × max_client_conn

Server pool headroom. The backend pool should be sized from concurrent transactions, and the alert placed where waiting starts to be sustained:

default_pool_size = ceil( transactions_per_second × mean_transaction_seconds ÷ 0.7 )
warn when: cl_waiting > 0 for longer than a scrape interval, or maxwait > 1 s

Database and user caps. These are the totals the database must accept:

Σ over pools ( pool_size ) ≤ max_db_connections ≤ database max_connections − reserved

An alert on current_connections per database at 85 % of max_db_connections catches the case where several pools share one host.

Memory. Each client connection costs PgBouncer memory for its buffers:

memory ≈ (max_client_conn × pkt_buf × 2) + (server_connections × pkt_buf × 2) + overhead

With the default 4 KB packet buffer and 1,000 clients, that is a few tens of megabytes — small, but worth checking before raising max_client_conn into the tens of thousands.

Exact remediation & configuration

Export the metrics. The Prometheus exporter publishes the console’s views; scrape it per PgBouncer instance with an instance label:

scrape_configs:
  - job_name: pgbouncer
    scrape_interval: 15s
    static_configs:
      - targets: ["10.0.1.20:9127", "10.0.1.21:9127", "10.0.1.22:9127"]
        labels: { service: orders }

Alert on each limit. The rules below assume the exporter’s common metric names; adjust to the exporter in use:

groups:
  - name: pgbouncer-limits
    rules:
      # 1. Client limit: refusals are invisible to PgBouncer's pool metrics, so watch the total
      - alert: PgBouncerClientLimitApproaching
        expr: sum by (instance) (pgbouncer_pools_client_active_connections
                               + pgbouncer_pools_client_waiting_connections)
              > 0.7 * on (instance) pgbouncer_config_max_client_conn
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.instance }}: client connections above 70 % of max_client_conn"

      # 2. Pool saturation: waiting clients sustained
      - alert: PgBouncerClientsWaiting
        expr: max by (instance, database) (pgbouncer_pools_client_waiting_connections) > 0
        for: 2m
        labels: { severity: warning }

      # 3. Wait time: the page-worthy signal
      - alert: PgBouncerMaxwaitHigh
        expr: max by (instance, database) (pgbouncer_pools_client_maxwait_seconds) > 2
        for: 1m
        labels: { severity: critical }

      # 4. Per-database cap
      - alert: PgBouncerDatabaseConnectionsHigh
        expr: pgbouncer_databases_current_connections
              > 0.85 * pgbouncer_databases_max_connections
        for: 5m
        labels: { severity: warning }

      # 5. The exporter itself
      - alert: PgBouncerExporterDown
        expr: pgbouncer_up == 0
        for: 5m
        labels: { severity: warning }

Alert on refusals from the log where the metric does not exist. max_client_conn refusals are logged but not always exported; a log-based counter closes the gap:

# promtail / vector rule
- match: 'no more connections allowed'
  labels: { alert_source: pgbouncer_client_limit }

Raise the right limit. Each symptom has one correct response:

[pgbouncer]
max_client_conn = 1000            ; raise when clients are refused and memory allows
default_pool_size = 11            ; raise when clients wait and the database has capacity
max_db_connections = 30           ; the cap across pools on one database
max_user_connections = 30         ; the cap per user
query_wait_timeout = 15           ; bound the wait rather than letting it grow
reserve_pool_size = 4
reserve_pool_timeout = 2

Apply with RELOAD; on the admin console — none of these requires a restart.

Keep the totals consistent. The sum of per-instance max_db_connections across every PgBouncer must fit the database’s usable connections, as covered in Deploying Highly Available PgBouncer.

Dashboards that make the limits obvious

Each limit should appear on the dashboard as a line, not as a number in a runbook.

Panel one: clients against the client limit. A stacked series of active and waiting clients with max_client_conn drawn as a threshold. Refusals happen at the line, and the distance to it is the headroom.

Panel two: server connections against the pool size. sv_active + sv_idle + sv_used per pool with pool_size + reserve_pool_size as the threshold shows whether the reserve is in use — a sign the regular pool is undersized.

Panel three: maxwait. The single most informative series PgBouncer publishes, because it is measured in the units users experience.

Panel four: per-database totals. current_connections against max_connections per database entry, which catches the cap that is easiest to forget.

Add two annotations: configuration reloads and deploys. A step change in any of these panels almost always follows one or the other.

Keep the limits in the dashboard, not only in the config. Exporting the configuration as metrics — most exporters publish pgbouncer_config_* series — means the threshold lines update themselves when someone changes a limit, which removes the classic failure where the dashboard still shows last year’s number.

Validation & verification

Each alert fires in a rehearsal. In staging, drive each limit in turn: open clients beyond 70 % of max_client_conn; hold transactions to fill the pool; lower max_db_connections below the pool sum. Each should fire its own alert and no others.

The client-limit alert precedes refusals. With the alert at 70 %, there should be several minutes between it firing and the first no more connections allowed entry in the log.

Thresholds track the configuration. Change a limit with RELOAD and confirm the dashboard’s threshold line moves, which proves the configuration metrics are being scraped.

Per-instance labels are present. With several PgBouncer instances, every alert must identify which one; an alert that aggregates across instances hides the one that is short.

The exporter’s own failure is covered. Stop an exporter and confirm PgBouncerExporterDown fires — otherwise every other alert silently stops working.

Frequently Asked Questions

Related

• PgBouncer Metrics Monitoring — the parent guide to the console’s views.
• Setting Up the PgBouncer Prometheus Exporter — exporting the series these alerts use.
• Diagnosing PgBouncer cl_waiting Spikes — investigating the alerts that fire on waiting.
• Configuring PgBouncer Pool Size and max_client_conn — choosing the limits these alerts watch.