Instrumenting HikariCP with the OpenTelemetry Java Agent

This guide is part of OpenTelemetry Connection Pool Tracing. The OpenTelemetry Java agent instruments HikariCP without touching the application: attach it to the JVM, point it at a Collector, and pool metrics and JDBC spans appear. The gap between that description and a working dashboard is usually four small things — a pool with a generated name, metrics that never leave the JVM because only tracing was configured, an export interval too long to see bursts, and semantic convention names that changed between agent versions:

# metrics arriving, but unusable
db_client_connection_count{pool_name="HikariPool-1", service_name="orders-api"}  12
db_client_connection_count{pool_name="HikariPool-2", service_name="orders-api"}   4
# after a restart, HikariPool-1 and HikariPool-2 swap meanings

The objective is an agent configuration that produces stable, named pool series, spans that show where connections are acquired, and an explicit choice of convention version so dashboards survive an upgrade.

Rapid incident diagnosis

Confirm the agent is attached and what it enabled:

jcmd $(pgrep -f orders-api) VM.command_line | tr ' ' '\n' | grep -i "javaagent\|otel"

Check that metrics — not just traces — are configured. A common outcome is a service with beautiful traces and no pool metrics, because only the traces exporter was set:

env | grep -E "OTEL_(TRACES|METRICS|LOGS)_EXPORTER|OTEL_EXPORTER_OTLP"

OTEL_METRICS_EXPORTER=none — sometimes set to reduce noise — silently removes every pool instrument.

Verify the instrumentation is active. The agent logs the instrumentations it applies at debug level; a targeted check is cheaper:

curl -s http://otel-collector:8889/metrics | grep -c "db_client_connection"

Zero means either no metrics pipeline or no pool instrumentation.

Look at the pool names. Generated names are the difference between a dashboard and a guessing game:

count by (pool_name) (db_client_connection_max{service_name="orders-api"})

HikariPool-1 and HikariPool-2 are assigned in construction order, which is not stable across restarts when pools are created concurrently.

Check which convention version is in use. Older agents emit db.client.connections.usage with a state attribute; newer ones emit db.client.connection.count with db.client.connection.state:

curl -s http://otel-collector:8889/metrics | grep -o "db_client_connections\?_[a-z_]*" | sort -u

Mathematical sizing or parameter formula

Export interval versus burst visibility. Pool gauges are sampled at the export interval, so a saturation shorter than the interval is invisible in the gauge:

p(burst observed) ≈ min(1, burst_duration ÷ export_interval)

An 8-second burst with a 60-second interval is seen about 13 % of the time; at 15 seconds, about 53 %. The histograms and counters — wait_time, timeouts — record every event regardless, which is why alerts should use them and dashboards use the gauges for context.

Series count. The agent’s pool instruments are few, so the cost is modest:

series_per_pool ≈ states(2) + max + idle.min + idle.max + pending + timeouts + 3 histograms × (buckets + 2)

With 15 buckets, about 58 series per pool. A fleet of 40 JVMs with two pools each is roughly 4,600 series.

Span volume. JDBC statement spans are one per statement, which is the largest part of the agent’s output:

spans_per_second ≈ statements_per_second + (1 per acquisition if data-source instrumentation is on)

A service issuing 3,000 statements per second produces 3,000 spans per second before sampling — the reason head sampling, or tail sampling in the Collector, matters more than the pool instrumentation does.

Overhead. The agent’s cost is dominated by span creation rather than by metric collection. Typical figures are a few percent of CPU for a database-heavy service, and it is worth measuring on a canary rather than assuming:

measure: p50 and p99 latency, CPU, and allocation rate, with and without the agent

Exact remediation & configuration

A complete agent configuration for a Spring Boot service:

JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"

OTEL_SERVICE_NAME=orders-api
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=commerce
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=none

OTEL_METRIC_EXPORT_INTERVAL=15000
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

OTEL_SEMCONV_STABILITY_OPT_IN=database
OTEL_INSTRUMENTATION_JDBC_DATASOURCE_ENABLED=true      # span around getConnection()
OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED=true

Name every pool. This is the single change that makes the metrics usable:

spring:
  datasource:
    hikari:
      pool-name: orders-primary
      maximum-pool-size: 16
      minimum-idle: 4

With several data sources, name each one after the database and role it serves — orders-primary, orders-replica — so a dashboard row is self-explanatory.

Decide about acquisition spans deliberately. OTEL_INSTRUMENTATION_JDBC_DATASOURCE_ENABLED=true adds a span per getConnection(). It is what makes pool wait visible in a trace, and it roughly doubles span volume on database-heavy services. Enable it where contention is being investigated, or permanently with tail sampling that keeps slow traces:

# Collector: keep slow and errored traces, sample the rest
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Suppress what you do not need. The agent instruments a great deal; disabling the noisiest parts reduces volume without losing the database picture:

OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
OTEL_INSTRUMENTATION_LOGBACK_APPENDER_ENABLED=false
OTEL_INSTRUMENTATION_SPRING_SCHEDULING_ENABLED=false

Keep Micrometer if it is already wired. Running both produces two series sets for the same pool; either disable the agent’s JDBC metrics or stop publishing the Micrometer ones, and choose one for alerting. The Micrometer route is described in Exposing HikariCP Metrics with Micrometer and Prometheus.

Add resource attributes that make grouping possible. deployment.environment and service.namespace cost nothing and let one dashboard serve every environment.

What the agent cannot see

The agent is powerful precisely because it needs no code changes, and that boundary is also its limitation.

Application intent. A span shows that getConnection() took 1.8 seconds; it cannot show which business operation was waiting or why that operation holds connections longer than others. A custom attribute on the request span — the endpoint, the tenant tier, the job type — is what makes the trace actionable, and it requires code:

Span.current().setAttribute("app.operation", "checkout.finalise");

Pool internals beyond the standard instruments. HikariCP’s getTotalConnections, creation timing and housekeeping are exposed through its own metrics interface; the agent publishes the conventional subset. Where a specific internal matters, Micrometer or a JMX exporter fills the gap.

Connections used outside JDBC. A native client, a driver the agent does not instrument, or a pool inside a library are invisible. The database’s own view — sessions by application_name — is the cross-check that catches them.

Work that holds a connection without querying. A span for getConnection() and spans for statements leave the gap between them unexplained; that gap is where an external call inside a transaction hides. Reading it requires looking at the waterfall rather than at any single span.

Running the agent alongside existing instrumentation

Most services that adopt the agent already have something: Micrometer metrics, a vendor APM agent, or hand-written timers. Running two systems over the same pool is the usual source of confusion afterwards.

Two agents on one JVM rarely work. A vendor APM agent and the OpenTelemetry agent both instrument the same classes, and the result ranges from duplicate spans to startup failures. Where both are required, use the vendor’s OpenTelemetry-compatible mode or its support for receiving OTLP, rather than attaching both.

Micrometer and the agent both publish pool metrics. The series have different names — hikaricp_connections_active against db_client_connection_count — so nothing breaks, but dashboards and alerts must pick one. Two practical rules: alert on one source only, and record which one in the runbook, because an incident is a bad time to discover that the alert watches the series nobody is looking at.

# If the agent is the source of truth, stop publishing the Micrometer pool metrics
management:
  metrics:
    enable:
      hikaricp: false

Traces and logs should share the context. With the agent attached, adding the trace and span ids to the log pattern makes a log line and a trace mutually reachable:

<pattern>%d{ISO8601} %-5level [%X{trace_id},%X{span_id}] %logger{36} - %msg%n</pattern>

The agent populates those MDC keys automatically for the supported logging frameworks, which costs nothing and is the single most useful addition to a log format in a traced service.

Keep the rollout reversible. The agent is attached by a JVM flag, so removing it is a deployment rather than a code change — which makes it reasonable to canary on one instance, compare latency and CPU, and expand or revert based on the measurement rather than on expectation.

Validation & verification

Metrics arrive with stable names. After a restart, the pool series must keep the same pool_name:

count by (pool_name) (db_client_connection_max{service_name="orders-api"})

Bursts are visible. Induce a short saturation in staging and confirm the histogram records it even when the gauges do not:

increase(db_client_connection_wait_time_seconds_count{pool_name="orders-primary"}[5m])

Acquisition spans appear when enabled. Find a slow trace and confirm a getConnection span with a duration matching the wait.

Overhead is acceptable. Compare a canary with and without the agent under identical load: p99 latency, CPU and allocation rate.

Convention names are pinned. Upgrade the agent in staging and confirm the metric names are unchanged; if they change, the opt-in is not set or the version moved past it.

Frequently Asked Questions

Related

• OpenTelemetry Connection Pool Tracing — the parent guide to the instruments and conventions.
• Collecting db.client.connection Metrics with the OpenTelemetry Collector — the pipeline these metrics flow into.
• Tracing Connection Acquisition Latency with OpenTelemetry Spans — what the acquisition spans are for.
• Exposing HikariCP Metrics with Micrometer and Prometheus — the alternative route for the same pool.