Profiling Connection Pool Contention with Java Flight Recorder

This guide is part of Java Connection Pool Benchmarks. Pool metrics say how long acquisition took on average; they do not say which code paths waited, what those threads were doing, or whether the time went to the pool, to a JDBC round trip or to a lock inside the application. Java Flight Recorder answers all three from a running production process, at an overhead low enough to leave enabled continuously:

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available,
request timed out after 3000ms.

That exception names the pool but not the cause. The objective is a recording that shows exactly where threads park waiting for connections, how long they wait, which call sites are responsible, and how that compares with the time spent inside the database.

Rapid incident diagnosis

Start a recording on the running process. No restart is needed:

PID=$(pgrep -f orders-api)
jcmd $PID JFR.start name=pool duration=120s filename=/tmp/pool.jfr settings=profile
jcmd $PID JFR.check

The profile settings enable thread-park events with a 10 ms threshold — enough to catch pool waits without recording every short park. For a service already running a continuous recording, dump the last interval instead:

jcmd $PID JFR.dump name=continuous filename=/tmp/pool.jfr maxage=10m

Summarise the events. jfr summary shows which event types dominate:

jfr summary /tmp/pool.jfr | head -25

jdk.JavaMonitorWait and jdk.ThreadPark counts in the thousands, with a large total duration, point at threads waiting rather than working.

Find the pool waits specifically. HikariCP parks threads in ConcurrentBag.borrow; DBCP2 and Tomcat JDBC park in their own structures:

jfr print --events jdk.ThreadPark --stack-depth 24 /tmp/pool.jfr \
  | grep -B3 -A18 "ConcurrentBag.borrow" | head -60

Each event carries a duration and the full stack, so the application method that asked for a connection is visible a few frames up.

Compare with time inside the database. JFR’s JDBC events — where the driver supports them — or the jdk.SocketRead events give the other half:

jfr print --events jdk.SocketRead /tmp/pool.jfr | grep -A4 "port = 5432" | head -30

Long socket reads mean slow queries; long parks in borrow with short socket reads mean the pool is the constraint.

Where a request's time goes, as seen by JFR A request's elapsed time divides into parking on connection acquisition, socket reads against the database, monitor waits inside the application, and actual CPU work, each visible as a distinct JFR event type. One slow request, by JFR event type ThreadPark (pool) 2,100 ms SocketRead (database) 320 ms MonitorWait (app lock) 90 ms CPU 60 ms the pool, not the database, owns most of this request's latency
JFR separates the three places a request can lose time: waiting for a connection, waiting on the database, and waiting on a lock inside the application.

Mathematical sizing or parameter formula

JFR gives the raw numbers to quantify contention rather than describe it.

Total time lost to acquisition over the recording:

pool_wait_total = Σ duration of ThreadPark events in the pool's borrow path
pool_wait_share = pool_wait_total ÷ (threads × recording_seconds)

A two-minute recording on a service with 200 request threads gives 200 × 120 = 24,000 thread-seconds. If parked-in-borrow events total 1,800 thread-seconds, 7.5 % of all thread time went to waiting for connections — a concrete number to compare against the cost of enlarging the pool.

Required pool size from observed demand. The recording also contains the data for Little’s Law, because each borrow’s hold time is the interval between acquisition and release:

in_flight = Σ hold_time ÷ recording_seconds
max_size  = ceil( in_flight ÷ 0.7 )

Overhead budget. Continuous recording with the default settings costs on the order of one percent of CPU; the profile settings cost more because the sampling interval is shorter and thresholds lower:

default  : ~1 % CPU, thread-park threshold 20 ms, suitable for always-on
profile  : ~2–3 % CPU, threshold 10 ms, suitable for a bounded investigation

Sizing the recording itself matters on busy services: a 120-second profile recording on a service with hundreds of threads can produce a file of tens of megabytes, which is fine locally and awkward to ship. maxsize bounds it:

jcmd $PID JFR.start name=pool maxsize=100m maxage=10m settings=profile
Quantifying contention from a recording Across 24,000 thread-seconds of capacity in a two minute recording, 1,800 thread-seconds were spent parked waiting for a connection, which is 7.5 percent of all thread time. Thread-seconds in a 120 s recording, 200 threads 1,800 parked 22,200 available for work 7.5 % of thread capacity spent waiting for connections compare with: the database's spare capacity, and the cost of raising max-pool-size a measured share makes the sizing decision an arithmetic one
Parked thread-seconds divided by total thread-seconds turns a vague "the pool feels slow" into a number that can be compared with the cost of more connections.

Exact remediation & configuration

Run a continuous recording in production. The cost is small and it means the next incident is investigated from data rather than from a reproduction:

java -XX:StartFlightRecording=name=continuous,maxage=30m,maxsize=200m,settings=default \
     -XX:FlightRecorderOptions=repository=/var/log/jfr \
     -jar orders-api.jar

In Kubernetes, mount a volume for the repository so a dump survives a container restart, and expose a way to trigger a dump — a management endpoint or a kubectl exec runbook step.

Add custom events around acquisition where the pool’s own frames are not enough. A small event records the call site, which JFR’s stack traces already imply but which is easier to query when named:

@Name("com.example.db.Acquire")
@Label("Connection Acquire")
@Category({"Application", "Database"})
class AcquireEvent extends jdk.jfr.Event {
    @Label("Pool") String pool;
    @Label("Call site") String callSite;
}

Connection acquire(DataSource ds, String callSite) throws SQLException {
    AcquireEvent e = new AcquireEvent();
    e.pool = "orders-pool";
    e.callSite = callSite;
    e.begin();
    try {
        return ds.getConnection();
    } finally {
        e.commit();          // duration is recorded automatically
    }
}

Custom events are cheap — a few tens of nanoseconds when the event is not enabled — and they make the analysis trivial:

jfr print --events com.example.db.Acquire /tmp/pool.jfr | grep -E "callSite|duration" | paste - - | sort -k4 -r | head

Analyse without the GUI. jfr print and jfr summary are enough for most questions, and they work in a terminal on a jump host:

# total parked time by top frame
jfr print --events jdk.ThreadPark --stack-depth 1 /tmp/pool.jfr \
  | awk '/duration/ {d=$3} /at /{print d, $0}' | sort -k1 -r | head -20

Act on what it shows. The recording points at one of three fixes: enlarge the pool when waits dominate and the database has headroom; fix the query when socket reads dominate; or restructure the code when application locks or long holds dominate. The sizing arithmetic is in Sizing Connection Pools for CPU-Bound versus IO-Bound Workloads.

Using JFR in benchmarks, not just incidents

JFR is at its most useful when a benchmark and production use the same instrumentation, because the numbers are then directly comparable.

Record the benchmark. Running a load test with the same continuous recording configuration produces a file that can be compared frame by frame with a production dump:

java -XX:StartFlightRecording=name=bench,duration=300s,filename=bench.jfr,settings=profile \
     -jar orders-api.jar &
k6 run --vus 200 --duration 5m checkout.js

Compare the shape, not just the totals. Two services can have the same mean acquisition time with very different distributions: one where every borrow waits two milliseconds, and one where 99 % are instant and 1 % wait a second. The second is the one that produces timeouts, and only the event-level data shows it.

Watch for measurement artefacts. A benchmark that runs on a laptop with four cores and a production service on a node with 64 will produce different pool sizes for pools whose defaults follow the core count, and different lock contention profiles. Pin the pool size explicitly in both, or the comparison measures the hardware rather than the change.

Keep a baseline recording. A JFR file from a healthy period is the most useful artefact during an incident: it answers “was this always like this?” in seconds. Store one per service per release, and the comparison becomes routine.

Question Event type What to look for
Are threads waiting for connections? jdk.ThreadPark in the pool’s borrow frames Total duration and count
Which call sites wait? Custom acquire event, or park stack traces Concentration in one endpoint
Is the database slow? jdk.SocketRead to the database port Long reads, high totals
Is the application locking? jdk.JavaMonitorEnter, jdk.JavaMonitorWait Contended monitors in data paths
Is the pool churning? jdk.SocketWrite bursts plus TLS handshakes Frequent new connections
Comparing benchmark and production recordings A recording from a load test and one from production are compared by event type, showing whether the benchmark reproduces the contention seen in production before any change is made. Benchmark same JFR settings, controlled load Production continuous recording, real traffic Compare by event type and by distribution not only by mean Comparable only if both use the same settings and pool sizes a benchmark that does not reproduce the production profile is measuring something else
The same recording configuration in the benchmark and in production makes the two directly comparable, which is what turns a profile into a decision.

Validation & verification

The recording captures the event you care about. Reproduce a pool wait deliberately — cap the pool at one connection under load — and confirm the parked events appear with the expected durations. If they do not, the threshold is too high: lower it with a custom settings file rather than switching to profile globally.

Overhead is acceptable. Measure request latency and CPU with and without the continuous recording on a canary instance. A default-settings recording should be within noise; if it is not, reduce maxsize or move to an event-specific settings file.

Dumps are obtainable during an incident. Practise it: jcmd JFR.dump on a production pod, copy the file out, open it locally. A procedure that has never been rehearsed tends to fail exactly when it is needed.

Findings match the metrics. The acquisition time distribution derived from JFR should agree with hikaricp_connections_acquire_seconds from Micrometer. A large disagreement usually means the metric is scoped differently — for example, measuring only successful acquisitions while JFR also records the ones that timed out.

Frequently Asked Questions

Does JFR require a commercial licence?
No. Flight Recorder is part of OpenJDK from Java 11 onward and is free to use in production. Older Oracle JDK 8 builds required a commercial flag, which is the source of the lingering belief that it is licensed.
Is settings=profile safe in production?
For bounded investigations, yes — a few minutes at a time. For always-on recording, default is the right choice; it still captures thread parks above 20 ms, which is where pool contention lives.
Can JFR show the SQL that was running?
Not by itself. The JDBC driver would have to emit events, and most do not. Pair JFR with the database’s own statement statistics, or add a custom event carrying the query identifier where that matters.
How does this compare with async-profiler?
async-profiler samples stacks including native frames and is excellent for CPU profiling. JFR is better for duration events — how long a thread was parked, and on what — which is exactly the question pool contention poses. Many teams use both.