Configuring HikariCP keepaliveTime Behind Firewalls and NAT

This guide is part of HikariCP Configuration Deep Dive. A NAT gateway, a stateful firewall or a network load balancer keeps a translation entry for each connection and discards it after an idle period — often without sending a reset to either end. Both sides still believe the connection exists. HikariCP hands it to a request, the driver writes a query into a socket that goes nowhere, and the request blocks until the TCP retransmission timeout expires, which can be minutes:

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
  Caused by: java.net.SocketTimeoutException: Read timed out

The distinguishing feature is timing: the errors cluster at the start of the working day, after a deployment freeze, or on a service with quiet periods — not under load. The objective is a configuration in which the pool proves a connection is alive before handing it out, and in which nothing in the network path ever removes a connection the pool still trusts.

Rapid incident diagnosis

Map the idle timeouts in the path. Every hop has one, and the smallest wins:

Compare the failure interval with those numbers. Enable HikariCP’s debug logging, which prints pool statistics every 30 seconds, and note the gap between the last activity and the failure:

logging.level.com.zaxxer.hikari.pool.HikariPool=DEBUG
logging.level.com.zaxxer.hikari.HikariConfig=DEBUG

A failure after roughly 350 seconds of idleness points at a NAT gateway; after four minutes, at an Azure load balancer.

Check whether the connection was idle in the pool or held by the application. A connection dropped while idle in the pool fails on the next borrow; one dropped while checked out fails mid-request. The first is what keepalives fix.

Confirm from the server that the session is gone. If PostgreSQL or SQL Server shows no session for the connection the pool believes it holds, the path dropped it:

SELECT count(*) FROM pg_stat_activity WHERE application_name = 'orders-api';
-- compare with hikaricp_connections from the application's metrics

A pool reporting 10 connections while the server shows 6 is four dead sockets waiting to be handed out.

Mathematical sizing or parameter formula

The settings must nest inside the smallest idle timeout in the path:

keepaliveTime   < min(all idle timeouts) ÷ 2
idleTimeout     < min(all idle timeouts) − keepaliveTime
maxLifetime     < min(server-side lifetime limits); and > keepaliveTime
validationTimeout ≤ connectionTimeout ÷ 2

Worked example behind an AWS NAT gateway at 350 seconds, with PostgreSQL’s idle_session_timeout at 20 minutes:

smallest idle timeout = 350 s
keepaliveTime = 120_000 ms     (120 s, comfortably under half of 350)
idleTimeout   = 180_000 ms     (3 min: idle connections above minimumIdle are closed first)
maxLifetime   = 1_500_000 ms   (25 min, inside the server's limits and typical LB lifetimes)

Why half. A keepalive at exactly the idle timeout races the middlebox; at half, two keepalives occur within every idle window, so a single lost packet does not lose the connection.

Keepalive cost. Each keepalive is one round trip per connection per interval:

keepalive_queries_per_second = pool_connections ÷ keepalive_interval_seconds

Ten connections at 120 seconds is 0.08 queries per second per process — negligible. Even a fleet of 50 processes generates four per second.

Connections at the minimum are the ones that matter. idleTimeout closes connections above minimumIdle; those at the minimum stay forever and are exactly the ones a middlebox will drop. If minimumIdle is zero, the pool empties during quiet periods and the problem disappears — at the cost of connection latency on the first request after idleness.

connections_at_risk = minimumIdle       (per process, during quiet periods)

Exact remediation & configuration

HikariCP, with TCP keepalives as a second layer:

spring:
  datasource:
    url: jdbc:postgresql://orders-db.internal:5432/orders?ApplicationName=orders-api&tcpKeepAlive=true&socketTimeout=30&loginTimeout=5
    hikari:
      minimum-idle: 2
      maximum-pool-size: 16
      keepalive-time: 120000        # must be less than max-lifetime
      idle-timeout: 180000
      max-lifetime: 1500000
      connection-timeout: 3000
      validation-timeout: 1500
      initialization-fail-timeout: -1

keepaliveTime makes HikariCP run its aliveness check — Connection.isValid(), or connectionTestQuery when configured — against each idle connection on that interval. It must be smaller than maxLifetime, and HikariCP disables it silently if it is not.

TCP keepalives are complementary, not redundant. The driver’s tcpKeepAlive=true asks the operating system to send keepalive probes, which also refresh middlebox state. Their default interval, however, is often two hours — far too long to help — so tune the socket options at the OS level if you rely on them:

# Linux: first probe after 120 s, then every 30 s, 3 failures before giving up
sysctl -w net.ipv4.tcp_keepalive_time=120
sysctl -w net.ipv4.tcp_keepalive_intvl=30
sysctl -w net.ipv4.tcp_keepalive_probes=3

In containers, set these through the pod’s securityContext.sysctls or a node-level tuning DaemonSet, because they are namespaced per pod on modern kernels.

Driver-specific socket timeouts turn an infinite block into an error. With PostgreSQL’s JDBC driver, socketTimeout=30 aborts a read that produces nothing for 30 seconds — the difference between a request that fails in half a minute and one that hangs for the TCP retransmission window. MySQL’s Connector/J has socketTimeout too, and SQL Server’s driver uses socketTimeout in milliseconds.

Where the middlebox cannot be avoided, consider making the pool empty during quiet periods:

      minimum-idle: 0          # no permanently idle connections to lose
      idle-timeout: 60000

The trade is that the first request after a quiet period pays connection establishment, which for a TLS connection to a distant database can be tens of milliseconds.

Verifying the fix and keeping it

Rehearse the drop. The cleanest test is to make the middlebox forget deliberately. On a staging environment, drop the connection state with an iptables rule on the path, or use a load balancer with a short idle timeout, then issue a request:

# On a Linux host acting as the NAT device, flush conntrack for the database port
sudo conntrack -D -p tcp --dport 5432

With the fix in place, the next request must succeed: HikariCP’s keepalive has already replaced any connection whose validation failed. Without it, the request hangs or fails with an I/O error — which is what makes the test meaningful.

Watch the pool’s own counters. HikariCP’s Micrometer metrics distinguish connections created from connections timed out; a rise in hikaricp_connections_creation during quiet periods means keepalives are finding dead connections and replacing them, which is the mechanism working:

rate(hikaricp_connections_creation_seconds_count{pool="orders-pool"}[15m])

A small, steady rate during idle hours is healthy. A rate near zero with occasional errors after idleness means keepalives are not running — usually because keepaliveTime exceeds maxLifetime and was disabled.

Confirm the pool and the server agree. A scheduled check comparing the pool’s connection count with the server’s session count catches silent drops before users do:

SELECT count(*) AS server_sessions FROM pg_stat_activity WHERE application_name = 'orders-api';

sum(hikaricp_connections{pool="orders-pool"})   # should match, allowing for scrape skew

Re-check after any network change. Moving to a different subnet, adding a service mesh, switching to a private endpoint or introducing a proxy all change the smallest idle timeout in the path. The settings above are derived from that number, so a change to it invalidates them.

Frequently Asked Questions

Related

• HikariCP Configuration Deep Dive — the parent guide to HikariCP’s parameters.
• Tuning HikariCP maxLifetime and idleTimeout — the lifetime settings keepalives sit inside.
• Tuning MySQL wait_timeout and Pool Idle Settings — the same ordering problem with a MySQL server-side timeout.
• Configuring SQLAlchemy pool_recycle for AWS RDS — the Python equivalent of retiring connections before the network does.