Handling Aurora Failover in Connection Pools

This guide is part of AWS Aurora Connection Scaling. Aurora promotes a replica in roughly 30 seconds. Applications routinely take five to fifteen minutes to recover, and the whole of that gap is connection pool behaviour.

The mechanism is specific. Aurora failover does not move an IP address — it repoints the Aurora cluster endpoint’s DNS record at the new writer. Your pool is holding TCP connections to the old writer’s address, which is still up, still accepting connections, and now a reader. Writes through those connections fail with cannot execute INSERT in a read-only transaction, or with ERROR: 25006, and the pool has no reason to think anything is wrong with the connection itself.

The three things that delay recovery

DNS caching. The Aurora cluster endpoint has a 5-second TTL, which is short. The JVM’s default DNS cache is not: with a security manager installed, networkaddress.cache.ttl defaults to caching successful lookups forever. Even without one, the default is 30 seconds, and many container base images and libraries cache longer.

Long-lived pooled connections. A connection established before the failover stays connected to the old host indefinitely. Nothing in a pool re-resolves DNS for an existing connection; re-resolution only happens when a new connection is opened.

Silent failure mode. The old writer accepts the connection and answers reads. Only writes fail, and they fail with an error the pool does not classify as a connection failure — so the connection is returned to the pool as healthy and handed out again.

The pool holds addresses, not names Before failover the pool's connections point at instance A, the writer. After failover DNS points the Aurora cluster endpoint at instance B, but the existing TCP connections still terminate on instance A, which is now a reader and rejects writes. DNS moves; established TCP connections do not before failover application pool 20 open connections instance A writer — writes succeed instance B reader after failover application pool same 20 connections instance A now a reader — writes fail instance B writer — DNS points here no traffic Recovery requires the pool to close those connections and open new ones, which only happens when a connection is retired by age, invalidated by an error, or the pool is deliberately reset.
The pool holds sockets to a specific instance. Only closing and reopening them follows the DNS change.

Fix one: make DNS re-resolution fast

On the JVM, set the caching TTLs explicitly. Do not rely on the defaults:

// as early as possible in main(), before any DataSource is created
java.security.Security.setProperty("networkaddress.cache.ttl", "5");
java.security.Security.setProperty("networkaddress.cache.negative.ttl", "1");

Or via JVM flags, which is more reliable in containers because it cannot be missed by a class-loading order problem:

-Dsun.net.inetaddr.ttl=5 -Dsun.net.inetaddr.negative.ttl=1

In Go and Node the resolver does not cache beyond the OS, so this is generally not an issue — but check for a caching layer in your container’s nsswitch configuration or a sidecar DNS proxy, both of which can add caching you did not ask for.

Fix two: cap connection age

A short connection lifetime is what forces new connections — and therefore new DNS lookups — on an ongoing basis. This is the single most effective setting:

spring:
  datasource:
    hikari:
      max-lifetime: 180000        # 3 minutes
      keepalive-time: 60000
      connection-timeout: 3000
      validation-timeout: 2000

Three minutes means that within three minutes of a failover, every connection has been replaced and re-resolved without any error handling being involved. That is a worst case, not an average: HikariCP staggers retirement with jitter, so most connections turn over sooner.

The same setting in Go and Node:

db.SetConnMaxLifetime(3 * time.Minute)
db.SetConnMaxIdleTime(60 * time.Second)
const pool = new pg.Pool({ maxLifetimeSeconds: 180, idleTimeoutMillis: 60_000 });

The trade-off is reconnect cost. A three-minute lifetime on a pool of 20 means about seven new connections a minute — trivial. Do not push it below a minute; at that point you are paying handshake cost constantly to shorten a recovery window that error handling should be covering anyway. The general reasoning is in tuning ConnMaxLifetime and ConnMaxIdleTime in Go.

Fix three: evict on the read-only error

Age-based recovery is a backstop measured in minutes. Detecting the specific error and evicting the connection immediately is what turns minutes into seconds.

PostgreSQL raises SQLSTATE 25006 (read_only_sql_transaction) when a write is attempted on a reader. That is an unambiguous signal that this connection is pointed at the wrong instance:

@Around("@annotation(org.springframework.transaction.annotation.Transactional)")
public Object evictOnReadOnly(ProceedingJoinPoint pjp) throws Throwable {
    try {
        return pjp.proceed();
    } catch (Exception e) {
        SQLException sql = findSqlException(e);
        if (sql != null && "25006".equals(sql.getSQLState())) {
            log.warn("write on a demoted instance — soft-evicting the pool");
            hikariPoolMXBean.softEvictConnections();
        }
        throw e;
    }
}

softEvictConnections() marks every connection for retirement: in-use connections are closed when returned, idle ones immediately. The pool refills from a fresh DNS lookup. One request fails and the rest of the fleet recovers within a second or two rather than within maxLifetime.

The Go equivalent is to close and recreate, or simply to let a short lifetime handle it — database/sql has no eviction API. In Node, pool.end() followed by creating a new pool works but is disruptive; a shorter maxLifetimeSeconds is usually the better trade there.

Fix four: use the right endpoint

Endpoint Resolves to Use for
Cluster (writer) endpoint current writer, updated on failover all writes, and reads needing the newest data
Reader endpoint round-robin across readers read-only workloads
Instance endpoint one specific instance, never moves diagnostics only

Instance endpoints in a production connection string are a common and serious mistake. They do not follow failover at all, so a pool pointed at an instance endpoint that gets demoted stays broken until someone changes configuration.

The reader endpoint has its own subtlety: it round-robins at DNS resolution time, so a pool that opens all its connections in the first second will land most of them on whichever reader DNS returned then. A short connection lifetime spreads them over time and produces a much better distribution across readers — a second, independent reason to cap connection age. Splitting read traffic onto that endpoint is covered in configuring Spring Boot read replica DataSource routing.

Three endpoints, three behaviours The Aurora cluster endpoint follows the writer across a failover. The reader endpoint round-robins across replicas at resolution time. An instance endpoint never moves and leaves a pool stranded after a demotion. Only two of the three endpoints survive a failover cluster endpoint always the current writer updated when promotion completes use it for every write path the default choice reader endpoint round-robins across readers resolved once per new connection cap connection age to spread load use it for read-only work instance endpoint pinned to one instance forever does not follow a promotion stranded after a demotion diagnostics only Naming a single instance in a production connection string is the one failure that no pool setting can recover from. The pool will keep reconnecting successfully to a host that is no longer the writer.
Cluster and reader endpoints follow the topology. An instance endpoint does not, and no pool setting rescues it.

Common failure patterns and remediation

Symptom Underlying cause Remediation
Writes fail for 10+ minutes after failover JVM DNS caching plus long connection lifetime Set networkaddress.cache.ttl; cap max-lifetime at 3 minutes
cannot execute INSERT in a read-only transaction persists Connection still bound to the demoted instance Evict on SQLSTATE 25006, or wait for the age cap
Recovery never happens without a restart Instance endpoint in the connection string Use the Aurora cluster endpoint
All connections land on one reader Reader endpoint resolved once at startup Cap connection age so resolution repeats
Failover storms the new writer Whole fleet reconnecting simultaneously Rely on pool lifetime jitter; stagger deploys
Health check passes while writes fail Health check runs a read Make the write path’s health check attempt a write
Aurora Serverless v2 rejects connections after failover Capacity dropped, so max_connections dropped Set a minimum capacity that covers the fleet’s connection count
Application recovers but latency stays high for minutes Cold buffer cache on the promoted instance Expected; not a pool problem

Verifying recovery time

The only way to know your recovery time is to cause a failover and measure it. Aurora supports this directly:

aws rds failover-db-cluster --db-cluster-identifier orders-prod

Run it during a load test and record three timestamps: when writes start failing, when the first write succeeds again, and when the error rate returns to zero. The gap between the second and third is the pool’s recovery tail, and it is the number these settings shorten.

Test it in staging on a schedule. Failover behaviour depends on the DNS cache configuration of whatever container image you happen to be running, and image upgrades change it silently. A quarterly game day catches that; an incident is a worse way to find out.

Instrument the failover path too. Aurora publishes an RDS event when a failover begins, and the AWS JDBC Driver for MySQL and PostgreSQL can shorten failover handling further by maintaining its own topology awareness — worth evaluating if your recovery target is under ten seconds.

Where recovery time actually goes Two timelines from the moment of failover. With defaults, DNS caching and long connection lifetime dominate, so writes recover after many minutes. With a short DNS TTL, a short connection lifetime and error-driven eviction, recovery follows shortly after promotion completes. Promotion is fast; the pool decides how long the outage lasts defaults promotion 30 s DNS still cached connections never retired — writes keep failing tuned promotion 30 s DNS 5 s evict on 25006 pool refilled against the new writer — writes succeeding Without eviction, the age cap still bounds recovery — at three minutes rather than at seconds. Both are enormous improvements on the default, which has no bound at all.
Aurora's 30-second promotion is rarely the problem. DNS caching and connection age are where the outage actually lives.

Operational boundary

These settings shorten recovery; they do not eliminate the failure. Some requests will fail during a failover, and the correct application behaviour is to retry idempotent writes with jittered backoff rather than to surface the error. Non-idempotent writes need an idempotency key so a retry is safe — that is application design, and no pool setting substitutes for it.

RDS Proxy changes the picture substantially: it holds the connections to the database, tracks the topology itself, and preserves client connections across a failover, which cuts application-visible failover time considerably. It costs money and adds a hop, and it introduces pinning behaviour of its own — the trade is covered in AWS RDS Proxy Connection Pooling.

Frequently Asked Questions

Why do writes fail but reads succeed after a failover?
Your pooled connections still reach the old writer, which is now a reader. Reads work there; writes are rejected with SQLSTATE 25006.
What connection lifetime should I use with Aurora?
Three minutes is a good balance — short enough to bound recovery, long enough that reconnect cost is negligible. Below one minute you are paying constant handshake cost for diminishing benefit.
Does the JVM really cache DNS forever?
With a security manager installed, successful lookups are cached indefinitely by default. Without one the default is 30 seconds. Either way, set it explicitly rather than relying on the environment.
Is softEvictConnections safe to call in production?
Yes. It retires connections gracefully — in-use ones are closed on return, not interrupted. The cost is a brief burst of reconnects.
Should I use the AWS JDBC driver?
It is worth evaluating if your recovery target is under ten seconds. It maintains cluster topology awareness and can fail over faster than DNS-based recovery allows.
Does RDS Proxy remove the need for this?
It substantially reduces application-visible failover time because the proxy holds the database connections and follows the topology. It does not remove the need for retry logic on writes.
What about Aurora Serverless v2?
The same connection behaviour applies, plus a capacity dimension: max_connections scales with capacity, so a scale-down after failover can reject connections. Set minimum capacity from your fleet’s connection count.