Configuring Oracle UCP for High Availability

This guide is part of HikariCP vs c3p0 vs DBCP2 Benchmark. Oracle’s Universal Connection Pool exists for one reason that generic JDBC pools cannot match: it participates in the database’s own high-availability machinery. When an Oracle RAC instance goes down or a Data Guard role change happens, the database publishes Fast Application Notification events, and UCP acts on them immediately — removing affected connections, redistributing work to surviving instances and, with Application Continuity, replaying in-flight work. A generic pool learns about the same event only when a query fails on a dead socket, typically after a TCP timeout:

java.sql.SQLRecoverableException: No more data to read from socket
java.sql.SQLException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

The objective is a UCP configuration that receives those events, validates what it hands out, bounds every wait so an outage cannot become an application hang, and distributes load across RAC instances rather than pinning to whichever instance was up at startup.

Rapid incident diagnosis

Check whether FAN is actually on. Without it, UCP is a generic pool with extra settings:

PoolDataSource pds = // ...
System.out.println("fastConnectionFailoverEnabled = " + pds.getFastConnectionFailoverEnabled());
System.out.println("ONS config = " + pds.getONSConfiguration());

With the Oracle 12c and later driver, auto-ONS usually configures itself from the connect descriptor; an explicit setONSConfiguration is needed only when the ONS nodes cannot be discovered.

Read the pool’s own statistics. UCP exposes them through the pool data source and through JMX:

UniversalConnectionPoolManager mgr = UniversalConnectionPoolManagerImpl.getUniversalConnectionPoolManager();
UniversalConnectionPoolStatistics s = mgr.getConnectionPool("orders-pool").getStatistics();
System.out.printf("available=%d borrowed=%d created=%d closed=%d avgWait=%dms peak=%d%n",
    s.getAvailableConnectionsCount(), s.getBorrowedConnectionsCount(),
    s.getTotalConnectionsCount(), s.getConnectionsClosedCount(),
    s.getAverageConnectionWaitTime(), s.getPeakConnectionsCount());

Borrowed at maxPoolSize with rising average wait is saturation. A large getConnectionsClosedCount immediately after an incident is UCP removing connections in response to a FAN down event, which is the pool working correctly.

Distinguish a role change from a node loss. In a RAC environment, gv$instance shows which instances are up, and the pool should have connections to each:

SELECT inst_id, instance_name, status FROM gv$instance ORDER BY inst_id;
SELECT inst_id, count(*) FROM gv$session WHERE program LIKE 'JDBC%' GROUP BY inst_id ORDER BY inst_id;

Connections concentrated on one instance after a node returns means runtime load balancing is not in effect — usually because the service is not configured with a goal, or because connections never recycle.

Look for hangs rather than errors. UCP’s connectionWaitTimeout bounds the borrow; without it, an outage that empties the pool leaves threads waiting. A thread dump showing threads in oracle.ucp.jdbc.PoolDataSourceImpl.getConnection with no timeout configured is the signature.

Failure detection with and without Fast Application Notification With FAN enabled, an instance going down publishes an event that UCP acts on within seconds, removing affected connections. Without FAN, the pool waits for TCP or query timeouts before it notices. Time from instance loss to pool recovery FAN enabled seconds: event-driven removal no FAN until TCP or query timeouts expire, per connection during that window, borrows hand out connections to an instance that is gone which is why validateConnectionOnBorrow matters even with FAN
With FAN, the pool learns about a node loss from the database in seconds. Without it, it discovers the same event only when a query fails on a dead socket.

Mathematical sizing or parameter formula

UCP’s sizing arithmetic is the familiar one, with an Oracle-specific twist: the pool spreads across RAC instances, so per-instance session limits matter as much as the total.

maxPoolSize = ceil( requests_per_second × queries_per_request × mean_query_seconds ÷ 0.75 )
per_instance_sessions ≈ maxPoolSize ÷ live_instances          (with runtime load balancing)

A service at 500 requests per second with 3 queries of 6 ms: 500 × 3 × 0.006 = 9 in flight; maxPoolSize = 12. Across a two-node RAC, each instance sees about 6 sessions from each application instance — and during a node outage, all 12 land on the survivor, which is the number the database’s sessions parameter must accommodate:

worst_case_per_instance = app_instances × maxPoolSize        (one node surviving)

Twelve application instances at maxPoolSize = 12 put 144 sessions on a single surviving node. If sessions on that node cannot take it while also serving other applications, either the pool or the instance sizing is wrong.

Timeouts must nest inside the request deadline:

connectionWaitTimeout < request_deadline − query_time − application_work
inactiveConnectionTimeout < shortest network idle timeout
maxConnectionReuseTime spreads reconnection so a cohort does not expire together

timeToLiveConnectionTimeout is the hard lifetime of a borrowed connection — it reclaims connections held too long, which is UCP’s abandoned-connection mechanism — and must sit above the longest legitimate borrow.

Validation cost. validateConnectionOnBorrow adds a round trip unless the driver can answer it locally; with the Oracle driver, the check is cheap because it uses a ping that avoids a full statement round trip in most cases. During a failover it is what prevents a dead connection from reaching the application before FAN has been processed, so its cost is worth paying in high-availability deployments.

Session distribution across RAC instances before and during an outage With two instances, twelve application instances of twelve connections each spread about 72 sessions per node. When one node fails, all 144 land on the survivor. Sessions per RAC instance normal, two nodes node 1 · 72 node 2 · 72 one node down survivor · 144 size the instance's sessions parameter for the outage case, not the normal case
During a node outage the surviving instance receives every application pool's full maximum, which is the number its session limit must accommodate.

Exact remediation & configuration

A production PoolDataSource for a RAC service, configured in code so every setting is explicit:

PoolDataSource pds = PoolDataSourceFactory.getPoolDataSource();
pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
pds.setURL("jdbc:oracle:thin:@//orders-scan.internal:1521/orders_svc");
pds.setUser("orders_app");
pds.setPassword(secret);
pds.setConnectionPoolName("orders-pool");

// Sizing
pds.setInitialPoolSize(4);
pds.setMinPoolSize(4);
pds.setMaxPoolSize(12);

// Waiting and lifetimes (seconds unless noted)
pds.setConnectionWaitTimeout(3);           // bounded borrow
pds.setInactiveConnectionTimeout(120);     // idle reclamation
pds.setTimeToLiveConnectionTimeout(300);   // reclaim a borrowed connection held too long
pds.setMaxConnectionReuseTime(1800);       // recycle so connections rebalance
pds.setAbandonedConnectionTimeout(120);    // abandoned-connection detection

// Validation and high availability
pds.setValidateConnectionOnBorrow(true);
pds.setFastConnectionFailoverEnabled(true);          // consume FAN events

// Statement caching, on the driver
pds.setConnectionProperty("oracle.jdbc.implicitStatementCacheSize", "50");

Fast Connection Failover requires the database service to publish FAN events and the application host to reach ONS. With the 12c and later driver and a SCAN-based connect descriptor, auto-ONS discovers the endpoints; otherwise set them explicitly:

pds.setONSConfiguration("nodes=rac1.internal:6200,rac2.internal:6200");

Runtime load balancing distributes borrows according to the load advisory the database publishes, which is what keeps a returning node from staying idle. It is enabled on the service side (-rlbgoal SERVICE_TIME on the database service) and consumed automatically by UCP when FAN is on.

Application Continuity, where licensed, replays an interrupted transaction transparently after a failure. It requires a service configured with -failovertype TRANSACTION, a driver that supports it and application code free of side effects that cannot be replayed. When it is available, the combination of FAN and replay turns most instance failures into a latency blip; when it is not, the application must handle recoverable errors itself.

Connection labelling is UCP’s mechanism for reusing connections whose session state matches what the borrower needs — an Oracle-specific alternative to resetting state on every borrow. Implement ConnectionLabelingCallback when the application sets session attributes such as the current schema or NLS settings, so the pool prefers a connection that already has them and avoids a round trip:

pds.registerConnectionLabelingCallback(new SchemaLabelingCallback());
Properties label = new Properties();
label.setProperty("schema", "orders_rw");
Connection c = pds.getConnection(label);

In Spring Boot, UCP is configured through properties with the pool data source type selected explicitly:

spring:
  datasource:
    type: oracle.ucp.jdbc.PoolDataSourceImpl
    oracleucp:
      connection-factory-class-name: oracle.jdbc.pool.OracleDataSource
      connection-pool-name: orders-pool
      initial-pool-size: 4
      min-pool-size: 4
      max-pool-size: 12
      connection-wait-timeout: 3
      inactive-connection-timeout: 120
      max-connection-reuse-time: 1800
      validate-connection-on-borrow: true
      fast-connection-failover-enabled: true

Rehearsing failover

A high-availability configuration that has never been tested is a hypothesis. Three rehearsals cover the failures UCP is meant to absorb.

Instance loss. With load running, stop one RAC instance:

srvctl stop instance -db orders -instance orders2 -failover

Expect: FAN down event within a second or two, a burst of getConnectionsClosedCount, borrows continuing against the surviving instance, and a brief error window only for operations that were in flight. With Application Continuity, even those should replay.

Planned relocation. Relocate the service rather than stopping the instance, which is the more common maintenance operation:

srvctl relocate service -db orders -service orders_svc -oldinst orders1 -newinst orders2

Expect: connections to the old instance drained gracefully as work completes, with no client-visible errors at all. Errors here usually mean validateConnectionOnBorrow is off, or the service was stopped rather than relocated.

Node return. Restart the stopped instance and watch the session distribution recover:

SELECT inst_id, count(*) FROM gv$session WHERE program LIKE 'JDBC%' GROUP BY inst_id;

Sessions should spread back over a few minutes as maxConnectionReuseTime recycles connections. If they do not, that setting is unset or too long — this is the same rebalancing problem that appears with reader endpoints in other databases, described in Routing Reads to Aurora Reader Endpoints with Connection Pools.

Rehearsal Healthy result Common cause of failure
Instance stop Seconds of errors, then normal FAN not enabled; ONS unreachable
Service relocation No client-visible errors Validation on borrow disabled
Node return Sessions rebalance within minutes maxConnectionReuseTime unset
Pool exhaustion Fast, bounded failure connectionWaitTimeout unset
Settings that cover each stage of an Oracle failover Fast Connection Failover handles notification, validate on borrow protects the window before the event is processed, connection wait timeout bounds the pool's behaviour when it empties, and maximum connection reuse time rebalances after recovery. FAN + ONS learn about it validate on borrow survive the gap wait timeout bound the empty pool reuse time rebalance after Four settings, four stages leaving any one unset turns a survivable event into an incident
Each setting covers a different part of the failover path: notification, what is handed out during the gap, and how work returns to a recovered node.

Validation & verification

FAN events are reaching the pool. Enable UCP logging and confirm the events appear during a rehearsal:

# logging.properties
oracle.ucp.level = FINE
oracle.ons.level = FINE

The log should record a service-down event followed by connection removals. No events means ONS is not reachable — check port 6200 from the application host.

Borrows are bounded. With the database stopped entirely, borrows must fail in about connectionWaitTimeout rather than hanging:

long t0 = System.nanoTime();
try (Connection c = pds.getConnection()) { /* ... */ }
catch (SQLException e) { System.out.printf("failed after %d ms: %s%n", (System.nanoTime()-t0)/1_000_000, e.getMessage()); }

Dead connections never reach the application. During the instance-stop rehearsal, count SQLRecoverableException occurrences in application logs. With validation on borrow and FAN enabled, these should be limited to operations that were in flight at the moment of failure.

Sessions stay inside the instance limit. During the single-node rehearsal, confirm the survivor’s session count stays below its limit:

SELECT resource_name, current_utilization, limit_value
FROM v$resource_limit WHERE resource_name IN ('sessions', 'processes');

Frequently Asked Questions

Is UCP worth it without RAC?
On a single instance, most of UCP’s advantages are unused, and a generic pool is simpler. With Data Guard role changes or any configuration that publishes FAN events, UCP’s event handling still pays.
Does validateConnectionOnBorrow duplicate FAN?
No — they cover different windows. FAN removes connections once the event arrives; validation protects the interval between the failure and the event, and covers failures that publish no event at all, such as a network partition.
How does UCP compare with HikariCP on raw acquisition latency?
HikariCP is generally faster on borrow and return, because its lock-free structure was built for exactly that. UCP trades some of that for features HikariCP does not have. On an Oracle database with RAC, the failover behaviour usually matters more than the microseconds.
Can labelling replace resetting session state?
It avoids the reset when a matching connection is available, but the callback must be correct: a labelling implementation that reports a match when the state differs hands the borrower a connection with the wrong session settings, which is a correctness bug rather than a performance one.