Fixing ActiveRecord ConnectionTimeoutError in Puma

This guide is part of Rails ActiveRecord Connection Pool. ConnectionTimeoutError is Rails’ pool-exhaustion signal, and in a Puma deployment it has three causes with very different fixes. Two of them are configuration mismatches that take one line to correct; the third is a genuine capacity problem. Telling them apart takes about two minutes.

ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool
within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use):
  app/controllers/orders_controller.rb:24:in `show'
  …

The message names the wait and the timeout, and says nothing about why. The database is almost always healthy when this fires — its own metrics will show low CPU and few active backends, which is what makes the error confusing the first time it appears.

Rapid Incident Diagnosis

Four readings, in order. Each rules out one cause.

1. Compare pool size to thread count. This is the first check because it is the most common cause and the cheapest to verify:

# From `rails runner` on an affected instance
puts ActiveRecord::Base.connection_pool.size          # => 5
puts Puma.stats_hash[:worker_status].first[:last_status][:max_threads]  # => 16

A pool of 5 serving 16 threads guarantees queueing at any concurrency above five, regardless of database capacity. The fix is one line in database.yml.

2. Read the pool’s own stat hash. ActiveRecord::Base.connection_pool.stat gives busy, idle, waiting and dead. A non-zero waiting with busy equal to size confirms exhaustion; dead above zero means threads are being killed while holding connections, which is a separate problem.

3. Check the database’s view. SELECT state, count(*) FROM pg_stat_activity WHERE application_name LIKE 'orders-web%' GROUP BY state. Predominantly active means the connections are working and the pool is genuinely too small. Predominantly idle in transaction means they are held by application code doing something else.

4. Check the fleet total against the allowance. If the per-process configuration is right but the fleet as a whole has exhausted its share, raising the pool makes it worse rather than better.

Three causes, distinguished by two readings Comparing pool size to thread count identifies a configuration mismatch. Backend state identifies whether connections are working or held. The fleet total identifies an allowance problem. pool < threads threads queue against each other pool 5, threads 16 → raise pool to threads + 1 costs nothing at the database connections held, not working backends idle in transaction slow render or external call → narrow the checkout scope a bigger pool only delays it fleet exhausted its allowance replicas × workers × pool too high FATAL: too many clients → fewer workers, more threads raising the pool makes it worse The left cause and the right cause call for opposite changes, which is why the diagnosis has to precede the fix.
Two of the three causes point toward a larger pool and one points firmly away from it, which is why raising the value before diagnosing is a coin flip.

Cause 1 — Pool Smaller Than Thread Count

Puma’s thread count and ActiveRecord’s pool size are configured in different files, and it is easy for them to drift apart — particularly when RAILS_MAX_THREADS is set for Puma but database.yml carries a hard-coded pool: 5 from an old generated file.

# config/database.yml — tie the two together explicitly
production:
  adapter: postgresql
  database: orders_production
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 }.to_i + 1 %>   # threads + 1
  checkout_timeout: 3
  idle_timeout: 300
  variables:
    application_name: <%= ENV.fetch("APP_TIER", "orders-web") %>
# config/puma.rb — the same variable drives both
max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }.to_i
threads max_threads_count, max_threads_count
workers ENV.fetch("WEB_CONCURRENCY") { 2 }.to_i
preload_app!

before_fork  { ActiveRecord::Base.connection_pool.disconnect! }
on_worker_boot { ActiveRecord::Base.establish_connection }

The + 1 covers Rails-internal threads and any periodic task in the process. The before_fork and on_worker_boot hooks matter independently: without them a preloaded application’s connections are inherited by every forked worker, which produces protocol corruption rather than a clean error.

Raising the pool here does not increase the fleet’s peak connection count beyond threads × workers × replicas, because the threads were already the concurrency bound — it removes queueing that was serving no purpose.

Cause 2 — Connections Held Through Non-Database Work

ActiveRecord checks out on the first query and holds until the end of the request. Everything after the last query — view rendering, JSON serialisation, an external API call — is time a connection is held for no database reason.

The tell is on the database side: idle in transaction backends, or idle backends whose count matches the thread count while the application reports waiting.

Two fixes, in order of effect:

Move external calls out of the request path or, if they must stay, out of any open transaction. A controller that calls a payment gateway between two queries holds a connection for the gateway’s full latency.

Return the connection explicitly before slow work where the structure allows it:

def show
  @order = Order.includes(:line_items).find(params[:id])   # all queries here
  ActiveRecord::Base.connection_pool.release_connection    # release before rendering

  render json: OrderSerializer.new(@order).as_json         # no connection held
end

That pattern is worth using sparingly — it is easy to release a connection that a later lazy-loaded association still needs, which produces a checkout in the middle of rendering and defeats the purpose. includes to eager-load everything first is what makes it safe.

Cause 3 — The Fleet Has Exhausted Its Allowance

If the per-process numbers are correct and the database is reporting too many clients, the fix is structural rather than a setting.

The most effective change in a Rails deployment is usually to shift the balance from workers to threads. Puma workers are separate processes, each with its own pool; threads share one. Moving from 4 workers × 5 threads to 2 workers × 10 threads keeps the same request concurrency while halving the pool count — and it usually reduces memory as well, since fewer copies of the application are loaded.

Configuration Request Concurrency Pools Connections per Replica
4 workers × 5 threads 20 4 24
2 workers × 10 threads 20 2 22
1 worker × 20 threads 20 1 21
2 workers × 10, pool 11 20 2 22

The gains look modest per replica and compound across the fleet: at 12 replicas, the first row is 288 connections and the third is 252, for identical serving capacity. On JRuby, where threads are genuinely parallel, the single-worker configuration is usually strictly better.

Beyond that, the remaining option is a multiplexing proxy, which removes the worker and replica count from the arithmetic entirely — covered in Configuring ActiveRecord with PgBouncer Transaction Mode.

Workers versus threads at equal concurrency Four workers of five threads and two workers of ten threads serve the same concurrency, but the first holds four pools per replica and the second holds two. 4 workers × 5 threads — 4 pools worker — pool 6 worker — pool 6 worker — pool 6 worker — pool 6 24 2 workers × 10 threads — 2 pools, same concurrency worker — pool 11 worker — pool 11 22 Two connections saved per replica, and one fewer copy of the application in memory On MRI the trade is bounded by the GIL; on JRuby the single-worker configuration is usually strictly better
Shifting concurrency from workers to threads reduces the pool count without reducing serving capacity, and the saving compounds across the fleet.

Validation and Verification

For the pool-versus-threads fix, confirm waiting returns to zero under the load that triggered the error:

# Sample every few seconds during a peak
s = ActiveRecord::Base.connection_pool.stat
puts "busy=#{s[:busy]} idle=#{s[:idle]} waiting=#{s[:waiting]} size=#{s[:size]}"
# healthy at peak: busy up to size, waiting 0

For the checkout-scope fix, compare the database’s state breakdown before and after. Backends should shift from idle in transaction toward active, and the total should fall for the same throughput.

For the worker-to-thread rebalance, verify both that the fleet’s connection count dropped and that request throughput and latency are unchanged — the change is only worth making if concurrency is genuinely preserved.

Waiting count before and after Before the change, threads wait for connections during every peak. After aligning the pool with the thread count the waiting series is flat at zero, while busy still reaches the pool size. 0 pool raised from 5 to 17 waiting, every peak flat at zero through the same peaks The fleet's peak connection count is unchanged — the threads were always the bound, and now they are not queueing for no reason.
Aligning the pool with the thread count removes queueing without raising peak connection use, because the thread count was already the concurrency ceiling.

Common Failure Patterns & Remediation

Symptom Root Cause Exact Fix Validation
ConnectionTimeoutError with database idle Pool smaller than Puma thread count pool = RAILS_MAX_THREADS + 1 stat[:waiting] returns to zero
Sidekiq raises it, web tier fine Worker tier inherited the web pool value Tier-specific environment variable Worker waiting zero at full concurrency
FATAL: too many clients after a scale-out Fleet total exceeds the allowance Fewer workers, more threads; or a proxy Backends under 80% of the limit at max scale
Error only during slow-dependency incidents Connections held through an external call Move the call outside the request or transaction Backends shift from idle-in-transaction to active
stat[:dead] non-zero Threads killed while holding connections Investigate request timeouts and OOM kills dead returns to zero
Every worker fails right after a deploy Preloaded connections inherited across the fork before_fork disconnect, on_worker_boot reconnect Workers start cleanly on every deploy
Connection count never falls at trough No idle_timeout configured Set it to a few minutes Trough backend count falls toward the idle floor

The second row is worth emphasising because it accounts for a large share of these reports. Sidekiq and Puma load the same database.yml, and a value that is correct for five Puma threads produces this error immediately in a Sidekiq process configured for ten concurrent jobs — with no change to the web tier at all, which is what makes it look like an unrelated problem.

Frequently Asked Questions

Why does the error appear only on some instances?
Usually because load is unevenly distributed, or because one instance has a long-running request holding connections. If it persists on a single instance across restarts, check whether that instance’s RAILS_MAX_THREADS differs — a per-instance environment override is a common cause.
Should checkout_timeout be raised to make the error go away?
No. It converts a fast failure into a slow one and holds Puma threads for longer, which reduces the instance’s capacity to serve anything else. If anything it should be lowered, so a shortage sheds load quickly rather than accumulating.
Does the error mean the database is overloaded?
Almost never. It means Rails could not obtain a connection from its own pool, which is a client-side condition. Check the database’s active backend count before assuming otherwise — it is usually low when this fires.
How does Sidekiq fit into this?
Sidekiq’s concurrency is its thread count, and it reads the same database.yml. A pool sized for five Puma threads will produce exactly this error in a Sidekiq process running ten. Drive the pool from a tier-specific environment variable so each process gets the right value.
Does reaping_frequency help?
It reclaims connections whose owning thread died, which addresses the dead count rather than waiting. Leaving it at the default is right; it is a safety net for killed threads, not a remedy for exhaustion.