Closing ORM Connections in Background Job Workers

This guide is part of ORM Connection Lifecycle Hooks. Web frameworks close database connections for you at the end of every request. Job workers have no equivalent boundary, and that single difference accounts for most of the connection leaks that appear in Celery, Sidekiq, RQ, Bull, and hand-rolled consumer loops.

The pattern is always the same. A web request has a well-defined lifecycle with framework hooks at both ends; the ORM registers there and returns the connection when the response is sent. A job worker has a loop. Nothing in that loop is a request, so nothing fires the hook, and the connection borrowed by job number one is still checked out when job number four thousand runs.

Why workers leak when web servers do not

Four properties of workers combine badly:

No request boundary. The ORM’s teardown is registered on a framework signal that never fires. Django’s request_finished closes connections above CONN_MAX_AGE; a Celery task is not a request.

Long-lived processes. A web worker is recycled after a few thousand requests by most process managers. A job worker can run for weeks, so a connection borrowed once is held for weeks.

Fork inheritance. Prefork job runners — Celery’s default, Sidekiq under some configurations, Resque — fork child processes. A pool created in the parent before the fork is inherited by every child, and multiple processes then share the same TCP sockets. The protocol corrupts in ways that produce baffling errors: results from another process’s query, SSL error: decryption failed, or a hang.

High process counts. A worker fleet is usually sized for throughput, not for connections. Sixteen workers with concurrency eight is 128 processes, and if each holds even one connection that is 128 backends before the web tier has asked for anything.

Django and Celery

Django’s connection handling assumes requests. Celery’s task_postrun signal is the equivalent boundary:

from celery.signals import task_postrun, worker_process_init
from django.db import close_old_connections, connections

@task_postrun.connect
def close_connections_after_task(**kwargs):
    # Honours CONN_MAX_AGE: closes only connections that have expired,
    # so short tasks still reuse a warm connection.
    close_old_connections()

@worker_process_init.connect
def reset_pool_after_fork(**kwargs):
    # Discard anything inherited from the parent process.
    for conn in connections.all():
        conn.close()

close_old_connections() rather than a blanket close is the right call: it respects CONN_MAX_AGE, so a worker processing many short tasks keeps a warm connection instead of reconnecting per task. Using connections.close_all() in task_postrun works but pays a full connection setup on every task, which for a task that runs in 40 ms is most of the runtime.

The worker_process_init handler is the fork fix, and it is not optional under Celery’s default prefork pool. Without it, every child inherits the parent’s connections and the sockets are shared.

If you have moved to Django’s built-in pool, the same signals apply but the reasoning shifts — the pool is per process, and close_old_connections returns the connection to it rather than closing a socket. Configuration for that mode is in enabling Django connection pooling with psycopg3.

SQLAlchemy workers

SQLAlchemy’s session is the unit that holds the connection, so the boundary is the session’s scope:

from contextlib import contextmanager

SessionLocal = sessionmaker(bind=engine, expire_on_commit=False)

@contextmanager
def job_session():
    session = SessionLocal()
    try:
        yield session
        session.commit()
    except Exception:
        session.rollback()
        raise
    finally:
        session.close()          # returns the connection to the pool

def handle_job(payload):
    with job_session() as s:
        s.add(Result(**payload))

The finally: session.close() is the whole fix. A session that goes out of scope without being closed eventually returns its connection when garbage collected, but “eventually” is not a guarantee you can size a pool against.

For forking runners, dispose the engine in the child rather than inheriting the parent’s pool:

from celery.signals import worker_process_init

@worker_process_init.connect
def dispose_engine(**kwargs):
    engine.dispose(close=False)   # abandon inherited connections without closing
                                  # the parent's sockets from the child

close=False matters here. A plain dispose() in a forked child closes sockets that the parent still believes it owns. close=False abandons the child’s references without touching the underlying connections, which is the correct semantics after a fork.

Rails and Sidekiq

ActiveRecord’s checkout is per thread, and Sidekiq runs a thread per concurrency slot. Sidekiq wraps each job in with_connection by default in recent versions, but explicit scoping is still worth it for jobs that do non-database work:

class ReportJob
  include Sidekiq::Job

  def perform(report_id)
    report = ActiveRecord::Base.connection_pool.with_connection do |_conn|
      Report.find(report_id)
    end

    # Deliberately outside the checkout: this is a slow HTTP call and
    # must not hold a database connection while it runs.
    payload = ExternalRenderer.render(report)

    ActiveRecord::Base.connection_pool.with_connection do
      report.update!(rendered_at: Time.current, payload: payload)
    end
  end
end

The pool size must be at least Sidekiq’s concurrency, or threads block waiting on each other:

# config/sidekiq.yml
:concurrency: 10

# config/database.yml
production:
  pool: <%= ENV.fetch("RAILS_MAX_THREADS", 10).to_i %>

Sidekiq’s own internal threads also need connections for a handful of operations, so a pool exactly equal to concurrency is tight. Concurrency plus two is a safer floor. The full arithmetic for the Rails side is in Rails ActiveRecord Connection Pool.

Node job runners

With BullMQ or a hand-rolled consumer, the pattern is a finally release, exactly as in an HTTP handler:

worker.on('active', () => metrics.inc('jobs_active'));

async function process(job) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    await handle(job.data, client);
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK').catch(() => {});
    throw err;
  } finally {
    client.release();     // never conditional, never skipped
  }
}

The .catch(() => {}) on the rollback is deliberate: if the connection is already broken, the rollback throws, and an unhandled rejection there masks the original error and skips the release.

Node workers do not fork by default, so the inheritance problem does not arise — but if you use the cluster module, create the pool in the worker process, never in the primary.

Bounding the worker fleet’s connections

Workers are usually the larger half of a fleet’s connection consumption, and the arithmetic is rarely done:

web:     6 hosts x 4 processes x pool 5    =  120
workers: 4 hosts x 8 processes x pool 2    =   64
beat / scheduler singleton                 =    2
migrations and operations                  =   25
                                    total  =  211  against a ceiling of 185

Two levers bring that down. Give workers a smaller pool than the web tier — they are throughput-bound, not latency-bound, so paying a connection setup occasionally is acceptable. And give them their own database role with a hard cap, so a runaway worker deployment cannot consume the web tier’s share:

ALTER ROLE app_worker CONNECTION LIMIT 70;

That role-level cap is the only containment that works regardless of what the worker configuration says, and it converts a fleet-wide outage into a worker-tier one. Deriving the total budget is covered in tuning PostgreSQL max_connections and superuser_reserved_connections.

Common failure patterns and remediation

Detecting it before it exhausts

The server-side signature of a worker leak is distinctive: many backends in state = 'idle' with no transaction open, attributed to the worker role or application name, with old backend_start timestamps.

SELECT application_name,
       usename,
       count(*)                                   AS conns,
       max(now() - backend_start)::interval(0)    AS oldest,
       count(*) FILTER (WHERE state = 'idle')     AS idle
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY 1, 2
ORDER BY conns DESC;

Set a distinct application_name per process type — orders-web, orders-worker, orders-beat — and this query names the culprit immediately. Without it, every connection looks the same and you are guessing. Reading the rest of that view is covered in interpreting pg_stat_activity wait events for pool issues.

Operational boundary

These hooks return connections at task boundaries. They do nothing about a task that holds a connection for its entire duration while doing slow non-database work — that is a scoping problem inside the task, and it is the more common cause of worker pool pressure once the teardown hooks are in place.

They also cannot help a worker fleet that is simply too large for the database. Past the point where process count times minimum pool size exceeds the ceiling, no amount of careful teardown helps, and the answer is a proxy or fewer processes.

Frequently Asked Questions

Related

• ORM Connection Lifecycle Hooks — the parent guide on where ORMs acquire and release.
• Enabling Django Connection Pooling with psycopg3 — how these hooks behave with a real pool.
• Rails ActiveRecord Connection Pool — pool size against Sidekiq concurrency.
• Tuning PostgreSQL max_connections and superuser_reserved_connections — budgeting the worker tier’s share.
• Interpreting pg_stat_activity Wait Events for Pool Issues — confirming a leak from the server side.