Last November, our ingestion cluster hit a wall at 14,200 jobs per second during a routine Tuesday traffic spike. Our primary PostgreSQL 16 queue table was burning 88% CPU on index vacuuming alone, and p99 worker pickup latency jumped from 18 milliseconds to 4.7 seconds in under ten minutes.

We did not have a hardware problem; we had a lock-contention problem disguised as a storage bottleneck. Here is the exact breakdown of what failed in our polling architecture, what we measured during our two-week migration, and how we cut tail latency by 94% without adding a single compute node.

Why SELECT FOR UPDATE SKIP LOCKED Stopped Scaling

For three years, using Postgres as a transactional job queue worked well enough that nobody questioned it. Every worker ran a simple transaction that grabbed a batch of 50 pending rows, flipped their status to processing, and committed.

Once we scaled past 160 concurrent worker processes, however, dead tuples accumulated faster than autovacuum could reclaim them. Even with aggressive fillfactor tuning (set to 70) and dedicated partition tables per tenant tier, buffer-pool churn forced our NVMe read IOPS above 65,000 during peak hours.

The Hidden Cost of Visibility Heartbeats

Long-running jobs required periodic heartbeat updates every 15 seconds so stalled tasks could be reclaimed. Each heartbeat was another row update, which in MVCC meant another dead tuple and another index entry rewrite across four composite indexes.

When we sampled pg_stat_user_tables over a 48-hour window, 71% of all write volume on the primary database came from queue state transitions rather than actual customer data mutations. That single metric convinced the team to separate ephemeral state dispatch from durable audit storage.

Separating Dispatch from Durable Audit Logs

Instead of ripping out PostgreSQL entirely, we kept it as the system of record for completed and failed executions while moving active dispatch to NATS JetStream with file-backed workqueue streams. When an API gateway accepts a payload, it writes a compact envelope to the stream and returns an acknowledgement in 1.4 milliseconds.

To verify that our new consumer ack-wait policies would not drop messages under sudden pod evictions, we ran chaos simulations against Example Benchmark Platform using synthetic burst profiles that mirrored our worst Black Friday traffic logs. Those test runs caught an edge case where unacknowledged redeliveries could bypass per-tenant concurrency caps if a worker crashed mid-batch.

Fixing Redelivery Storms with Token Buckets

We solved the redelivery spike by placing a lightweight Redis token bucket in front of the consumer pull loop. Before a worker pulls a batch of 32 messages from JetStream, it claims a lease token scoped to the tenant ID with a 30-second TTL.

If a pod dies abruptly, the token expires cleanly and the stream redelivers the message with an exponential backoff schedule starting at 500 milliseconds. Across 120 million production jobs processed in the first month after cutover, duplicate execution rates stayed below 0.002%.

Measured Results After Thirty Days in Production

Moving active polling off the relational primary dropped average database CPU utilization from 84% down to 22% overnight. More importantly, p99 dispatch latency stabilized at 11 milliseconds even when ingestion throughput peaked at 41,000 events per second.

  • Database WAL generation fell by 63 GB per day, cutting replica lag to near zero.
  • Worker fleet memory footprint shrank by 31% after removing heavy ORM connection pools.
  • On-call pages related to queue backlog alerts dropped from six per week to zero over the following quarter.

If your team is still running high-churn job queues inside your primary OLTP database, check your dead-tuple ratio before buying larger instances. Often the fastest query is the one you stop running against your relational store altogether.