Designing a media optimization queue on Postgres

When building Cloudizero, our media asset optimization SaaS, the initial architectural instinct was to introduce an external message broker like Redis (BullMQ) or RabbitMQ to manage transcoding workloads.

However, operating multiple stateful storage backends in early-stage products adds operational complexity, backup fragmentation, and transactional failure modes. We chose to leverage PostgreSQL’s native capabilities using FOR UPDATE SKIP LOCKED.

The Core Queue Mechanism

By combining transactional state updates with SKIP LOCKED, multiple worker nodes can safely claim distinct pending jobs concurrently without lock contention:

WITH next_job AS (
  SELECT id
  FROM media_jobs
  WHERE status = 'pending' AND scheduled_at <= NOW()
  ORDER BY priority DESC, created_at ASC
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
UPDATE media_jobs
SET status = 'processing', started_at = NOW()
FROM next_job
WHERE media_jobs.id = next_job.id
RETURNING media_jobs.*;

Advantages in Production

1. Atomic Transactions: An asset upload record and its corresponding background job are committed in a single atomic transaction. No orphaned jobs or phantom uploads.
2. Simplified Infrastructure: Zero external broker daemon to monitor, backup, or patch.
3. Auditability: Complete operational history, retry counts, and worker error logs live in standard relational tables accessible via SQL.

For pipelines processing thousands of jobs per hour, Postgres with SKIP LOCKED delivers bulletproof reliability and near-zero operational maintenance.