Architecture
Background Jobs and Queues: Designing Work That Happens Later
Updated August 18, 2016By the CalliArc team
Key takeaway
Every job must be idempotent and safe to retry, because it will run twice. Pass identifiers rather than objects, keep jobs small, use separate queues by priority, and alert on queue depth and age — a silently growing backlog is the most common failure and the least visible.
Sending an email, generating a report, or calling a third-party API inside a web request ties your response time to systems you don't control. Moving that work to a queue is straightforward; making it reliable requires a few deliberate decisions.
Design of the job itself
- Pass an identifier, not a serialised object. By the time the job runs, the record may have changed — and it should act on the current state.
- Make it idempotent: running it twice must produce the same outcome as running it once. At-least-once delivery is the norm.
- Keep jobs small and single-purpose. A job that does five things fails partway through and leaves you with no clean retry.
- Set an explicit timeout, or one stuck job occupies a worker indefinitely.
- Handle the case where the referenced record has been deleted before the job runs.
Retries and failure
- Exponential backoff with jitter, and a maximum attempt count.
- Distinguish retryable failures (a timeout, a rate limit) from permanent ones (invalid data) — retrying a permanent failure twenty times wastes capacity and delays everything else.
- A dead letter queue for exhausted jobs, with an alert. Jobs failing silently into nowhere is the classic production surprise.
- Make failures inspectable and replayable once the underlying problem is fixed.
Queue structure
- Separate queues by priority and by latency expectation. A bulk import must not delay password reset emails.
- Separate slow, long-running work from fast work so one doesn't starve the other.
- Scale workers per queue independently.
- Consider ordering requirements explicitly — most queues don't guarantee order, and jobs that assume it will produce rare, confusing bugs.
Operating it
- Alert on queue depth and, more importantly, on the age of the oldest item. Depth alone hides a slowly draining backlog.
- Track job duration and failure rate per job type.
- Make sure a deployment drains or safely interrupts in-flight jobs rather than killing them mid-write.
- Have a way to pause a queue — when a third party is failing, stopping is better than burning through retries.
- Schedule cleanup of completed job records, which grow surprisingly fast.