Why an application puts work in a message queueSeparate accepting a job from finishing its effect

Split the immediate response from the slow task

A producer creates a job message describing work to perform. A queue or broker holds and distributes messages under its configured behavior. A worker receives a message and performs the operation. The browser can receive an accepted-job response with a job identifier, then check status later. It should not be told that an export exists before the worker has produced and checked it.

The message should identify the job and the required work without casually embedding secrets or large private files. Where inputs are stored, who may access them and how long they remain are separate product and security decisions. A queue is not permission to retain user uploads.

Buffering buys time, not infinite capacity

If producers briefly outpace workers, a queue can absorb a burst while workers catch up. If incoming work continuously exceeds processing capacity, the backlog keeps growing. Users then wait longer even though the submission endpoint looks fast. Observe queue depth and age of the oldest job, not just how quickly jobs are accepted.

Backpressure, admission limits and additional processing capacity are possible responses, depending on the task. More workers may not help if they all compete for the same database, external quota or CPU bottleneck. Define a maximum acceptable delay and explain it in the user experience.

Acknowledgements describe a delivery boundary

In a manual-acknowledgement design, the worker acknowledges a delivery when it has completed the required handling. If the worker disappears before acknowledging, a broker may redeliver under its documented behavior. A publisher confirmation and a consumer acknowledgement answer different questions: accepted by the broker versus handled by a consumer.

Durable queue configuration and persistent messages address parts of recovery, but are not a blanket guarantee that data can never be lost. Broker replication, publisher confirms, failure timing and operational recovery matter. Read the selected broker’s contract rather than treating a drawing of a queue as proof of durability.

Redelivery can repeat a completed effect

Suppose worker A saves a report but crashes before its acknowledgement reaches the broker. Worker B receives the job again. If it blindly sends another email or appends another charge, the system duplicates an effect even though the queue behaved as designed. The application needs a stable job or operation identifier and a suitable idempotency/reconciliation strategy.

Our in-memory fixture deliberately models completion followed by lost acknowledgement. The replacement worker sees the same job id, finds the completed result and avoids a second effect. That is a policy demonstration, not a crash-safe transaction. A real implementation must make its durable state and side effect agree across failures and concurrent workers.

Ordering depends on more than insertion order

Multiple workers can finish tasks in a different order from the order they received them. Redelivery and priority can further change observed behavior. If sequence matters, define the scope—perhaps one customer’s operations—and use supported partitioning or ordering mechanisms. Do not promise that one queue automatically gives every downstream effect a global chronological order.

Some tasks can be retried after a transient error; others fail repeatedly because the input is invalid. Use bounded attempts and a review or dead-letter path where appropriate. A poison job should not consume all capacity forever, and moving it aside is not the same as completing it successfully.

Practice the job-state conversation

Our fixture reports queued, processing, completed and duplicate-suppressed transitions for a fictional job. A separate failed job reaches a bounded review state after two attempts. The client-facing message should distinguish “waiting,” “working,” “ready” and “needs attention.” A job id alone should not expose another person’s report; access checks remain necessary.

Limits

No RabbitMQ server runs in this exercise. The local model is volatile and does not test persistence, broker failover, network partitions or exactly-once processing. It demonstrates why those concerns must be specified. The official RabbitMQ materials explain acknowledgement and confirmation mechanisms for a real implementation.

References

Sources used to check the explanations and examples are listed below.

1. RabbitMQ: work queues and acknowledgements
2. RabbitMQ: consumer acknowledgements and publisher confirms

Found a mistake? Send a correction with this lesson’s title, URL and disputed detail. No login needed.