Dead Letter Queue

A separate queue where messages that can't be processed are moved aside for inspection and manual handling.

A dead letter queue (also known as a poison queue) is where messages go when they can't be processed successfully. Instead of retrying forever or silently dropping the message, you move it aside so it doesn't block other messages.

The typical setup combines retries with a dead letter queue. When a message handler fails, the system retries it a few times. If it still fails after all retries, it's acknowledged on the original topic and published to the dead letter queue. This prevents a single broken message from blocking the entire subscription.

In Watermill, the PoisonQueue middleware handles this. You can also customize which errors trigger the move: temporary errors (like network timeouts) should be retried, while permanent errors (like invalid data) should go straight to the dead letter queue.

Moving messages is the easy part, but managing them after is more complex. You need to:

1. Inspect the failed messages to understand what went wrong.
2. Fix the underlying issue (a bug in the handler, missing data, schema mismatch).
3. Requeue the messages back to the original topic for reprocessing, or delete them if they're no longer relevant.

One trade-off to keep in mind: using a dead letter queue affects message ordering. A message moved to the queue and later requeued will be processed out of its original order. If strict ordering matters, you may need a different strategy.

Setting up alerts for new messages on the dead letter queue topic is a good practice. You don't want failed messages sitting there unnoticed.