Message Ordering
Controlling the sequence in which messages are delivered and processed in an event-driven system.
Message ordering is about controlling the sequence in which messages are delivered and processed. In distributed systems, the order you publish messages is not necessarily the order consumers receive them.
Consider a subscription system that publishes SubscriberSubscribed and SubscriberUnsubscribed events. If these arrive out of order, a read model built from them can end up in an incorrect state: it might try to unsubscribe someone who hasn't been subscribed yet.
The obvious solution is to order every message. But ordering everything has two serious downsides. First, performance drops because messages must be processed one by one. Second, and worse, a single failing message blocks all other messages behind it.
The practical approach is to keep ordering as narrow as possible. Instead of ordering all messages globally, order them within a small scope. For example, order events for one customer, one aggregate, or one entity. Kafka achieves this through partitions. Google Cloud Pub/Sub uses ordering keys. The idea is the same: messages that must be ordered share a key, while unrelated messages are processed independently and in parallel.
This connects to the aggregate pattern from Domain-Driven Design. An aggregate defines the smallest transactional boundary in your domain. It's often a natural fit for the ordering scope: events within one aggregate need ordering, but events across different aggregates don't.
There are cases where you can avoid ordering altogether. If each event updates a different part of the state, order doesn't matter. You can also use version numbers or timestamps to detect and handle out-of-order delivery. But these approaches require custom logic for each handler.
References
- Watermill 1.4 Released (Event-Driven Go Library) — Explains how a single failing message blocks other messages in most Pub/Subs, which is crucial if you care about message ordering.
- Watermill 1.3 released, an open-source event-driven Go library — Introduces handler groups for subscribing to multiple event types on one topic while preserving message ordering. Also covers a fix for potential message loss in watermill-sql related to transaction ordering.
- Shipping an AI Agent that Lies to Production: Lessons Learned — Lists message ordering as one of the key challenges when building an event-driven AI mentor system with Watermill and Google Cloud Pub/Sub.
- Synchronous vs Asynchronous Architecture — Covers message ordering as an anti-pattern when done too broadly. Discusses partition-based ordering in Kafka and Google Cloud Pub/Sub ordering keys. Key advice: keep ordering as narrow as possible.
- Event-Driven Architecture: The Hard Parts — Explains how proper message ordering prevents a single failing message from blocking the entire system. Discusses the connection between narrow ordering and the aggregate pattern from DDD.