Outbox Pattern
Storing events in the same database transaction as data changes to guarantee delivery.
The Outbox Pattern solves the dual-write problem: how do you update your database and publish an event atomically, without a distributed transaction?
The answer is to not publish the event at all. Instead, store it in an "outbox" table in the same database, within the same transaction as your data change. A separate process (the forwarder) reads from the outbox table and publishes the events to the message broker.
This guarantees that if the data change is committed, the event will eventually be published. If the transaction rolls back, no event is stored. You get atomicity without distributed transactions.
The trade-off is that events are delivered asynchronously, with a small delay. Combined with at-least-once delivery semantics, consumers must handle duplicate events (idempotency).
References
- Distributed Transactions in Go: Read Before You Try — Explains the outbox pattern in depth: storing events in the same database transaction as data changes, then asynchronously forwarding them to a Pub/Sub. Covers the dual-write problem and why ignoring it is one of the most common mistakes.
- Watermill v1.2 released — Introduces the Forwarder Component, which enables the outbox pattern by publishing messages within database transactions and forwarding them to a Pub/Sub.
- Watermill 1.3 released, an open-source event-driven Go library — Fixes a low chance of losing messages in watermill-sql due to how transactions work. Recommends using new tables if you use watermill-sql for the outbox pattern.
- Watermill 1.4 Released (Event-Driven Go Library) — References examples on distributed transactions where the outbox pattern is used with Watermill to reliably publish events.
- SQLite Pub/Sub, Quickstart, and more — Watermill 1.5 Released — Releases watermill-sql v4 with support for packages beyond database/sql, including pgx and ORMs, with transaction support key for the Outbox pattern.
- Durable Background Execution with Go and SQLite — Explains that all state changes in one event handler must apply within the same database transaction. The outbox pattern is essential when making additional changes to the database together with publishing an event.
- Watermill: from a hobby project to 8k stars on GitHub — Discusses how Watermill grew to support the transactional outbox pattern and other event-driven patterns as key features of the library.
- Event-Driven Architecture: The Hard Parts — Covers the outbox pattern as the safest way to publish events without losing data. Discusses why not using the outbox pattern is one of the simplest ways to miss events.
- Watermill Forwarder Component
