Idempotency
Ensuring that processing a message more than once produces the same result as processing it once.
An operation is idempotent if running it multiple times produces the same result as running it once. In event-driven systems, it's very often a requirement.
The reason is how message brokers work. Most brokers guarantee at-least-once delivery: if something goes wrong (a network issue, a crash before acknowledgement, etc.), the broker will redeliver the message. Your handler might process the same event twice. If your handler is not idempotent, the second processing can corrupt your data.
Common strategies for making handlers idempotent:
- Track processed event IDs. Store the ID of every event you handle. Before processing, check if you have seen it before.
- Use idempotency keys in API calls. Pass a unique key with each request so the receiving service can detect duplicates.
- Make operations naturally idempotent. Setting a value to "active" is idempotent. Incrementing a counter is not.
Even if duplicate delivery seems unlikely, it is actually quite common, even in smaller systems. Building idempotency into your handlers from the start saves you from subtle, hard-to-debug data inconsistencies later.
References
- Using MySQL as a Pub/Sub — Explains that when using SQL databases as Pub/Subs, you need to handle event deduplication yourself. A simple approach is to make sure your event handlers are idempotent.
- Durable Background Execution with Go and SQLite — Covers idempotency as one of two key properties for durable execution. Shows how to prove idempotency with event duplication tests and make handlers tolerate repeated processing.
- Live website updates with Go, SSE, and htmx — Mentions that with at-least-once delivery, you may receive the same message twice. Instead of working around it, design your handlers to be idempotent so processing the message twice has the same effect as processing it once.
- Event-Driven Architecture: The Hard Parts — Discusses idempotent handlers as the solution to duplicate message delivery. Covers practical approaches like using ON CONFLICT DO NOTHING for inserts and idempotency keys for external API calls.
- AMA #1: Clean Architecture, Learning, Event-Driven, Go — Mentions making endpoints idempotent when discussing event-driven architecture patterns.
- Watermill Deduplicator Middleware
