Message Broker
Infrastructure that sits between publishers and subscribers, routing messages between services.
A message broker (also known as a Pub/Sub) is the infrastructure component that sits between publishers and subscribers, handling message delivery, persistence, and routing.
In essence, it is a middleman. Publishers send messages to the broker, and the broker delivers them to interested subscribers. This decoupling means services don't need to know about each other directly.
Message brokers vary widely in their trade-offs. Kafka provides persistent storage with message replay. Google Cloud Pub/Sub handles scaling automatically. RabbitMQ offers flexible routing with exchanges and queues. You can even use an SQL database like PostgreSQL as a message broker when you don't want to add extra infrastructure.
Choosing the right message broker depends on your throughput needs, ordering guarantees, persistence requirements, and operational complexity. Like choosing a database engine, there is no "best" general solution.
Message brokers are critical infrastructure. When your messaging infrastructure goes offline, your services won't work properly. Consider availability, maintenance, backups, and security the same way you would for a database.
References
- Using MySQL as a Pub/Sub — Shows how SQL databases can serve as Pub/Subs using Watermill, covering persistent events, transactional outbox, and database synchronization.
- Introducing Watermill - Go event-driven applications library — Introduces Watermill's Pub/Sub abstraction with pluggable implementations for Kafka, Google Cloud Pub/Sub, and more.
- Distributed Transactions in Go: Read Before You Try — Explains how Pub/Sub enables event-driven communication between services without distributed transactions.
- Live website updates with Go, SSE, and htmx — Uses Watermill and Google Cloud Pub/Sub to stream real-time updates via SSE. Explains the Pub/Sub abstraction, publisher/subscriber interfaces, and how message brokers route events to multiple service replicas.
- SQLite Pub/Sub, Quickstart, and more — Watermill 1.5 Released — Announces SQLite Pub/Sub support in Watermill, removing the need for a separate message broker. Covers 13 officially supported Pub/Subs and GoChannel context propagation.
- Watermill 1.4 Released (Event-Driven Go Library) — Introduces AWS SNS/SQS Pub/Sub, a universal requeuer for handling failed messages across all Pub/Subs, delayed message delivery via PostgreSQL, and the AMQP v3 release.
- Watermill 1.3 released, an open-source event-driven Go library — Covers 12 Pub/Sub implementations including Kafka, Redis Streams, NATS, and RabbitMQ. Introduces request-reply pattern working on top of any Pub/Sub.
- Watermill v1.2 released — Adds 4 new Pub/Subs (Bolt, NATS Jetstream, Redis Stream) and the Forwarder component for forwarding messages from one Pub/Sub to another, such as SQL to Kafka.
- Golang CQRS, Metrics and AMQP - Watermill v0.3.0 released — Adds AMQP (RabbitMQ) Pub/Sub supporting both fanout and queue modes. Also introduces HTTP Publisher as a Pub/Sub and GoChannel updates.
- Watermill v0.2.0 released — Adds Google Cloud Pub/Sub and NATS Streaming implementations. Replaces Kafka driver with Sarama and introduces message context propagation.
- The Go libraries that never failed us: 22 libraries you need to know — Recommends Watermill for messaging, highlighting its support for 10+ Pub/Subs including Kafka, Google Cloud Pub/Sub, NATS, and RabbitMQ. Contrasts it with low-level message broker libraries.
- Durable Background Execution with Go and SQLite — Demonstrates using SQLite Pub/Sub with Watermill for durable execution. Explains how writing events to a persistent Pub/Sub provides durability without a separate message broker.
- Synchronous vs Asynchronous Architecture — Discusses when to use message queues and event-driven architecture for async communication. Covers message broker trade-offs, including infrastructure complexity, debugging, and local development.
- Event-Driven Architecture: The Hard Parts — Explores message broker reliability, at-least-once delivery, and the outbox pattern. Discusses how message brokers are designed not to lose messages and the challenges of ordering and idempotency.
- Watermill: from a hobby project to 8k stars on GitHub — Shares the story of building Watermill with 12 officially supported Pub/Subs like Kafka, RabbitMQ, NATS, and Postgres. Discusses the difference between Pub/Sub and message queue.
- Watermill Pub/Sub Implementations
