Consumer Group

A mechanism that lets multiple instances of a service share the work of processing messages from a topic.

A consumer group is a way to distribute message processing across multiple instances of a service, so each message is handled by only one instance in the group.

When you scale a service horizontally (say, 3 replicas of your order processor), you don't want all 3 instances to process the same message. Consumer groups solve this: the message broker assigns each message to exactly one member of the group. The work is spread across all instances automatically.

Each message goes to exactly one instance within the group. The broker handles the assignment.

The concept exists in most message brokers, though the naming varies. Kafka calls them consumer groups. Google Cloud Pub/Sub uses subscriptions. RabbitMQ has queues with competing consumers. The idea is always the same: multiple consumers cooperate so that each message is processed once (per group).

Consumer groups also enable independent processing pipelines. Two different services can each have their own consumer group on the same topic. The order service and the notification service both receive every OrderPlaced event, but within each service, only one instance handles it.

This is the difference between fan-out (across groups) and load balancing (within a group). Every group gets every message. Within each group, only one instance handles it.

When a consumer joins or leaves a group (for example, during a deploy or autoscaling), the broker rebalances the message partitions across the remaining members. This rebalancing can cause brief processing delays, and in some brokers it may lead to duplicate delivery. That's why at-least-once delivery semantics and idempotent handlers matter: rebalancing is a normal part of operating consumer groups.

The consumer group also tracks which messages have been processed. In Kafka, this is stored as an offset per partition. If a consumer crashes and restarts, it picks up where it left off. This tracking is what makes consumer groups reliable for long-running workloads.