Partitioning

Splitting a topic into smaller segments so messages can be processed in parallel while preserving order within each segment.

Partitioning is how message brokers split a topic into independent segments, so multiple consumers can process messages in parallel without breaking ordering guarantees.

In a single-partition topic, all messages are processed sequentially by one consumer. This keeps the order, but it limits throughput. If one message fails, everything behind it waits. Partitioning solves this by dividing the topic into multiple queues. Each partition is processed independently, so a failure in one doesn't block the others.

The way messages are routed to partitions is through a partition key (sometimes called an ordering key). You set the key when publishing, and the broker uses a hashing algorithm to assign the message to a specific partition. Messages with the same key always land in the same partition and are delivered in order. Messages with different keys can be processed in parallel.

Events for order-1 always go to the same partition. Events for order-2 and order-3 may land in different partitions, so they're processed in parallel.

Choosing the right partition key is a design decision that depends on your use case. A common choice is the entity ID. For example, using the order ID as the key ensures all events for a given order are processed in sequence, while events for different orders are handled concurrently. If you pick a key with low cardinality (only a few distinct values), you'll end up with hot partitions that limit scaling.

The trade-off is between ordering and throughput. Broad keys (like a tenant ID shared by many messages) give you strong ordering across many events but poor parallelism. Narrow keys (like individual entity IDs) give you high parallelism with ordering scoped to a single entity.

The number of partitions also limits how many consumers can work simultaneously. In Kafka, each partition is assigned to exactly one consumer within a consumer group. If you have more consumers than partitions, some consumers sit idle. If you have fewer, some consumers handle multiple partitions.

With 5 partitions and 3 consumers, some consumers handle more than one partition. Adding a 4th consumer would rebalance the load. Adding a 6th would leave one idle.

Different message brokers implement this differently. Kafka has explicit partitions configured per topic. Google Cloud Pub/Sub uses ordering keys. The concept is the same: route related messages together so they stay in order, while unrelated messages flow in parallel.