Kafka

A distributed append-only log used as a message broker: messages survive being read, and many consumers process them independently.

Kafka is a Message Broker built around one idea: a distributed append-only log. Publishers write messages to the end of the log. Consumers read from any position they choose, and reading a message doesn't delete it.

This is what sets Kafka apart from most brokers. A typical queue treats a message as something to deliver and forget. Kafka keeps messages on disk, in order, for as long as you configure: days, months, or forever. A new service can start today and process events published last year. After fixing a bug, you can rewind a consumer and reprocess everything it got wrong.

Each Topic is split into partitions, and the partition is the unit of both ordering and parallelism. Kafka guarantees order within one partition, not across the topic. Messages published with the same partition key (an order ID, a user ID) always land in the same partition, so events about one entity arrive in the order they happened.

Consumers track their progress with offsets: each consumer remembers how far into each partition it has read. Offsets belong to the consumer, not the broker, so any number of consumers can read the same topic without affecting each other. A Consumer Group spreads a topic's partitions across instances of one service, which means scaling out is adding members to the group.