What it is
Kafka is a distributed log. Producers append events to a topic, the topic is split into partitions that each hold an ordered sequence, consumers read them in groups at their own pace, and retention means yesterday is still there when somebody needs to ask it a new question. It is what you reach for when several systems need to know the same thing happened, in the same order, without being wired directly to each other.
What we do with it
- Event-driven design that stays comprehensible. Topics, keys, partitioning and ownership decided deliberately, and a schema registry so a change to an event is a reviewed version rather than a surprise in somebody else's consumer.
- Ordering and exactly-once questions, answered. Idempotent producers, transactions, offset management, and the outbox pattern - so that a duplicate becomes a non-event rather than a support ticket.
- Throughput and latency work. Partition counts, batching, compression, replication settings, and consumers that keep up when a backlog arrives at once.
- Stream processing on the same platform. Kafka Streams or ksqlDB where the transform between two topics is all the service ever needed to be.
- Operations, which is most of the work. Replication and in-sync replicas, retention and tiered storage, upgrades, and monitoring that tells a lagging consumer apart from a broken one.
- Anywhere it runs. Self-managed, MSK on AWS, Event Hubs on Azure, or Confluent Cloud - the same design, on whichever of them you already pay for.
How it starts
Tell us which systems need to know about which events, and in what order they matter. For a broker that is already running, we start with the topics, the consumer groups and the lag - the numbers usually say where the problem is before anybody reads the code.
