What it is
Kafka Connect is the part of Kafka that does the plumbing. It is a framework of source and sink connectors that move data between Kafka and the systems that hold it - a database, an object store, a search index, a warehouse - so that offsets, retries, scaling and restarts are the framework's business rather than a hand-written service per integration. A pipeline that would otherwise be three applications becomes a configuration, a schema and a running Connect cluster.
What we do with it
- Change data capture from systems that cannot be changed. Debezium on MySQL, PostgreSQL, SQL Server or Oracle, so a database that predates everybody in the room becomes a live stream without an edit to the application that owns it.
- Sinks that keep a copy honest. Elasticsearch, object storage, a warehouse or another database, kept in step continuously instead of by a nightly export nobody trusts.
- Configuration that is deliberate. Converters and schemas, single message transforms, topic names and partition keys, and the dead letter queue that stops one bad record stopping the pipeline.
- Custom connectors when the system you need is not covered. Written, packaged, versioned and supported properly rather than living in somebody's home directory.
- Scaling and the unglamorous work. Task counts, rebalances, offset resets, and moving a running set of connectors between clusters or between Kafka versions without losing a message.
- Monitoring. Connector and task state, lag, and the retried records that are invisible until somebody looks for them - which is the difference between a pipeline and a hope.
How it starts
Name the systems on either side and what has to stay true between them: how often, how much, and what happens to a record that cannot be delivered. From there we will propose either a connector configuration, a small custom connector, or an honest "this does not need Kafka at all".
