data engineering

Exploring Confluent Kafka: a developer's experience in real-time data processing

Tracking job-run status across distributed systems by treating Kafka topics as tables instead of reaching for a database.

Read the full piece

Job status tracked through Kafka rather than a database. Three job-runner producers publish JSON messages containing job_id, timestamp and status into a job-status topic split across three partitions, whose retention holds the history. On the right, consumers filter by job ID, filter by time range, and track their position by offset. At the bottom, a crossed-out status table with its polling loop shows the layer that is removed: offsets replace rows, and partitions provide parallelism. — The database and its polling loop are the parts that disappear — retention becomes the store.

01 The short version

Tracking the status of job runs across distributed systems usually means a database table and a lot of polling. I wanted to see how far an event-driven approach would go instead.

The experiment treats Kafka topics as the store rather than as transport. Producers publish JSON status messages carrying a job ID, a timestamp and a status; consumers poll the topic and filter by job ID or time range to reconstruct what happened.

No conventional database is involved. Kafka's retention holds the history, partitioning gives parallel consumption, and offsets track what each consumer has already seen.

"Treating topics as tables" is the phrase I kept coming back to — it isn't right for every workload, but for high-throughput status streams it removes an entire layer.

02What you'll take away

Producers publish structured JSON — job ID, timestamp, status — so consumers can filter without extra lookups.
Partitioning is what makes it scale: consumers work in parallel instead of contending over one table.
Offsets, not rows, are how you track progress through the stream.
Kafka's own retention becomes the persistence layer, which is what removes the database from the picture.