Decouple services by passing messages asynchronously through an append-only log.
A traditional Queue deletes a message once read. A Streaming Platform (like Kafka) is an immutable append-only log. Messages are written to Partitions. Consumers don't delete data; they just advance a Cursor (Offset).
To scale reading, we use a Consumer Group. Each partition is assigned to exactly one consumer in the group. If a consumer is too slow, Lag builds up (the distance between the latest message and the cursor). You can fix this by adding more consumers to rebalance the partitions!
# The Broker assigns partitions to consumers.
# Max concurrency = number of partitions!
def consume_loop(consumer_id, assigned_partitions):
while True:
for p in assigned_partitions:
# Read from the current cursor
messages = broker.fetch(p, offset=p.current_offset)
for msg in messages:
process(msg)
# Advance the cursor (commit offset)
p.current_offset += 1