Real-Time Event Streaming
Process continuous streams of events as they happen, not in nightly batches.
Streaming systems treat data as an unbounded sequence of events — clicks, payments, sensor readings — and process each one within milliseconds or seconds.
A typical pipeline ingests events into a log such as Kafka, transforms and aggregates them with a stream processor, and writes results to stores, dashboards and alerts.
Diagram
Key concepts
- Event time vs processing time
- When the event happened versus when it was processed. Late events are normal.
- Windowing
- Group events into tumbling, sliding or session windows to compute aggregates such as “orders per minute”.
- Watermark
- The processor’s estimate of how far event time has progressed, used to close windows.
- Change data capture (CDC)
- Stream every database change as an event (e.g. Debezium) without changing the application.
- Stateful processing
- Joins and aggregates keep state, checkpointed so the job can recover after failure.
Trade-offs
Benefits
- Insights and reactions in seconds
- One event log feeds many independent consumers
- Replay history to rebuild views or fix bugs
Costs
- Harder to reason about than batch jobs
- Late, duplicate and out-of-order events
- Operating stream clusters takes expertise
Use it when
- Fraud detection, monitoring, live analytics, recommendations
- Syncing data between systems in near real time
Common mistakes
- Using processing time for business metrics and getting wrong numbers when events arrive late
- Unbounded state growth in joins
- Streaming everything when an hourly batch would meet the need
Interview questions
Q1What is the difference between batch and stream processing?
Batch processes a bounded dataset on a schedule (hourly, nightly). Streaming processes unbounded data continuously as each event arrives.
Q2How do you handle events that arrive late?
Use event-time windows with watermarks and an allowed lateness. Late events can update results or go to a side output for correction.
Q3Why is CDC useful?
It turns an existing database into an event stream without code changes, so search indexes, caches and warehouses stay in sync.
Related lessons
Messaging & Microservices
Message Brokers: RabbitMQ vs Kafka vs ActiveMQ
Queues and logs for decoupling services — and how the three popular brokers differ.
Cloud & DeliveryObservability: Logs, Metrics & Traces
Know what your system is doing, why it is slow and where it failed.
APIs & SecurityAPI Architecture Styles
REST, GraphQL, gRPC, WebSockets, webhooks and SOAP — what each is for.