Rules for Low-Latency Systems

Low latency is rarely about one clever trick. It comes from removing work from the critical path: fewer hops, less waiting, less data and fewer surprises at the tail.

The rules below apply whether you build trading systems, real-time games or an ordinary API that must answer in under 100 ms.

Diagram

How it works

1. Measure first: trace a request and find where the time goes.
2. Remove or parallelise network calls on the critical path.
3. Keep hot data in memory and use connection pools.
4. Move non-essential work (emails, analytics, resizing) to background queues.
5. Compress and trim payloads; return only the fields the client needs.
6. Avoid locks and garbage-collection pressure in hot code; reuse buffers.
7. Deploy close to users and keep services in the same region as their data.
8. Set timeouts everywhere so one slow dependency cannot stall the request.

Key concepts

Critical path

The chain of work the user must wait for. Everything else should happen in parallel or later.

Connection pooling

Reuse database and HTTP connections instead of paying for handshakes on every call.

Batching

Combine many small reads or writes into one round trip.

Pre-computation

Calculate expensive results ahead of time (materialised views, denormalised read models).

Tail latency

Optimise p99, not just the average. Hedged requests and timeouts help.

Trade-offs

Interview questions