Latency Numbers & the Memory Hierarchy

Every data access lands somewhere in the memory hierarchy, and each level is roughly an order of magnitude slower than the one before. Knowing the rough numbers lets you spot an impossible design in seconds.

The figures below are order-of-magnitude guides on modern hardware, not benchmarks. What matters is the ratio: RAM is about 100× faster than an SSD read, and a same-datacentre round trip is about 1,000× faster than crossing an ocean.

Diagram

Approximate access times. Use them for estimates, then measure the real system.

How it works

Estimate request volume: daily users × actions per user ÷ 86,400 seconds.
Estimate data size: records per day × bytes per record × retention days.
Place hot data in the fastest level you can afford (memory cache), cold data on disk or object storage.
Count network hops on the critical path; each one adds latency and a failure point.

Key concepts

Latency: Time for one operation to complete, e.g. one request round trip.
Throughput: How many operations complete per second. High throughput does not imply low latency.
Tail latency: The slowest requests (p95, p99). Users notice the tail, and fan-out calls amplify it.
Locality: Data accessed together should be stored together so caches stay effective.

Trade-offs

Benefits

Fast sanity checks during design and interviews
Explains why caching and batching work
Highlights where to spend optimisation effort

Costs

Real numbers vary by hardware, cloud and load
Easy to over-optimise the wrong layer without measuring

Use it when

Back-of-the-envelope capacity planning
Deciding whether a cache, CDN or replica in another region is worth it

Common mistakes

Chatty services making dozens of sequential network calls per request
Treating averages as the user experience — look at p99
Forgetting serialisation and TLS handshakes add to network time

Interview questions

Q1A page makes 20 sequential calls to a service in the same datacentre. What is the minimum latency?

About 20 × 0.5 ms = 10 ms of network alone, before any processing. Running independent calls in parallel or batching them is the first fix.

Q2Why is reading from RAM so much faster than reading from an SSD?

RAM is electrically close to the CPU with no storage controller or file system involved. An SSD read goes through the controller, flash pages and an I/O stack — tens of microseconds.

Q3Why does tail latency matter more in microservices?

If one request fans out to 10 services, the response waits for the slowest. A 1% chance of a slow call per service becomes roughly a 10% chance per request.