Rate Limiting
Cap how many requests a client can make so one caller can’t overwhelm everyone else.
A rate limiter counts requests per client — by API key, user, IP or tenant — and rejects or delays those above a limit, usually with HTTP 429 Too Many Requests.
It protects services from abuse and runaway clients, enforces fair use between customers and keeps costs predictable for expensive operations such as AI calls.
Diagram
Key concepts
- Token bucket
- Tokens refill at a fixed rate; each request spends one. Allows short bursts up to the bucket size. Very common.
- Leaky bucket
- Requests drain from a queue at a constant rate, smoothing bursts into a steady flow.
- Fixed window
- Count requests per calendar minute. Simple, but allows double bursts at window edges.
- Sliding window
- Counts over a rolling period, fixing the edge problem at a little more cost.
- Distributed limiting
- With many API instances, counters live in a shared store such as Redis so limits are global.
Trade-offs
Benefits
- Protects availability for all customers
- Stops brute-force and scraping
- Controls cost of expensive downstream calls
Costs
- Shared counters add latency and a dependency
- Limits that are too strict hurt real users
- Per-IP limits punish users behind shared NAT
Use it when
- Public APIs and login endpoints
- Multi-tenant SaaS to enforce plan quotas
- Calls to paid third-party services
Common mistakes
- Not returning
Retry-After, so clients retry immediately - Limiting only at the edge while internal callers can still flood a service
- Failing closed when the counter store is down and blocking everyone
Interview questions
Q1Why is the token bucket so popular?
It enforces an average rate while still allowing short bursts, it needs only two values per client (tokens and last refill time) and it is easy to implement atomically in Redis.
Q2Where should the rate limiter live?
Usually at the API gateway or edge for global limits, plus finer limits inside services for expensive operations.
Q3What should happen if Redis, which stores the counters, is unavailable?
Decide per endpoint. Most APIs fail open (allow traffic, alert), while sensitive endpoints like login may fall back to a local in-memory limit.
Related lessons
APIs & Security
API Gateway
A single front door that routes, secures and shapes traffic to your back-end services.
APIs & SecurityAPI Security Essentials
Defence in depth for APIs: transport, identity, authorization, input and abuse controls.
Scalability & PerformanceLoad Balancing
Spread traffic across healthy servers so no single machine becomes the bottleneck.