Resilience: Timeouts, Retries & Circuit Breakers

In a distributed system something is always failing. Resilience patterns accept this and limit the damage: they bound how long you wait, how often you retry and how much load you send to a struggling dependency.

The classic failure they prevent is the cascade: a slow database makes service A’s threads wait, A becomes slow, so B waits on A, and soon everything is down.

Diagram

The three states of a circuit breaker.

Key concepts

Timeout: Every remote call gets a deadline. Without one, a hung dependency holds resources forever.
Retry with backoff + jitter: Retry transient failures a few times, waiting longer each time with randomness.
Circuit breaker: After repeated failures, stop calling the dependency for a while and fail fast.
Bulkhead: Separate pools of threads or connections per dependency so one can’t exhaust them all.
Fallback: A degraded but useful response: cached data, default values, “recommendations unavailable”.
Load shedding: Reject low-priority work early when overloaded, to protect critical paths.

Example

A resilience pipeline with Polly (.NET)

builder.Services.AddHttpClient<CatalogClient>()
    .AddResilienceHandler("catalog", b => b
        .AddTimeout(TimeSpan.FromSeconds(2))
        .AddRetry(new() { MaxRetryAttempts = 3, BackoffType = DelayBackoffType.Exponential, UseJitter = true })
        .AddCircuitBreaker(new() { FailureRatio = 0.5, MinimumThroughput = 20,
                                   BreakDuration = TimeSpan.FromSeconds(30) }));

Trade-offs

Benefits

Stops cascading failures
Gives failing dependencies time to recover
Users get fast errors or fallbacks instead of hanging pages

Costs

Thresholds need tuning per dependency
Retries add load and can amplify an outage if uncontrolled
Fallbacks must be tested like any other feature

Use it when

Every call across a network: databases, services, third-party APIs

Common mistakes

Retrying at every layer — 3 retries × 3 layers = 27 calls
Retrying non-idempotent operations
Timeouts longer than the caller’s own timeout

Interview questions

Q1Why add jitter to retries?

Without it, thousands of clients that failed at the same moment retry at the same moment, creating repeated load spikes. Jitter spreads them out.

Q2What is the bulkhead pattern?

Isolating resources — like the watertight compartments of a ship — so a flood of slow calls to one dependency can’t consume the threads or connections needed for others.

Q3How should timeouts relate across a call chain?

Each inner timeout should be shorter than the outer one, so the caller still has time to handle the failure. Propagating a deadline makes this automatic.