Observability: Logs, Metrics & Traces
Know what your system is doing, why it is slow and where it failed.
Monitoring tells you that something is wrong; observability helps you understand why. It rests on three signals: logs, metrics and distributed traces, tied together by a shared correlation ID.
OpenTelemetry has become the standard way to instrument code once and send telemetry to any back end — Azure Monitor, Grafana, Datadog, Splunk or others.
Diagram
Key concepts
- Structured logs
- JSON events with fields (tenant, user, order ID) that can be queried, not just grepped.
- Metrics
- Numeric time series: request rate, error rate, latency percentiles, queue depth.
- Distributed tracing
- A trace follows one request across services; each span records a step and its duration.
- Golden signals
- Latency, traffic, errors and saturation — the first four things to watch for any service.
- SLI / SLO
- A service-level indicator (e.g. % of requests under 300 ms) and the objective you promise (99.9%).
Trade-offs
Benefits
- Faster incident detection and resolution
- Data-driven performance work
- Evidence for capacity planning and SLOs
Costs
- Telemetry volume can become expensive — sample and set retention
- Sensitive data can leak into logs
Use it when
- Every production service, before launch rather than after the first incident
Common mistakes
- Logging secrets, tokens or personal data
- Alerting on every error instead of on user impact and SLO burn
- No correlation ID, so logs from different services can’t be joined
Interview questions
Q1Logs, metrics or traces — which would you check first for a latency spike?
Metrics to confirm scope and timing, then traces of slow requests to see which span grew, then logs for that service and time window for the cause.
Q2What is an error budget?
The amount of unreliability an SLO allows. With a 99.9% monthly SLO, about 43 minutes of failure is “budget”. When it runs out, reliability work takes priority over features.
Q3How do you control telemetry cost?
Sample traces (keep all errors and slow requests), drop noisy debug logs in production, aggregate metrics and set retention by data type.
Related lessons
Messaging & Microservices
Resilience: Timeouts, Retries & Circuit Breakers
Contain failures so one slow dependency doesn’t take the whole system down.
Messaging & MicroservicesReal-Time Event Streaming
Process continuous streams of events as they happen, not in nightly batches.
Cloud & DeliveryCI/CD Pipelines
Automate build, test and deployment so every change reaches production safely and quickly.