Observability Explained: Logs, Metrics, and Traces
Key takeaway
Metrics tell you something is wrong, traces tell you where, and logs tell you why. Instrument all three around the user-facing operations that matter, alert only on symptoms users would notice, and treat retention as a cost decision — most teams keep far too much for far too long.
Monitoring answers questions you thought of in advance. Observability is the ability to answer the ones you didn't — which is what an outage actually consists of.
The three signals and what each is for
- Metrics — cheap numeric time series. Request rate, error rate, latency percentiles, queue depth. Best for alerting and trends; useless for explaining a single request.
- Traces — the path of one request across services, with timing per step. The fastest way to find which component is slow.
- Logs — detailed events with context. Expensive at volume, and where the actual explanation lives.
What to instrument first
- The four signals for every user-facing endpoint: traffic, error rate, latency (p50/p95/p99), and saturation of the underlying resource.
- Business events that matter — orders placed, payments failed, signups completed. A drop here is often the first true sign of an outage.
- Dependency calls, so you can distinguish your slowness from a provider's.
- A correlation id propagated across every service and included in every log line. Without it, distributed logs are unusable.
Alerting discipline
Alert on symptoms, not causes: high CPU is not an incident, users receiving errors is. Every alert must be actionable and have a runbook; anything else trains the on-call engineer to ignore the pager. Review alerts monthly and delete the ones nobody acted on — an alert that has fired forty times with no action is actively harmful.
Controlling the bill
- Sample traces rather than capturing every request, keeping all errors and slow requests.
- Log at info in production, debug on demand — and never log payloads containing personal or card data.
- Set retention by usefulness: detailed logs for days to weeks, aggregated metrics for a year.
- Structure logs as key-value data so you can query rather than grep; unstructured logs at volume are storage, not observability.