Observability Explained: Logs, Metrics, and Traces

Monitoring answers questions you thought of in advance. Observability is the ability to answer the ones you didn't — which is what an outage actually consists of.

The three signals and what each is for

• Metrics — cheap numeric time series. Request rate, error rate, latency percentiles, queue depth. Best for alerting and trends; useless for explaining a single request.
• Traces — the path of one request across services, with timing per step. The fastest way to find which component is slow.
• Logs — detailed events with context. Expensive at volume, and where the actual explanation lives.

What to instrument first

• The four signals for every user-facing endpoint: traffic, error rate, latency (p50/p95/p99), and saturation of the underlying resource.
• Business events that matter — orders placed, payments failed, signups completed. A drop here is often the first true sign of an outage.
• Dependency calls, so you can distinguish your slowness from a provider's.
• A correlation id propagated across every service and included in every log line. Without it, distributed logs are unusable.

Alerting discipline

Alert on symptoms, not causes: high CPU is not an incident, users receiving errors is. Every alert must be actionable and have a runbook; anything else trains the on-call engineer to ignore the pager. Review alerts monthly and delete the ones nobody acted on — an alert that has fired forty times with no action is actively harmful.

Controlling the bill

• Sample traces rather than capturing every request, keeping all errors and slow requests.
• Log at info in production, debug on demand — and never log payloads containing personal or card data.
• Set retention by usefulness: detailed logs for days to weeks, aggregated metrics for a year.
• Structure logs as key-value data so you can query rather than grep; unstructured logs at volume are storage, not observability.