Logs, Metrics and Traces: Which One Answers Which Question

What each is for

Metrics tell you a problem exists. Logs tell you what it was. Traces tell you where it was, which matters once a request crosses several services.

Metrics: cheap, aggregated, alertable

Numbers over time. Small to store, fast to query, and the right basis for alerting.

• Start with the four golden signals: latency, traffic, errors, saturation.
• Record percentiles rather than averages — p50, p95, p99. Averages hide the bad experiences.
• Use counters for events, gauges for current values, histograms for distributions.
• Watch cardinality. A label containing a user ID or request ID creates a series per value and will become the largest cost in your observability bill.

Cardinality is the trap. It is easy to add a label that seems useful and multiply your metric count by ten thousand.

Logs: detailed, expensive, indispensable

1. Log structured JSON, not free text, so fields can be queried rather than pattern-matched.
2. Include a request identifier on every line, and propagate it across services.
3. Use levels properly: error for things needing attention, warn for the unexpected but handled, info for significant events, debug off in production.
4. Never log credentials, tokens, card numbers or personal data.
5. Set retention deliberately — 30 days hot, longer in cheap archive if needed.
6. Sample high-volume, low-value logs rather than dropping the category entirely.

Log volume grows with traffic and is billed by volume on every platform. It is the observability line item most likely to surprise you.

Traces: for distributed systems specifically

A trace follows one request across services, showing where time was spent. In a monolith this is largely answerable from logs and profiling. Across five services it is the only practical way to find the slow hop.

• Use OpenTelemetry. It is the standard, and it avoids committing to one vendor’s agent.
• Sample. Tracing every request is expensive and unnecessary; one to ten per cent plus all errors is typical.
• Propagate context through every service boundary, or traces break into fragments.
• Add span attributes for the things you would want to filter on later.

Adopt in this order

1. Provider metrics and three alerts. Free, fifteen minutes, catches most outages.
2. Application metrics: request rate, error rate, latency percentiles.
3. Structured logging with request identifiers, shipped off the server.
4. Dashboards for the handful of things you check during an incident.
5. Tracing, once you have more than one service and a latency question you cannot answer.
6. Profiling, when you need to know which function is slow rather than which service.

Teams frequently start at step five because it is the most interesting, and end up with excellent traces and no alerting.

Controlling the bill

• Cap metric cardinality; review label sets before they ship.
• Set log retention by importance — audit logs long, debug logs short.
• Sample traces and high-volume logs.
• Drop health check requests from logs and metrics; they dominate both.
• Aggregate at the source where possible rather than shipping raw events.
• Review observability spend alongside infrastructure spend; it is often a surprising share.

Common questions

Where to go next

See monitoring a cloud server and audit logging.