/blog

10 min read

Clock skew bit us twice. Here's what we log now

Two incidents, one cause: machines that disagreed about the time. The fields we added to every log line afterwards.

The first time, a cache entry expired before it was written. The second time, an audit trail showed a refund issued before the payment it refunded. Both times, two hosts disagreed about the time by a few hundred milliseconds.

Wall clocks are for humans

Use a monotonic clock to measure durations and timeouts. In Go, time.Since already does; in other languages you may need to ask for it by name.

start := time.Now()
doWork()
elapsed := time.Since(start) // monotonic, safe across NTP jumps

What every log line carries now

  • ts: the wall-clock time, in UTC, with milliseconds.
  • host and clock_offset_ms from the last sync check.
  • trace_id, so ordering comes from causality, not timestamps.

Order by cause, not by clock

When two events on different machines need an order, pass a version or sequence number along with the request. A timestamp from another host is a hint, never a fact.

$ subscribe

Want the next one?

Leave your email and new posts from Stack & Heap come to your inbox.

$ subscribe --email

Get new posts by email

Leave your address and new posts from Stack & Heap come to your inbox.

Rather write back? hello@stackandheap.example