Skip to content

Stop instrumenting what you planned

Stop instrumenting what you planned

Event schemas get written during design and never revisited, so the data describes the product you intended rather than the one people use. A short argument for instrumenting late.

Event schemas are usually written at the same time as the spec, by the same person, before anyone has touched the thing. That timing is the problem. You end up with a precise record of the journey you drew on a whiteboard and no record at all of the one people actually take.

The symptom is an argument, not a gap

You rarely notice this as missing data. You notice it as a meeting where two people read the same funnel and disagree about what it means, because the funnel encodes an assumption nobody remembers making. The step that everyone skips was never given an event, so it does not appear as a drop. It appears as nothing.

Instrument late, then backfill

The version that works: ship the feature with coarse instrumentation, watch real sessions for a week, then write the schema against what you saw. It feels backwards and it is slower to start. It also means the first serious question you ask of the data is answerable.

Coarse does not mean careless. Page views, a handful of interaction events, and one identifier that survives a reload will tell you where the real path diverges from the drawn one. That is enough to write a good schema and not enough to write a wrong one.

The part that is genuinely hard

Backfilling is where this gets expensive, and there is no trick. You either accept a gap in the series or you replay raw logs into the new schema. We support the replay because the alternative is a chart with a discontinuity nobody can explain six months later, and an unexplained discontinuity quietly discredits every number beside it.

Written by

Julian Vandermolen

Last updated