Observation Before Explanation
The most common modelling error is not a bad model. It is explaining a system before describing it.
There is a strong pull, on meeting a new system, to explain it immediately. An explanation feels like progress. But an explanation offered before the system has been described carefully is usually a description of your expectations wearing the system's name.
So we try to hold the explanation for as long as we can stand to. First: what actually happens? Under what conditions? How often, how large, with what exceptions? Description is unglamorous, and it is where most of the real information turns out to be.
Observation comes before interpretation. The order is not a preference; it is what keeps the interpretation honest.
The discipline pays off later. A model built on a careful description can be tested against the parts of the description it did not use — a free holdout set you get for nothing. A model built on a hunch tends to explain everything and predict nothing.
- Write down what happened before writing down why.
- Record the exceptions, especially the ones that are inconvenient.
- Note the conditions of observation, not just the observations.
- Keep the first description; it is the only version not yet contaminated by the model.