The Spreadsheet That Lied to Me (Politely)
A demand forecast, a very clean chart, and the meeting where I learned a model can be accurate and still wrong.
Years before I wrote any machine learning code, I sat in a planning meeting looking at a forecast everyone liked. The chart was smooth and the error bars were tight. It told us to cut a product line that, as it turned out, was about to have its best season in years. The forecast wasn’t wrong, exactly. It was answering a question none of us had asked.
I still think about that meeting. A forecast that fits last year well is assuming next year will look like last year. Ours had picked up a promotion schedule, a factory shutdown, and one unusually cold winter, and treated all three as the trend. The model understood the past fine. We hadn’t thought hard enough about what the past meant.
What fixed it wasn’t a better algorithm. It was asking a few plain questions before the chart went on a slide. Compared to what? Over what period? What happened in that data that won’t happen again? A lot of analytics, I think, is just not accepting the first number until you understand the story under it.
Machine learning didn’t change that for me; it just raised the stakes. A gradient-boosted model can be very confident about a pattern that stopped existing. So now anything I ship comes with a short list of what the model can’t know. It makes the demo a little less impressive. It makes the decision better, which is what we were there for.