Field notes
Short write-ups on things we've learned building ML systems. Full posts live right here on this page — expand the ones that interest you.
Here's a pattern we see constantly. A team wants a model. We look at the data and find duplicate records, inconsistent labels, three date formats, and a CSV export process that silently drops rows. The model was never the problem.
The uncomfortable truth is that data cleanup is 60–80% of most ML projects, and there's no shortcut. The good news: cleanup work pays off even if you never train a model. Better data helps your dashboards, your analysts, and every future project.
Our advice: budget for data work explicitly. If a vendor's proposal doesn't mention it, they either haven't looked at your data or they're planning to bill you for the surprise later.
Before any model, we always build the naive answer first. Predicting demand? Try "same as last week." Classifying tickets? Try keyword matching. These take an afternoon.
The baseline does two jobs. It gives you a number the fancy model has to beat, and it sometimes turns out to be good enough on its own — in which case, congratulations, you just saved a project's worth of budget.
We've had engagements end at the baseline stage. Nobody was upset about it. That's what an honest process looks like.
People ask us this weekly, so here's the rough decision tree we use. First: can careful prompting of a hosted model hit your quality bar? Test it on a hundred real examples, not five. If yes, stop — it's the cheapest option to run and iterate on.
If prompting plateaus below your bar, or the per-request cost is too high at your volume, or your data can't leave your infrastructure, then fine-tuning a smaller open model starts making sense. It costs more upfront and less per request.
The mistake we see: teams jump to fine-tuning for prestige reasons. Measure first. The boring option wins more often than anyone admits.
A model is a snapshot of the world at training time. The world moves. Customer behaviour shifts, document formats change, a new product line appears — and accuracy quietly drops while the dashboard stays green.
This is why we treat monitoring as part of the deployment, not an add-on. Log predictions, sample them for review, track the distribution of inputs, and alert when it drifts from what the model was trained on.
A reasonable rule of thumb: if nobody has looked at your model's real-world accuracy in three months, you don't know what your model is doing. Cheap monitoring beats expensive surprises.