Most of us here have shipped at least one model that was accurate the week it went out and quietly wrong three months later. This is a writeup of the pattern that fixes that without a retraining cron job you have to babysit, and it is simpler than the MLOps stack you are probably picturing.
Drift Is A Deployment Problem, Not A Modelling One
A model trained once is a photograph of the world on the day you exported the dataset. Your support categories shift, your customers start using new words, the traffic pattern that counted as an anomaly in March is Tuesday in August. Nothing about the model got worse. The world moved.
The usual answer is a retraining pipeline: a scheduler, an export job, a training container, a promotion step, a rollback plan. That is real work, and it is why a lot of small teams just never retrain. The model sits there degrading and nobody owns it.
Every Prediction Becomes A Row
The alternative is to make the serving path and the training path the same path. When a pipeline serves a prediction, it appends that input to the step's dataset, labelled with the step's own output, and once N new rows have accumulated the step rebuilds itself from the complete dataset.
That is the whole mechanism. There is no scheduler, because the trigger is traffic. A busy pipeline might retrain every 500 rows while a quiet one retrains every 20, and each pipeline sets its own interval. Live training in ML Prediction Engine is one implementation of exactly this, a per-pipeline switch rather than a subsystem.
It suits some model classes far better than others. Clusterers start following this month's themes instead of last quarter's. Anomaly detectors absorb the new normal, which is usually what you want and occasionally exactly what you do not. Classifiers are the risky case, and that is the next section.
The Feedback Loop Is Real, So Look At It
Training on your own outputs is a feedback loop, and feedback loops amplify whatever bias is already there. A classifier that leans slightly toward one label will produce slightly more of that label, then learn from those rows, then lean a little harder.
The mitigation is not clever, it is visibility. The appended rows have to land in the same readable dataset as the hand-added ones, so you can open the table, see what the model has been teaching itself, and delete the rows that are wrong. If your training data is a blob you cannot inspect, live training turns a slow drift problem into a fast one.
This is the argument for keeping the dataset as a plain file you own rather than an opaque managed store. The ML Prediction Engine approach is a jsonL file per model step, one JSON object per line, readable with any tool and diffable when you want to know what changed. That property is what makes the loop safe to leave running.
When Not To Turn It On
Leave it off when a wrong prediction is expensive and you have no human in the loop to catch it. Leave it off when your labels come from an authority outside the model, like a human reviewer or a downstream system, because in that case you should be appending the reviewer's label, not the model's guess. And leave it off for the first few weeks of any new pipeline, while the dataset is still small enough that a handful of self-generated rows can swing it.
Takeaway
Retraining does not need to be a platform. If the serving call already has the input in hand, appending it to the dataset and rebuilding on a row count is a few lines of plumbing, and it turns a model that decays into one that tracks. The only hard requirement is that you can still read the training data afterwards, because the day the model starts behaving strangely, that table is the entire debugging story.
