There is no universal interval, and any article offering one is guessing on your behalf. What can be measured is what each choice costs, and those figures are more surprising than the usual advice suggests.
Researchers at Delft University of Technology measured the energy consumption of eight retraining configurations across three production failure-prediction datasets from Backblaze, Google and Alibaba. Three findings stand out:
- Retraining only on recent data, rather than the full accumulated history, cut energy use by around 25% with no meaningful accuracy penalty
- Retraining on a drift signal rather than a fixed schedule cut it by up to 40% over a projected year
- Some drift-triggered configurations consumed more energy than a fixed schedule, and one produced a model no better than never retraining at all
The specific figures come from Random Forest models, and the authors are clear that deep learning workloads may behave differently. The pattern is the useful part, and it holds regardless of architecture.
That spread is the actual subject. Cadence is a resource decision with a measurable price attached, and most teams set it by intuition. Our guide to AI model lifecycle management sets out where retraining sits in the wider lifecycle. This post covers what the decision costs and how to size it against your own data.
Measuring your own decay rate
The defensible way to set an interval is to derive it from historical data rather than pick a round number. Evidently’s ML observability course sets out the procedure:
- Train on an older slice of your labelled history, holding back everything after a chosen cut-off.
- Apply that model forward against successive batches of newer data at a fixed step, daily, weekly or monthly depending on the workload.
- Record where performance crosses the threshold you consider acceptable. That crossing point is your decay window.
- Set the schedule inside that window with margin, rather than at its edge.
The same sweep answers a second question worth asking at the same time: how much new data the model needs before a retrain is worth running at all. Cadence and data volume are separate constraints, and a schedule firing before enough new samples have accumulated produces a model differing from its predecessor mainly by noise.
Teams with steady data arrival can treat volume as a trigger in its own right, retraining once accumulated new records pass a threshold rather than when a date arrives.
The output is a measured property of the system rather than a preference, which matters when the interval has to be justified to an auditor or a risk committee. It also surfaces the cases where no schedule is safe, because the decay window turns out shorter than the time needed to validate and ship a replacement.
Decay rates vary far more than published advice implies:
- Days – Fraud detection and real-time bidding, where adversaries or markets adapt continuously
- Weeks – Recommendation and demand models tracking seasonal or promotional behaviour
- Months to a year – Models of slow-moving physical or industrial processes
Averaging across that range produces a number fitting nobody.
The data window is the cheapest change available
Most discussion of retraining treats it as a frequency question, which leaves the more consequential variable unexamined: what data goes into the retrain.
Full-history retraining fits the model on everything accumulated to date.
Sliding-window retraining fits it on the most recent period and discards the oldest.
The Delft measurements found the sliding window consumed significantly less energy across nearly every configuration tested, around 25% lower for periodic retraining, while accuracy held. On one dataset it scored marginally higher, since dropping stale samples helped rather than hurt.
The exception is instructive. Full-history retraining performed better only on the dataset with the shortest collection period, where the model still needed volume more than recency. Where history is short, keep it. Where history is long and the world has moved, carrying old data forward costs compute to preserve patterns that no longer apply.
The difference compounds, which is the part that matters for capacity planning. A full-history retrain works on a training set growing every cycle, so each run costs more than the last and the annual bill curves upward as the model ages.
A sliding window holds the training set at roughly constant size, so cost per run stays flat and the capacity requirement can be sized once. Two models on identical schedules can therefore follow completely different cost trajectories, and the divergence only becomes visible a year or two into production.
What a drift trigger actually saves
Trigger-based retraining fires when monitoring detects a distribution shift or a performance drop, and the reasoning is sound: retrain when something changed rather than when the calendar says so.
The Delft results show the saving is real but conditional. Over a projected year, drift-triggered retraining reduced energy by roughly 40% on one dataset, 10% on another and 7% on the third. The detector itself was cheap, accounting for under 4% of combined training and detection energy, so monitoring overhead is not the constraint.
The failure modes are where this gets useful, because they run in opposite directions:
- Too insensitive – On the Alibaba dataset, one configuration registered no drift, triggered no retraining and produced a model performing at least 4% below the alternatives, effectively matching a model never retrained at all.
- Too sensitive – On the Google dataset, two configurations triggered often enough to consume more energy than a fixed schedule would have, and one ended up 1% worse on accuracy. The team pays for responsiveness and receives churn.
The standard remedy is a minimum interval between retrains, enforced independently of the trigger. A cooldown puts a ceiling on how often the pipeline can fire regardless of what the detector reports, capping the cost of an over-sensitive signal and stopping the deployed model changing faster than anyone can evaluate it.
Requiring drift to persist across more than one monitoring window before it escalates does similar work at the detection end.
Neither failure announces itself. Both look like a working trigger from the outside, which is why the detector needs validating against historical drift before it is trusted to control a retraining pipeline.
The specifics of detector selection and threshold setting are covered in our post on model drift.
The label problem underneath every trigger
Both approaches assume you can tell whether the model is still right, and that assumption is where many teams get stuck.
Measuring accuracy requires ground truth, and ground truth frequently arrives long after the prediction. A loan either defaults or it does not, on its own timeline. A flagged transaction is confirmed or reversed weeks later. Any trigger waiting on confirmed accuracy is late by construction.
This also constrains the empirical method above, which needs labelled historical batches to establish a decay window. Teams without reliable labels cannot run that experiment and cannot build an accuracy trigger, which leaves them with input-side signals indicating the world has changed without confirming the model is wrong.
Those signals justify investigation, though rarely an automatic retrain on their own, which is the practical reason most teams keep a scheduled floor underneath whatever trigger they build.
Rebuild or update in place
Once the decision to retrain is made, a second question follows: fit a new model from scratch, or update the existing one with new data.
- Full rebuild. Clears accumulated bias and permits architecture or feature changes an incremental update cannot reach. Costs more, and discards prior learning unless older data is mixed back in.
- Incremental or continual update. Cheaper, and shortens the loop between spotting a problem and shipping a fix. Patches the model rather than reshaping it, and repeated uncurated updates can accumulate bias quietly.
Many classical algorithms do not support incremental fitting at all and must be refitted from the combined dataset regardless. Most production teams run both, using incremental updates for routine freshness and periodic full rebuilds as a reset.
Retraining is a deployment
Whatever cadence and method a team settles on, a retrain is a release, and every release can regress. A model looking stronger on recent data can be worse on the long tail, or can have learned from a corrupted batch nobody inspected.
Before any candidate replaces the incumbent:
- Compare the two on a curated evaluation set reflecting long-term behaviour, not just the last few weeks
- Make promotion conditional on a genuine improvement rather than mere completion of the run
- Keep a rollback path to the previous version, tested rather than assumed
- Record which data version, code commit and detector state produced the candidate
Automating the retrain without automating that gate removes the person who would have caught the regression. Where the gate lives, who approves promotion and what evidence gets recorded is a governance question, covered in our guide to governance tools for enterprise AI model lifecycle management.
Retraining is one part of keeping a deployed system honest over its whole life, which is the discipline behind Neurotechnology Cloud’s applied AI solutions: monitoring, drift detection and controlled retraining built into hosted models from the first deployment, on GPU infrastructure sized for the retraining load the model will actually generate.
