MLOps Best Practices: Scaling ML Models in Production

A machine learning model that works in a notebook is roughly ten percent of a machine learning system. The other ninety percent is the part that decides whether it still works in six months, and it is the part that stalls most projects.

This is a working guide to that ninety percent: what to build, in what order, and which of the widely-repeated best practices are worth the effort at your stage.

The failure mode you are actually defending against

Traditional software fails loudly. A bug throws an exception, a service returns a 500, an alert fires and someone investigates.

Machine learning fails quietly. The model keeps returning confident predictions with normal latency and a zero error rate. The predictions are simply worse than they were, because the world moved and the model did not. Nobody notices until a business metric shifts and someone spends three weeks tracing it back.

Every MLOps practice below exists to convert a silent degradation into a visible signal.

Hold that as the test. If a practice does not eventually help you detect or recover from silent degradation, it is probably ceremony, and you can defer it.

Reproducibility comes first

Before deployment, before monitoring, before any of the interesting work: you need to be able to rebuild any model you have ever shipped. Without that, every subsequent practice is built on sand, because you cannot compare a degraded model against the one that worked.

• Code - the training pipeline in git, not a notebook. Notebooks are for experiments; the artefact that produces a shipped model has to be a script that runs unattended.
• Data - the exact training set. Not "the customers table as of roughly December". A content hash or an immutable snapshot, because the table has changed since and you cannot reconstruct it.
• Environment - pinned dependencies and a container image. A model trained against one version of a library and served against another is a class of bug that is genuinely painful to find.

The test is simple and worth actually running: pick a model in production, and rebuild it from scratch on a clean machine. If the output differs, you do not have reproducibility, you have a story about reproducibility.

MLflow, SageMaker Model Registry, Azure ML or Vertex AI all do this. So does a well-disciplined object store with a metadata table, and for a team shipping its first two models that is a legitimate choice.

What matters is not the tool but that every model carries its lineage: which code, which data, which parameters, what it scored on the held-out set, who approved it and when. When a model misbehaves, this record is the first thing you reach for.

Deployment: pick the simplest pattern that fits

There are three serving patterns, and teams routinely choose a harder one than their problem requires.

Start at the top of that table and move down only when the business requirement genuinely forces it. A surprising number of real-time endpoints serve predictions that are consumed by a dashboard someone reads once a day. Batch would have been cheaper to build and far cheaper to operate.

Probably not at first, and this is worth being blunt about because it is where a lot of MLOps budget disappears.

Kubernetes solves real problems: heterogeneous workloads, GPU scheduling, multi-team isolation, fine-grained autoscaling. It also introduces a permanent operational burden that a small team pays for every week. If you are serving two or three models at moderate volume, a managed container service - Cloud Run, Azure Container Apps, ECS Fargate - handles it with a fraction of the overhead.

The signals that you have genuinely outgrown managed containers are worth naming: GPU inference where you need to pack multiple models onto one card, more than roughly a dozen models with different scaling profiles, or a hard requirement to run the same stack across clouds. Short of those, the simpler platform wins.

Every model deployment should be reversible in seconds without a rebuild. The standard progression:

1. Shadow mode - the new model scores real traffic, its predictions are logged and discarded. You get a live comparison against the incumbent at zero risk.
2. Canary - a small share of real traffic, with automated rollback if the guard metrics move.
3. Progressive rollout - widening the share as the evidence holds.

Shadow mode is the step that gets skipped and the one that pays for itself fastest. Offline evaluation tells you how the model performs on the data you chose; shadow mode tells you how it performs on the data you actually get, which is never quite the same distribution.

Autoscaling: the part that surprises people

Scaling ML inference does not behave like scaling a web service, for three reasons that all bite at once.

A web container starts in a second. A container that must load a multi-gigabyte model into memory - or worse, onto a GPU - can take a minute or more. Scale-from-zero is therefore usually a false economy for anything user-facing: you save on idle compute and pay in timeouts during exactly the traffic spike you scaled up for.

The practical answers are keeping a warm floor of instances, loading the model from a local disk cache rather than object storage on every start, and scaling on a leading indicator like queue depth rather than a lagging one like CPU.

Inference is often memory-bound or GPU-bound while CPU sits comfortably low. Autoscaling on CPU will happily let a service saturate and queue while reporting thirty percent utilisation. Scale on request queue depth or on p95 latency against your actual SLO.

Most inference runtimes are dramatically more efficient processing sixteen requests together than one at a time - frequently several times the throughput on the same hardware. Dynamic batching, where the server waits a few milliseconds to accumulate a batch, is the highest-leverage optimisation available for GPU serving.

The trade-off is a small latency floor added to every request. If your SLO is 500ms, a 20ms batching window is invisible and roughly triples your throughput. If your SLO is 30ms, it is not available to you.

Monitoring: four layers, in order of value

Build these in sequence. Each is useful on its own, and skipping ahead to the sophisticated one while missing the basic one is a common and expensive mistake.

Latency, error rate, throughput, saturation. Ordinary application monitoring, and it catches the ordinary failures - which are still the majority of your incidents. If this is not solid, nothing above it matters.

Track the distribution of incoming features against the training distribution. Population Stability Index or a Kolmogorov-Smirnov test per feature is enough to start, and either will catch the most common real-world failure: an upstream system changed a field, and a feature that was a percentage is now a fraction.

Data drift monitoring catches integration breakage as much as it catches genuine distribution shift, and integration breakage is far more frequent.

Track the distribution of predictions themselves. A fraud model that suddenly flags eight percent of transactions where it used to flag one percent has told you something important, and it has told you before any labels arrive.

This layer is cheap - it needs no ground truth - and it is the one most teams are missing.

The one everybody wants first and can rarely build first, because it needs labels, and labels arrive late. A churn model predicting ninety days out cannot be evaluated for ninety days.

Where labels are delayed, use proxies - agreement with human reviewers on a sampled subset, downstream business metrics, or the rate at which predictions are manually overridden. An override rate climbing steadily is a real signal, and it is available immediately.

Retraining: schedule or trigger?

Scheduled retraining is simpler and it is the right default. Monthly or quarterly, with the same pipeline, automatically evaluated against the incumbent, promoted only if it wins on the held-out set.

Triggered retraining - firing on a drift threshold - sounds better and behaves worse in practice. Drift metrics are noisy, thresholds are hard to set, and the common outcome is retraining on a week of anomalous data and shipping a model that is worse. If you do use triggers, gate them on a human approving the promotion.

One rule regardless of approach: a retrained model is a new model. It goes through the same shadow and canary progression as the original. "It is just a retrain" is how a bad model reaches production without review.

The feature store question

Feature stores solve one genuine and painful problem: training-serving skew, where the feature computed during training differs subtly from the one computed at inference. The classic version is an aggregate that includes data from after the prediction point in training and cannot at serving time, which produces a model that scores beautifully offline and disappoints in production.

They are also a significant piece of infrastructure. For a team with one or two models, defining features once in shared code that both paths import solves the same problem for a fraction of the effort. The store earns its place when several teams need the same features, or when point-in-time correctness across many entities has become genuinely hard to reason about.

A realistic maturity path

If you are starting from a model in a notebook, this is the order that has worked:

1. Training as a script in git, running unattended, producing a versioned artefact.
2. A registry entry recording code, data and metrics for every model.
3. Batch or endpoint serving in a container, deployed through CI like any other service.
4. Layer 1 and Layer 3 monitoring - service health and prediction distribution. Both are cheap and together they catch most incidents.
5. Shadow deployment for the next model version.
6. Layer 2 drift monitoring on the features that matter most.
7. Scheduled retraining with automated evaluation and human promotion.
8. Layer 4 accuracy tracking once labels are available at useful latency.

Steps one to four take most teams a few weeks and eliminate the majority of production incidents. The rest is genuine improvement rather than firefighting, and it can be paced against how much the model actually matters to the business.

What to take from this

MLOps has an unhelpful reputation as a large platform you adopt. In practice it is a set of habits that each make silent failure visible, and you can adopt them one at a time in the order above.

The teams we see struggling are rarely the ones with the least sophisticated tooling. They are the ones who cannot rebuild a model from three months ago, and therefore cannot tell whether today's problem is new.