MLOps as Three Levels, and the One Thing to Build Next

Somebody left. You have inherited three models. One runs as a scheduled job on a machine nobody can name, one lives behind an endpoint that a mobile app depends on, and the third exists only as a notebook with a cell that reads final_final_v2. None of them has a test. All of them are, as far as anyone can tell, working.

The instinct at this point is to go shopping. There are several hundred products that describe themselves as MLOps platforms, each promising to solve the thing you have not yet defined. Resist it for an afternoon. The useful question is not which tool. It is which of three levels you are on, because the answer determines a single next thing to build, and building that one thing in the right order costs a fraction of what building the wrong one costs.

What is MLOps, and why did it need a name?

The Software Engineering Institute at Carnegie Mellon defines it plainly in its introduction to MLOps as a set of practices for streamlining and automating the lifecycle of machine learning models in production, sitting where machine learning, DevOps and data engineering overlap. That definition is accurate and slightly useless on its own, because it does not say why the overlap needed a separate word.

The honest origin is a paper rather than a product launch. In 2015, Sculley and nine colleagues at Google published Hidden Technical Debt in Machine Learning Systems at NIPS. Its argument was that machine learning systems are unusually cheap to build and unusually expensive to keep, and it named the reasons: boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration sprawl and changes in the external world.

Two findings from that paper do more work than the rest. The first is the ratio. A mature system, the authors wrote, might end up being at most 5 percent machine learning code and at least 95 percent supporting code, most of it glue written to move data into and out of general purpose packages. The second is what they called CACE, changing anything changes everything. No input to a trained model is really independent of the others, so altering one feature can shift the weight and the usefulness of every remaining feature. The same applies to hyperparameters, sampling, thresholds and data selection. It also applies to how you split the data, which is where forecasting projects fail silently: a random train test split on a time series leaks the future into training.

Those two facts explain the discipline. If 95 percent of the system is plumbing, then plumbing is the job. And if changing anything changes everything, then you cannot verify a change by reading the diff. You have to run it and measure it. The Wikipedia entry on the history of MLOps traces the term's appearance to the mid 2010s for exactly this reason, as projects began moving out of experimentation and into production and discovered that the second half was the expensive one.

What does this have to handle that ordinary operations does not?

Five things, and each one prevents a specific failure that a healthy deployment pipeline would sail straight past. This is the table worth internalising, because it explains why you cannot simply point your existing continuous integration setup at a model and declare the problem solved.

The row that surprises people is the first one. A machine learning system can be in the middle of a serious failure while every dashboard is green, because the failure is a slow statistical divergence and not an exception. Google's architecture guidance calls this model decay and treats it as the normal condition rather than an incident, which is the correct framing and a genuinely different mental model from the one operations engineering usually runs on.

The three levels, and the one thing each one adds

Google Cloud's reference document on continuous delivery and automation pipelines in machine learning sets out three levels of maturity. It is the most widely reproduced model in the field and it is worth reading in the original, because most secondary summaries drop the part that matters: what is actually being deployed changes at every level.

Level 0: a manual process

Every step is driven by hand. Data analysis, preparation, training and validation happen inside interactive notebooks run by a person. The trained model is passed to an operations team as a static artefact. Releases happen a few times a year. Testing lives in the notebook. What gets deployed is the model, as a prediction service, and nothing else.

The symptom that tells you that you are here: somebody has to be available for the system to improve. If a person going on holiday means no model can be retrained, you are at level 0 regardless of how sophisticated the model itself is.

The single next thing to build: not a pipeline. Monitoring. Level 0's defining hazard is that nobody knows the model has decayed, and you cannot justify pipeline work you cannot measure the need for. Log the input feature distributions and the prediction distribution, then compare this week against the training window. A modest weekly report on that comparison earns its keep faster than any orchestration tool.

What skipping it costs: a model that has been wrong for eight months and a stakeholder who found out before you did. This is the most common failure in the field and it costs trust, which is more expensive to rebuild than any pipeline.

Level 1: pipeline automation

The unit of deployment changes. Instead of shipping a trained model, you ship the pipeline that produces trained models, and it runs continuously in production on fresh data. The Google model calls this continuous training, and it brings four components with it.

Data validation runs before training and rejects the run rather than training on bad input. It checks for schema skew, meaning unexpected, missing or malformed features, and value skew, meaning a statistical shift large enough to indicate the world has moved. Model validation runs after training and compares the candidate against the model currently serving, across segments rather than in aggregate, and refuses to promote a model that is worse. Metadata management records which pipeline version ran, when, with what parameters, over which data, producing which metrics, with pointers back to previous versions so a rollback is a lookup rather than an excavation. A feature store is optional and does one important thing: it makes the feature computed at training time and the feature computed at serving time the same object, which is the only reliable cure for training and serving skew.

The symptom that tells you that you are here: you can retrain without a human, but shipping a new kind of pipeline still means a person copying files. Retraining is automatic, changing the recipe is not.

The single next thing to build: the trigger. Level 1 without a considered trigger is a scheduled job wearing a costume. The interesting triggers are not the calendar. They are new data arriving, measured performance degradation and a detected shift in the input distribution. Pick the one that matches how your data actually behaves, and write down the threshold.

What skipping it costs: compounding manual work. Every model you add multiplies the retraining burden, so the team's capacity to improve anything falls as the portfolio grows. This is the level where organisations quietly stop adding models and nobody records why.

Level 2: continuous integration and delivery of the pipeline

The unit of deployment changes again. Now the pipeline components themselves are built, tested and deployed automatically. A commit triggers a build that tests feature engineering logic, verifies that training converges, checks numerical stability so a NaN cannot reach production, validates each component's artefacts and then tests the components against each other. Delivery adds infrastructure compatibility checks, prediction service API tests, latency and throughput testing, and progressive rollout by canary or split traffic.

The symptom that tells you that you are here: the bottleneck has moved from operations to ideas. Nobody is waiting on a deploy. They are waiting on someone to have a hypothesis worth testing.

The single next thing to build: at this level the honest answer is usually nothing more in this direction. Level 2 is where the marginal return on automation drops sharply for most teams, and effort is better spent on data quality or on problem selection.

What skipping it costs: very little, if you have fewer than about five models and a slow rate of experimentation. Level 2 is priced for organisations running many pipelines and changing them weekly. Building it early is the most common expensive mistake in this space, and Google's own document says as much: the levels are meant to be adopted gradually, not as a package.

Where does generative AI change this picture?

Substantially, and this is the part most maturity models have not caught up with. If your system calls a hosted model through an API rather than training its own, there is no training pipeline. No continuous training, no feature store, no retraining trigger, no model validation in the sense above. Roughly two thirds of the machinery described so far simply does not apply.

What survives is the part that was always about production rather than about training. Three things replace the pipeline.

Evaluation takes the place of model validation, and it is harder rather than easier, because there is no accuracy number waiting to be computed. You need a fixed set of inputs, an agreed way to judge outputs, and the discipline to run it before and after every change. We set out a workable method for that in building an evaluation you can actually re-run, and the point that carries over from MLOps is the one about segments: an aggregate score that improved can easily hide a category that got worse.

Prompt and configuration versioning takes the place of model lineage. The prompt, the model identifier, the temperature and the tool definitions are now the artefact that determines behaviour, and they change far more often than model weights ever did. CACE applies here with full force. Reordering two sentences in a system prompt can change the output distribution as thoroughly as retraining would, which is why the prompt belongs in version control alongside the code that sends it.

Observability takes the place of drift monitoring, and the thing that drifts is not your data. It is the vendor's model. A provider can update a model behind a stable name, and your system's behaviour changes without a single commit on your side. That is a genuinely new failure mode with no equivalent in the classical picture, and the only defence is to keep running your evaluation set on a schedule rather than only when you change something.

If you are self hosting an open weight model instead of calling an API, you land in between. The training pipeline is still absent, but serving is now yours, which brings back capacity planning, batching and version pinning. Our walkthrough of serving models with a vLLM inference server covers that operational surface, and it is closer to classical infrastructure work than to anything in the maturity model above.

How do you tell tooling from theatre?

By asking which of the five concerns in the first table a tool actually addresses, and refusing to accept more than one answer at a time. A great deal of what is sold as an MLOps platform is a workflow scheduler with a model registry attached, which is genuinely useful at level 1 and does nothing at all for drift or skew.

Two questions cut through most sales conversations. First: if my input distribution shifts by ten percent next month, which part of this notices? If the answer involves a dashboard nobody is rostered to look at, it does not notice. Second: if I need to reproduce the model that served traffic on a specific date, what do I run? A satisfying answer names a stored pipeline version, a data snapshot and a parameter set. An unsatisfying one names a person.

The 2021 paper Using AntiPatterns to avoid MLOps mistakes by Muralidhar and colleagues catalogues the ways enterprise machine learning operations go wrong in practice, and its most useful observation is that many failures are not technical errors at all. They come from insufficient knowledge of the surrounding context, meaning the team automated something correctly that should not have been automated in that form. No platform purchase fixes that category.

The order that works, and why it is boring

Start by writing down what is in production. Not what should be, what is. Three models, where each one runs, what data each one reads, what depends on each output, and who finds out first if it breaks. Undeclared consumers, one of the debts Sculley's paper named, are the single most common source of unpleasant surprises, and the only way to find them is to ask.

What should the first monitoring actually record?

Less than people expect, and none of it requires a platform. Four things, recorded weekly, cover the majority of what goes wrong.

Record the distribution of each input feature, as a handful of summary statistics rather than a full histogram: mean, median and the share of missing values will do. Record the distribution of the predictions themselves, because a model that starts predicting one class 90 percent of the time has told you something is wrong long before anyone measures accuracy. Record the volume, since a feature that silently stopped arriving looks identical to a feature whose value happens to be null. And record the delay before you learn the truth, which is the number almost nobody writes down: if a fraud model's real label arrives 60 days after the prediction, then your accuracy dashboard is a report on the model you were running two months ago, and any conclusion you draw from it today is out of date by construction.

That last one deserves emphasis because it decides which of the earlier triggers is even available to you. If ground truth takes weeks, performance degradation cannot be your retraining trigger, and you have to fall back on input distribution shift as a proxy. Teams routinely design a trigger they cannot compute in time and then wonder why it never fires.

Then add monitoring to whichever model has the most consequential output. One model, one week of data, one comparison against the training distribution. This is a day of work and it converts the whole subject from anxiety into evidence.

Only then build a pipeline, and build it for the model where the monitoring says the drift is real. A pipeline built for a stable model is a maintenance liability with no return. A pipeline built for a drifting one pays for itself on the first automatic retrain.

The thing to notice about that order is that it inverts the way the field is usually sold. Platforms are marketed level 2 first, because that is where the impressive diagrams live. The work that actually reduces risk runs the other way, and it starts with a spreadsheet and a question about who is reading the output. If your systems are increasingly assembled from hosted models and connected tools rather than trained weights, the same instinct applies to how you expose them, which is the reasoning behind how we think about the MCP server surface: define the boundary and the contract before you automate anything across it.

Three inherited models, then. Do not buy anything this week. Find out what depends on them, watch the one that matters most, and let the evidence tell you which level you are actually on.