Your Data Pipeline Fails at the Seam, Not the Stage

Every team can draw the pipeline. Ingest, clean, engineer features, store, train, serve, monitor. Seven boxes, arrows between them, a diagram that fits on one slide and survives three reorgs.

The diagram is not where the failures happen. They happen on the arrows.

A model that scored well in a notebook and behaves differently in production is usually not a modeling problem. It is a pipeline problem with a specific shape: the same feature was computed twice, by two pieces of code, and the two versions disagreed. The training job computed one definition. The serving path computed another. Nobody noticed because each one was internally consistent.

The stages are commodity. The contracts between them are the system.

The seam that matters

Call it training-serving skew, and treat it as the default state rather than an edge case.

In training, a feature like "average order value over the last 90 days" is computed in a batch job over historical data. In serving, the same feature has to be computed for one customer, right now, from an operational store that may not have the same history, the same timezone handling, or the same definition of "order."

Two implementations of one definition will drift. The first divergence is usually small: a null handling rule, a rounding choice, a filter that excludes test accounts in one path and not the other. The model does not fail loudly. It degrades, and the degradation looks like a business trend.

The fix is unglamorous. One definition of each feature, in one place, used by both paths. That is what a feature store is for, and a feature store is also a commitment: the batch and online paths compute from the same logic, and a test asserts that they agree on a sample. Without the test, the store is a database with better marketing.

Seven stages, one contract each

The standard list is fine. What is missing from most versions is the contract each stage owes the next one.

The last row is the one teams skip. What does the service do when the model is unavailable, or when an input arrives with a field missing? "It throws an error" is an answer, and it is usually the wrong one. A fallback rule that is written down and tested is a design decision. A fallback that emerges during an incident is an accident.

Backfills break more models than drift does

Data drift gets the attention. A backfill gets the incident.

Any correction to history, a late-arriving dataset, a fixed bug in a transformation, a replayed source system, changes the distribution the next model sees. If the training pipeline reads from a table that gets backfilled, then retraining after a backfill produces a model trained on different data than the one it replaces, and nobody approved that change.

Two practices reduce this to a manageable problem. Freeze the training set: when a model is trained, record the exact snapshot, and make retraining read from a snapshot rather than from whatever the table holds today. And treat a backfill as a change with a review, not as a maintenance task, because it changes the inputs to a decision system.

Point-in-time correctness belongs here too. A feature that joins to a dimension table using the current value will leak future information into a historical row. The model looks excellent offline and fails in production, because in production it cannot see the future. This is the most common way a team ships a model that has already been beaten by its own evaluation.

What to instrument

Monitoring usually starts with infrastructure: CPU, memory, latency, error rate. Those are necessary and they will not tell you the model stopped working.

Four signals belong next to them.

Freshness: how old is the newest record in each input, and when did it last arrive late. Null rate per field, because a field that quietly stops populating is a silent failure. Distribution shift on the top features, compared against the training snapshot rather than against yesterday. And prediction distribution, because a model whose outputs collapse toward one value has usually lost an input.

Add one more that is cheap and underused: the count of predictions where an input was imputed or defaulted. A rising imputation rate is an upstream failure arriving through the front door.

Alerts belong on the business metric as well as the model metric. If the pipeline is healthy and the outcome is wrong, the pipeline is not the problem, and the team needs to find out before the quarterly review does.

The pipeline is the record

There is a second reason to build the contracts carefully, and it is not performance.

When someone asks why a decision was made, the answer comes from the pipeline. Which data snapshot, which transformation version, which model artifact, which inputs. A pipeline that can reproduce a prediction from six weeks ago is a pipeline that can answer the question. One that cannot is a system that produces confident numbers with no history.

That is the difference between a demo and something an auditor will accept. The model is a component. The pipeline is the evidence.

The seven-stage diagram is a useful checklist and a poor design document. What determines whether the system holds up is the set of agreements between the stages: one feature definition, one snapshot per model, one schema per boundary, one fallback per service.

Those agreements are boring to specify and expensive to skip. Teams that write them down spend their incidents on infrastructure. Teams that do not spend them explaining why the model was right last quarter.

Start with the seam between training and serving. It is the one that fails quietly, and quiet failures are the ones that reach a customer.