Every machine learning team hits the same obstacle. A model performs well during training and staging, but after a few weeks of production, it begins to make nonsensical decisions. No one has altered any code. What has actually changed is the data.

Data debt refers to this state, a gap dating the model’s training data and the production data the model is encountering. Like technical debt, it creates no immediate failures. It results in a slow, tedious decline of the model’s decisions. By the time the degradation is noticed, the cause is impossible to determine. Upstream systems have changed multiple times and no one has documented the changes.

In this piece, I present methods for addressing data debt before it becomes costly. The methods I present are data debt prevention measures and not abstract ideas of data debt containment.

Where Data Debt Comes From

Data debt accumulates from a small, limited set of factors in a typical data production system.

Schema drift— an upstream team alters a field in such a way that the feature pipeline must adapt to the change, or, it breaks without a warning.
Null-rate creep— a field that previously was fully populated begins to show up empty, due to a broken integration or a policy change (ex. a form field becomes optional).
Distribution shift— the underlying distribution moved, but values remained valid and structurally unchanged. Examples of such distribution shifts are changes in customer demographics, new marketing channels affecting user behaviour, and evolving fraud behaviour.

Each of these is detectable. The problem is that most teams don't wire in detection until after an incident forces them to.

Building the Detection Layer

The goal is a checkpoint that runs on every batch (or every N minutes for streaming) and answers three questions: is the schema what I expect, are the nulls where I expect them, and has the distribution moved more than it should have?

1. Schema Drift Detection

```
import great_expectations as gx
context = gx.get_context()
validator = context.sources.pandas_default.read_csv("batch_2026_08_07.csv")

Column-level expectations

validator.expect_table_columns_to_match_ordered_list(
column_list=["user_id", "transaction_amount", "merchant_category", "timestamp"]
)
validator.expect_column_values_to_be_of_type("transaction_amount", "float64")
validator.expect_column_values_to_be_of_type("timestamp", "datetime64[ns]")
results = validator.validate()
if not results["success"]:
alert_on_schema_drift(results)
```
The key design decision: run this before the batch enters your feature store, not after. A schema check that fires after features are already computed tells you the damage is done.

2. Null-Rate Monitoring

Static null checks ("fail if nulls > 0") are usually too rigid for real data. What you want is a rolling baseline, compare today's null rate against a trailing window, not a fixed number.

```
import pandas as pd
def null_rate_check(df: pd.DataFrame, column: str, baseline_rate: float, tolerance: float = 0.05):
current_rate = df[column].isna().mean()
delta = abs(current_rate - baseline_rate)
if delta > tolerance:
raise DataQualityAlert(
f"{column} null rate moved from {baseline_rate:.2%} to {current_rate:.2%}"
)
return current_rate

baseline_rate computed as a trailing 30-day average, stored alongside your pipeline metadata

null_rate_check(batch_df, "merchant_category", baseline_rate=0.02, tolerance=0.03)
```

In Great Expectations, the equivalent expectation is:

validator.expect_column_proportion_of_unique_values_to_be_between( "merchant_category", min_value=0.0, max_value=0.05 )

But for null-rate specifically, expect_column_values_to_not_be_null with a mostly= parameter lets you set a tolerance rather than a hard fail:

validator.expect_column_values_to_not_be_null("merchant_category", mostly=0.97)
3. Distribution Shift — KS-Test and PSI

This is where most teams stop, because schema and nulls are easy to reason about and distribution shift is not. But it's usually the one that actually breaks your model, since a feature can be fully populated and correctly typed while still meaning something completely different than it did in training.

Kolmogorov-Smirnov test works well for continuous features when you want a statistical significance test:

from scipy.stats import ks_2samp def check_distribution_shift(reference: pd.Series, current: pd.Series, alpha: float = 0.05): statistic, p_value = ks_2samp(reference, current) drifted = p_value < alpha return {"statistic": statistic, "p_value": p_value, "drifted": drifted} result = check_distribution_shift( reference=training_df["transaction_amount"], current=batch_df["transaction_amount"] )

Population Stability Index (PSI) is more common in production because it gives you a continuous severity score instead of a binary pass/fail, which makes it easier to set alerting tiers:

import numpy as np def calculate_psi(reference: pd.Series, current: pd.Series, bins: int = 10): breakpoints = np.linspace(0, 100, bins + 1) bin_edges = np.percentile(reference, breakpoints) bin_edges[0], bin_edges[-1] = -np.inf, np.inf ref_counts = np.histogram(reference, bins=bin_edges)[0] / len(reference) cur_counts = np.histogram(current, bins=bin_edges)[0] / len(current) # avoid division by zero / log(0) ref_counts = np.clip(ref_counts, 1e-6, None) cur_counts = np.clip(cur_counts, 1e-6, None) psi = np.sum((cur_counts - ref_counts) * np.log(cur_counts / ref_counts)) return psi psi_score = calculate_psi(training_df["transaction_amount"], batch_df["transaction_amount"])

The commonly used thresholds:

For categorical features, run PSI on the category proportions instead of percentile bins, and pair it with a chi-squared test if you want a formal significance check alongside the severity score

Wiring This Into a Real Pipeline with Evidently AI

Writing every check by hand doesn't scale past a handful of features. Evidently AI is built for exactly this — it generates a full drift report across every column with a single call, which is much more practical once you're monitoring dozens or hundreds of features.

from evidently.report import Report from evidently.metric_preset import DataDriftPreset, DataQualityPreset report = Report(metrics=[ DataDriftPreset(), DataQualityPreset(), ]) report.run(reference_data=training_df, current_data=batch_df) report.save_html("drift_report.html") drift_result = report.as_dict() drifted_columns = [ metric["result"]["column_name"] for metric in drift_result["metrics"] if metric["metric"] == "ColumnDriftMetric" and metric["result"]["drift_detected"] ] if drifted_columns: trigger_retraining_review(drifted_columns)

This is the piece most teams are missing: a single scheduled job that runs schema checks, null monitoring, and drift detection together, and routes the output somewhere a human actually sees it — not just a log line that scrolls past.

A Minimal Production Checkpoint

Putting it together, a reasonable checkpoint that runs on every batch looks like this:

```
def data_quality_checkpoint(batch_df: pd.DataFrame, reference_df: pd.DataFrame, config: dict):
results = {}

# 1. Schema
results["schema"] = validate_schema(batch_df, config["expected_schema"])

# 2. Null rates
results["nulls"] = {
    col: null_rate_check(batch_df, col, config["null_baselines"][col])
    for col in config["monitored_columns"]
}

# 3. Distribution shift
results["psi"] = {
    col: calculate_psi(reference_df[col], batch_df[col])
    for col in config["numeric_columns"]
}

severity = classify_severity(results)  # e.g. "ok" / "warn" / "block"
if severity == "block":
    halt_pipeline_and_alert(results)
elif severity == "warn":
    log_and_notify(results)

return results

```

The classify_severity step matters more than it looks — most teams either alert on everything (and get ignored within a week) or alert on nothing until it's a full outage. A three-tier system (ok / warn / block) with PSI and null-rate thresholds tuned per feature, not globally, is usually the right middle ground.

Where Retraining Fits In

Detection alone can't fix something — it's only the start. The feature's importance to the model influences your decision after a "warn" or "block" signal:

High-importance feature drifted → treat as a blocking issue, pause automated decisioning if the model is high-stakes (credit, fraud), and route to a retraining review.
Low-importance feature drifted → log it, but don't halt the pipeline; feed it into your next scheduled retraining cycle instead.

This is also where model explainability tools (SHAP, feature importance rankings) become useful as an input to your data quality system, not just a reporting artefact. Knowing which features actually move your model's predictions tells you which drift signals deserve a "block" and which are noise.

The Takeaway

Data debt has no clearest discovery point. It appears where the assumption is made that the successful run of a pipeline means no changes to the data. Successful schema checks identify structural changes. Successful null-rate checks identify missing data. Subtle changes may go undetected and are more difficult to check. Individually, each of these checks is not challenging to implement. The failure is the belief that these can be added after something has gone wrong in production.

Newspaper

Data Debt in Production ML Pipelines: Detection and Remediation at Scale

Column-level expectations

baseline_rate computed as a trailing 30-day average, stored alongside your pipeline metadata