Model Testing and Validation

A machine learning model is software whose behaviour is learned from data rather than written as rules. A fraud detector, a product recommender, a document classifier and a demand forecaster are all models. Testing them uses familiar QA instincts (what is the oracle, where are the edge cases, what changed) applied to an artefact that is statistical: it will be wrong some of the time by design, and the question is how often, for whom and in what way.

This lesson covers the established practices for validating a trained model, and then contrasts them with evaluating LLM-based applications.

ML model testing vs LLM application evaluation

The two are related but not the same:

	Trained ML model (classic ML)	LLM application
What you test	A model your team trained for a specific task	A system built on a general-purpose model you usually do not train
Output	A label, score or number	Free text, structured output or actions
Oracle	Labelled ground truth for each example	Often fuzzy: rubrics, references, human or model judges
Core metrics	Accuracy, precision, recall, F1, AUC, error measures, calibration	Task success, faithfulness, relevance, safety, format validity
Main levers	Data, features, training, thresholds	Prompts, retrieval, model choice, guardrails

If your team trains models, this lesson applies directly. If you build on hosted LLMs, see LLM Evaluation; many ideas below (held-out data, slicing by group, drift) still carry over.

Data splits and leakage

A model must be judged on data it did not learn from. The standard approach divides labelled data into:

• Training set: used to fit the model.
• Validation set: used to choose settings (hyperparameters, decision thresholds, which model to keep).
• Test set: used once, at the end, to estimate real-world performance. If you keep tuning against it, it stops being a fair estimate.

Cross-validation (rotating which part of the data is held out) gives a more stable estimate on smaller datasets.

Data leakage is when information that would not be available at prediction time leaks into training, making the model look far better in testing than in production. Testers are well placed to spot it:

• Duplicates across splits: the same customer, document or image appears in training and test.
• Time leakage: training on future data to predict the past. For time-dependent problems, split by time (train on earlier, test on later).
• Target leakage: a feature that encodes the answer, such as a “chargeback_filed” field in a fraud model when chargebacks happen after fraud is confirmed.
• Group leakage: multiple records from one user in both train and test; split by user instead.

from sklearn.model_selection import train_test_split

# Stratify keeps the class balance similar in each split, which matters for rare classes.
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, stratify=y_temp, random_state=42
)

A fixed random_state makes the split reproducible, so results can be compared across runs.

Overfitting and underfitting

• Overfitting: the model memorises training data, including its noise. Training performance is excellent; validation performance is noticeably worse.
• Underfitting: the model is too simple to capture the pattern. Training and validation performance are both poor.

The check is simple and should be automated: record training and validation metrics for every model version, and flag a large gap. An unusually perfect test score is also a warning sign; it more often points to leakage than to a brilliant model.

Metrics that matter

Accuracy (the share of correct predictions) is misleading when classes are imbalanced. If 1% of transactions are fraud, a model that always says “not fraud” is 99% accurate and useless.

For classification, start from the confusion matrix:

	Predicted positive	Predicted negative
Actually positive	True positive (TP)	False negative (FN)
Actually negative	False positive (FP)	True negative (TN)

• Precision = TP / (TP + FP): when the model says positive, how often is it right?
• Recall = TP / (TP + FN): of the real positives, how many did the model find?
• F1 = the harmonic mean of precision and recall, a single number that is only high when both are.

Which matters more is a product decision. For fraud blocking, a false positive blocks a genuine customer; a false negative lets fraud through. For a medical screening triage, missing a case (low recall) is usually worse than an extra review. Write the trade-off down as an acceptance criterion, for example: “recall at least 0.85 with precision at least 0.60 on the test set”.

from sklearn.metrics import classification_report, confusion_matrix, precision_score, recall_score

y_pred = model.predict(X_test)
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))

assert recall_score(y_test, y_pred) >= 0.85, "recall below acceptance threshold"
assert precision_score(y_test, y_pred) >= 0.60, "precision below acceptance threshold"

Many models output a probability and use a threshold (often 0.5 by default) to decide the label. Changing the threshold trades precision against recall, so the threshold is part of what you test and version. Precision-recall curves and ROC AUC summarise performance across thresholds.

For regression models (predicting numbers), use error measures such as mean absolute error and check errors per segment, not just overall.

Calibration

A model is well calibrated if its probabilities mean what they say: of all cases it scores at 0.8, about 80% should be positive. Calibration matters whenever scores are shown to people or used for decisions such as “review anything above 0.7”.

from sklearn.calibration import calibration_curve

probs = model.predict_proba(X_test)[:, 1]
frac_positive, mean_predicted = calibration_curve(y_test, probs, n_bins=10)
for predicted, actual in zip(mean_predicted, frac_positive):
    print(f"predicted {predicted:.2f} -> actual {actual:.2f}")

Large gaps between predicted and actual in a bin indicate miscalibration. Calibration methods exist to correct it; your job is to check it and make it part of the acceptance criteria.

Fairness and bias checks

Overall metrics can hide poor performance for specific groups. Slice every key metric by relevant segments: region, language, device, age band, and protected characteristics where it is lawful and appropriate to use them for evaluation.

import pandas as pd
from sklearn.metrics import recall_score, precision_score

df = pd.DataFrame({"group": groups_test, "y": y_test, "pred": y_pred})
for name, part in df.groupby("group"):
    print(
        name,
        "n=", len(part),
        "precision=", round(precision_score(part.y, part.pred, zero_division=0), 3),
        "recall=", round(recall_score(part.y, part.pred, zero_division=0), 3),
    )

What to look for:

• Different error rates between groups, for example a much lower recall for one region.
• Small sample sizes: a group with 30 examples gives a noisy estimate; report counts with metrics and collect more data before drawing conclusions.
• Proxy features: a postcode can act as a proxy for ethnicity or income even if those are not features.

Fairness has several formal definitions that cannot all be satisfied at once, so which one applies is a policy decision for the product, legal and ethics stakeholders. Testers make the disparities visible and make sure the agreed criteria are checked on every model version. Open-source libraries such as Fairlearn provide group metrics and mitigation methods.

Robustness

A robust model gives sensible outputs when inputs are slightly different from what it saw in training. Test ideas:

• Perturbations that should not change the answer: typos, casing, extra whitespace, synonyms, reordered fields, small image rotations or brightness changes.
• Boundary and invalid inputs: missing values, values outside training ranges, empty text, extremely long inputs, unexpected categories. The surrounding system should validate or handle them, and the model should not produce confident nonsense.
• Invariance and directional tests: changing a customer’s name should not change a credit decision; increasing a transaction amount should not lower the fraud score, all else equal.

def test_name_change_does_not_change_prediction(model, sample_applications):
    for app in sample_applications:
        original = model.predict_proba([app.features()])[0, 1]
        renamed = model.predict_proba([app.with_name("Alex Example").features()])[0, 1]
        assert abs(original - renamed) < 0.01, f"prediction depends on name for {app.id}"

These behavioural tests read like ordinary unit tests and run in CI against each new model version. Adversarial robustness (inputs crafted to fool a model) is a specialised security area; tools exist for it, including Garak for probing LLMs, but treat this as specialist work.

Drift in production

A model validated today degrades as the world changes:

• Data drift: input distributions shift (new customer segments, a new app version sending different values, seasonal patterns).
• Concept drift: the relationship between inputs and the right answer changes (fraudsters change tactics).
• Upstream changes: a field changes units or starts arriving empty because of a bug in another system. This is the most common and most preventable.

Monitoring:

• Validate inputs against expected schemas, ranges and null rates before they reach the model.
• Compare live feature distributions with the training data using statistical tests or measures such as the population stability index, per feature, on a schedule.
• Track prediction distributions (for example, the share of transactions flagged).
• Measure real performance when ground truth arrives (chargebacks, user corrections), which may be days or weeks later.

Open-source tools such as Evidently generate drift and data quality reports comparing a reference dataset with current data, which can run as scheduled jobs and feed alerts.

A QA checklist for a model release

• Splits are reproducible and free of duplicate, time, target and group leakage.
• Training versus validation gap is within an agreed limit.
• Test-set metrics meet written acceptance criteria (for example precision, recall, calibration).
• Metrics are sliced by agreed segments with sample counts; disparities are reviewed.
• Behavioural tests (invariance, directional, invalid inputs) pass.
• The new version is compared against the current production model on the same test set; no significant regressions in any key segment.
• Input validation and drift monitoring are configured, with owners for alerts.
• A rollback path to the previous model version exists and has been tried.

Practice exercise

A team presents a new loan-default model: “accuracy 97% on the test set, much better than the old model’s 91%”. Default rate in the data is about 4%. The test set was created by randomly sampling rows from three years of data, and some customers have several loans. What questions do you ask and what checks do you require before release?

Show a model answer

Questions and concerns:

• Accuracy is misleading at a 4% positive rate: predicting “no default” for everyone scores about 96%. Ask for the confusion matrix, precision, recall and F1 for the default class, plus precision-recall curves, compared with the old model on the same test set.
• Leakage: random row sampling across three years mixes time periods and puts the same customer in both training and test. Require a time-based split (train on earlier, test on the most recent period) and grouping by customer. Check for features that are only known after default (such as collections flags).
• Threshold: which threshold produced the labels, and does it match the business’s cost of false approvals versus false rejections?
• Calibration: if probabilities drive pricing or manual review, check calibration curves.
• Fairness: slice metrics by agreed segments (and protected characteristics where lawful for evaluation), with sample counts; involve compliance, since lending is regulated.
• Robustness: invariance tests (changing name or other irrelevant fields does not change the score), directional tests (higher debt-to-income should not lower risk), invalid input handling.
• Operations: input validation, drift monitoring on key features and prediction rates, performance monitoring as defaults are observed, model and data versioning, and a tested rollback to the old model.

Release only when the model beats the current one on the agreed metrics under a leakage-free split, meets fairness criteria and passes behavioural tests.

Key takeaways

• Model testing is statistical: define acceptable error rates, for whom, and which errors matter more.
• Use separate train, validation and test sets; hunt for duplicate, time, target and group leakage.
• Prefer precision, recall, F1 and confusion matrices over accuracy on imbalanced data; test thresholds and calibration.
• Slice metrics by groups to expose fairness problems, with sample sizes, and involve policy owners in choosing criteria.
• Behavioural tests check robustness; drift monitoring (for example with Evidently) catches degradation after release.
• Testing a trained ML model differs from evaluating an LLM application, but held-out data, slicing and monitoring apply to both.