Data Quality for Machine Learning: A Practical Checklist

When a model disappoints, teams often reach for a more powerful algorithm. Experienced practitioners look at the data first. Missing values, mislabelled examples, duplicates, stale records and samples that do not match real use cause more failures than the choice of model.

Google’s Rules of Machine Learning, a widely read engineering guide, makes a similar point. It advises starting with a simple model and a solid pipeline, logging metrics early and watching for dropped or stale data in the pipeline. It also suggests that model complexity should fit the amount of data you have. This article turns that outlook into a checklist for data quality in machine learning, usable by a small team building its first model and an organisation with a data platform.

Quality depends on the task

The same dataset can be good for one purpose and poor for another. A customer table with a few missing postcodes may be fine for churn prediction and unusable for delivery routing. Start by writing what the model must predict, for whom and under what conditions. Then ask what data would be needed to do that reliably.

The main dimensions

• Completeness: are values missing, and is the missingness random or systematic? If one group has more missing data, the model will perform worse for it.
• Accuracy: do values match reality? Sensor drift, typing errors and unit mix-ups all hurt.
• Consistency: do the same facts agree across sources and formats, such as date formats, currencies and category names?
• Timeliness: is the data fresh enough, and available at prediction time?
• Uniqueness: are there duplicate records or near duplicates, which can inflate evaluation scores?
• Validity: do values fall within allowed ranges and formats?
• Representativeness: does the sample reflect the population and conditions where the model will run, including rare cases?
• Label quality: are the target values correct, consistent and defined clearly?

Labels deserve special attention

For supervised learning, labels are the answer key. Problems include:

• Inconsistent labelling: different people apply different rules.
• Ambiguous definitions: “fraud”, “urgent” or “positive” mean different things to different reviewers.
• Noisy labels: random errors from rushed work.
• Proxy labels: a stand-in for what you truly want, such as “ticket closed” for “customer satisfied”.
• Delayed labels: outcomes known only after months, which slows learning.

Write a labelling guide with examples, train labellers, measure agreement between them and review disagreements. Send a sample for expert review.

Leakage: the silent score inflator

Leakage happens when training data contains information that will not be available at prediction time, or that reveals the answer. Examples include using a field filled in after the outcome, such as a cancellation reason in a churn model, or random splits that put records from the same customer in both training and test sets. Leakage gives great test scores and poor real results. Check each feature: would we know this at the moment we predict?

Bias and fairness

Data reflects the past, including its unfairness. Check performance across relevant groups, such as regions, age bands or languages where it is lawful and appropriate to do so. Look for under-represented groups and for features that act as proxies for protected characteristics. Document findings and mitigation. Legal rules on automated decisions differ by country and sector, so take advice for high-impact uses.

Profile and clean

1. Profile the data: counts, distributions, missing rates, ranges, unique values and relationships. Plot it.
2. Handle missing values with a documented rule: remove, fill using a sensible method or add an indicator. Avoid hiding systematic gaps.
3. Deal with outliers: decide whether they are errors or real rare events.
4. Standardise formats and units.
5. Deduplicate, including near duplicates.
6. Reconcile sources and fix conflicts.
7. Record each step, so the process can be repeated on new data. Keep raw data unchanged and apply cleaning in later layers.

Our article on AI data integration covers connecting and preparing data from many sources.

Split data properly

• For time-based problems, train on earlier periods and test on later ones, to mimic real use.
• Keep related records, such as all rows for one customer or device, in the same split.
• Hold out a final test set that is used only once.
• Make sure the test set reflects the data the model will see, including rare classes.

Automate data tests

Treat data checks like code tests:

• Schema checks: columns, types and allowed values.
• Range and rule checks: ages are positive, end dates follow start dates.
• Volume and freshness checks: row counts and update times.
• Distribution checks: compare new data with the training data to detect drift.
• Referential checks: keys exist in related tables.

Run them whenever data arrives, and stop the pipeline on severe failures. Alert owners, and track results over time.

Document the data

Create a simple data sheet for each dataset: source, collection method, time range, owner, known issues, intended use, limits and privacy status. Document labels and cleaning steps. This helps colleagues, auditors and your future self.

Privacy, consent and rights

Check that you are allowed to use the data for training. Respect consent, purpose limitation and retention rules, and handle personal data carefully through minimisation, pseudonymisation and access controls. Rules differ by country, so involve privacy specialists. Remember that some people have rights to access or delete their data, which affects stored training sets.

Monitor after launch

Data changes. Customer behaviour shifts, sensors age and upstream systems alter formats. Monitor:

• Input distributions and missing rates.
• Feature drift against training data.
• Prediction distributions.
• Model performance, using delayed labels where possible.
• Data pipeline health and freshness.

Set thresholds that trigger investigation or retraining. Collect feedback from users and add the errors you find to the training and test sets. Our guide to MLOps basics covers how to run models dependably.

Common mistakes

• Starting modelling before profiling the data.
• Ignoring label quality.
• Leakage through careless features and splits.
• One-off cleaning with no repeatable process.
• Assuming training data matches production data.
• No ownership for data quality.

A practical start

1. Pick the model’s purpose and the key decisions it supports.
2. Profile the data and list issues by impact.
3. Fix the top issues, and automate checks for them.
4. Build a simple baseline model, and learn from errors.
5. Improve data before adding model complexity.

How SoftSapien supports data readiness

SoftSapien helps teams assess data quality, build cleaning and validation pipelines, design labelling processes and monitor models in production. Explore our data science and big data services.

To review your data before you build, go to the contact page and chat with us, send a message or schedule a free consultation call.