Data Quality Management: A Practical Guide

Data quality management is the set of rules, measurements and routines that keep the data in your systems fit for the decisions and processes that depend on it. In practice it means agreeing what good looks like for each field that matters (a valid customer email, a complete ship-to address, an order date that is never in the future), checking the data against those rules on a schedule, publishing the results, and fixing root causes instead of patching symptoms. You do not need a large team or a platform license to start. You need six dimensions to describe problems precisely, written rules for the data you depend on most, a scorecard, and an owner for each rule.

The six data quality dimensions, with examples

Data quality dimensions give you a shared vocabulary. Without them every complaint sounds the same (the numbers are wrong), and you cannot tell a missing field from a stale feed. The six below cover almost every problem you will meet. Some frameworks add integrity (do foreign keys resolve) and reasonableness (does the value make sense in context); treat those as extensions of the same list.

Dimension	Question it answers	Example failure	Example rule
Accuracy	Does the value reflect the real world?	A customer’s billing state is NJ in the CRM, but every invoice is paid from DE.	Billing address matches the most recent paid invoice; a monthly sample is checked against source documents.
Completeness	Is anything required missing?	Leads arrive from the website with no phone number, so sales cannot call them.	Phone or email is populated for every lead with status New or later.
Consistency	Do copies of the same fact agree across systems?	A SKU is Active in the ERP and Discontinued on the web store.	Product status in the ERP and the e-commerce catalog must match for every SKU, checked nightly.
Timeliness	Is the data current enough for its use?	The inventory dashboard shows last night’s snapshot while the warehouse ships against live counts.	The inventory table is refreshed within the window the warehouse team agreed, and the dashboard shows the refresh time.
Uniqueness	Does each real-world thing appear once?	The same hospital exists as three customer records with three credit limits.	One customer record per tax ID; a fuzzy match on name plus postal code flags candidates for review.
Validity	Does the value follow the agreed format and allowed values?	A date of birth entered as 13/02/1990 in a US system; a state spelled New Jersy.	State is a two-letter USPS code; dates are stored as ISO 8601; email passes a syntax check before save.

Validity and completeness are the easiest to enforce because a form or an API can reject bad input at entry, so fix those first. Accuracy is the hardest to automate because it needs a trusted reference, so you usually measure it by sampling. Consistency and uniqueness problems come from the same fact being created in two systems, which is why they overlap with master data management and data lineage: you need to know which copy is the source of truth and how the others derive from it.

Writing data quality rules

A data quality rule is a testable statement about one data element. Vague goals (clean customer data) produce arguments; a rule produces a pass or a fail and a person who has to act on it. Keep every rule to this shape.

• Data element: system, table and field (CRM, Account, Billing State).
• Dimension: one of the six above.
• Rule logic: a plain-English sentence and the SQL or tool configuration that implements it.
• Scope: which rows the rule applies to (active customers only, orders from the last 90 days).
• Threshold and severity: the pass rate that triggers action, and whether a failure blocks the load, raises a warning or is informational. Some rules are zero-tolerance (duplicate tax IDs); most allow a small tolerance you set and revisit.
• Owner and maintainer: a business owner who decides what the rule means, and an engineer who keeps the check running.
• Action on failure: who gets the ticket, and what they do.

Do not try to cover every field. Pick one business process (quote to cash, procure to pay, hire to retire) and list the ten to twenty fields it cannot run without. Those are your critical data elements, and your first rules belong to them.

Example rules in SQL

Most teams can express their first rules as queries that return the failing rows. These run on SQL Server, Snowflake, Databricks SQL or a Microsoft Fabric warehouse with minor syntax changes.

-- Completeness: every active customer has a billing email
SELECT customer_id
FROM dim_customer
WHERE is_active = 1 AND (billing_email IS NULL OR billing_email = '');

-- Validity: billing state must be a USPS state code
SELECT customer_id, billing_state
FROM dim_customer
WHERE billing_state NOT IN (SELECT state_code FROM ref_us_state);

-- Uniqueness: one customer record per tax ID
SELECT tax_id, COUNT(*) AS copies
FROM dim_customer
WHERE tax_id IS NOT NULL
GROUP BY tax_id
HAVING COUNT(*) > 1;

If you transform data with dbt, the same rules live next to the models as tests (unique, not_null, accepted_values, relationships) and run in your CI pipeline. Whatever the tool, write the plain-English version of every rule first and have the business owner approve it, then version the implementation in git alongside the pipeline code so a rule change is reviewed like a code change.

Measuring and reporting data quality

A rule that runs but is never read changes nothing. Store every run’s result in a small table, then build a scorecard on top of it. The table needs only rule_id, run_at, rows_checked, rows_failed and pass_rate; everything else is derived.

Roll the results up by dimension (is completeness or timeliness the bigger problem), by system (does the CRM or the ERP produce most failures) and by owner (so the weekly review has a name next to every red cell). The scorecard should fit on one screen: the critical rules with pass rate against threshold, the trend over recent weeks, and the count and age of open issues. Power BI on the results table is enough for most teams.

Keep an issue log next to the scorecard and tag each issue with a root-cause category: entry error, missing validation at the source, integration mapping error, timing, duplicate creation, or an upstream change nobody announced. After a quarter, the distribution tells you where to spend money. If most failures are entry errors, add validation to the form; if most are mapping errors, fix the integration, not the data.

Keep the cadence small. Checks run every night. A steward and an engineer spend twenty minutes a week on new failures. Leadership sees the scorecard monthly, with the business cost attached where you can state it: returned mail, rejected claims, late invoices, hours spent reconciling.

Data quality tools

The tool matters less than the rules, the owners and the review habit. Still, the right tool removes friction. The options sort into tiers: build your own with SQL and a scheduler, open-source frameworks inside your pipelines, platform-native features, and commercial platforms that add profiling, cleansing and stewardship workflows.

Tool	Type	Best fit	Notes
SQL rules plus a scheduler (SQL Server Agent, Azure Data Factory, cron)	Build your own	Any team with one engineer and a database	No license cost; a results table plus a Power BI scorecard covers the basics.
dbt tests (dbt Core, dbt Cloud)	Open source / SaaS	Teams transforming data in Snowflake, Databricks, SQL Server, BigQuery or Fabric	Tests live beside the models and run in CI.
Great Expectations, Soda	Open source / SaaS	Python or YAML-centric pipelines on Airflow, Azure Data Factory or Databricks	Expectation suites or SodaCL checks, profiling and generated documentation.
Snowflake data metric functions	Platform-native	Snowflake customers	System metrics for nulls, duplicates and freshness, plus custom metrics scheduled on tables.
Databricks pipeline expectations and Unity Catalog monitoring	Platform-native	Databricks customers	Expectations can warn, drop the row or fail the pipeline.
Azure Data Factory Assert transformation, Fabric Dataflows	Platform-native	Microsoft data stacks	Row-level assertions inside mapping data flows; column profiling in Power Query.
Microsoft Purview Data Quality	Governance platform	Organizations that catalog Fabric, Azure and SQL assets in Purview	Rules, suggested rules and quality scores per asset, linked to the catalog and lineage.
Informatica Data Quality, Ataccama ONE, Collibra Data Quality	Commercial platform	Larger or regulated estates with many sources, ERP and legacy systems	Profiling, standardization, address validation and cleansing at scale.

If you also need a catalog, a business glossary or policy workflows, Data Governance Tools Compared: Purview, Collibra, Alation, Atlan covers how the governance platforms bundle quality with lineage and stewardship.

A starter process for data quality management: the first 90 days

This is the sequence we use with organizations of every size. The steps are the same for a founder with two systems and for a division with twenty; only the tooling and the number of rules change.

1. Days 1 to 10: pick one process and one owner. Choose the process whose data problems cost the most visible money. Name the business owner and the technical maintainer.
2. Days 10 to 20: list the critical data elements. Ten to twenty fields, each with its system of record and an agreed definition.
3. Days 20 to 30: profile the data. Counts, null rates, distinct values, min and max, top values. Profiling tells you which dimensions are failing before you guess.
4. Days 30 to 50: write and approve the rules. One rule per element and dimension that matters. Expect ten to fifteen rules, not fifty.
5. Days 50 to 65: automate the checks. Scheduled SQL, dbt tests, platform-native metrics or a framework, depending on your stack. Every run writes to the results table.
6. Days 65 to 75: publish the scorecard and the issue log. Make it visible to the people who enter the data, not only to those who consume it.
7. Days 75 to 90: fix at the source and hold the first reviews. Required fields and pick lists in the CRM, validation in the web form, duplicate checks at entry. Hold the weekly review and the first monthly readout, then pick the second process.

A twelve-person online retailer on Shopify and QuickBooks Online profiles exports in Power Query, keeps eight rules in a shared sheet and reviews them on Fridays. A 250-person distributor on Dynamics 365 Business Central and Salesforce runs dbt tests against a SQL Server warehouse and publishes a Power BI scorecard. A division of a manufacturer on SAP S/4HANA with Snowflake uses data metric functions for freshness and duplicates, Informatica for address standardization and Purview for the catalog. The practices are identical; the budget is not.

Once the first domain is stable, what keeps it stable is governance: named owners, definitions people can find, and a review rhythm. Data Governance Best Practices: 12 That Actually Stick covers how to keep that going without a large program.

Where PS3G fits

PS3G helps organizations of every size set up data quality management as part of data governance services: an assessment of your critical data, written rules with owners, the scorecard, and the stewardship rhythm that keeps it alive. When the fix is technical, our data engineering team builds the pipelines, warehouses and tests on SQL Server, Snowflake, Databricks or Microsoft Fabric, with the checks wired into the pipeline rather than bolted on afterward. A founder usually needs a short assessment and a few rules; a larger company usually needs the full starter process and a dedicated engineer for a few months. The engineering is the same; the engagement is scoped to what you need.

Related reading

• Data Governance Tools Compared: Purview, Collibra, Alation, Atlan
• Data Lineage: What It Is, Why It Matters and How to Capture It
• Data Governance Best Practices: 12 That Actually Stick
• Data governance services

Frequently asked questions

What is data quality management in simple terms?

Data quality management is deciding what good data looks like for the fields your business depends on, checking the data against those rules on a schedule, and fixing the causes of failures. It combines written rules, automated checks, a scorecard people can see, and named owners. It is a routine, not a one-time cleanup project.

What are the six data quality dimensions?

Accuracy (does the value match reality), completeness (is anything required missing), consistency (do copies agree across systems), timeliness (is the data fresh enough for its use), uniqueness (does each real thing appear once) and validity (does the value follow the agreed format and allowed values). Some frameworks add integrity and reasonableness.

Which data quality tools are free or low cost?

SQL rules on a scheduler cost nothing beyond the database you already run. dbt Core, Great Expectations and Soda Core are open source and run inside your pipelines. Snowflake, Databricks, Azure Data Factory and Microsoft Fabric include native checks. Commercial platforms from Informatica, Ataccama, Collibra and Microsoft Purview add profiling, cleansing and stewardship workflows for larger estates.

Who should own data quality in a small company?

The person who feels the pain of bad data in a given process should own the rules for it: the operations lead for orders, the controller for customers and invoices, the sales manager for leads. One technical person maintains the checks. In a small company that is often two people and a weekly twenty-minute review, and that is enough to start.

How is data quality management different from data governance?

Data governance decides who owns which data, what it means and what policies apply. Data quality management is one of the practices governance puts in place: it measures whether the data meets the agreed standard and drives the fixes. You can run quality checks without a formal governance program, but the rules stick better when owners and definitions already exist.