What is Data Quality?

How it works

Quality is usually broken into dimensions, each with typical checks:

Dimension	Example check
Completeness	Required columns are not null; expected row counts arrived
Uniqueness	The primary key or grain has no duplicates
Validity	Values are in an accepted set or range, match a format
Consistency	Foreign keys exist in their dimension; totals reconcile with the source
Freshness	The newest row is recent enough for the schedule

Each check is a query that returns failing rows; zero means pass. Checks run where the data changes: on landing (schema and volume), after each transformation (keys, relationships, business rules) and before publishing (reconciliation, freshness). Severity decides the action: an error stops the pipeline or quarantines the batch, a warning alerts the owner but lets the data through.

Tools package this pattern: dbt tests (not_null, unique, accepted_values, relationships, source freshness), Great Expectations and Soda, Databricks expectations in declarative pipelines, and data observability platforms that also learn normal volumes and distributions to catch anomalies nobody wrote a rule for.

Example

Five checks over a small load, each returning the number of failing rows. Four fail, so this batch should not be published (PostgreSQL 16; the freshness check uses a fixed “now” to keep the output stable).

CREATE TABLE orders (order_id int, customer_id int, status text, amount numeric, loaded_at timestamp);
INSERT INTO orders VALUES
  (1, 10, 'shipped', 40, '2026-10-22 06:00'), (2, NULL, 'shipped', 25, '2026-10-22 06:00'),
  (2, 11, 'pending', 25, '2026-10-22 06:00'), (3, 12, 'shiped', -5, '2026-10-22 06:00');

SELECT 'customer_id not null' AS check_name, count(*) AS failing_rows FROM orders WHERE customer_id IS NULL
UNION ALL
SELECT 'order_id unique', count(*) FROM (SELECT order_id FROM orders GROUP BY order_id HAVING count(*) > 1) d
UNION ALL
SELECT 'status accepted values', count(*) FROM orders WHERE status NOT IN ('pending', 'shipped', 'cancelled')
UNION ALL
SELECT 'amount >= 0', count(*) FROM orders WHERE amount < 0
UNION ALL
SELECT 'loaded in last 24h', (max(loaded_at) < timestamp '2026-10-22 12:00' - interval '24 hours')::int FROM orders;

       check_name       | failing_rows
------------------------+--------------
 customer_id not null   |            1
 order_id unique        |            1
 status accepted values |            1
 amount >= 0            |            1
 loaded in last 24h     |            0

Common mistakes

• Checking only at the end, so bad data has already spread to every downstream table.
• Writing hundreds of warnings nobody owns, until alerts are ignored.
• Testing columns but not the grain, so duplicate rows pass every not-null and value check.
• Having no plan for failures: no quarantine table, no owner, no rerun process.