What is a Data Catalog?

How it works

The word covers two related layers:

• Technical catalogs (metastores) tell engines what a table is: its schema, location, format and partitions. Hive Metastore, the AWS Glue Data Catalog, Unity Catalog, Snowflake’s information schema and Iceberg REST catalogs play this role. Without one, every engine would have to be told where each table’s files are.
• Discovery and governance catalogs add what humans need: descriptions, owners, glossary terms, tags such as PII, usage statistics, quality status and lineage. Examples include Unity Catalog’s Catalog Explorer, Microsoft Purview, AWS SageMaker Catalog (formerly DataZone), Alation, Collibra, DataHub and OpenMetadata.

Modern platforms merge the two. Unity Catalog, for instance, holds the three-level namespace (catalog.schema.table), grants permissions on it, records lineage automatically and lets teams tag and document assets in one place. Metadata gets into a catalog by crawling (Glue crawlers infer schemas from files), by integration (dbt pushes model and column descriptions), or by people documenting what they own. The catalog is only as useful as that metadata is complete and current.

Example

The smallest catalog is the database’s own: table and column comments stored with the schema and queried from information_schema and the comment functions (PostgreSQL 16).

CREATE TABLE sales.orders (order_id int PRIMARY KEY, customer_email text, amount numeric(10,2));
COMMENT ON TABLE sales.orders IS 'One row per order. Owner: data-platform team. Source: shop DB via CDC.';
COMMENT ON COLUMN sales.orders.customer_email IS 'PII: customer email, masked for analysts.';
COMMENT ON COLUMN sales.orders.amount IS 'Order total in INR, after discounts.';

SELECT c.column_name, c.data_type,
       coalesce(col_description('sales.orders'::regclass, c.ordinal_position), '') AS description
FROM information_schema.columns c
WHERE c.table_schema = 'sales' AND c.table_name = 'orders'
ORDER BY c.ordinal_position;

SELECT obj_description('sales.orders'::regclass) AS table_description;

  column_name   | data_type |                description
----------------+-----------+-------------------------------------------
 order_id       | integer   |
 customer_email | text      | PII: customer email, masked for analysts.
 amount         | numeric   | Order total in INR, after discounts.

                           table_description
------------------------------------------------------------------------
 One row per order. Owner: data-platform team. Source: shop DB via CDC.

Common mistakes

• Launching a catalog without owners, so descriptions are never written or kept current.
• Documenting tables in a separate wiki instead of in the catalog or the code (dbt YAML) that feeds it.
• Tagging PII in the catalog but not enforcing access or masking from those tags.
• Letting each engine keep its own metastore, so the same table has different schemas in different tools.