What is a Data Lake?

How it works

A lake is mostly folders of files: Parquet for analytics, plus JSON, CSV, logs, images or anything else a team needs to keep. Storage costs little and scales without limits you are likely to hit, and the same files can be processed by Spark, Trino, DuckDB, pandas or a warehouse that reads external tables.

Lakes are usually organised in layers (raw, cleaned, curated, or bronze, silver and gold) and in folders partitioned by a column such as date, for example s3://lake/events/dt=2026-10-05/. A catalog such as the AWS Glue Data Catalog records which folders form which table and what their schema is, so engines can query them by name.

The weakness is reliability. A folder of files is not a table:

• There are no transactions, so a job that fails halfway leaves readers seeing part of its output.
• Nothing enforces the schema, so a bad file can break every query on the folder months later.
• Updating or deleting one row means rewriting files yourself.

Without layers, owners and a catalog, a lake drifts into a “data swamp” that nobody trusts. Open table formats such as Delta Lake and Apache Iceberg fix the reliability problems by adding a transaction log or metadata layer over the files; that combination is called a lakehouse.

Example

Writing a small date-partitioned dataset as Parquet files and reading the folder back as one dataset with PyArrow 25 (in the PySpark virtualenv). The partition column comes from the folder names.

import os, tempfile
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq

lake = os.path.join(tempfile.mkdtemp(), "events")
table = pa.table({
    "event_id": [1, 2, 3],
    "user": ["a", "b", "a"],
    "dt": ["2026-10-04", "2026-10-05", "2026-10-05"],
})
pq.write_to_dataset(table, lake, partition_cols=["dt"])

print(sorted(os.listdir(lake)))
events = ds.dataset(lake, format="parquet", partitioning="hive")
print(events.to_table(filter=ds.field("dt") == "2026-10-05").num_rows)

['dt=2026-10-04', 'dt=2026-10-05']
2

Common mistakes

• Dumping files with no layers, owners or catalog entries, which is how a lake becomes a swamp.
• Letting many jobs write tiny files, so every query spends its time opening objects.
• Assuming a folder overwrite is atomic; readers can see a mix of old and new files.
• Storing personal data in raw folders without access control or retention rules.