What is a generator and why can it help with large datasets?

Detailed explanation

import sys

squares_list = [i * i for i in range(1_000_000)]
squares_gen = (i * i for i in range(1_000_000))

print(sys.getsizeof(squares_list) > 1_000_000, sys.getsizeof(squares_gen) < 1_000)
print(sum(squares_gen) == sum(squares_list))

True True
True

The list object alone takes megabytes; the generator object is a few hundred bytes regardless of how many values it will produce, and gives the same sum.

Where it helps in data engineering

• Reading large files line by line.
• Iterating over database cursors or paginated APIs.
• Chaining transformation stages that each handle one record.
• Feeding a loader in fixed-size batches.

The single-use trap

gen = (x for x in [1, 2, 3])
print(list(gen), list(gen))

[1, 2, 3] []

The second pass is silently empty.

Generators feeding batched writes

The usual production shape: a generator reads records lazily, a helper groups them into fixed-size batches, and each batch is written in one call.

import sqlite3
from itertools import islice

def read_rows(n):
    for i in range(n):                     # stands in for csv.reader(f) or a database cursor
        yield (i, f"user_{i % 7}")

def in_batches(iterable, size):
    it = iter(iterable)
    while batch := list(islice(it, size)):
        yield batch

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE events (id INTEGER PRIMARY KEY, user_name TEXT)")
calls = 0
for batch in in_batches(read_rows(25_000), 10_000):
    with conn:
        conn.executemany("INSERT INTO events VALUES (?, ?)", batch)
    calls += 1
print(calls, "batches,", conn.execute("SELECT count(*) FROM events").fetchone()[0], "rows")

3 batches, 25000 rows

At most one batch of 10,000 rows is in memory, and the database sees three round trips instead of 25,000.

Trade-offs

• Single pass only: if you need two passes (for example, a total and then each row’s share of it), either read the source twice or store an intermediate result.
• No length or random access: len(gen) fails; counting consumes it.
• Lifetime of resources: a generator reading a file keeps the file open until it is exhausted or closed, so consume it inside the with block that opened the file.
• Debugging: errors surface where the value is consumed, not where the generator was created, which can make tracebacks less obvious.

Common mistakes

1. Converting a generator to a list “to be safe”, which removes the benefit.
2. Calling len() on it.
3. Consuming it twice and not noticing.