PySpark vs pandas vs Polars: When to Use Each

“Should this job be pandas, Polars or Spark?” comes up in design reviews and in interviews, and the honest answer is rarely “the fastest one”. The three libraries share the word DataFrame but run on different execution models, fail in different ways when data grows, and cost different amounts to run and operate. This lesson compares them on the things that decide the choice, then runs one identical task in all three so you can see the differences in code and output rather than take them on trust.

It assumes you know pandas basics and lazy evaluation in Spark. It does not repeat them.

The short version

	pandas	Polars	PySpark
Execution	Eager: each line runs immediately	Eager or lazy; lazy queries are optimised before they run	Lazy: transformations build a plan, actions run it
Parallelism	Core mostly single-threaded	Multi-threaded on one machine	Distributed across executors (or threads in local mode)
Where data lives	In the memory of one Python process	In memory, or streamed in batches by the streaming engine	Partitions spread over executors, spilled to disk when needed
In-memory format	NumPy arrays, plus Arrow-backed types (strings by default in 3.0)	Apache Arrow	Spark’s own row and columnar formats in the JVM; Arrow at the Python boundary
Startup	Import time	Import time	A JVM and session start (seconds), plus cluster start on real clusters
Index	Row index and MultiIndex	No index	No index
Natural home	Notebooks, small jobs, glue code, tests	Single-node ETL and analytics	Data that does not fit one machine, lakehouse tables, streaming

Everything below is the reasoning behind each row.

Execution model

pandas is eager. df[df["status"] == "paid"] builds a boolean array and a new DataFrame the moment it runs. There is no query plan, so pandas cannot skip a column you never use later or push a filter into the file reader for you; you do that by hand with columns= and similar arguments. Vectorised operations run in compiled code, but the Polars documentation’s comparison puts it fairly: some pandas operations are multi-threaded, the core is single-threaded.

Polars has both modes. The eager API (pl.read_parquet, DataFrame methods) feels like pandas. The lazy API (pl.scan_parquet, LazyFrame) records the query and optimises it when you call collect() or a sink_* method: predicate pushdown applies filters while reading, projection pushdown reads only the needed columns. The Polars docs recommend the lazy API in most cases. Work runs on all cores of one machine.

Polars 2.0 (October 2026) changed the default engine: LazyFrame.collect() now resolves engine="auto" to the streaming engine, which processes data in batches, where 1.x used the in-memory engine. Eager DataFrame operations still use the in-memory engine, and sink_* already streamed. The streaming engine does not guarantee row order for group_by, joins or unpivot, so code that relied on incidental order needs an explicit sort or maintain_order=.

PySpark is lazy and distributed. Transformations build a logical plan; Catalyst optimises it; an action (write, count, show, collect) runs it as stages of tasks over partitions on executors. A groupBy shuffles rows across the network so each key ends up in one place. The same code runs in local[*] mode on a laptop (driver and executors in one JVM, tasks on threads) and on a cluster of hundreds of machines.

Memory and data size

Avoid “pandas up to N GB, Spark above”. The limit depends on what each engine has to hold at its peak, so reason from that:

• pandas holds every column you load, as full in-memory arrays, plus intermediate results: the boolean mask, the filtered copy, the group-by output. The pandas scaling guide warns that even datasets that are a sizable fraction of memory become unwieldy because some operations make intermediate copies. Compressed Parquet or CSV on disk is also much smaller than the decoded arrays. The levers are loading fewer columns, efficient dtypes and processing in chunks.
• Polars in eager mode has the same in-memory ceiling for whatever it materialises, though its Arrow columns and lazy pushdown usually mean less is loaded in the first place. Lazy queries on the streaming engine process batches, so a filter plus group-by on a file larger than RAM can work when the group-by result itself fits. Some operations are not implemented for streaming and fall back to the in-memory engine; check the physical plan if memory matters.
• PySpark splits data into partitions and each executor works on a few at a time. Total data can be far larger than any machine; the limits move to per-partition and per-task memory (a skewed key, a huge group in a pandas UDF), to shuffle volume, and to the driver, which must not receive large results through collect() or toPandas().

So the questions to ask are: how big is the working set (columns needed after pruning, decoded), how big is the largest intermediate (a join or a wide group-by), does it grow every month, and does one machine with that much RAM exist in your budget. To measure instead of guess, check df.memory_usage(deep=True) in pandas or df.estimated_size() in Polars on a sample, and watch the peak resident memory of a real run.

One task in all three

The task: read an orders file, keep paid orders from 2026, then per country count orders, sum revenue and average the amount, and write the result to Parquet. First, generate a sample of one million synthetic orders:

import tempfile
from pathlib import Path

import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

DATA = Path(tempfile.mkdtemp(prefix="frames-"))
ROWS = 1_000_000
rng = np.random.default_rng(42)

start = np.datetime64("2025-01-01T00:00:00", "s")
seconds = rng.integers(0, 730 * 24 * 3600, ROWS)  # spread over 2025 and 2026
orders = pa.table({
    "order_id": np.arange(1, ROWS + 1, dtype="int64"),
    "customer_id": rng.integers(1, 50_001, ROWS),
    "country": rng.choice(["DE", "FR", "GB", "IN", "US"], ROWS),
    "status": rng.choice(["paid", "refunded", "pending"], ROWS, p=[0.8, 0.1, 0.1]),
    "amount": np.round(rng.gamma(2.0, 30.0, ROWS), 2),
    "ordered_at": (start + seconds).astype("datetime64[us]"),
})
pq.write_table(orders, DATA / "orders.parquet")
print(orders.num_rows, orders.schema.names)

1000000 ['order_id', 'customer_id', 'country', 'status', 'amount', 'ordered_at']

pandas

import pandas as pd

df = pd.read_parquet(
    DATA / "orders.parquet",
    columns=["country", "status", "amount", "ordered_at"],  # read only what the task needs
)
paid_2026 = df[(df["status"] == "paid") & (df["ordered_at"] >= "2026-01-01")]
result_pd = (
    paid_2026.groupby("country", as_index=False)
    .agg(orders=("amount", "size"), revenue=("amount", "sum"), avg_amount=("amount", "mean"))
    .sort_values("country", ignore_index=True)
    .round({"revenue": 2, "avg_amount": 2})
)
result_pd.to_parquet(DATA / "revenue_pandas.parquet", index=False)
print(result_pd)

  country  orders     revenue  avg_amount
0      DE   79532  4768214.89       59.95
1      FR   79471  4770571.15       60.03
2      GB   80502  4824510.41       59.93
3      IN   80144  4808448.24       60.00
4      US   80053  4828930.36       60.32

Every line ran as soon as it was reached. The column pruning happened only because columns= was passed; without it pandas would load all six columns.

Polars

import polars as pl

query = (
    pl.scan_parquet(DATA / "orders.parquet")
    .filter((pl.col("status") == "paid") & (pl.col("ordered_at") >= pl.datetime(2026, 1, 1)))
    .group_by("country")
    .agg(
        pl.len().alias("orders"),
        pl.col("amount").sum().round(2).alias("revenue"),
        pl.col("amount").mean().round(2).alias("avg_amount"),
    )
    .sort("country")
)
query.sink_parquet(DATA / "revenue_polars.parquet")  # nothing has run until this line

with pl.Config(float_precision=2):
    print(pl.read_parquet(DATA / "revenue_polars.parquet"))

shape: (5, 4)
┌─────────┬────────┬────────────┬────────────┐
│ country ┆ orders ┆ revenue    ┆ avg_amount │
│ ---     ┆ ---    ┆ ---        ┆ ---        │
│ str     ┆ u32    ┆ f64        ┆ f64        │
╞═════════╪════════╪════════════╪════════════╡
│ DE      ┆ 79532  ┆ 4768214.89 ┆ 59.95      │
│ FR      ┆ 79471  ┆ 4770571.15 ┆ 60.03      │
│ GB      ┆ 80502  ┆ 4824510.41 ┆ 59.93      │
│ IN      ┆ 80144  ┆ 4808448.24 ┆ 60.00      │
│ US      ┆ 80053  ┆ 4828930.36 ┆ 60.32      │
└─────────┴────────┴────────────┴────────────┘

The query is built from expressions (pl.col("amount").sum()), not from lambdas or index alignment. Note the sort("country"): with streaming as the default in Polars 2, a group_by returns groups in no particular order, so sort whenever order matters. Note also that pl.len() returns u32, which matters later. explain() shows what the optimiser did:

print(query.explain())

SORT BY [col("country")]
  AGGREGATE[maintain_order: false]
    [len().alias("orders"), col("amount").sum().round().alias("revenue"), col("amount").mean().round().alias("avg_amount")] BY [col("country")]
    FROM
    simple π 2/2 ["country", "amount"]
      Parquet SCAN [/tmp/frames-hczf4x10/orders.parquet]
      PROJECT 4/6 COLUMNS
      SELECTION: (col("ordered_at") >= 2026-01-01 00:00:00) & (col("status") == "paid")
      ESTIMATED ROWS: 1000000

PROJECT 4/6 COLUMNS is projection pushdown and SELECTION inside the scan is predicate pushdown: the same pruning you did by hand in pandas, done for you, plus the filter applied while reading.

PySpark

from pyspark.sql import SparkSession, functions as F

spark = (
    SparkSession.builder.master("local[*]")  # driver and executors in this one process
    .appName("frames-comparison")
    .config("spark.sql.shuffle.partitions", "8")  # the default of 200 is sized for clusters
    .getOrCreate()
)
spark.sparkContext.setLogLevel("ERROR")

result_sp = (
    spark.read.parquet(str(DATA / "orders.parquet"))
    .filter(
        (F.col("status") == "paid")
        & (F.col("ordered_at") >= F.lit("2026-01-01").cast("timestamp_ntz"))
    )
    .groupBy("country")
    .agg(
        F.count("*").alias("orders"),
        F.round(F.sum("amount"), 2).alias("revenue"),
        F.round(F.avg("amount"), 2).alias("avg_amount"),
    )
    .orderBy("country")
)
result_sp.write.mode("overwrite").parquet(str(DATA / "revenue_spark"))
result_sp.show()
print(sorted(p.name.split("-")[0] for p in (DATA / "revenue_spark").iterdir() if not p.name.startswith(".")))

+-------+------+----------+----------+
|country|orders|   revenue|avg_amount|
+-------+------+----------+----------+
|     DE| 79532|4768214.89|     59.95|
|     FR| 79471|4770571.15|     60.03|
|     GB| 80502|4824510.41|     59.93|
|     IN| 80144|4808448.24|      60.0|
|     US| 80053|4828930.36|     60.32|
+-------+------+----------+----------+

['_SUCCESS', 'part']

Two differences from the single-node tools. The output path is a directory of part-* files plus a _SUCCESS marker, because each task writes its own file; downstream readers must point at the directory. And show() is a second action, so Spark ran the plan twice (once to write, once to show); in a real job you would read back what you wrote or cache deliberately. The scan pushes the filter and prunes columns just like Polars:

import contextlib, io

buf = io.StringIO()
with contextlib.redirect_stdout(buf):
    result_sp.explain(mode="formatted")
for line in buf.getvalue().splitlines():
    if line.startswith(("ReadSchema", "PushedFilters")):
        print(line)

PushedFilters: [IsNotNull(status), IsNotNull(ordered_at), EqualTo(status,paid), GreaterThanOrEqual(ordered_at,2026-01-01T00:00)]
ReadSchema: struct<country:string,status:string,amount:double,ordered_at:timestamp_ntz>

The sample file stores timestamps without a time zone, which Spark reads as timestamp_ntz; casting the literal to the same type keeps the comparison exact and pushable.

Same answer, different types

for name in ["revenue_pandas.parquet", "revenue_polars.parquet", "revenue_spark"]:
    t = pq.read_table(DATA / name)
    print(f"{name:24}", ", ".join(f"{f.name}:{f.type}" for f in t.schema))

expected = pl.read_parquet(DATA / "revenue_pandas.parquet")
for name in ["revenue_polars.parquet", "revenue_spark"]:
    got = pl.read_parquet(DATA / name if name.endswith(".parquet") else DATA / name / "*.parquet")
    print(name, got.cast({"orders": pl.Int64}).equals(expected))

revenue_pandas.parquet   country:large_string, orders:int64, revenue:double, avg_amount:double
revenue_polars.parquet   country:large_string, orders:uint32, revenue:double, avg_amount:double
revenue_spark            country:string, orders:int64, revenue:double, avg_amount:double
revenue_polars.parquet True
revenue_spark True

The values agree; the schemas do not. The Polars count is unsigned 32-bit, and string columns come out as large_string from pandas 3 and Polars but string from Spark. If a downstream table has a fixed schema, cast explicitly before writing rather than relying on each library’s defaults.

Timings from one run (not a benchmark)

To give a feel for fixed costs, the same task (without the average and sort) ran on a larger generated file: 20 million rows, 339 MB of Parquet, on a 4 vCPU, 15 GB sandbox VM, each library in a fresh process, twice.

	Task time	Other cost	Peak memory of the Python process
pandas 3.0.6	9.6 s and 10.9 s	none	about 1.9 GB
Polars 2.0.0 (lazy, sink_parquet)	0.7 s and 0.8 s	none	about 0.5 GB
PySpark 4.2.0, local[*]	13.7 s and 14.4 s	12.6 s and 14.7 s to start the session	not comparable (work happens in the JVM)

Read it narrowly. On one machine with data that fits in memory, a multi-threaded engine with pushdown beats both an eager single-threaded library and a distributed engine paying JVM, planning and shuffle overhead. pandas peaked at more than five times the file size on disk for a four-column read. None of this says anything about Spark on a cluster with data no single machine can hold, which is the case it exists for. Measure your own workload before deciding.

APIs and moving data between them

pandas has the largest API and ecosystem: plotting, statistics, Excel, SQL readers, and almost every Python data library accepts a pandas DataFrame. pandas 3.0 made two changes that matter when you compare it with the others: Copy-on-Write is always on (chained assignment no longer modifies the original), and string columns use a dedicated str dtype, backed by PyArrow when it is installed.

Polars expressions (pl.col, pl.when, .over() for window functions) compose and run in parallel, and the docs stress that Polars is strict about types where pandas converts loosely. There is no index, so code that relies on index alignment, reset_index or MultiIndex must be rewritten, not translated.

PySpark’s DataFrame API (F.col, F.when, Window) looks close to Polars expressions, and Spark SQL compiles to the same plans. The pandas API on Spark (import pyspark.pandas as ps) offers pandas-style syntax on top of Spark for porting code:

import pyspark.pandas as ps

psdf = ps.read_parquet(str(DATA / "orders.parquet"))
paid = psdf[psdf["status"] == "paid"]
print(paid.groupby("country", as_index=False).agg(revenue=("amount", "sum")).sort_values("country").head(2))

  country     revenue
3      DE  9560709.19
0      FR  9585353.44

This sums all years (there is no date filter here). The index labels 3 and 0 show that the group order before sort_values differs from what pandas would produce. It is not 100% pandas-compatible, it attaches a default index that adds overhead unless you set index_col, and the Spark docs recommend switching to the PySpark API where possible. Treat it as a migration bridge rather than the final form. (This block is skipped by the page’s automated example check and was run by hand on pandas 2.3.3 and 3.0.6 with the same output; it also printed PandasAPIOnSparkAdviceWarning about the default index.)

Arrow is the interchange layer. Polars stores data in Arrow format, pandas 3 can import any object that implements the Arrow PyCapsule interface with DataFrame.from_arrow(), and since Spark 4.0 createDataFrame() accepts a PyArrow Table and DataFrame.toArrow() returns one. Spark also uses Arrow for toPandas() and createDataFrame(pandas_df) through spark.sql.execution.arrow.pyspark.enabled, which the 4.2 docs describe as enabled by default.

small = query.collect()                    # a Polars DataFrame
as_pandas = pd.DataFrame.from_arrow(small)  # pandas 3: import through the Arrow PyCapsule interface
print(type(as_pandas).__name__, as_pandas.dtypes.to_dict())

DataFrame {'country': <StringDtype(na_value=nan)>, 'orders': dtype('uint32'), 'revenue': dtype('float64'), 'avg_amount': dtype('float64')}

Handing the same result to Spark fails, because Spark has no unsigned integer types:

spark.createDataFrame(small.to_arrow())

pyspark.errors.exceptions.base.PySparkTypeError: [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] uint32 is not supported in conversion to Arrow.

Cast first, and the round trip is exact:

sdf = spark.createDataFrame(small.with_columns(pl.col("orders").cast(pl.Int64)).to_arrow())
sdf.printSchema()
back = pl.from_arrow(sdf.toArrow())  # toArrow() collects to the driver: small results only
print(back.equals(expected))

root
 |-- country: string (nullable = true)
 |-- orders: long (nullable = true)
 |-- revenue: double (nullable = true)
 |-- avg_amount: double (nullable = true)

True

toArrow() and toPandas() both pull every row to the driver, so use them on aggregated or sampled results, never on the full table. For large hand-offs, write Parquet (or a Delta or Iceberg table) from one engine and read it from the other.

Two version notes. Polars 2.0 removed the older DataFrame Interchange Protocol (__dataframe__); pl.from_dataframe() now accepts only objects that support the Arrow PyCapsule interface. And the PySpark 4.2 installation page lists pandas >=2.2.0,<3.0.0 as the supported range for Spark SQL, Spark Connect and the pandas API on Spark. pip does not enforce that upper bound, and with pandas 3 installed PySpark prints a FutureWarning that it does not yet fully support pandas 3. The examples above worked on both, but in production Spark environments pin pandas below 3 until your Spark version documents support.

I/O and formats

All three read and write CSV, JSON and Parquet; the differences are in how.

• pandas reads a file fully into memory (read_csv can read in chunks with chunksize) and writes one file. Parquet goes through PyArrow.
• Polars has eager readers (read_*) and lazy scanners (scan_parquet, scan_csv, scan_ndjson) that push filters and column selection into the read, and sink_* methods that write a lazy result in a streaming way. In 2.0, pl.read_csv is implemented as scan_csv(...).collect(), which removed a few parameters such as n_threads and batch_size.
• PySpark reads directories and partitioned layouts (country=DE/), splits large files across tasks, writes one file per task, and has the widest connector set: JDBC, Kafka, Delta Lake and Iceberg tables, cloud object stores at scale. See reading and writing data and writing efficient output.

For pipelines, the common pattern is Parquet or a table format as the contract between steps, so each step can use whichever engine fits.

Testing and local development

• pandas and Polars need nothing but pip install. Tests run in milliseconds, and both ship frame comparison helpers: pandas.testing.assert_frame_equal and polars.testing.assert_frame_equal.
• PySpark needs a Java runtime (Java 17 or later for Spark 4) and a session per test run; start one session per test module or session with a pytest fixture rather than one per test, set a small spark.sql.shuffle.partitions, and use pyspark.testing.assertDataFrameEqual and assertSchemaEqual. Expect seconds of overhead per test run, not milliseconds.
• A practical habit for any of them: keep transformations as functions that take and return DataFrames, so tests can feed a tiny in-memory frame. The pytest lesson covers the structure.

Cluster cost and operations

pandas and Polars run wherever Python runs: a laptop, an Airflow worker, a container, a serverless function. The cost is one machine, sized by peak memory. Scaling means a bigger machine, which works until the largest affordable instance is not enough.

Spark needs a cluster manager (Kubernetes, YARN, standalone) or a managed platform, executors sized for memory and cores, shuffle storage, monitoring through the Spark UI, and upgrades of the whole runtime. In return you get horizontal scale, task retries when a node dies, and integration with lakehouse tables and streaming. The common waste is the reverse of pandas: clusters running for jobs that would finish faster on one node, or all-purpose clusters left idle. When a pipeline’s input shrinks after filtering or aggregation, it is often cheaper to hand the small result to a single-node step than to keep it on the cluster. Executor memory and tuning covers sizing.

When to choose each

If the situation is mostly…	Leaning	Why
Exploration, small files, reshaping API responses, tests, libraries that expect pandas	pandas	Largest ecosystem, no setup, everyone on the team can read it
Single-machine ETL where the working set fits one machine, or streams through it in batches	Polars	Multi-threaded, lazy pushdown, streaming engine, no cluster
Data that does not fit the largest machine you can afford, or grows toward that	PySpark	Partitions, shuffles and retries across many machines
Lakehouse tables (Delta, Iceberg), Kafka or other streaming sources, a platform already running Spark	PySpark	Native connectors and the team’s operational tooling
A pandas codebase that has outgrown one machine	PySpark (pandas API on Spark as a bridge), or Polars if one bigger machine suffices	Port the heavy steps first; rewrite to the native API over time
Warehouse data that is already queryable	Neither: SQL in the warehouse	No data movement; see when not to use pandas

Mixing them is normal: Spark for the heavy joins over the lake, Polars or pandas for the small aggregated output, pandas in tests and notebooks.

Migration tips

pandas to Polars

1. Remove index logic: replace set_index, reset_index and alignment-based assignment with explicit joins and columns.
2. Replace apply and lambdas with expressions; pl.when().then().otherwise() covers most conditional logic.
3. Start with the lazy API on the heaviest step, check explain(), and sort explicitly wherever order matters, especially after upgrading to Polars 2.
4. Expect stricter types: count columns are u32, and mixed types raise instead of becoming object.

pandas to PySpark

1. Profile first: if one bigger machine or Polars would do, that is less to operate.
2. The pandas API on Spark can run existing code with few changes; set index_col when converting and watch for operations that collect to one partition.
3. Move the hot paths to native DataFrame functions; pandas UDFs are the escape hatch for logic with no built-in function.
4. Remove toPandas() on large data and every pattern that loops over rows on the driver.

Between Polars and PySpark

1. The expression APIs map closely (pl.col("x").sum().over("k") and F.sum("x").over(Window.partitionBy("k"))), so the logic usually ports line by line.
2. Cast unsigned integers before handing Arrow data to Spark, and agree on time zone handling for timestamps.
3. Prefer a Parquet or table hand-off over in-memory conversion for anything larger than a result set.

Common mistakes

1. Choosing Spark for a few hundred megabytes because “it scales”, and paying session start, shuffle and cluster costs on every run.
2. Quoting a fixed size threshold instead of reasoning about the working set, intermediates and growth.
3. Assuming the pandas API on Spark behaves exactly like pandas, or using it without an index column.
4. Calling toPandas() or collect() on a large Spark DataFrame and crashing the driver.
5. Relying on group or join output order in Polars 2 lazy queries.
6. Letting each library pick output types and then failing a schema check downstream.
7. Installing pandas 3 into a PySpark 4.2 environment without noticing it is outside the documented supported range.

Interview angle

For “pandas, Polars or Spark?”, answer with the framework: data size now and in a year, the size of the largest intermediate, where the data lives, latency and schedule, team skills, and what the platform already runs. Then recommend for the scenario given. A strong answer mentions execution models (eager, lazy single node, lazy distributed), driver memory limits in Spark, Arrow as the bridge, and the operational cost of a cluster. Expect follow-ups such as “how would you migrate a pandas job that no longer fits?” and “why might Polars be faster than Spark on one machine?”. Practise with Q38 in the Top 50 PySpark interview questions and processing a large CSV with limited memory.

Key takeaway

pandas is eager and in memory, Polars is lazy and multi-threaded on one machine, PySpark is lazy and distributed. Pick by working-set size, growth, where the data lives and what you can operate, not by benchmark headlines; exchange data through Arrow or Parquet with explicit types; and measure your own workload before committing to a cluster.