What is the difference between a transformation and an action in Spark?

Detailed explanation

Spark builds a logical plan as you chain transformations. Nothing runs yet. When you call an action, Spark optimises the whole plan (for example pushing filters down and pruning columns), splits it into stages and runs it as tasks on the cluster.

Transformations (lazy)	Actions (trigger a job)
select, filter, withColumn	count, collect, show, take
join, groupBy().agg()	write...save() / saveAsTable
distinct, repartition	foreach, toPandas

Narrow and wide

• Narrow transformations (filter, select) let each output partition depend on a single input partition, so no data moves.
• Wide transformations (groupBy, most joins, distinct) need data with the same key together, which causes a shuffle and a new stage.

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[2]").appName("tva")
         .config("spark.sql.shuffle.partitions", "4").getOrCreate())

orders = (spark.range(0, 100_000)                       # nothing runs yet
          .select(F.col("id").alias("order_id"),
                  (F.col("id") % 50).alias("customer_id"),
                  (F.col("id") % 500).alias("amount")))
big = orders.filter("amount > 100").select("customer_id", "amount")
by_customer = big.groupBy("customer_id").agg(F.sum("amount").alias("total"))  # still lazy

print(by_customer.orderBy("customer_id").first())       # action: a job runs

Row(customer_id=0, total=420000)

Only the last line starts work. Spark then sees the whole chain, so it can apply the filter before the aggregation and pre-aggregate (partial_sum) on each partition before the shuffle that groupBy needs.

Trade-offs and pitfalls

• Errors surface late. A bad cast or a division error appears at the action, often many lines after the transformation that caused it. A missing column is caught earlier, when the DataFrame is defined, because analysis is eager.
• Recomputation. Every action re-runs the plan from the source. cache() the DataFrame if several actions reuse it, and unpersist() afterwards.
• count() is not a full check. The optimiser prunes columns the action does not need, so a count() can succeed on a DataFrame whose columns would fail when written.

Common mistakes

1. Thinking read or filter runs immediately. (Reading can run a small job to infer a schema or list files, but the data itself is read only by an action.)
2. Calling several actions on the same expensive DataFrame without caching, so Spark recomputes it each time.
3. Using collect() on large data, pulling everything to the driver.
4. Treating cache() as an action: it is lazy, and the cache fills on the next action.