Cloud Data Cost Optimisation and FinOps for Data Engineers

Cloud data platforms make it easy to start and easy to overspend. You pay for what you use, but “use” includes every byte a query scans, every hour a cluster or warehouse runs while idle, and every copy of data nobody deletes. Cost optimisation is the engineering work of paying only for what creates value. FinOps is the practice that makes cost visible and shared between engineering, finance and the business, so it is managed continuously rather than after a surprise bill.

Where the money goes

Cost driver	How it is billed	Typical waste
Compute for queries and jobs	Per second or hour of a warehouse or cluster, or per byte scanned	Idle warehouses, oversized clusters, full table scans
Storage	Per GB per month, by storage class	Old raw data in hot storage, duplicate copies, endless snapshots and time travel
Data movement	Per GB leaving a region or a cloud	Cross-region pipelines, exports to other clouds
Managed services and licences	Per connector, per row synced, per seat	Syncing tables nobody uses
Streaming	Per shard, partition, broker or throughput unit	Capacity sized for peak, all day

Scan less: the biggest lever

Engines that charge per byte read (and engines that charge for time, which scanning consumes) reward reading less data. This estimate uses an assumed price of $5 per TB scanned:

# Rough monthly cost of one dashboard query on a scan-priced engine (you pay per byte read).
PRICE_PER_TB = 5.00          # assumed on-demand price in USD per TB scanned; check your own engine's rate
table_tb = 2.0               # one year of events, about 2 TB
days, columns = 365, 40      # partitioned by day, 40 columns
runs_per_month = 500 * 30    # the dashboard refreshes this query 500 times a day

def monthly(tb_per_run):
    return tb_per_run * PRICE_PER_TB * runs_per_month

full_scan = table_tb                              # row format, no partitions: read everything
one_day = table_tb / days                         # partition pruning: read only yesterday
one_day_two_cols = one_day * 2 / columns          # columnar format too: read only 2 of 40 columns

for label, tb in [("full scan", full_scan), ("+ partition pruning", one_day), ("+ columnar, 2 columns", one_day_two_cols)]:
    print(f"{label:24} {tb * 1024:9.2f} GB per run   ${monthly(tb):>10,.2f} per month")

# Compute billed by the hour: an always-on warehouse vs one that suspends when idle.
RATE_PER_HOUR = 4.00         # assumed price of one warehouse in USD per running hour
busy_hours_per_day = 6
always_on = 24 * 30 * RATE_PER_HOUR
auto_suspend = busy_hours_per_day * 30 * RATE_PER_HOUR
print(f"always on: ${always_on:,.0f}/month   auto-suspend: ${auto_suspend:,.0f}/month")

full scan                  2048.00 GB per run   $150,000.00 per month
+ partition pruning           5.61 GB per run   $    410.96 per month
+ columnar, 2 columns         0.28 GB per run   $     20.55 per month
always on: $2,880/month   auto-suspend: $720/month

The numbers are illustrative, but the ratios are real: filtering on the partition column and storing data in a columnar format turned a ruinous query into a cheap one, without changing the result. In practice:

• Partition or cluster large tables on the column most queries filter by, usually a date. See partitioning and clustering.
• Store analytics data as Parquet (or in a warehouse’s native columnar format) and select only the columns you need. See data file formats.
• Avoid SELECT * in dashboards and scheduled queries.
• Pre-aggregate with incremental models or materialised views when many users ask the same question.
• Cache results where the engine supports it, and stop dashboards refreshing more often than the data changes.

Size, suspend and schedule compute

• Auto-suspend and auto-resume warehouses and interactive clusters, so you stop paying minutes after the last query. The example above shows the effect of paying for 6 busy hours instead of 24.
• Right-size. Start small and scale up only when queries queue or spill to disk. Bigger is faster but not always cheaper.
• Separate workloads (loading, transformation, BI, data science) on separate warehouses or clusters, so one heavy job does not force everything onto a large size.
• Job clusters, not always-on clusters, for scheduled Spark work: they start for the job and stop after it.
• Spot or preemptible instances for fault-tolerant batch work such as Spark executors, at a large discount, with retries for when instances are reclaimed.
• Committed or reserved capacity for the steady baseline you are sure to use, once usage is stable.

Storage tiering and clean-up

• Lifecycle policies move old objects to cheaper storage classes and delete what retention rules allow. Raw data from three years ago rarely needs hot storage.
• Shorten time travel and snapshot retention on large, frequently rewritten tables, and run clean-up (such as VACUUM on Delta tables).
• Fix small files. Millions of tiny files slow queries and raise request costs; compact them.
• Delete unused tables and copies. Query history and lineage show which tables nobody reads.

FinOps: making cost everyone’s job

The FinOps Foundation describes a repeating cycle:

1. Inform: make costs visible. Tag every resource with team, project and environment; build dashboards of cost per team, pipeline and query.
2. Optimise: act on the biggest items first, using the levers above.
3. Operate: set budgets and alerts, review costs regularly, and build cost checks into design reviews.

Two common ways to share costs:

• Showback: each team sees what it costs, without being billed.
• Chargeback: each team’s budget is actually charged.

A useful habit for data engineers is a unit cost: cost per dashboard, per pipeline run, or per thousand events processed. It shows whether cost grows faster than the business.

Common mistakes

1. No tags, so nobody knows which team or pipeline spends the money.
2. Optimising small items first while one always-on cluster costs more than everything else.
3. Buying commitments before usage is stable, then paying for capacity you do not use.
4. Cutting cost by removing tests or monitoring, which costs more when a bad number reaches a decision.
5. Treating cost as a one-off project instead of a regular review.

What interviewers look for

For “how would you reduce our data platform costs?” or “our warehouse bill doubled, what do you do?”:

• Start by finding where the cost is: tags, billing exports, query history.
• The main levers in order of impact: scan less, suspend and right-size compute, tier and clean storage, then spot and commitments.
• A concrete example with numbers, like the estimate above.
• How you would keep it under control: budgets, alerts, showback and unit costs.

Practise with the data engineering interview questions, and see Snowflake cost optimisation for one platform in depth.

Key takeaway

Most cloud data cost comes from compute that scans too much or runs while idle, and from storage nobody cleans up. Partition and use columnar formats so queries read less, suspend and right-size compute, tier and delete old data, and use spot and commitments where they fit. FinOps makes this continuous: tag everything, show each team its costs, and review them regularly.