How would you cut the AWS bill of a data platform without breaking SLAs?

Detailed explanation

A good answer is a process, not a list of tricks: make spend visible, fix the largest drivers first, check that SLAs still hold, then add guardrails so it does not drift back.

1. Make spend visible

• Tags (team, pipeline, env) on buckets, Glue jobs, EMR clusters, Redshift workgroups, Athena workgroups; activate them as cost allocation tags. Enforce with tag policies or IaC checks.
• Cost Explorer grouped by service and tag, and Data Exports (the Cost and Usage Report) into S3 for SQL analysis in Athena.
• Service-level views: S3 Storage Lens for bucket growth, noncurrent versions and incomplete uploads; Athena workgroup metrics for bytes scanned; Glue and EMR job histories for DPU-hours and idle time.

2. Fix the usual drivers

Area	Pricing model	Typical waste	Fix
S3 storage	Per GB-month by storage class, plus requests and retrievals	Raw and intermediate copies kept forever, old object versions, incomplete multipart uploads	Lifecycle transitions and expirations, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload, Intelligent-Tiering for unknown access patterns
S3 requests	Per request	Millions of tiny files read by every query	Compaction to larger files
Athena	Per byte scanned (or per DPU-hour with reservations)	JSON or CSV scans, missing partitions, SELECT *	Parquet or Iceberg, partition projection, workgroup scan limits, result reuse
Glue	Per DPU-hour, per second	Oversized fixed worker counts, dev endpoints and sessions left open	Auto scaling, smaller workers, Flex for non-urgent jobs, session idle timeouts
EMR	EC2 plus an EMR charge, per second	Clusters left running, On-Demand only	Transient clusters, auto-termination, Spot task nodes, Graviton, managed scaling
Redshift	Provisioned: node-hours; Serverless: RPU-hours (per second, 60-second minimum) plus storage	High base capacity, clusters running overnight with no queries	Right-size base RPUs and set max RPU-hour usage limits; pause or resize provisioned clusters; Reserved nodes for steady clusters
Network	NAT gateway hourly plus per GB processed; inter-AZ and inter-Region transfer per GB	Spark jobs in private subnets reading S3 through a NAT gateway	S3 and DynamoDB gateway endpoints (no additional charge), co-locate compute and data in one Region
CloudWatch Logs	Per GB ingested and stored	Debug logging and “never expire” retention	Log levels, retention policies, metric filters instead of storing everything
Commitments	Savings Plans, Reserved Instances or nodes	Paying On-Demand rates for a steady base	Commit only for the always-on base after right-sizing

3. Guardrails

AWS Budgets with alerts per team, Cost Anomaly Detection, Athena per-query scan limits, Redshift Serverless usage limits, EMR auto-termination, and IaC defaults (lifecycle rules and log retention in every module) keep the savings.

Example

Lifecycle rules for a raw zone: move to Standard-IA after 30 days, to Glacier Flexible Retrieval after 90, expire after two years, clean up versions and failed uploads:

{
  "Rules": [
    {
      "ID": "raw-zone-tiering",
      "Status": "Enabled",
      "Filter": {"Prefix": "raw/"},
      "Transitions": [
        {"Days": 30, "StorageClass": "STANDARD_IA"},
        {"Days": 90, "StorageClass": "GLACIER"}
      ],
      "Expiration": {"Days": 730},
      "NoncurrentVersionExpiration": {"NoncurrentDays": 7},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 3}
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration --bucket acme-lake-raw \
  --lifecycle-configuration file://raw-lifecycle.json

Infrequent Access classes are not always cheaper. They bill at least 128 KB per object and at least 30 days of storage. This model counts billed GB-days for one day’s landing of small JSON files that are deleted after 10 days, in Standard versus Standard-IA:

KB, GB = 1024, 1024 ** 3

def billed_gb_days(objects, avg_kb, days_kept, min_object_kb=0, min_days=0):
    per_object_kb = max(avg_kb, min_object_kb)
    return objects * per_object_kb * KB / GB * max(days_kept, min_days)

objects, avg_kb, days = 2_000_000, 20, 10          # 2 million 20 KB files, kept 10 days
standard = billed_gb_days(objects, avg_kb, days)
standard_ia = billed_gb_days(objects, avg_kb, days, min_object_kb=128, min_days=30)
print(f"Standard    : {standard:8.1f} GB-days at the Standard rate")
print(f"Standard-IA : {standard_ia:8.1f} GB-days at the IA rate ({standard_ia / standard:.0f}x more GB-days)")

# The same volume compacted into 128 MB Parquet files and kept 400 days
compacted = billed_gb_days(objects * avg_kb // (128 * 1024), 128 * 1024, 400, 128, 30)
print(f"compacted, 400 days in IA: {compacted:8.1f} GB-days, objects: {objects * avg_kb // (128 * 1024)}")

Standard    :    381.5 GB-days at the Standard rate
Standard-IA :   7324.2 GB-days at the IA rate (19x more GB-days)
compacted, 400 days in IA:  15250.0 GB-days, objects: 305

For short-lived small files, Standard-IA bills 19 times the GB-days, which wipes out its lower rate; the same data compacted into large files and kept for a year is exactly what IA and Glacier classes are for. Apply the same thinking to Intelligent-Tiering: objects under 128 KB are never moved to cheaper tiers, and every monitored object carries a monitoring fee.

Trade-offs and pitfalls

• Cutting without measuring: optimising a minor service while a NAT gateway or forgotten cluster dominates.
• Breaking SLAs: shrinking Glue workers or Redshift base capacity can push jobs past their deadline; compare run times before and after.
• Spot for everything: drivers, core nodes and tight-SLA jobs need On-Demand capacity; Spot interruptions cost reruns.
• Commitments too early: a Savings Plan bought before right-sizing locks in waste.
• Glacier for data that is still queried: retrieval fees and restore delays; keep queryable tables in Standard or Intelligent-Tiering.
• Deleting raw data you need for replay: agree retention with data owners and compliance first.

Follow-up answers

• Intelligent-Tiering costing more: many objects under 128 KB (no tiering, still Frequent Access), or millions of objects whose per-object monitoring fee exceeds the storage saving; compact first, or use lifecycle rules when the access pattern is known.
• NAT gateway: it charges per GB processed, so Spark or Glue in private subnets reading terabytes from S3 through it pays per GB on top of everything else. A gateway endpoint for S3 removes that charge and keeps traffic on the AWS network.
• Redshift Serverless vs provisioned: Serverless bills RPU-seconds only while queries run, which suits intermittent or spiky use; a provisioned cluster with Reserved nodes is usually cheaper when it is busy most hours of the day. Compare RPU-hours used per day with the cluster’s always-on hours.
• Accountability: tags and cost categories per team, monthly reports in the team’s own dashboard, budgets with alerts to the owning team, and cost as a review item for new pipelines.