Kubernetes vs Docker: What Each Does and How They Work Together

Docker packages an application with everything it needs (code, libraries, settings) into an image, and runs that image as a container on one machine. Kubernetes runs and manages many containers across many machines: it decides where each one runs, restarts it when it crashes, scales it up and down, and connects containers to each other.

So the comparison is not really either-or. Docker is how you build and run a container; Kubernetes is how you operate containers at scale. A typical flow is: build an image with Docker, push it to a registry, and let Kubernetes run it.

Side by side

	Docker	Kubernetes
Main job	Build images and run containers	Schedule and manage containers across a cluster
Scope	One machine	Many machines (a cluster)
Key objects	Image, container, volume, network, Dockerfile	Pod, Deployment, Service, Job, CronJob, ConfigMap, Secret
Restarts and scaling	Basic restart policies; manual scaling	Automatic restarts, rolling updates, autoscaling
Multi-container apps	Docker Compose on one machine	Native, across machines
Best for	Local development, CI, single-server apps	Production platforms with many services or jobs
Learning curve	Gentle	Steep

Docker in one paragraph

You write a Dockerfile that starts from a base image, copies your code, installs dependencies and sets the command to run. docker build turns it into an image; docker run starts a container from it. Because the image contains everything, it runs the same on your laptop, in CI and on a server. Docker Compose starts several containers together (say, PostgreSQL and Airflow) on one machine. The Docker lesson covers this hands-on.

Kubernetes in plain words

• Cluster: a group of machines (nodes) managed together.
• Pod: the smallest unit Kubernetes runs: one or a few containers that share a network address.
• Deployment: “keep three copies of this pod running, and replace them gradually when I ship a new image”.
• Service: a stable network name for a group of pods, so others can reach them as pods come and go.
• Job and CronJob: run a container until it finishes, once or on a schedule. These are the objects batch data work uses most.
• ConfigMap and Secret: settings and credentials passed into containers.

You describe what you want in YAML files, and Kubernetes keeps working to make the cluster match them. Here is a nightly load job described as a CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-orders-load
spec:
  schedule: "15 2 * * *"            # 02:15 every day
  concurrencyPolicy: Forbid          # never start a run while the previous one is still going
  jobTemplate:
    spec:
      backoffLimit: 2                # retry a failed run twice
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: loader
              image: registry.example.com/data/orders-loader:1.4.0   # built with Docker
              args: ["--date", "yesterday"]
              resources:
                requests: { cpu: "500m", memory: "1Gi" }
                limits:   { memory: "2Gi" }
              envFrom:
                - secretRef: { name: warehouse-credentials }

The image is built with Docker. Kubernetes decides which machine runs it, gives it the CPU and memory requested, injects the credentials, retries failures and records the result.

“Kubernetes dropped Docker”: what that meant

In 2022, Kubernetes 1.24 removed its built-in adapter for the Docker Engine (“dockershim”). Clusters now run containers with runtimes such as containerd or CRI-O. This did not stop Docker images from working: images built with Docker follow the open OCI standard and run on Kubernetes as before. You still build with Docker; the cluster just uses a different engine to run the result.

Where data engineers meet Kubernetes

• Spark on Kubernetes: the driver and executors run as pods, created per job. See deploying Spark.
• Airflow on Kubernetes: Airflow itself is often deployed with a Helm chart; the KubernetesExecutor runs each task in its own pod, and the KubernetesPodOperator runs any container image as a task. See Airflow executors.
• Model serving: online prediction services run as Deployments behind a Service, with autoscaling. See model serving.
• Managed versions: most teams use a managed cluster (Amazon EKS, Azure AKS, Google GKE) rather than running Kubernetes themselves.

Do you need Kubernetes?

Often not at first. For a few scheduled scripts, a managed scheduler or a serverless job service is simpler. Kubernetes pays off when you run many services and jobs, need autoscaling and isolation, and have (or can borrow) a platform team to run it. Docker, on the other hand, is worth learning on day one: it is used in almost every data team for local development and CI.

Common mistakes

1. Comparing them as alternatives. Docker builds and runs containers; Kubernetes orchestrates them.
2. Adopting Kubernetes for a handful of cron jobs and spending more time on the cluster than on the pipelines.
3. No resource requests and limits, so one job starves others or gets killed for using too much memory.
4. Using the latest image tag in production, so you never know which version ran.
5. Putting secrets in images instead of Kubernetes Secrets or a secrets manager.

What interviewers look for

• One sentence each, and how they work together.
• The main Kubernetes objects, especially Pod, Deployment, Service, Job and CronJob.
• A data example: Spark or Airflow tasks running as pods.
• When Kubernetes is worth its complexity and when a simpler service is better.

Practise with the data engineering interview questions.

Key takeaway

Docker packages and runs containers on one machine; Kubernetes schedules, scales and heals containers across a cluster. You build images with Docker and run them on Kubernetes. Data engineers meet Kubernetes through Spark, Airflow and model serving, usually on a managed cluster, and should learn Docker first.