Immutable infrastructure with Docker explained

Immutable infrastructure is a deployment philosophy: once a runtime instance is deployed, it is never modified in place. Updates, patches, and config changes happen by building a new instance (a new image, a new VM, a new container) and replacing the old one. Docker enforces this naturally because the image is read-only and containers are cheap to recreate. The result is reproducible, predictable infrastructure.

#Theory

#TL;DR

• Mutable: patch a running server in place (SSH, apt upgrade, edit configs). Each server's state diverges over time. "Snowflake servers."
• Immutable: build the new artifact (image), deploy, retire old. Each instance is identical. "Phoenix servers."
• Docker makes immutable easy: image = artifact, container = instance, redeploy = stop+start.
• State must be externalized: DB on a managed service or persistent volume, files in object storage, sessions in Redis. The container itself is disposable.
• Rollback is just "deploy the previous tag."
• Combines naturally with blue-green, canary, rolling updates.

#The mental shift

Mutable mindset:

• "Server prod-1 has been running for 3 years. We have applied 47 patches."
• Configs live on disk; everyone has SSH access; over time, no two servers are identical.
• New version = SSH in, run installer, restart service.
• Drift between staging and production is a constant fight.

Immutable mindset:

• "This release is image tag 1.7.3. Every running container is from the same digest."
• Configs come from environment variables or mounted files at start time.
• New version = new image, new container, same image runs in dev/staging/prod.
• Drift is impossible because nothing modifies the running instance.

#What goes outside the container

For immutable to work, anything that needs to change at runtime must live outside:

If any of these is on the container's writable layer, immutable is broken: replacing the container loses data.

#Why immutable wins

1. Reproducibility. A given image tag, run anywhere, behaves the same way. Dev = staging = prod.
2. Easy rollback. "Deploy 1.7.2 again." One command. No restoration of system state.
3. No config drift. Every container of the same tag is bit-identical.
4. Auditability. What is running? docker inspect shows the image digest. Trace it back to a git SHA via build labels.
5. Forces good hygiene. You cannot SSH and "just fix it" — pushing back into the image build process produces a permanent fix.
6. Pairs with deployment patterns. Blue-green, canary, rolling — all assume identical replicas you can swap.

#Trade-offs

• Cost of state externalization. Setting up Postgres, Redis, S3, etc. is more upfront work than slapping data on the local disk.
• Build pipeline must be solid. If your CI is flaky, every change costs you.
• Image size. A 2 GB image is fine in mutable land but slows immutable redeploys. Optimize image size aggressively.
• Cold-start cost. A new container starts cold; warm caches are gone. Mitigate with startup warmup or readiness probes.

#Examples

#Mutable vs Immutable: same change

Fix: bump the API's max body size from 1 MB to 5 MB.

Mutable workflow:

Immutable workflow:

#Externalizing state

Good:

Bad (state inside container):

#Immutable + blue-green

Blue and green are both immutable: blue runs 1.0, green runs 2.0, neither is mutated mid-flight.

#Immutable + canary

Each copy is immutable; the rollout is a series of replacements, not in-place upgrades.

#Immutable + rolling update (Swarm)

#Pinning by digest for true immutability

A tag like myorg/app:1.0 can be re-pushed (someone could overwrite it). A digest cannot. For maximum reproducibility (and security), pin by digest in production.

Build systems can lock the digest into the deploy manifest:

#Real-world usage

• Microservices on Kubernetes: every Pod is a container from a specific image; no kubectl exec to fix bugs in prod.
• Serverless / containers-as-a-service (Cloud Run, Fargate): platform enforces immutability — you cannot SSH in.
• Compose-managed dev environments: redeploy on every change; the container is throwaway.
• CI build agents: each job runs in a fresh container.
• Hardened production environments: container images are signed and scanned, deployed via GitOps (Argo CD, Flux). The whole pipeline is immutable end-to-end.

#Anti-patterns to avoid

SSH-ing into running containers to fix things

The fix lives until the container restarts, then is gone. Worse, it diverges from your image. Push the fix into the repo, rebuild, redeploy.

Storing data in the container's writable layer

When the container is replaced, that data is gone. Always use volumes or external services.

Re-using the same tag for new builds

Use meaningful tags (semver, git SHA, timestamp). Reserve latest for dev convenience; never deploy from latest in prod.

Treating the host as immutable but containers as mutable

If you docker exec into containers to patch them, you have not gained much. The whole stack should follow the discipline.

#Common mistakes

Forgetting that volumes break the model

A volume is mutable state. That is fine — it is the externalized data. But understand: when you redeploy, the new container reuses the same volume. If a deploy migration corrupts data, the next deploy inherits the corruption. Treat volumes (and DB schemas) with care.

Configs baked into the image

Now the same image is not portable across environments. Mount or env-inject configs at run time.

Long-lived containers with hot-reload

If your dev workflow is docker exec api npm run reload, you have made the container mutable. For dev, that is fine. For prod, never.

#Follow-up questions

Q: Is immutable infrastructure only achievable with containers?

A: No. AMIs, Packer-built images, terraform-recreated servers all enable it. Containers make it cheaper because rebuild + redeploy is fast.

Q: What about secrets that change without redeploy?

A: Mount them from a secrets manager (Vault, AWS, GCP). The container reads them on startup or refreshes them periodically. The image itself contains no secrets.

Q: Does immutable mean I can never restart a container?

A: A restart is fine — it is not modifying the image, just rerunning it. "Immutable" refers to the image and on-disk filesystem of the running container, not its lifecycle.

Q: (Senior) How do you reason about long-lived in-memory state (cache warmups, leader-election) under immutable?

A: Two strategies. (1) Make state externally durable: leader-election via a shared service (etcd, ZooKeeper); cache backed by Redis. (2) Embrace cold starts: each new container warms up from cold. Combine with rolling deploys so traffic shifts gradually, no thundering herd. Real systems use both: warm caches when it matters, externalize when correctness depends on it.

Q: (Senior) How does immutable infrastructure interact with regulated environments (PCI, HIPAA)?

A: Beautifully. Auditors love it. Every running version maps back to a signed image (cosign/Notary), which maps back to a CI build, which maps back to a git commit. Provenance is end-to-end. Patching is a code change with a PR, review, and CI artifact, not someone SSH-ing into prod and editing a config. Most compliance frameworks now explicitly favor immutable + GitOps as a stronger control surface than mutable patch-management.