What is the overlay2 storage driver and when to change it?

Storage drivers in Docker manage how image layers and the writable container layer are combined on the host filesystem. overlay2 is the modern default, backed by the Linux kernel's OverlayFS. It is fast, well-supported, and used by virtually every recent Docker install. Other drivers (btrfs, zfs, vfs, devicemapper, fuse-overlayfs) exist for niche cases.

#Theory

#TL;DR

• overlay2 = kernel OverlayFS + Docker integration. Fast and stable.
• Stacks N read-only image layers + 1 writable container layer via union mount.
• Copy-on-write: modifying a file from a lower layer copies it to the upper layer first.
• Default since Docker 17.06+. Requires kernel 4.x and a backing FS that supports xattrs (ext4, xfs with ftype=1).
• Other drivers exist; overlay2 wins for general use.
• Switching driver invalidates all images and containers in /var/lib/docker.

#How OverlayFS works

OverlayFS combines two directories into a virtual filesystem:

When the container reads a file, the kernel walks the stack top-down: upperdir first, then each lowerdir. First match wins. When the container writes:

• New file: created in upperdir.
• Modify existing file from a lowerdir: kernel copies the entire file from lowerdir to upperdir, then modifies the copy. Original lowerdir file is untouched.
• Delete a file from a lowerdir: kernel writes a special whiteout marker in upperdir. The file appears gone, even though it still exists in the lower layer.
• Delete a directory: marker is an opaque directory xattr.

Copy-on-write is fast: file metadata is copied lazily, pages are mapped on demand. But touching a 5GB file copies 5GB.

#Where it lives on disk

The l/ directory contains short symlinks because mount commands have a length limit on the lower= argument.

#Other drivers and when to consider them

btrfs (Btrfs filesystem)

• If your /var/lib/docker is on a btrfs filesystem and you want native subvolume snapshots.
• Useful if you already use btrfs cluster-wide.
• Slightly faster image build/extract for some workloads.
• Avoid if you do not already run btrfs; do not switch FS just for this.

zfs (ZFS filesystem)

• Same idea as btrfs but for ZFS.
• Strong snapshot/clone story; production-grade on FreeBSD, OpenZFS on Linux.
• Avoid unless you run ZFS for other reasons.

fuse-overlayfs (userspace overlay)

• For rootless Docker where the user's UID cannot do real overlay mounts.
• Slower than overlay2 (userspace FUSE) but usable.
• Default for rootless setups on older kernels; modern kernels (>=5.13) allow real overlay2 in user namespaces, removing the need.

devicemapper (block-level COW)

• RHEL/CentOS 7-era default before overlay2 was reliable on those distros.
• Two modes: loopback (slow, ext4 file backed, never use for prod), direct-lvm (LVM thin pool, manageable).
• Deprecated in 2023; do not pick for new installs.

vfs (no COW)

• A full copy of every layer. Massive disk usage. Use only for testing or when no other driver works.

#How to check what you have

Backing Filesystem: extfs (ext4) and Supports d_type: true are required. xfs needs to be created with ftype=1 to support overlay2.

#Examples

#Switching to overlay2 (from devicemapper or vfs)

This is destructive. All images/containers in /var/lib/docker are erased.

#Inspecting a layer

Useful for forensics on a stopped container's FS.

#Disk usage breakdown

overlay2's space efficiency comes from layer sharing: 50 containers sharing the same image base store one copy of each lower layer plus 50 small upper layers.

#Real-world usage

• Default on every modern Linux: kernel 4.x + ext4/xfs gets overlay2 out of the box.
• Docker Desktop: runs Docker inside a Linux VM that uses overlay2.
• Container clusters (Swarm/K8s): every node uses overlay2 unless explicitly configured.
• Rootless Docker on old kernels: fuse-overlayfs.
• Hyper-converged ZFS or btrfs storage stacks: the matching driver, for snapshot integration.

#When changing the driver is actually justified

1. You already deeply use ZFS/btrfs for the host filesystem and want unified snapshot management.
2. You hit overlay2 limits (very rare): max stack depth ~127 lowerdirs, kernel-version-specific quirks, exotic xattr requirements.
3. Rootless Docker on a kernel that does not allow user-namespace overlay mounts.

Otherwise, leave overlay2. The performance/stability gap is real for non-default drivers.

#Common mistakes

Switching driver without wiping /var/lib/docker

Docker refuses to start if metadata for one driver exists and you ask for another. Either wipe the dir or use a separate data-root.

Using xfs without ftype=1

overlay2 needs xfs formatted with ftype=1. If your xfs is ftype=0, you cannot use overlay2 on that filesystem. Reformat or use a different mount point.

Loopback devicemapper

On old systems you might find devicemapper in loopback mode (a sparse file). It is slow and corrupts under pressure. Migrate to overlay2 immediately.

Confusing storage driver with Docker volumes

The storage driver manages image and container layers. Docker volumes are separate; their data lives in /var/lib/docker/volumes/ regardless of driver.

#Follow-up questions

Q: Why was the original overlay driver replaced?

A: Original overlay had inode exhaustion problems with many layers and deeper compatibility issues. overlay2 reuses inodes more aggressively and is the recommended successor.

Q: Does the storage driver affect runtime performance?

A: Slightly. overlay2 is near-native for read; writes that touch big lower-layer files are slow due to copy-up. For heavy file-write workloads, prefer volumes or bind mounts to bypass the storage driver entirely.

Q: Can I have multiple Docker daemons with different drivers?

A: Yes, run them with separate --data-root directories. Rare in practice.

Q: (Senior) How do you reason about disk space when many containers share lower layers?

A: docker system df -v shows the apparent vs shared size. The first container that pulls the base layer consumes the full size; subsequent containers add only their writable upper layer. When the last container using a layer is removed, the layer becomes prunable. So 100 containers from nginx:alpine cost roughly nginx_size + 100 * upper_layer_writes, not 100 * nginx_size.

Q: (Senior) What is metacopy and when does it matter?

A: metacopy=on lets overlay2 copy-up only metadata (chmod, chown) without copying the file's data. Saves space and time for permission-only changes. Off by default for compatibility; turn it on if you understand the tradeoff (slightly looser COW semantics around hardlinks).