How to troubleshoot network problems between Docker containers?

docs.questions.sections.docker~5 min read

Network troubleshooting in Docker means walking from DNS down to packet inspection until you find where the connection breaks. Most issues are in the top three layers (wrong network, missing DNS, blocked port). Reach for tcpdump and iptables only when those fail.

Theory

TL;DR

Containers reach each other only if on the same network.
The default bridge network has no DNS between containers; user-defined bridges do.
Host firewall (UFW, firewalld, custom iptables) often interferes with the docker0 bridge.
--network=host shares the host's network namespace; no isolation.
Overlay networks (Swarm) need UDP 4789 (VXLAN) and an MTU lower than the host's.
Diagnostic order: layer-by-layer from name resolution down to packets.

Diagnostic checklist

What network is each container on?
Is DNS resolving the peer's name?
Is the peer's IP reachable (ping)?
Is the target port open (nc, telnet)?
Is the target service actually listening on that port and binding to the right interface?
Is a firewall (host or in-container) dropping packets?
Is anything wrong at layer 2/3 (MTU, MAC address conflict)?

Network types and their gotchas

Default bridge (no name)

Auto-assigned to containers without --network.
No DNS between containers. Names do not resolve. You must use --link (deprecated) or IPs.
Avoid for anything beyond toy use.

User-defined bridge (docker network create mynet)

DNS works: containers can reach each other by container name.
Isolated from the default bridge.
Default for Compose projects (one bridge per project).

host network (--network=host)

Container shares host's network namespace.
No port mapping needed; no isolation.
Useful for performance-sensitive cases or system tools.
Linux-only behavior; on Docker Desktop (Mac/Windows) it does not mean host networking, it means host of the VM.

Overlay (Swarm)

Spans multiple Docker hosts via VXLAN.
Needs UDP 4789 between hosts for VXLAN traffic, TCP/UDP 7946 for control plane.
MTU mismatch is the classic overlay bug: VXLAN encapsulation adds 50 bytes; if host MTU is 1500, container MTU should be ≤ 1450.

none (--network=none)

Loopback only. Used for sandboxing.

Examples

Step 1: confirm both containers are on the same network

bash

docker inspect app --format '{{json .NetworkSettings.Networks}}' | jq
# {
#   "my-app-default": {
#     "IPAddress": "172.18.0.3",
#     "Aliases": ["app", "abc123"]
#   }
# }

docker inspect db --format '{{json .NetworkSettings.Networks}}' | jq
# {
#   "some-other-network": {
#     "IPAddress": "172.19.0.5"
#   }
# }

Different networks: connect them, or move one container.

bash

docker network connect my-app-default db
# now db is on both networks

Step 2: DNS

bash

docker exec app sh -c 'getent hosts db'
# 172.18.0.5      db

If nothing resolves: you are probably on the default bridge. Move to a user-defined network.

bash

docker exec app cat /etc/resolv.conf
# nameserver 127.0.0.11        ← embedded Docker DNS

127.0.0.11 is the embedded resolver Docker injects into user-defined networks. Missing means default-bridge issue.

Step 3: IP-level reachability

bash

docker exec app ping -c 3 db
# PING db (172.18.0.5): 56 data bytes
# 64 bytes from 172.18.0.5: seq=0 ttl=64 time=0.080 ms

No response: networking layer is broken. Check iptables:

bash

sudo iptables -L DOCKER-USER -nv
sudo iptables -L DOCKER -nv

UFW often resets these chains; add explicit allows or disable UFW interference.

Step 4: port-level reachability

bash

docker exec app nc -vz db 5432
# Connection to db 5432 port [tcp/postgresql] succeeded!

Connection refused: the service is not listening on that port. Operation timed out: a firewall (host or in-container) is dropping the packet.

bash

# Inside the db container
docker exec db ss -tlnp
# State    Recv-Q   Send-Q   Local Address:Port
# LISTEN   0        128      127.0.0.1:5432   ← bound to localhost, not container interface!

Classic mistake: Postgres bound to 127.0.0.1 instead of 0.0.0.0 is unreachable from peer containers. Fix the service config: listen_addresses = '*' for Postgres.

Step 5: packet capture (when nothing else helps)

Find the bridge name:

bash

docker network inspect my-app-default --format '{{.Id}}'
# 4f7c9... (first 12 chars become the bridge interface name suffix)
ip link show | grep br-
# br-4f7c9...

Capture:

bash

sudo tcpdump -i br-4f7c9 -nn 'host 172.18.0.5 and port 5432'

Run a request from app, watch the packet flow. SYN with no SYN-ACK = firewall drop or service not listening. Reset = service refused.

Step 6: container-level firewall

Some images ship with iptables rules (security-hardened distros):

bash

docker exec db iptables -L -nv
# Chain INPUT (policy DROP 0 packets, 0 bytes)

policy DROP blocks everything not explicitly allowed. Either fix the in-container rules or use a base image without them.

Common scenarios and fixes

"Cannot connect to db from app" — Compose project

yaml

services:
  app:
    image: myorg/app
  db:
    image: postgres:16

Compose auto-creates <project>_default network. Both services on it. app reaches db by name. Should just work.

If broken: confirm both came up (docker compose ps), then:

bash

docker compose exec app ping db
docker compose exec app nc -vz db 5432

"Host cannot reach container's published port"

bash

docker run -d -p 8080:80 nginx
curl localhost:8080  # works
curl 192.168.1.5:8080  # fails from another machine

Check the host firewall: ufw status, firewall-cmd --list-all, or iptables -L. Open port 8080.

"Container can hit external internet but not other container"

Likely on the default bridge (no DNS) or different networks. Move to a user-defined bridge:

bash

docker network create mynet
docker run --network=mynet --name=app myorg/app
docker run --network=mynet --name=db postgres:16

"Random connection drops or slow Swarm overlay"

MTU mismatch. Check host MTU:

bash

ip link show eth0 | grep mtu
# mtu 1500

VXLAN overhead = 50 bytes. Containers in overlay should run at MTU 1450 (or 1400 to be safe across cloud NAT). Set per-network:

bash

docker network create -d overlay \
    --opt com.docker.network.driver.mtu=1450 \
    swarm-net

"`--network=host` works on Linux, fails on Mac"

Docker Desktop runs Linux in a VM. host networking targets the VM, not your Mac. To reach localhost on the Mac from a container, use host.docker.internal.

Useful tools

nicolaka/netshoot: a debug image with dig, nc, tcpdump, iperf, mtr, tshark. Run inside the same network as the misbehaving container:

bash

docker run --rm -it --network=container:app nicolaka/netshoot
# Now you have full diagnostic toolkit in app's network namespace

docker logs <container> for the service's own log; often the failure is on the listening side.
docker exec <container> cat /etc/hosts to see what the container sees as its own name.

Real-world usage

"App cannot reach DB": 90% of the time, networks misaligned or DNS is missing. Apply steps 1-2.
Intermittent failures: usually MTU on overlay, or DNS TTL with restarting containers.
Production outages: tcpdump on the bridge, then iptables. Get a packet capture before restarting anything.

Common mistakes

Putting services on the default bridge

No DNS. Always create a user-defined network or rely on Compose's auto-network.

Binding a service to 127.0.0.1 inside a container

Reachable only from within that container. Bind to 0.0.0.0 so other containers can reach it on the bridge.

Modifying host iptables manually

Docker rewrites iptables on every restart. Custom rules belong in the DOCKER-USER chain (Docker leaves it alone).

Using --link

Deprecated. Sets up /etc/hosts entries, no DNS. Use user-defined networks.

Follow-up questions

Q: Why does the default bridge not have DNS?

A: Historical. The default bridge predates Docker's embedded DNS server. User-defined bridges added DNS but old behavior was kept for compatibility.

Q: What is 127.0.0.11?

A: Docker's embedded DNS resolver. Each container in a user-defined network has it as its nameserver and queries are resolved to other containers' IPs.

Q: How do I see what packets are dropped at the host firewall?

A: Add a logging rule before the DROP rule: iptables -I DOCKER-USER -j LOG --log-prefix 'docker-drop: '. Then dmesg -w shows the drops in real time.

Q: (Senior) How do you debug an overlay network that works locally but loses packets across hosts?

A: Confirm UDP 4789 (VXLAN) is allowed between hosts (cloud security groups often block it). Run tcpdump -i any port 4789 on both hosts during a failing request. Check MTU; lower it to 1400 if VXLAN is going through any tunnel that adds more overhead. Inspect docker network inspect <overlay> for peer counts; mismatched peers means Swarm gossip is unhealthy.

Q: (Senior) How do you detect MAC address conflicts on a custom bridge?

A: docker network inspect <net> lists each container's MAC. Conflicts come from manually setting --mac-address. The kernel logs bridge: received packet on br0 with own address as source address on conflicts. Fix by removing manual MAC assignment.

Short Answer

Interview ready

Premium

A concise answer to help you respond confidently on this topic during an interview.

Comments

No comments yet