This post documents a real debugging session that started with a simple symptom — example.com wouldn't load — and ended up uncovering several separate, unrelated problems layered on top of each other. It's written to walk through the actual diagnostic process, not just the final fix, since the process is usually the more useful part to learn from.
Starting point: an Ingress setup (from a previous post) was working, then stopped working after some changes were made mid-session, including installing a different CNI (Calico) on top of the cluster's existing one. This is the story of tracking that down.
The Symptom
❯ curl example.com
curl: (7) Failed to connect to example.com port 80 after 21007 ms: Could not connect to server
A connection timeout like this can mean a lot of different things: DNS pointing to the wrong place, a firewall silently dropping packets, a service that's down, or a machine that simply can't reach the target network at all. The rest of this post is the process of ruling each of those out, one at a time.
Step 1: Confirm What the Hostname Actually Resolves To
Before assuming anything about networking, the first thing to check is whether example.com is even resolving to the IP it's supposed to:
❯ getent hosts example.com
10.96.101.127 example.com
This confirmed the local /etc/hosts entry was working correctly — example.com resolved to 10.96.101.127, matching the Ingress controller's IP from earlier setup. So the problem wasn't DNS/hosts resolution. The IP itself was the next thing to check.
Step 2: Test the IP Directly, Bypassing the Hostname
If the hostname resolves correctly but the connection still fails, the next step is to test the raw IP directly — this removes DNS/hosts from the equation entirely and asks: "is this address reachable at all?"
❯ curl -v http://10.96.101.127
* Trying 10.96.101.127:80...
* connect to 10.96.101.127 port 80 from 10.0.2.15 port 57576 failed: Connection refused
* Failed to connect to 10.96.101.127 port 80 after 21003 ms: Could not connect to server
Two useful details came out of this:
Connection refused, not a timeout. This is a meaningful difference. A timeout usually means packets are going out and nothing is answering. "Refused" happens quickly and means something did respond — but actively said "no."- The source IP was
10.0.2.15. This is VirtualBox's default address for a guest machine in NAT networking mode, which confirmed this was being tested from inside a VM.
The actual explanation: 10.96.101.127 is a Kubernetes ClusterIP. A ClusterIP isn't a real address that exists anywhere on the network — it's a virtual IP that only works because of routing rules (iptables/IPVS) that Kubernetes' kube-proxy installs inside the specific machine running the cluster. No other machine — not a VM, not a laptop on the same network — can ever reach a ClusterIP directly, because that address doesn't exist anywhere outside that one machine's own routing table.
This meant the original approach (pointing /etc/hosts at the ClusterIP) was never going to work from a separate machine, regardless of any other fix.
Step 3: Find a Real, Reachable Address Instead
Since ClusterIPs don't work across machines, the next step was finding an address that does — specifically, the IP of one of the actual Docker containers backing the kind cluster's nodes, combined with the Ingress controller's NodePort (a real port opened on the node itself, unlike a ClusterIP).
❯ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
691fe58f2715 kindest/node:v1.35.0 "/usr/local/bin/entr…" 2 days ago Up 18 minutes 127.0.0.1:38079->6443/tcp cka-new-control-plane
aae68ca53b7b kindest/node:v1.35.0 "/usr/local/bin/entr…" 2 days ago Up 18 minutes cka-new-worker2
b51bce23b84f kindest/node:v1.35.0 "/usr/local/bin/entr…" 2 days ago Up 18 minutes cka-new-worker
75634b82514e kindest/node:v1.35.0 "/usr/local/bin/entr…" 2 days ago Up 18 minutes cka-new-worker3
None of the worker nodes had any ports published to the host — only the control-plane's Kubernetes API port was mapped. So the node's actual IP address on Docker's internal network was needed instead:
❯ docker inspect cka-new-worker | grep -A 15 '"kind"'
...
"IPAddress": "172.18.0.5",
...
A discrepancy showed up here worth noting: this IP (172.18.0.5) didn't match what kubectl get nodes -o wide had shown earlier in the session for this same node (172.18.0.3). Docker assigns container IPs dynamically, and since these containers showed Up 18 minutes (meaning they had recently restarted), the IP had simply changed. This is a good general reminder that hardcoding container IPs is fragile — they aren't guaranteed to stay the same across restarts.
Testing the NodePort against this IP, however, still failed:
❯ curl -v http://172.18.0.5:31003 -H "Host: example.com"
* Trying 172.18.0.5:31003...
* connect to 172.18.0.5 port 31003 from 172.18.0.1 port 53748 failed: Connection timed out
So even with a real, reachable-in-theory IP and the correct port, nothing answered. This meant the problem was something deeper than just "wrong IP" — worth investigating the cluster's internal health next.
Step 4: Ruling Out the Obvious Culprits
At this point, several standard checks were run to narrow down where the actual fault was.
Was the Ingress controller pod even healthy?
❯ kubectl get pods -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE
ingress-nginx-controller-7d65c586d6-kvn9s 1/1 Running 3 (20m ago) 16h 10.244.1.2 cka-new-worker2
It was Running, but had restarted 3 times recently — worth remembering, but not conclusive on its own.
Was kube-proxy (the component responsible for NodePort/ClusterIP routing) healthy on every node?
❯ kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide
NAME READY STATUS RESTARTS AGE NODE
kube-proxy-2mlbd 1/1 Running 4 (25m ago) 2d18h cka-new-worker
kube-proxy-jb2rt 1/1 Running 4 2d18h cka-new-worker2
kube-proxy-m2vq9 1/1 Running 4 2d18h cka-new-control-plane
kube-proxy-rnlwc 1/1 Running 4 2d18h cka-new-worker3
All Running, though cka-new-worker had also restarted recently.
Was there a conflict between two CNI plugins (since Calico had been installed on top of the cluster's default kindnet networking earlier)?
❯ docker exec cka-new-worker2 ls /etc/cni/net.d/
10-kindnet.conflist
Only one CNI config was present — no conflict there, at least on this node.
Did basic pod-to-pod networking inside the cluster even work at all?
❯ kubectl run curl --rm -it --image=curlimages/curl -- curl http://hello-world
curl: (6) Could not resolve host: hello-world
This was the first real breakthrough. This same command had worked earlier in the setup process — now it couldn't even resolve the Service's name. This shifted the investigation toward DNS specifically.
Step 5: Investigating the DNS Failure
First, check whether restarting kube-proxy (to clear out any stale routing rules) fixes it:
❯ kubectl delete pods -n kube-system -l k8s-app=kube-proxy
pod "kube-proxy-2mlbd" deleted from kube-system namespace
pod "kube-proxy-jb2rt" deleted from kube-system namespace
pod "kube-proxy-m2vq9" deleted from kube-system namespace
pod "kube-proxy-rnlwc" deleted from kube-system namespace
After waiting for the new pods to come back Running:
❯ kubectl run curl --rm -it --image=curlimages/curl -- curl http://hello-world
curl: (6) Could not resolve host: hello-world
No change. This ruled out stale kube-proxy rules as the cause.
Check whether the cluster's internal DNS service (CoreDNS) has healthy endpoints behind it:
❯ kubectl get endpoints -n kube-system kube-dns
NAME ENDPOINTS AGE
kube-dns 10.244.0.2:9153,10.244.0.3:9153,10.244.0.2:53 + 3 more... 2d18h
This looked healthy — CoreDNS pods were properly registered.
Check whether a real pod is even configured to use the right DNS server:
❯ kubectl exec -it hello-world-c8b6f8dbd-9t9n6 -- sh
# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
This was also correct — 10.96.0.10 is the standard internal DNS Service IP.
So far: correct DNS configuration, healthy-looking CoreDNS, no stale kube-proxy rules — yet DNS lookups still failed. The next step was testing whether this was actually a DNS-specific issue, or something more fundamental with networking between nodes.
Step 6: Isolating DNS vs. General Networking
Test a plain HTTP connection to a Service, bypassing DNS entirely (using the raw ClusterIP):
# python3 -c "import urllib.request; print(urllib.request.urlopen('http://10.96.74.127').read())"
b'Hello, World!'
This worked. So general Service routing (HTTP, TCP) was fine — the problem was specific to DNS.
Test raw connectivity to the DNS port itself (port 53), both against the Service IP and a CoreDNS pod's IP directly:
# python3 -c "
import socket
try:
s = socket.create_connection(('10.96.0.10', 53), timeout=3)
print('connected')
s.close()
except Exception as e:
print('failed:', e)
"
failed: timed out
# python3 -c "
import socket
try:
s = socket.create_connection(('10.244.0.2', 53), timeout=3)
print('connected')
s.close()
except Exception as e:
print('failed:', e)
"
failed: timed out
Both failed — even a direct connection attempt to a CoreDNS pod's own IP, with no DNS resolution or Service routing involved at all, still timed out.
The key detail that explained this: checking which node each pod involved was running on showed that the working test (10.96.74.127, resolving to the hello-world pod) happened to be a pod on the same node the test was run from. The failing tests (both CoreDNS IPs) were both on the control-plane node — a different node entirely. This pointed toward a cross-node networking problem, not a DNS-specific one.
Step 7: Confirming Cross-Node Networking Was Broken
Checked whether the Ingress controller's NodePort Service had a networking policy that might explain uneven behavior across nodes:
❯ kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.spec.externalTrafficPolicy}'
Local
This turned out to be a real, separate finding: externalTrafficPolicy: Local means kube-proxy will only forward NodePort traffic on a node if that node happens to be running a copy of the target pod locally — every other node will actively drop the traffic rather than forward it elsewhere. This fully explained the earlier NodePort timeout, since the Ingress controller pod only ran on cka-new-worker2, and the test had hit a different node.
Confirmed by checking iptables on the failing node:
❯ docker exec cka-new-worker3 iptables-save | grep -i "53\|drop" | head -30
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "ingress-nginx/ingress-nginx-controller:http has no local endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 31003 -j DROP
This line confirms it directly: any NodePort traffic to port 31003 on this node is explicitly dropped, because this node has no local copy of the Ingress controller pod.
Fixing the NodePort routing issue was straightforward — hit the correct node directly:
❯ curl -v http://172.18.0.4:31003 -H "Host: example.com"
* Connected to 172.18.0.4 (172.18.0.4) port 31003
< HTTP/1.1 504 Gateway Time-out
The connection succeeded this time — but nginx itself returned a 504 Gateway Time-out. This meant the request reached the Ingress controller fine, but the Ingress controller couldn't reach its own backend (the hello-world Service). And, notably, the Ingress controller pod and the hello-world pod were running on two different nodes — another cross-node hop, timing out in exactly the same pattern as the DNS failures.
This was the real conclusion: same-node pod-to-pod traffic worked consistently throughout this session. Cross-node pod-to-pod traffic (DNS lookups to CoreDNS, and now the Ingress controller reaching its backend) consistently failed. Along the way, other explanations were ruled out — no competing CNI configs, no blocking NetworkPolicy, correctly configured DNS, correct iptables NAT rules, and Calico's own control-plane pods weren't even running:
❯ kubectl get pods -n calico-system -o wide
No resources found in calico-system namespace.
The most likely explanation: installing Calico mid-way through the cluster's life (on top of the existing kindnet CNI, without removing it) had left the routes between nodes in a broken, inconsistent state — even after Calico itself was no longer running. Untangling that kind of mixed-CNI networking state by hand is generally not worth the time; the standard, reliable fix is to rebuild the cluster from scratch.
Step 8: Rebuilding the Cluster
❯ kind delete cluster --name cka-new
To make future testing easier, a kind config file was created to forward the Ingress controller's expected ports straight from the container to localhost on the host machine — removing the need to look up container IPs at all going forward:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: cka-new
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 31003
hostPort: 31003
protocol: TCP
- containerPort: 30437
hostPort: 30437
protocol: TCP
- role: worker
- role: worker
- role: worker
❯ kind create cluster --name cka-new --config kind-config.yaml
This time, Calico was intentionally left out — the default kindnet CNI had been working fine before Calico was introduced mid-session, and nothing about the setup actually required Calico's features.
Reapplying the application and Ingress manifests:
❯ kubectl apply -f deployment.yaml -f service.yaml -f ingress.yaml
❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/aws/deploy.yaml
Step 9: A Slow Pod Isn't Always a Stuck Pod
While waiting for the Ingress controller to come up, it sat in ContainerCreating for several minutes, which raised the question of whether something was actually wrong:
❯ kubectl get pods -n ingress-nginx -w
ingress-nginx-controller-7d65c586d6-fz54r 0/1 ContainerCreating 0 2m59s
Checking the pod's events showed the actual cause:
❯ kubectl describe pod -n ingress-nginx ingress-nginx-controller-7d65c586d6-fz54r
...
Events:
Warning FailedMount 4m41s (x8 over 5m45s) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
Normal Pulling 3m30s kubelet Pulling image "registry.k8s.io/ingress-nginx/controller:v1.15.1@sha256:..."
This looked alarming at first, but both issues were self-resolving, expected steps of the install process rather than actual problems:
- The
FailedMount warnings happened because two setup Jobs (ingress-nginx-admission-create and ingress-nginx-admission-patch) needed to finish first — they're responsible for generating the certificate secret the controller pod was waiting to mount. Until those Jobs completed, that secret simply didn't exist yet.
- The image pull itself was just slow — checking
kubectl get events showed the two setup Jobs' images had each taken over a minute to pull, which suggested a slow registry pull rather than anything broken locally.
Waiting it out confirmed this:
❯ kubectl get pods -n ingress-nginx
ingress-nginx-controller-7d65c586d6-fz54r 1/1 Running 0 7m34s
Lesson here: not every ContainerCreating or warning event means something is broken — checking the actual Events section (via kubectl describe pod) before assuming failure saves a lot of unnecessary troubleshooting.
Step 10: Fixing the Service Configuration Correctly
With the controller running, the externalTrafficPolicy and Service type still needed to be set correctly, to avoid re-hitting the earlier Local-policy and LoadBalancer-pending issues.
The first attempt patched only externalTrafficPolicy:
❯ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
❯ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
ingress-nginx-controller LoadBalancer 10.96.61.252 <pending> 80:32087/TCP,443:32290/TCP
This revealed two remaining issues: the Service was still type: LoadBalancer (stuck <pending>, same problem as before), and the NodePorts had landed on different numbers (32087/32290) than what kind-config.yaml was forwarding (31003/30437).
Fixing both at once, by explicitly setting the type to NodePort and pinning the exact port numbers:
❯ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","protocol":"TCP","nodePort":31003},{"name":"https","port":443,"targetPort":"https","protocol":"TCP","nodePort":30437}]}}'
❯ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
ingress-nginx-controller NodePort 10.96.61.252 <none> 80:31003/TCP,443:30437/TCP
This matched the port forwarding configured in kind-config.yaml.
Step 11: Confirming Everything Works
❯ curl -v http://localhost:31003 -H "Host: example.com"
* Trying [::1]:31003...
* connect to ::1 port 31003 from ::1 port 49216 failed: Connection refused
* Trying 127.0.0.1:31003...
* Connected to localhost (127.0.0.1) port 31003
< HTTP/1.1 200 OK
Hello, World!
This confirmed everything was working correctly: the port forwarding in kind-config.yaml, the Cluster traffic policy correctly routing regardless of which node the request landed on, and the Ingress controller successfully reaching the hello-world backend (no more 504 Gateway Time-out).
The ::1 (IPv6) connection attempt failing before falling back to IPv4 is expected — kind-config.yaml only forwards the port over IPv4, so curl's dual-stack attempt simply skips it and retries over IPv4 instead. Not an error to fix.
Final step — update /etc/hosts, this time pointing at 127.0.0.1 instead of any container IP, since port forwarding now handles the routing:
127.0.0.1 example.com
❯ curl http://example.com:31003
Hello, World!
Why the Port Still Has to Be Specified
Running curl http://example.com without a port defaults to port 80 — but nothing is mapped to host port 80 in this setup, only 31003 and 30437. Without a port, the connection fails immediately:
curl: (7) Failed to connect to example.com port 80 after ...: Connection refused
This is expected: the Ingress controller is exposed via a NodePort Service, and NodePorts are, by design, high-numbered ports (typically in the 30000–32767 range) — they were never going to answer on plain port 80 without an explicit mapping.
To make curl http://example.com work without a port number, kind-config.yaml could instead map host port 80 directly to the NodePort:
extraPortMappings:
- containerPort: 31003
hostPort: 80
protocol: TCP
This requires a cluster rebuild to take effect, and on Linux, binding to port 80 may require elevated (sudo) privileges since ports below 1024 are privileged. For local testing and practice environments, simply including the NodePort in the URL is generally the more common and simpler approach — production clusters solve the "no port number" problem differently anyway, using a real external load balancer or a bare-metal solution like MetalLB in front of the cluster.