Module 5.3: DNS Deep Dive

Цей контент ще не доступний вашою мовою.

Complexity: [MEDIUM]

Time to Complete: 40 minutes

Prerequisites

Before starting this module, you should have completed:

CKA Module: DNS in Kubernetes - Core DNS concepts, Service discovery basics
Module 5.1: Cilium - Networking fundamentals
Basic understanding of DNS (A records, CNAME, TTL)

What You’ll Be Able to Do

After completing this module, you will be able to:

Configure CoreDNS in Kubernetes with custom forwarding rules, caching, and zone delegation
Implement DNS-based service discovery patterns for cross-cluster and external service resolution
Monitor DNS query performance and troubleshoot resolution failures in Kubernetes clusters
Optimize DNS caching strategies and NodeLocal DNSCache for high-throughput applications

Why This Module Matters

The VP of Engineering stared at the dashboard. Average API latency: 500ms. The application code took 12ms. Where were the other 488 milliseconds going?

A mid-size SaaS company had been chasing this ghost for weeks. Their microservices were fast in isolation, but every cross-service call added mysterious overhead. They profiled the code. They tuned the database. They upgraded the network. Nothing helped.

Then a junior engineer ran strace on a simple HTTP call and saw it: five DNS queries fired before the actual resolution succeeded. Every single service-to-service call triggered five failed lookups before the sixth one returned an answer.

The culprit? Kubernetes’ default ndots:5 setting. When their Go service called payments-api:8080, the resolver dutifully tried payments-api.default.svc.cluster.local, then payments-api.svc.cluster.local, then payments-api.cluster.local, then payments-api.us-east-1.compute.internal, then payments-api.ec2.internal—all failures—before finally resolving the bare name. Five unnecessary round trips. On every request. Across 200 microservices making thousands of calls per second.

The fix was a two-line YAML change. Latency dropped from 500ms to 45ms overnight.

This module teaches you how to master CoreDNS configuration, eliminate DNS-related performance problems, and automate external DNS management—because DNS is the invisible backbone that can silently destroy your cluster’s performance.

Did You Know?

DNS is the #1 hidden performance killer in Kubernetes. The default ndots:5 setting means a lookup for api.stripe.com generates 5 failed queries before succeeding—multiplied across every outbound call in your cluster.
CoreDNS handles billions of queries daily across Kubernetes clusters worldwide. It replaced kube-dns in Kubernetes 1.13 and processes every single service discovery request in your cluster.
NodeLocal DNSCache can reduce DNS latency by 10x. By running a DNS cache on every node, queries that normally take 5-10ms round-trip to CoreDNS drop to sub-millisecond local lookups.
external-dns was born from frustration. Engineers were tired of manually updating Route53 every time they created an Ingress. The project now supports 30+ DNS providers and manages millions of records automatically.

CoreDNS Corefile Deep Dive

CoreDNS is configured through a file called the Corefile, stored in a ConfigMap named coredns in the kube-system namespace. Let’s look at the default and then customize it.

The Default Corefile

k get configmap coredns -n kube-system -o yaml

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
        max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

Plugin Ordering Matters

CoreDNS processes plugins in the order they appear in the Corefile. This is critical—a misconfigured order can silently break DNS resolution.

Plugin	Purpose	When to Customize
`errors`	Log errors to stdout	Always keep first
`health`	Health check endpoint on :8080	Rarely changed
`ready`	Readiness check endpoint on :8181	Rarely changed
`kubernetes`	Resolve cluster Services/Pods	Zone or TTL tuning
`prometheus`	Expose metrics on :9153	Leave enabled
`forward`	Forward unresolved queries upstream	Custom upstreams, split DNS
`cache`	Cache DNS responses	Increase for performance
`rewrite`	Rewrite queries before processing	Domain aliasing, migrations
`hosts`	Serve records from inline entries	Static overrides, testing
`file`	Serve records from a zone file	Advanced custom zones
`loop`	Detect and halt forwarding loops	Always keep
`reload`	Auto-reload Corefile on changes	Always keep
`loadbalance`	Randomize A/AAAA record order	Always keep

Customizing the Corefile

Split DNS: Route internal domains to a private DNS server

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health { lameduck 5s }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf { max_concurrent 1000 }
        cache 30
        loop
        reload
        loadbalance
    }
    corp.internal:53 {
        errors
        cache 60
        forward . 10.0.0.53 10.0.0.54
    }

This routes *.corp.internal queries to your corporate DNS servers while everything else goes through the normal resolution path.

Rewrite plugin: Migrate service names without breaking callers

rewrite name old-payment-svc.default.svc.cluster.local new-payment-svc.default.svc.cluster.local

This transparently redirects lookups for the old service name to the new one—invaluable during service migrations.

Hosts plugin: Override specific records

hosts {
    10.0.0.100 legacy-db.corp.internal
    10.0.0.101 legacy-cache.corp.internal
    fallthrough
}

The fallthrough directive is essential—without it, any query not matched by the hosts block will return NXDOMAIN instead of continuing to the next plugin.

DNS Performance Optimization

The ndots Problem

The ndots setting in /etc/resolv.conf controls when the resolver treats a name as fully qualified. The Kubernetes default is ndots:5, meaning any name with fewer than 5 dots gets the search domains appended first.

What happens when your app calls api.stripe.com (2 dots, less than 5):

1. api.stripe.com.default.svc.cluster.local  → NXDOMAIN (wasted)
2. api.stripe.com.svc.cluster.local          → NXDOMAIN (wasted)
3. api.stripe.com.cluster.local              → NXDOMAIN (wasted)
4. api.stripe.com.us-east-1.compute.internal → NXDOMAIN (wasted)
5. api.stripe.com.ec2.internal               → NXDOMAIN (wasted)
6. api.stripe.com.                           → SUCCESS (finally!)

Six queries instead of one. Multiply that by every external API call across every pod.

Fix 1: Set ndots to 2 in your pod spec

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
  containers:
    - name: app
      image: my-app:latest

With ndots:2, names with 2+ dots (like api.stripe.com) are tried as absolute names first. Internal service names like my-svc still get the search domains appended.

Fix 2: Use FQDNs with trailing dots in your application

# In your app config, use trailing dots for external domains:
STRIPE_API_HOST: "api.stripe.com."
DATABASE_HOST: "db.us-east-1.rds.amazonaws.com."

The trailing dot tells the resolver “this is already fully qualified, do not append search domains.” Zero wasted queries.

Fix 3: Reduce search domains

spec:
  dnsConfig:
    searches:
      - default.svc.cluster.local
      - svc.cluster.local
    options:
      - name: ndots
        value: "2"

NodeLocal DNSCache

NodeLocal DNSCache runs a lightweight DNS caching agent on every node as a DaemonSet. Instead of pods sending queries across the network to CoreDNS, they hit a local cache first.

Why it matters:

Cache hits resolve in <1ms instead of 5-10ms network round trips
Reduces load on CoreDNS pods dramatically
Eliminates conntrack race conditions that cause intermittent DNS failures on busy nodes (the infamous EAGAIN bug)

Deploy NodeLocal DNSCache:

# Get your cluster DNS IP
CLUSTER_DNS=$(k get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')

# Apply the NodeLocal DNSCache manifest (Kubernetes official addon)
# Replace __PILLAR__CLUSTER__DNS__ and __PILLAR__DNS__DOMAIN__ in the manifest
curl -s https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml \
  | sed "s/__PILLAR__CLUSTER__DNS__/$CLUSTER_DNS/g" \
  | sed "s/__PILLAR__DNS__DOMAIN__/cluster.local/g" \
  | sed "s/__PILLAR__LOCAL__DNS__/169.254.20.10/g" \
  | k apply -f -

Verify it is running:

k get pods -n kube-system -l k8s-app=node-local-dns

After deployment, pods on each node will resolve DNS through the local cache at 169.254.20.10 before falling through to CoreDNS.

external-dns: Automatic DNS Record Management

external-dns watches Kubernetes resources (Ingress, Service, Gateway) and automatically creates DNS records in your cloud provider. No more manual Route53 clicks.

How It Works

Ingress/Service/Gateway created
        ↓
external-dns detects the resource
        ↓
Reads annotations for hostname/TTL
        ↓
Creates/updates DNS record in provider (Route53, CloudFlare, etc.)
        ↓
Resource deleted → DNS record cleaned up

Installation

# Add the Helm repo
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm repo update

# Install for AWS Route53
helm install external-dns external-dns/external-dns \
  --namespace external-dns \
  --create-namespace \
  --set provider.name=aws \
  --set policy=sync \
  --set registry=txt \
  --set txtOwnerId=my-cluster \
  --set domainFilters[0]=example.com

The txtOwnerId is critical in multi-cluster setups—it prevents one cluster from deleting records owned by another.

Provider Configuration

Provider	Auth Method	Key Setting
AWS Route53	IRSA / IAM role	`provider.name=aws`
Cloudflare	API token	`--set cloudflare.apiToken=<token>`
Google Cloud DNS	Workload Identity / SA key	`provider.name=google`
Azure DNS	Managed Identity / SP	`provider.name=azure`

Annotations for Controlling Records

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    # Tell external-dns what hostname to create
    external-dns.alpha.kubernetes.io/hostname: app.example.com
    # Set a custom TTL (default is usually 300)
    external-dns.alpha.kubernetes.io/ttl: "60"
    # Create an alias record instead of CNAME (AWS-specific)
    external-dns.alpha.kubernetes.io/alias: "true"
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app
                port:
                  number: 80

For Services (LoadBalancer type):

apiVersion: v1
kind: Service
metadata:
  name: my-api
  annotations:
    external-dns.alpha.kubernetes.io/hostname: api.example.com
    external-dns.alpha.kubernetes.io/ttl: "120"
spec:
  type: LoadBalancer
  ports:
    - port: 443
      targetPort: 8443
  selector:
    app: my-api

external-dns reads the annotation, waits for the LoadBalancer to get an external IP/hostname, and creates the DNS record pointing to it.

DNS Debugging Toolkit

When DNS breaks, everything breaks. Here is your troubleshooting arsenal.

Quick Debug Commands

# Spin up a debug pod with DNS tools
k run dns-debug --image=registry.k8s.io/e2e-test-images/agnhost:2.39 \
  --restart=Never -- sleep 3600

# Test cluster DNS resolution
k exec dns-debug -- nslookup kubernetes.default.svc.cluster.local

# Test external resolution
k exec dns-debug -- nslookup google.com

# Detailed query with dig (shows timing and query path)
k exec dns-debug -- dig +search +all payments-api.default.svc.cluster.local

# Check what resolv.conf looks like inside the pod
k exec dns-debug -- cat /etc/resolv.conf

# Cleanup
k delete pod dns-debug --now

CoreDNS Metrics

CoreDNS exposes Prometheus metrics on port 9153. The key ones to monitor:

# Query rate - how busy is CoreDNS?
rate(coredns_dns_requests_total[5m])

# Error rate - are queries failing?
rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m])

# Cache hit ratio - is caching effective?
coredns_cache_hits_total / (coredns_cache_hits_total + coredns_cache_misses_total)

# Latency - how fast are responses?
histogram_quantile(0.99, rate(coredns_dns_request_duration_seconds_bucket[5m]))

Common DNS Failure Modes

Symptom	Likely Cause	Fix
`SERVFAIL` on all external queries	Upstream DNS unreachable	Check `forward` plugin config and network policies
Intermittent `EAGAIN` / `connection refused`	conntrack table full on busy nodes	Deploy NodeLocal DNSCache
Resolution works, but takes 5+ seconds	ndots causing search domain expansion	Set `ndots:2` or use FQDNs with trailing dots
New Service not resolving	CoreDNS not watching the namespace	Check CoreDNS logs, ensure `kubernetes` plugin config
`NXDOMAIN` for valid Service	Wrong namespace in query	Use full `svc-name.namespace.svc.cluster.local`
DNS works in pods but not in init containers	Init container ran before CoreDNS was ready	Add `dnsPolicy: Default` or init container retry logic

Common Mistakes

Mistake	Why It Happens	How to Fix
Leaving `ndots:5` as default	Nobody reads the default resolv.conf	Set `ndots:2` in pod spec or use FQDNs with trailing dots
Editing CoreDNS ConfigMap without `reload` plugin	Expecting changes to apply instantly	Ensure `reload` is in the Corefile (it is by default)
Missing `fallthrough` in hosts/file plugins	Queries not matched by the plugin return NXDOMAIN	Always add `fallthrough` unless you want to terminate resolution
Setting DNS cache TTL too high	Stale records after Service recreation	Use 30s for cluster-internal, up to 300s for external
external-dns without `txtOwnerId`	Multiple clusters overwrite each other’s records	Always set a unique owner ID per cluster
Not monitoring CoreDNS metrics	DNS degradation goes unnoticed until apps fail	Alert on SERVFAIL rate and p99 latency
Forgetting `dnsPolicy` when using `hostNetwork`	Pods with hostNetwork skip cluster DNS by default	Set `dnsPolicy: ClusterFirstWithHostNet`

Quiz

Question 1: Your pod calls api.payment.example.com. With the default ndots:5, how many DNS queries will be made before the name resolves?

Show Answer

Six queries. The name api.payment.example.com has 3 dots, which is less than 5. The resolver appends each search domain first (generating 5 NXDOMAIN responses), then tries the bare name as a last resort. Setting ndots:2 or appending a trailing dot (api.payment.example.com.) would resolve it in one query.

Question 2: You add a custom server block to the CoreDNS Corefile for corp.internal but queries to app.corp.internal return SERVFAIL. What should you check first?

Show Answer

Check that the forward directive in the corp.internal block points to a reachable DNS server. Also verify that no NetworkPolicy is blocking CoreDNS pods from reaching the corporate DNS server IP. Run k logs -n kube-system -l k8s-app=kube-dns to see the actual error CoreDNS is returning.

Question 3: After deploying NodeLocal DNSCache, how do pods know to use the local cache instead of the CoreDNS ClusterIP?

Show Answer

NodeLocal DNSCache uses a link-local address (169.254.20.10) and modifies the node’s iptables/ipvs rules to intercept traffic destined for the kube-dns ClusterIP. Pods continue to use the same ClusterIP in their /etc/resolv.conf, but the traffic is transparently redirected to the local cache. If the local cache is down, traffic falls through to CoreDNS normally.

Question 4: You deploy external-dns with policy=sync and notice DNS records disappearing. What is happening?

Show Answer

The sync policy means external-dns will delete any DNS records in the managed zone that do not correspond to a current Kubernetes resource. If you have manually created records or records from another source, sync will remove them. Use policy=upsert-only if you want external-dns to create and update records but never delete them. Alternatively, use domainFilters to restrict which domains external-dns manages.

Hands-On Exercise: DNS Mastery Lab

Objective

Customize CoreDNS, optimize DNS performance, and set up external-dns record management.

Setup

# Create a kind cluster
kind create cluster --name dns-lab

# Deploy two test services
k create namespace app-team
k run web --image=nginx --port=80 -n app-team
k expose pod web --port=80 -n app-team
k run api --image=nginx --port=80 -n app-team
k expose pod api --port=80 -n app-team

Part 1: CoreDNS Customization (10 min)

View the current CoreDNS Corefile:

k get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}'

Add a custom hosts entry that maps legacy-db.corp.internal to 10.96.0.100:

k edit configmap coredns -n kube-system

Add before the forward line:

hosts {
    10.96.0.100 legacy-db.corp.internal
    fallthrough
}

Wait for CoreDNS to reload (check logs):

k logs -n kube-system -l k8s-app=kube-dns -f --tail=5

Verify the custom entry resolves:

k run dns-test --image=busybox:1.36 --restart=Never -- nslookup legacy-db.corp.internal
k logs dns-test

Success criteria: nslookup returns 10.96.0.100 for legacy-db.corp.internal.

Part 2: DNS Performance Optimization (15 min)

Check the default resolv.conf inside a pod:

k run check-dns --image=busybox:1.36 --restart=Never -n app-team -- cat /etc/resolv.conf
k logs check-dns -n app-team

Note the ndots:5 and the search domains listed.

Create a pod with optimized DNS settings:

# Save as optimized-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: optimized-app
  namespace: app-team
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
    searches:
      - app-team.svc.cluster.local
      - svc.cluster.local
  containers:
    - name: app
      image: busybox:1.36
      command: ["sleep", "3600"]

k apply -f optimized-pod.yaml

Compare DNS behavior—verify internal resolution still works:

k exec optimized-app -n app-team -- nslookup web
k exec optimized-app -n app-team -- nslookup api.app-team.svc.cluster.local

Verify external resolution is faster (no wasted queries):

k exec optimized-app -n app-team -- nslookup google.com

Success criteria: Internal Services resolve correctly. External names resolve without search domain expansion.

Part 3: external-dns Simulation (10 min)

Since external-dns requires a real DNS provider, we will simulate the setup and verify the configuration:

Install external-dns in dry-run mode:

helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm repo update
helm install external-dns external-dns/external-dns \
  --namespace external-dns \
  --create-namespace \
  --set provider.name=aws \
  --set policy=upsert-only \
  --set registry=txt \
  --set txtOwnerId=dns-lab \
  --set dryRun=true \
  --set logLevel=debug

Create a Service with external-dns annotations:

# Save as annotated-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: public-api
  namespace: app-team
  annotations:
    external-dns.alpha.kubernetes.io/hostname: api.example.com
    external-dns.alpha.kubernetes.io/ttl: "120"
spec:
  type: LoadBalancer
  ports:
    - port: 80
  selector:
    app: api

k apply -f annotated-svc.yaml

Check external-dns logs to see it detecting the annotated resource:

k logs -n external-dns -l app.kubernetes.io/name=external-dns --tail=20

Success criteria: external-dns logs show it detected the annotated Service and would create an A/CNAME record for api.example.com.

Cleanup

kind delete cluster --name dns-lab

Cross-References

CKA DNS Module - Covers foundational DNS concepts: Service discovery, dnsPolicy, and basic CoreDNS configuration that this module builds upon
Module 5.1: Cilium - Network fundamentals and CNI context
Module 5.2: Service Mesh - mTLS and service-to-service communication
Module 1.1: Prometheus - Monitoring CoreDNS metrics

Next Module

Continue to the next networking module or explore Module 1.1: Prometheus to set up monitoring for the CoreDNS metrics discussed in this module.

Module 5.3: DNS Deep Dive

Complexity: [MEDIUM]

Time to Complete: 40 minutes

Prerequisites

What You’ll Be Able to Do

Why This Module Matters

Did You Know?

CoreDNS Corefile Deep Dive

The Default Corefile

Plugin Ordering Matters

Customizing the Corefile

DNS Performance Optimization

The ndots Problem

NodeLocal DNSCache

external-dns: Automatic DNS Record Management

How It Works

Installation

Provider Configuration

Annotations for Controlling Records

DNS Debugging Toolkit

Quick Debug Commands

CoreDNS Metrics

Common DNS Failure Modes

Common Mistakes

Quiz

Hands-On Exercise: DNS Mastery Lab

Objective

Setup

Part 1: CoreDNS Customization (10 min)

Part 2: DNS Performance Optimization (15 min)

Part 3: external-dns Simulation (10 min)

Cleanup

Further Reading

Cross-References

Next Module