DNS (Domain Name System) associates domain names with typed resource data, including but not limited to IPv4 and IPv6 addresses. In datacenters, DNS serves critical roles beyond simple lookups: service discovery, load distribution, and failover orchestration. Understanding DNS behavior during outages can mean the difference between graceful degradation and cascading failures.

# Internal vs External DNS

Datacenters typically run separate DNS infrastructure for internal and external queries.

External DNS: Resolves public-facing names (www.example.com) for internet clients. Hosted on public DNS servers (Route53, Cloudflare DNS, etc.). Returns public IP addresses (load balancers, CDN endpoints).

Internal DNS: Publishes internal service names (for example, db.internal or cache.prod.dc1) for authorized internal clients. Answers can use private address space such as 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16. Access is normally restricted by network and DNS policy rather than by DNS itself.

# Split-Horizon DNS

Split-horizon (split-brain) DNS returns different answers depending on who's asking. Same hostname resolves differently for internal vs external clients.

Why use split-horizon:

• Avoid hairpinning: internal servers access services directly, not through external load balancers
• Reduce latency: skip NAT, load balancers, and public internet path for internal traffic
• Security: hide internal topology from external queries

Configuration: A split-DNS deployment selects a view using policy—often the query source network, but potentially also the server address, query type, or recursion context—and returns the answer defined for that view.

# Service Discovery Patterns

In modern microservice architectures, service discovery finds instances of a service dynamically. DNS is one approach, but not the only one.

DNS-Based Service Discovery

Query: web-service.prod.internal
Response: 10.0.1.10, 10.0.1.11, 10.0.1.12 (A records)

Client or local resolver orders the addresses according to its selection policy
Application attempts an address, possibly through a proxy or load balancer, and may retry others

Advantages: Basic DNS lookup uses ubiquitous client APIs and can publish multiple addresses without a service-specific client library.

Limitations: Caching delays changes according to TTL, clients vary in address selection and retry behavior, and basic A/AAAA records do not themselves carry weights or health. However, authoritative DNS platforms can integrate weighted policies and health checks, and systems such as Kubernetes can publish ready endpoints through DNS.

Dedicated Service Discovery Systems

Systems like Consul, etcd, ZooKeeper, or Kubernetes Services provide dynamic service discovery with health checks and real-time updates.

Advantages: Fast updates (no TTL delays), health-aware (excludes failed instances), richer metadata (version tags, weights).

Disadvantages: Requires client libraries, adds dependency on registry service, more complex than DNS.

# TTL and Caching Behavior

DNS responses include a TTL (Time to Live) that tells clients/resolvers how long to cache the answer. TTL is critical for failover speed and load balancing.

Query: api.example.com
Response: 203.0.113.10, TTL=60 seconds

A resolver may cache this answer for up to 60 seconds from receipt
Clients querying that resolver see the remaining TTL
Resolvers may evict the answer before that limit

Low TTL (5-60 seconds):

• Pros: Fast failover, quick DNS-based load balancing changes
• Cons: Higher DNS query load, more latency (frequent lookups)

High TTL (300-3600 seconds):

• Pros: Lower DNS load, less latency (fewer lookups)
• Cons: Slow failover, clients may hit stale IPs for hours

Best practice: Use low TTL (60s) for services that need fast failover. Use high TTL (300-600s) for stable infrastructure. Plan failovers accounting for "longest TTL" delay.

# DNS During Outages and Failover

DNS is often involved in failover scenarios, but it's not instant. Understanding DNS propagation delays prevents false confidence in failover plans.

Scenario: Datacenter Failover

Actual recovery time: 60+ seconds, not "instant."

Problem: Clients that cached old IP (before failover) wait for TTL to expire. During this time, they attempt to connect to failed datacenter.

Mitigation strategies:

• Use anycast IP addresses (same IP advertised from multiple datacenters via BGP)
• Use load balancers that do health checks and remove failed backends
• Combine DNS failover with application-level retries to alternate endpoints
• Pre-position clients with multiple IP addresses (A records) and client-side failover

DNS as Dependency in Outages

Lesson from Facebook 2021: A maintenance command first disconnected Facebook's backbone. DNS locations could no longer reach the datacenters, declared themselves unhealthy, and withdrew their own BGP advertisements. The DNS servers remained operational but unreachable, so clients could not query them.

Implication: DNS depends on underlying network reachability, and health logic can turn a backbone failure into route withdrawal. Map that dependency and keep recovery access independent of the network and DNS services being repaired.

# Key Takeaways

• Split-horizon DNS returns different answers for internal vs external queries
• DNS-based service discovery is simple but slow to update; dedicated systems (Consul, K8s) offer health-aware, real-time discovery
• TTL determines failover speed—low TTL (60s) enables faster recovery but increases DNS load
• DNS failover isn't instant; plan for "longest TTL" delay in recovery time objectives
• DNS depends on network routing; loss of backbone reachability can make serving locations withdraw routes, leaving operational DNS servers unreachable
• Use anycast or load balancers with health checks for faster failover than DNS alone provides