[dns-operations] Best practices for Linux/UNIX stub resolver failover

[Chuck Anderson]

On Fri, May 02, 2014 at 09:04:41AM +0000, Brett Carr wrote:
> 
> On 22 Apr 2014, at 20:04, Chuck Anderson <cra at WPI.EDU> wrote:
> 
> > Is it really expected that the first DNS server listed in
> > /etc/resolv.conf should never go down?  Operationally speaking, who
> > can actually rely on listing multiple nameservers in /etc/resolv.conf
> > and using libc's failover mechanism in any kind of production server?
> > Because the failover behavior in libc is atrocious--each new or
> > existing process has to re-do the failover after timing out, and even
> > long-running processes have to call res_init() to re-read resolv.conf.
> > It seems that the only sensible way to run a datacenter (or a network
> > full of Linux workstations for that matter) is to either:
> > 
> > 1. Make sure the first nameserver listed in resolv.conf never goes
> >   down by using Anycast DNS or some other failover mechanism like
> >   VRRP or CARP on the DNS server side.
> > 
> > or:
> > 
> > 2. Use a local DNS daemon on every server with forwarders configured
> >   to the network's nameservers, and fix resolv.conf to 127.0.0.1.
> > 
> 
> Not an expert either but I remember having bad experiences of this on Solaris 8/10 and early versions of Linux many years ago but I was under the impression that modern versions of *nix had this covered, certainly the quick test I just did on a RHEL6 server makes me think this is no longer an issue.
> 
> The server I have has a resolv.conf configured as follows:
> 
> nameserver	192.168.1.244
> nameserver	192.168.1.1
> 
> If I stop the resolver running on 192.168.1.244 and then generate a query I see the following in a tcpdump
> 
> 09:57:01.348084 IP 192.168.1.173.47658 > 192.168.1.244.domain: 35734+ A? pool.ntp.org. (30)
> 09:57:01.348160 IP 192.168.1.173.47658 > 192.168.1.244.domain: 61935+ AAAA? pool.ntp.org. (30)
> 09:57:01.348223 IP 192.168.1.173.42169 > 192.168.1.1.domain: 35734+ A? pool.ntp.org. (30)
> 09:57:01.348267 IP 192.168.1.173.42169 > 192.168.1.1.domain: 61935+ AAAA? pool.ntp.org. (30)
> 
> This is very quick failover to the second resolver.
> 
> Or did I mis-understand the problem.

This is a common misunderstanding.  Simply shutting down the DNS
server software but leaving the IP responding otherwise doesn't show
the problem, because the server will immediately respond with an ICMP
Port Unreachable or TCP RST causing the stub resolver to fail over
immediately.  To see the poor failover behavior, you have to take the
server down completely, or try putting in a nameserver IP address that
isn't responding at all (i.e. choose an unused IP on your network).
Then the stub resolver will not receive any response at all, and will
have to wait for a timeout period (configurable, 5 seconds by
default).  The issue with glibc's stub resolver is that for each new
query an application makes it tries the first server again--there is
no state kept to remember that the first server is down.  And if you
reorder the entries in resolv.conf (using something like a nsfailover
cron job [1]), a long-running process/thread won't notice unless the
application has been programmed to call res_init() or you are using a
patched glibc [2] as apparently Debian/Ubuntu have.

[1] https://github.com/kvz/nsfailover

[2] https://sourceware.org/ml/libc-alpha/2012-12/msg00416.html