[dns-operations] Load balancing DNS queries across many machines

[Michael Sinatra]

On 07/16/09 19:32, Matthew Dempsky wrote:
> Let me make an analogy with HTTPS.
> 
> When HTTPS was first introduced, web servers didn't have native
> support for it, so a reasonable solution was to setup an HTTPS-to-HTTP
> proxy server that accepted HTTPS requests, handled the SSL protocol,
> and forwarded the encapsulated request to a plain HTTP server.  Even
> today this is a popular solution.
> 
> However, one problem with using a forwarder like this is that the HTTP
> server sees the TCP connection came from a machine on the network
> instead of the actual client, so log files store the wrong IP address
> and server-side code that uses IP address might break.  For a lot of
> users this might not be much of a problem, but for cases where it
> matters, people have de facto standardized on the X-Forwarded-For
> header field to communicate the client IP address to the HTTP server.
> 
> Admins that need their HTTP servers to see the IP address still have
> to update their HTTP server code, but the change to make an HTTP
> server understand X-Forwarded-For is much simpler than the change
> required to make it understand SSL.

s/HTTPS/IPv6 HTTP/
and
s/HTTP/IPv4 HTTP/

and you have a situation many people face now, since many enterprise web 
servers are behind IPv4-only load-balancers or other IPv4-only 
technologies.  So yes, I see what you're talking about.

> Similarly, for DNSCurve, most authoritative DNS servers don't have
> native support for it, so a reasonable solution is a DNSCurve-to-DNS
> proxy server that accepts DNSCurve requests, handles the DNSCurve
> protocol, and forwards the encapsulated request to a plain DNS server.
>  And for similar reasons, we can expect this might stay a popular
> solution even after DNS servers start natively supporting DNSCurve.
> 
> Part of my original email was to basically ask if anyone has an
> equivalent to X-Forwarded-For for DNS yet.
> The other part was that when a DNSCurve-to-DNS proxy server receives a
> DNS response from a backend server, it needs to know what key to
> encrypt the response with.  The direct analog to the HTTPS-to-HTTP
> proxy server is to have the DNSCurve-to-DNS proxy server just maintain
> some extra state in memory to keep track of
> keys/nonce/encoding-format/etc.  However, because having to keep state
> like this is not ideal, it would be nicer to just encode the state in
> a cookie that the proxy server sends along to the backend DNS server.
> Then the forwarder can just rely on the backend server echoing the
> cookie.  (In the HTTPS-to-HTTP case, because the protocols are TCP,
> the server already has to maintain some state in memory, so the extra
> state for SSL isn't as much of a concern.)


That's cool, but it doesn't solve the redundancy/reliability issue. 
Because you asked about how we deal with load-sharing, I assumed you 
were interested in solving that issue.  Even if you have a client 
address and cookie in the packet payload, the packet itself won't be 
received by the failover forwarder if the main forwarder goes down 
before the backend has answered.  That situation is no different than if 
you didn't encode the client address and cookie.  It's a pretty small 
issue, but I thought you might be concerned.

Therefore, the only advantage of doing that is for query logging.  If I 
am in control of both the forwarder and the backend, I have them both 
syslog to the same server and I am done.  If I am not in control of 
both, then it's no different that if someone sets up a caching 
forwarding resolver on my net and forwards to my nameservers.  If I see 
a problem coming from that host, I email the contact for the host and 
tell him/her to determine who is causing the actual problem.


> This would similarly require some (small) additional changes to the
> backend server for support (comparable to EDNS PING).

Oh, I am not going to comment on EDNS-PING.  I am just an ops guy.

> To be compatible with existing DNS servers, we'll need DNSCurve-to-DNS
> proxy servers that keep state in memory anyway.  But for high traffic
> sites where keeping state might be problematic or where the DNS
> servers want to know the client's IP address, the backend protocol
> will need to be extended.  This can either be DNS + some new EDNS
> options or some new (non-backwards-compatible, but simpler) DNS +
> extended header protocol.
> 
> It's no problem to have multiple backend protocols (e.g., how the web
> world has HTTP, SCGI, FCGI, and probably others), but before inventing
> anything new, I was just curious if anyone had already tackled this
> matter.

I see.  So the question was really, has anyone tried any kind of 
stateful load-sharing protocol for DNS beyond anycast or CARP that might 
have already done what you want to do.  My personal answer (which I 
think I explained somewhat in my previous message) is "probably not." 
Because anycast and other simple redundancy mechanisms work so well for 
DNS, there is little need to do anything that replicates state, since 
there is very little state to replicate.  But I'll let others answer if 
they have implemented such a method.

michael