[dns-operations] Query merging and/or prefetching [was: Re: Who Ignores TTLs ?]

[Simon Leinen]

Florian Weimer writes:
>> We very often see a massive amount of request from a single address,
>> for a single domain name. Sometimes 200-300 queries within a few
>> hundred milliseconds (with an average delta of 3 milliseconds between
>> queries). The load vaporizes as soon as the resolver receives the
>> first response we send. This is not a DDoS, and normally, this will
>> be hidden in the noise and thunder of the regular load.

> I would expect such behavior for a really popular name if the resolver
> does not collapse queries with an identical question section.

Hey, that makes a lot of sense!

So, which recursive name server implementations *do* perform such
merging/collapsing? (And if so, how? :-)

It sounds like a win to avoid these short-term query bursts, especially
for name servers serving large client populations.  This is how I would
probably approach this (but fortunately I'm not a DNS implementor):

Before issuing a new query, check whether there is an identical
outstanding query.  If so, make the current thread-of-execution wait for
and consume the outstanding query's result instead of issuing the new
query.

But I can see a few devils lurking in the details.  For example:

Should we match the query before or after deciding which server to send
it to? "Before" could be more efficient because it would allow us to
re-use information from failed attempts of the ongoing query.

Should this optimization be avoided when the outstanding query has been
outstanding for too long? I'd probably issue a new query anyway if the
old query has been outstanding for more than a second or so, just
because "things may have changed" and it seems harmless to send an extra
query every second.

...

On the other hand, it may be that most of the potential benefits of this
"query merging" could be achieved by the recursive nameserver
prefetching such "really popular names" before they expire.  Of course
this potentially creates extra traffic, but is simpler to implement;

Olaf mentioned in <A80A7B02-4782-4D80-B654-7ADC4325AE10 at NLnetLabs.nl>
how Unbound does it.  Sounds simple and effective!
-- 
Simon.