[dns-operations] Forcing BIND to randomly expire records from cache ahead of time

[Mark Pettit]

Thanks, those are some rather interesting (and one funny) ideas.

On Thursday, July 3, 2014, Damian Menscher <damian at google.com> wrote:

> Rather than address your question, I'm going to argue that the real
> problem you should fix is the synchronized lookups.  Some ideas for
> smoothing it out a bit:
>
>   - Have you considered just turning off NTP?  ;)
>   - Can the servers delay their start time by a random offset (but fixed
> for each server) from the top of the minute?
>   - Can the servers cache the result rather than looking it up so often?
>  Then you'll be fine as long a they don't get synchronized to the TTL.
>
> Damian
>
>
> On Thu, Jul 3, 2014 at 2:06 PM, Mark Pettit <mark at pettit.org
> <javascript:_e(%7B%7D,'cvml','mark at pettit.org');>> wrote:
>
>> Hi, folks.
>>
>> I have an issue with BIND cache timeouts, and I was hoping someone else
>> might have some idea how to fix this.
>>
>> Here's the situation: we have a large number of servers that do a huge
>> number of DNS lookups at the top of every minute. The TTL for the records
>> they're looking up is 3600.
>>
>> What we've noticed is that on a host with a recently-restarted copy of
>> BIND, we see huge spikes in DNS latency every 61 minutes. This makes
>> logical sense, given the behavior of the DNS lookups.
>>
>> What is more interesting is that on hosts that have been running BIND for
>> a very long time (on the order of months), the spikiness is not visible.
>>
>> Our speculation is that over time, due to the interaction between the
>> 3600 TTL and the "once every minute" lookup behavior, cache misses become
>> randomly distributed throughout the hour, and don't cause the spiky
>> behavior that is observed initially.
>>
>> One of our ideas to resolve this is to randomize the TTLs in the zone
>> files, causing them to expire out of cache at different times, thus forcing
>> more-rapid distribution of cache misses across the hour.
>>
>> However, this would involve some massive edits to our zone files, and
>> isn't really ideal.
>>
>> What *would* be ideal would be if we could tell BIND to randomly expire
>> some small percentage of cached entries ahead of the actual TTL expiration.
>> This would serve the same purpose as assigning "random" TTLs to the actual
>> records in the zone files.
>>
>> Does BIND have a config option like this? Has anyone else ever
>> encountered this issue, and if so, how did you address it?
>>
>> Thanks for any advice, and I hope everyone has a fantastic Fourth of July
>> weekend.
>>
>> Mark Pettit
>>
>>
>> _______________________________________________
>> dns-operations mailing list
>> dns-operations at lists.dns-oarc.net
>> <javascript:_e(%7B%7D,'cvml','dns-operations at lists.dns-oarc.net');>
>> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>> dns-jobs
>> <https://lists.dns-oarc.net/mailman/listinfo/dns-operationsdns-jobs>
>> mailing list
>> https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20140703/c7ec6c8d/attachment.html>