<div dir="ltr">Rather than address your question, I'm going to argue that the real problem you should fix is the synchronized lookups. Some ideas for smoothing it out a bit:<div><br><div> - Have you considered just turning off NTP? ;)</div>
<div> - Can the servers delay their start time by a random offset (but fixed for each server) from the top of the minute?</div></div><div> - Can the servers cache the result rather than looking it up so often? Then you'll be fine as long a they don't get synchronized to the TTL.</div>
<div><br></div><div>Damian</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 3, 2014 at 2:06 PM, Mark Pettit <span dir="ltr"><<a href="mailto:mark@pettit.org" target="_blank">mark@pettit.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hi, folks.</span><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">I have an issue with BIND cache timeouts, and I was hoping someone else might have some idea how to fix this.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Here's the situation: we have a large number of servers that do a huge number of DNS lookups at the top of every minute. The TTL for the records they're looking up is 3600.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">What we've noticed is that on a host with a recently-restarted copy of BIND, we see huge spikes in DNS latency every 61 minutes. This makes logical sense, given the behavior of the DNS lookups.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">What is more interesting is that on hosts that have been running BIND for a very long time (on the order of months), the spikiness is not visible.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Our speculation is that over time, due to the interaction between the 3600 TTL and the "once every minute" lookup behavior, cache misses become randomly distributed throughout the hour, and don't cause the spiky behavior that is observed initially.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">One of our ideas to resolve this is to randomize the TTLs in the zone files, causing them to expire out of cache at different times, thus forcing more-rapid distribution of cache misses across the hour.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">However, this would involve some massive edits to our zone files, and isn't really ideal.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">What *would* be ideal would be if we could tell BIND to randomly expire some small percentage of cached entries ahead of the actual TTL expiration. This would serve the same purpose as assigning "random" TTLs to the actual records in the zone files.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>Does BIND have a config option like this? Has anyone else ever encountered this issue, and if so, how did you address it?</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Thanks for any advice, and I hope everyone has a fantastic Fourth of July weekend.</div><span class="HOEnZb"><font color="#888888"><div style="font-family:arial,sans-serif;font-size:13px">
<br></div>
<div style="font-family:arial,sans-serif;font-size:13px">Mark Pettit<br></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div></font></span></div>
<br>_______________________________________________<br>
dns-operations mailing list<br>
<a href="mailto:dns-operations@lists.dns-oarc.net">dns-operations@lists.dns-oarc.net</a><br>
<a href="https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs" target="_blank">https://lists.dns-oarc.net/mailman/listinfo/dns-operations<br>
dns-jobs</a> mailing list<br>
<a href="https://lists.dns-oarc.net/mailman/listinfo/dns-jobs" target="_blank">https://lists.dns-oarc.net/mailman/listinfo/dns-jobs</a><br></blockquote></div><br></div>