[dns-operations] peak DNS traffic increases

[Jeroen Massar]

On 2020-03-27 11:55, bert hubert wrote:
[..]
> Several of our ISP customers have informed us they are seeing >25% increases
> in peak resolver DNS traffic, plus remarkable shifts in DNS access patterns. 
> The usual 'waves' are all gone. This increase is far bigger than the
> concurrent increase in bandwidth use.

I was looking[1] at that too in the last few days, and digging a bit what might be actually causing it, but as the patterns seem uniform and nothing sticks out (no weird no domains etc either), it does not look like nothing malicious, just normal user traffic.

E.g. there are a lot of cases where some customer has an older netgear device that still tries time-g.netgear.com, and due to implementation it does that at near linerate since that hostname is long gone. We got a rule for that hostname though, thus customer support get a ticket for it and contact the customer and then they help them resolve the issue, which typically means the link of the customer is not full anymore and their whole experience is much better.

But none of that kind of traffic, all looks normal on our (AS15600) side.

[..]
> PS: Without tooting my own horn too much, a relatively 'no thinking'
> performance increase can typically be had by putting a dnsdist with a small
> cache in front of a setup.

Bert, you and the whole awesome PowerDNS team are not tooting any horn wrongly by having given the world dnsdist, great product, like all from PowerDNS.
Thus: Thanks Bert and team!

See also slide 5/6 of this preso which contains our current over-engineered setup, because we all like to sleep sometimes don't we:
 https://www.swinog.ch/wp-content/uploads/2019/05/Managing-sleep-with-a-resilient-DNS-infrastructure.pdf

Yes, it is the only codepath that is not unique... that is how much I trust the engineering there, which I think says something ;)


And folks, do please all keep safe. This is the time where for me it demonstrates that doing home office is really useful and I hope that corporations around the world realize they can do mostly without offices (except for in-person meetings) which reduces traffic and thus is good for nature. Of course, not for everybody, YMmV.

Greets,
 Jeroen


[1] FTR: For debugging, we dnstap the BIND instance, which thus gives a sample of the traffic, source IPs get anonimized (zero out the subnet of the network), the collected data is deleted directly after analysis and dnstap is stopped. We do also sometimes on single instances, manually, run 'dnstop' to just quickly peek what kind of traffic is happening on the box.