[dns-operations] Domains delegated to blackhole consuming all recursive slots

[Ondřej Caletka]

Hello list,

for a few weeks we are seeing that our recursive nameservers are
returning SERVFAILs irregularly. By analysing the logs, it looks like
the default BIND quota of 1000 concurrent ongoing recursions is being hit.

Analysing output of `rndc recursing` shows that there is a lot of
queries like this:

; client 195.113.226.114#50037: 'cxqhupitwhmb.biantai666.cbi1.net'
requesttime 1395667641
; client 195.113.226.114#39096: 'ilydadqjobypmz.biantai666.cbi1.net'
requesttime 1395667641
; client 195.113.226.114#35832: 'kxszsnwtufqbob.biantai666.cbi1.net'
requesttime 1395667641
; client 195.113.226.114#33908: 'cropapebglizol.biantai666.cbi1.net'
requesttime 1395667641
; client 195.113.226.114#34537: 'qfsxyhmzedwlsd.biantai666.cbi1.net'
requesttime 1395667641

According to recent thread: "Sporadic but noticable SERVFAILs in
specific dual stack nodes in an anycast resolving farm", I assume that
this is some kind of C&C communication of a botnet. The problem is that
the authoritative servers responsible for such C&C domain are now
somehow blackholed on IP level (or just choked under the amount of traffic).

By not getting any answer, the query will stay in recursing state for
very long time, eventually filling up the limit of 1000 concurrent
recursions. Increasing the limit is possible but there is a risk of
reaching limits on another level (like number of open file descriptors).

I'm now working this around by defining the zones with longest recursion
times as authoritative with no data. But this have to be checked
manually to be sure no legitimate domain (like in-addr.arpa) would be
accidentally blocked.

There should be better solution. Something like cache for unreachable
nameservers so the non-responding nameserver would be considered dead
for a couple of minutes.

Am I missing something?

Best regards,

Ondřej Caletka,
CESNET

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5563 bytes
Desc: Elektronicky podpis S/MIME
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20140324/7c58b086/attachment.bin>