[dns-operations] Domains delegated to blackhole consuming allrecursive slots

[Stephan Lagerholm]

Hi Ondrej,

I don't have a solution for your Bind environment but I do have some more information. You are experiencing what we internally named the "Chinese water torture attack". It is not botnet C&C it is an attack on the authoritative servers using your recursive DNS. 

The attacker is using open resolvers around the world including in your network and they are sending domains that are guaranteed cache misses by appending 1-16 chars of junk in front of the domain they are attacking. You can verify that the source IP is an open resolver:
Dig @195.113.226.114 www.google.com +short 

The open resolver is probably some low end router / DSL modem that is configured with your DNS as a forwarder. So all requests end up in your recursive DNS. As you say, it gets really nasty once the Authoritative servers gets knocked over as the recursive DNS retries multiple times. The queries gets stuck and eventually resources gets depleted. 

Ultimately, you need to close access to the open resolvers in your network. Short term increasing various settings to not get overwhelmed is your only option. Let me know if I can be of any further assistance

Thanks /Stephan







> -----Original Message-----
> From: dns-operations-bounces at lists.dns-oarc.net [mailto:dns-operations-
> bounces at lists.dns-oarc.net] On Behalf Of Ondrej Caletka
> Sent: Monday, March 24, 2014 10:46 AM
> To: dns-operations at mail.dns-oarc.net
> Subject: [dns-operations] Domains delegated to blackhole consuming
> allrecursive slots
> 
> Hello list,
> 
> for a few weeks we are seeing that our recursive nameservers are
> returning SERVFAILs irregularly. By analysing the logs, it looks like
> the default BIND quota of 1000 concurrent ongoing recursions is being
> hit.
> 
> Analysing output of `rndc recursing` shows that there is a lot of
> queries like this:
> 
> ; client 195.113.226.114#50037: 'cxqhupitwhmb.biantai666.cbi1.net'
> requesttime 1395667641
> ; client 195.113.226.114#39096: 'ilydadqjobypmz.biantai666.cbi1.net'
> requesttime 1395667641
> ; client 195.113.226.114#35832: 'kxszsnwtufqbob.biantai666.cbi1.net'
> requesttime 1395667641
> ; client 195.113.226.114#33908: 'cropapebglizol.biantai666.cbi1.net'
> requesttime 1395667641
> ; client 195.113.226.114#34537: 'qfsxyhmzedwlsd.biantai666.cbi1.net'
> requesttime 1395667641
> 
> According to recent thread: "Sporadic but noticable SERVFAILs in
> specific dual stack nodes in an anycast resolving farm", I assume that
> this is some kind of C&C communication of a botnet. The problem is that
> the authoritative servers responsible for such C&C domain are now
> somehow blackholed on IP level (or just choked under the amount of
> traffic).
> 
> By not getting any answer, the query will stay in recursing state for
> very long time, eventually filling up the limit of 1000 concurrent
> recursions. Increasing the limit is possible but there is a risk of
> reaching limits on another level (like number of open file
> descriptors).
> 
> I'm now working this around by defining the zones with longest
> recursion times as authoritative with no data. But this have to be
> checked manually to be sure no legitimate domain (like in-addr.arpa)
> would be accidentally blocked.
> 
> There should be better solution. Something like cache for unreachable
> nameservers so the non-responding nameserver would be considered dead
> for a couple of minutes.
> 
> Am I missing something?
> 
> Best regards,
> 
> Ondřej Caletka,
> CESNET
> 
> 
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2014.0.4336 / Virus Database: 3722/7229 - Release Date:
> 03/21/14