[dns-operations] Interesting messages in our logs

[Lyle Giese]

Just to flush out the details here, in case anyone is wondering.  We 
have a small number of domains that are DNSSEC signed, but those under 
attack are not signed.

In the past two days, I am seeing RRL kicking in heavily for queries for 
host names or subdomains in the form:

<variable>.example.com

 From IPv4 and IPv6 Google ip addresses.  At the same time, but I see a 
few of the 'no more TCP clients: quota reached' messages. Again, after 
the RRL limit kicking in, rolling over to TCP is expected.

I am seeing the 'attack' first against one domain for a period of only a 
few(less than 5) minutes.  And then the next day, another flurry of 
activity against another domain lasting about 4 minutes.

I am not sure what the goal is of the attackers yet.  But in bouncing 
the queries through Google does a pretty good job of hiding their 
identity from me.

I have logs with time stamps I can provide to Google if they are 
interested.

In this the amount of queries against one domain did catch my attention 
in reviewing logs, but the roll over to TCP causing the no more TCP 
clients: quota reached' made it more interesting to follow up on.  Esp 
to hear how the other side reacts to the TCP errors.  I guess Google 
would have to answer that question.

Lyle Giese
LCR Computer Services, Inc.

On 11/01/14 22:45, Paul Vixie wrote:
>
>
>> Lyle Giese <mailto:lyle at lcrcomputer.net>
>> Saturday, November 01, 2014 5:37 PM
>>
>> Now on the TCP side, I am seeing 'no more TCP clients: quota 
>> reached'.  I am still not clear on what the other side is told or 
>> what it thinks is going on at this point.  My thought process/logic 
>> is that a TCP Reset should be handled above the application layer and 
>> that an application could not create/generate that.  Of course, I can 
>> be wrong on this point and would welcome corrections.
>>
>> Now that I have written this, I suspose it's within reason that an 
>> application can tell the upper layers, I am to busy to handle that 
>> request, drop the TCP session and those layers use the RST function 
>> to accomplish that.
>
> yes. in general, when you receive a tcp connection that puts you over 
> your application layer quota, you close() it, possibly shutdown()'ing 
> it first. this will send a FIN immediately, and send an RST if 
> non-meta data keeps coming.
>>
>> But it sounds more and more like an DoS attempt against my 
>> authoritative servers by exhausting TCP resources.  And they were 
>> utilizing Google's servers to assist in that process. Not at all sure 
>> what the heck they would be trying to accomplish hitting up my servers.
>
> my intuition is that this is not a TCP attack per se. if you look at 
> the QNAME's google is sending you, they will probably be pseudo-random 
> subdomains of the zones you are authoritative for. and if your zone is 
> signed with DNSSEC, and has no wildcard, then you may find that 
> google's EDNS buffer size is not large enough to contain the responses 
> you are sending, so, you're sending TC=1 for other reasons, and google 
> is coming back with TCP a lot more often than RRL could possibly cause 
> them to do.
>
> in other words TCP is a side show, the real thrust of this attack is 
> likely elsewhere in your mix.
>
>> And I would assume that it would take a lot more to accomplish a DoS 
>> attack against Google's servers this way!  But it's an excellent way 
>> for the attackers to hide their identity from me as I see only 
>> Google's IP addresses in the requests.
>
> google has been doing their own version of RRL for a long time, longer 
> than DNS RRL has been specified, and they do a pretty good job. 
> however, pseudo-random subdomains are the attack du jour, and google's 
> rate limiting machinery might not be coping with it well. even so, it 
> would take a botnet-sized client population hammering google public 
> dns with pseudo-random subdomains of your zones to get google to 
> actually launch that number of simultaneous queries toward your 
> authority servers.
>
> i recommend analyzing your QNAMEs so that you can isolate the target.
>
> google's dns people are certainly monitoring this thread and it's my 
> bet that they'll be in touch with you privately.
>
> -- 
> Paul Vixie

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20141102/f8796a07/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20141102/f8796a07/attachment.jpg>