[dns-operations] DoS with amplification: yet another funny Unix script

[paul vixie]

second omnibus reply, different thread, same basic topic. i am very glad
that vernon and i spent a year talking about this before spending a year
implementing, testing, and documenting, and that we did it early enough
to inform the inevitable debate about "well since we can't stop ip
spoofing, what will we all do about DNS amplified reflected DDoS?"

On 9/11/2012 7:08 PM, Colm MacCárthaigh wrote:
> Where I work we concentrate on writing very good firewalls. Sometimes
> these rules even have to parse DNS, just as the DNS server must ...
> which causes duplication of work. We do this for several reasons;
> ...

i hope everyone within range of your voice translated "where you work"
to "amazon". amazon's ability to integrate the knowledge of "all dns
domain names for where there are authoritative servers inside this
network" into the configuration of its firewalls makes you guys
specialists and it leans your solutions toward amazon-specific (as well
as "infinitely cool stuff").

but before we hear from other authority dns providers who can do the
same thing, let me ask, please, no. this thread is on the topic of DoS
with amplification and this forum is about dns operations. noting that
someone could achieve great results by outsourcing to some dns authority
provider who does this well in-house using various commercial and
proprietary solutions would be off-topic in the extreme.

as to the purported advantages, i have more to say:

> An in-the-path firewall actually has access to more data than the DNS
> server alone does. For example, it can build up a simple profile of
> expectation values for IP TTLs on a per-source network basis. It can
> use all IP data for that profile; DNS, HTTP, whatever it's seen. Those
> expectation values can then be used to detect and reject spoofed
> packets, in combination with other statistical scores. That's just one
> simple example - there are many more.

on this simple example (which i don't consider representative of the
full spectrum of advantages to this approach, mind you) i'd like to +1
what vernon said -- dns servers can certainly learn the inbound TTL. i'd
also like to note that networks do change their IP TTL's for good
reasons, like changing their transit providers or adding a new peering
connection somewhere. but i already know that amazon's approach allows
for a small number of different IP TTL's per netblock, to allow for
exactly this kind of normal churn. (i am mentioning it only for the
gallery and the archive.)

> ... During real attacks, if a packet makes it to the dns server, the game is already lost.

i hope you meant to qualify that with "here at amazon, ..." because i
know of many deployments where that is absolutely not the case. here i'm
thinking of most root and tld name servers.

> When the primary goal is to mitigate the impact of reflection attacks
> on their potential targets, it makes some sense to filter responses.
> That way you can implement a rate-limit that takes bandwidth into
> account, rather than simply PPS. Definitely agreed about that :)

protecting the server is a tertiary goal for DNS RRL. the most important
thing to protect is the commons, and response rate limiting is going to
have to become the default for all wide area UDP speakers not just DNS,
in the same way that TCP syn flood protection is the default for all IP
stacks nowadays. because other network operators allow spoofed-source IP
to exit their network, we are all at risk of being made accomplice to
untraceable unstoppable too-cheap-to-meter completely devastating DDoS
attacks. that has got to be our first priority.

second, the outbound responses often congest the amplifier's own network
traffic, as well as costing money, and costing in terms of phone calls
to the NOC from victims asking "can you please stop burying me in
responses to questions i'm not asking?" this is a reasonable secondary
priority.

third, it's possible that the server's CPU cycles or kernel context
switching capacity could be overloaded by such an attack, and it's
reasonable to want to protect against this. though it's easier, by using
a combination of local and wide area anycasting, plus upgrading name
server CPU and memory backplane on an annual basis, for most DNS
operators to ensure that their CPU and context switch speed are never
the low hanging bottlenecks.

On 9/11/2012 8:22 PM, Vernon Schryver wrote:
>> From: =?ISO-8859-1?Q?Colm_MacC=E1rthaigh?= <colm at stdlib.net>
>>>  Any firewall rule that doesn't compute DNS responses about as good as a DNS server is simplisitic.
>> With the greatest of respect; that thinking is itself simplistic.
>> Where I work we concentrate on writing very good firewalls. Sometimes
>> these rules even have to parse DNS, just as the DNS server must ...
> That "just as the DCC server must" is false.

vernon almost certainly meant "DNS server" here. but i agree, so i'll
quote more:

> For example, I doubt that those firewalls do enough DNS computing to recognize and limit a stream of responses generated from a single wildcard before the responses
> have been transmitted by the DNS server.  They probably doesn't even recognize pernicious but simple NXDOMAIN cases.  They might but probably don't notice that a stream of responses are approximately identical referrals from authoritative servers or approximately identical recursion from recursive servers.  I think DNS rate limiting must do all of that while not slowing other high volume traffic.

this is "it" in a nutshell. colm's firewalls know the full list of DNS
properties ("zone names") that the servers are authoritative for, and
they know which ones have wildcards. they really can count what i was
calling "vastly similar" responses to distant netblocks. so perhaps when
i say "firewalls can't do this" i should qualify it by saying "most
firewalls, including anything like iptables and ipfw, can't do this."
because if you're as big as amazon you can certainly afford to do
in-path at line-rate almost all of what DNS RRL does in-server. colm,
i'm sorry i implied otherwise.

paul