<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I'm puzzled by the explanation of Socket Overloading in<br> <a href="https://sites.google.com/site/hayashulman/files/NIC-derandomisation.pdf" target="_blank">https://sites.google.com/site/hayashulman/files/NIC-derandomisation.pdf<br>
</a>I understand it to say that Linux on a 3 GHz CPU receiving 25,000<br>packets/second (500 bytes @ 100 Mbit/sec) spends so much time in<br>interrupt code that low level packet buffers overflow.<br></blockquote><div><br>
</div><div><br></div><div>Just to clarify, the attacker ran (two to three sync-ed hosts, and the burst was split among those hosts).</div><div>The packets rate is not the only factor, and actually burst concentration is much more significant. Specifically, when the packets in the burst have no (or almost no) interpacket delay, the impact is different; e.g., when running the same evaluation with a single attacking host (even on same LAN) no loss was incurred - even if the attacker transmitted constantly, since both the attacking host, and the (store-and-foward) switch that the attacker's host was connected to, also introduced delays between packets (due to their own interrupts and delays), thus `spreading` the burst and reducing its impact.</div>
<div><br></div><div>I will be happy to have your thoughts on this additional piece of info, i.e., the significance of burst volume concentration (=no, or low, interpacket delay in arriving burst), that may have been not clear from the writeup in the paper (I will clarify this in the paper too - thanks).<br>
</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
That puzzles me for reasons that might be summarized by considering<br>my claim of 20 years ago that ttcp ran at wirespeed over FDDI with<br>only 40-60% of a 100 MHz CPU.<br><a href="https://groups.google.com/forum/#!topic/comp.sys.sgi.hardware/S0ZFRpGMPWA" target="_blank">https://groups.google.com/forum/#!topic/comp.sys.sgi.hardware/S0ZFRpGMPWA<br>
</a><a href="https://www.google.com/search?q=ttcp" target="_blank">https://www.google.com/search?q=ttcp<br></a>
Those tests used a 4 KByte MTU and so about 3K pps instead of 25K pps.<br>The FDDI firmware and driver avoided all interrupts when running at<br>speed, but I think even cheap modern PCI Ethernet cards have "interrupt<br>
bursting." Reasonable network hardware interrupts the host only when<br>the input queue goes from empty to not empty or the output queue goes<br>below perhaps half full full, and then only interupts after a delay<br>
equal to perhaps half a minimum sized packet on the medium. I wouldn't<br>expect cheap PCI cards to be that reasonable, or have hacks such as<br>ring buffer with prime number lengths to avoid other interrupts.<br>Still, ...<br>
IRIX did what I called "page flipping" and what most call "zero copy I/O"<br>for user/kernel-space copying, but modern CPUs are or can be screaming<br>monsters while copying bytes which should reduce that advantage. It<br>
would be irrelevant for packets dropped in the driver, but not if the<br>bottleneck is in user space such as overloaded DNS server.<br>
That old ttcp number was for TCP instead of UDP, which would be an<br>advantage for modern Linux.<br>
So I would have guessed, without having looked at Linux network<br>code for many years, that even Linux should be using less than 20%<br>of a 3 GHz CPU doing not only interrupts but all of UDP/IP.<br></blockquote><div><br>
</div><div>Thanks for this input, and for the reference.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
100MHz/3GHz * 60% * 25000 pps /3000 pps = 17%<br>
Could the packet losses have been due to the system trying to send<br>lots of ICMP Port-Unreachables? I have the confused impression that<br>Socket Overloading can involve flooding unrelated ports.<br></blockquote><div>
<br></div><div><br></div><div>But, why would ICMP errors cause loss? </div><div>Inbound packets have higher priority over outbound packets.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
How was it confirmed that kernel interrupt handling was the cause<br>of the packet losses instead of the application (DNS server) getting<br>swamped and forcing the kernel to drop packets instead of putting<br>them into the application socket buffer? Were giant application<br>
socket buffers tried, perhaps with the Linux SO_RCVBUFFORCE?<br>(probably a 30 second change for BIND)<br></blockquote><div><br></div><div><br></div><div>This a good question. So, this evaluation is based on the following observation: when flooding closed ports, or other ports (not the ones on which the resolver expects to receive the response) - no loss was incurred, but all connections experience an additional latency; alternately, when flooding the correct port - the response was lost, and the resolver would retransmit the request after a timeout. </div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
25K qps is not a big queryperf number, which is another reason why I<br>don't understand how only 25K UDP qps could swamp a Linux kernel. Just<br>now the loopback smoke-test for RPZ for BIND 9.9.4 with the rpz2 patch<br>
reported "24940 qps without RPZ" on a 2-core 2.4 GHz CPU running FreeBSD 9.0.<br>
What about the claims of Gbit/sec transfer speeds with Linux?<br><a href="https://www.google.com/search?q=linux+gigabit+ethernet+speed" target="_blank">https://www.google.com/search?q=linux+gigabit+ethernet+speed<br></a>
I'm not questioning the reported measurements; they are what they are.<br>However, if they were due to application overload instead of interrupt<br>processing, then there might be defenses such as giant socket buffers.</blockquote>
<div><br></div><div>I just want to clarify, that I appreciate your questions/comments, and questioning the results is of course an important contribution to the research. Maybe the writeup requires clarification, e.g., I will check if the text clearly explains the setup and evaluation, and maybe the evaluation results were misinterpreted, I do not rule that out, I could have missed out on something; either way, I will need to check both possible options. </div>
<div>So, I really appreciate the questions and data that you provided. </div><div><br></div><div>I used the default buffers in OS and resolver. So, you think that it could be that the loss was on the application layer?... </div>
<div><br></div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I am not at liberty to disclose location or vendor, but I'm aware of linux boxes handling 20k PPS mixed UDP/TCP at an average 2% CPU. They aren't even modern boxes although a bit newer than the dual core that Vernon mentions below. In short, I agree completely with everything Vernon said here. I suspect outdated information or some other factor was involved.</blockquote>
<div><br></div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Just to reinforce Vernon and Jo's points, we have DNS servers running<br>
Linux at ARIN pushing 25~30k packets per second. Overall CPU<br>utilization (across all cores) is under 10%. Interrupt rates tend to be<br>around 15~18k per second.</blockquote><pre><br></pre></div>
<div>Thanks for your input. One of the main factors of the attack is `burst concentration`. This is specifically impacted by the inter-packet delays of inbound traffic, and the amortised rate is not the factor. In particular, even when sending the burst at a higher rate, from a single host, the impact was much less significant, e.g., even no loss; the cause for this are the delays added by the interrupts of the sending system, and the routers enroute (that further add their own interrupts). Thus, the impact of the resulting volume is negligible, and does not result in packets' loss. But, if the volume is split between a number of synced hosts, the impact is significantly enhanced, and typically results in packets' loss. What are your thoughts on this? Does this clarify the results?</div>
</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Oct 23, 2013 at 10:56 PM, Haya Shulman <span dir="ltr"><<a href="mailto:haya.shulman@gmail.com" target="_blank">haya.shulman@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I see I'm stupid for not seeing that in the first message. I did search<br>for 'http' but somehow didn't see the URL. But why not simply repeat<br>the URL for people like me? Why not the URL of the paper at the<br>
beginning instead of a list of papers?<br><a href="https://sites.google.com/site/hayashulman/files/NIC-derandomisation.pdf" target="_blank">https://sites.google.com/site/hayashulman/files/NIC-derandomisation.pdf</a></blockquote>
<div><br>
</div></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">I did not realise that this was the problem, I thought that for some reason you could not download from my site, indeed, using the url would have been more convenient, sorry.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
By searching for "DNSSEC" with my PDF viewer, I found what I consider<br>too few references to the effectiveness of DNSSEC against the attacks.<br>There is nothing about DNSSEC in the abstract, a list of DNSSEC problems<br>
early, and a DNSSEC recommendation in the conclusion that reads to me<br>like a concession to a referee. Others will disagree.</blockquote><div><br></div><div><br></div><div>Ok, thanks for this comment, please clarify which paper you are referring to, and I will check if appropriate references could be added. </div>
<div><br></div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> - forwarding to third party resolvers.<br>
<blockquote> I agree so strongly that feels like a straw man. I think<br> forwarding to third pary resolvers is an intolerable and<br> unnecessary privacy and security hole. Others disagree.<br> - other mistakes<br>
that I think are even worse than forwarders.<br> - DNSSEC<br> Perhaps that will be denied, but I challenge others to read those<br> papers with their litanies of DNSSEC issues and get an impression<br> of DNSSEC other than "sow's ear sold as silk." That was right<br>
for DNSSEC in the past. Maybe it will be right forever. I hope<br> not, but only years will tell. As far as I can tell from a quick<br> reading, the DNSSEC issues are valid, but are sometimes backward<br> looking, perhaps due to publication delays. For example, default<br>
verifying now in server software and verifying by resolvers such<br> as 8.8.8.8 should help the verifying situation.<br><br></blockquote></blockquote><div><br></div><div>Agreed and noted, thank you. </div><div><br>
</div><div>p.s. Can you please cc me when sending responses related to me? Thank you in advance!</div><div><br></div><div>--</div><div><span style="font-size:14.545454025268555px;font-family:Calibri,sans-serif">Best Regards,</span></div>
<div>
<div><span style="font-size:14.545454025268555px;font-family:Calibri,sans-serif">Haya Shulman</span></div><div><span style="font-size:14.545454025268555px;font-family:Calibri,sans-serif">Technische Universität Darmstadt</span></div>
</div></div></div><div><div class="gmail_extra"><div dir="ltr"><p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">FB Informatik/EC SPRIDE</span></font></p>
<p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">Mornewegstr. 30</span></font></p><p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">64293 Darmstadt</span></font></p>
<p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">Tel. <a href="tel:%2B49%206151%2016-75540" value="+4961511675540" target="_blank">+49 6151 16-75540</a></span></font></p>
<p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px"><a href="http://www.ec-spride.de" target="_blank">www.ec-spride.de</a></span></font></p>
</div>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">Best Regards,</span></font></p>
<p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">Haya Shulman</span></font></p><p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">Technische Universität Darmstadt</span></font></p>
<p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">FB Informatik/EC SPRIDE</span></font></p><p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">Mornewegstr. 30</span></font></p>
<p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">64293 Darmstadt</span></font></p><p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px">Tel. <a href="tel:%2B49%206151%2016-75540" value="+4961511675540" target="_blank">+49 6151 16-75540</a></span></font></p>
<p style="margin:0cm 0cm 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.545454025268555px"><a href="http://www.ec-spride.de" target="_blank">www.ec-spride.de</a></span></font></p></div>
</div></div>