[dns-operations] DNS server benchmarking sanity check

Tue Aug 16 17:20:22 UTC 2016

> On Aug 16, 2016, at 5:52 AM, Costantino Andrea (Con) <andrea.costantino at h3g.it> wrote:
> 
> Hello guys,
> 
> newbie here. Read a lot, post not that much.
> 
> I would just advice not to use iptables at all when during performance tests.
I second that but if you see the details in Anand Buddhdev's mail you can see he is not tracking the UDP DNS packets, so it should do the trick.

-A PREROUTING -i p1p1 -p udp -m udp --dport 53 -j NOTRACK
-A OUTPUT -o p1p1 -p udp -m udp --sport 53 -j NOTRACK

Thank you.

> 
> I had some recursive DNSes for customers that were running IPTables for protection (small set of rules), and, with decent HW at that time (HP DL380G6) I found that one CPU was almost overwhelmed with kernel stuff during IPTables evaluation (I did non have multiqueue on that HW/SW driver couple for those NICs).
> Actually I had the limit for IPTables that was starting to drop packets way sooner than the BIND limit on that HW, since a single CPU was almost 100% with something in the 30k pkts/s.
> 
> With multiqueue you get a lot more sharing of packets on the CPU cores, but it's not perfect and IPTables will run in multiple core, but probably will burn out faster than the nameserver processes.
> 
> I think you should try to measure some other way. Pkts/s on the switch is usually rather good as raw estimate, if it's a single authoritative server. Without EDNS you will get 1 pctk query vs. 1 pctk answer.
> 
> When you start losing you'll notice that switch port statistics show packet_input < packet_output (so you're not answering to all requests).
> 
> And if you implement correctly (dedicated management channel) it will be a completely out of band channel :)
> 
> 
> Ciao,
> A.
> 
> 
> -----Messaggio originale-----
> Da: dns-operations [mailto:dns-operations-bounces at dns-oarc.net] Per conto di Anand Buddhdev
> Inviato: lunedì 15 agosto 2016 08:47
> A: dns-operations at dns-oarc.net
> Oggetto: Re: [dns-operations] DNS server benchmarking sanity check
> 
> On 15/08/16 02:42, Mark Delany wrote:
> 
> Hello Mark and others,
> 
>> In general, for systems with many cores, I could not find a way to
>> fully utilize the CPUs due to locking and serializing thru the kernel.
>> 
>> Increasing the number of threads to match the cores and even binding
>> to CPUs and using unique SO_REUSEPORT sockets still showed a decline
>> starting at around 8 threads/cores... as best I recall.
> 
> I'm still running my tests, and this should not yet be taken as a conclusion, but of all the name servers I'm testing (bind, knot, nsd, powerdns, yadifa), nsd is winning.
> 
> On my hardware (10 cpus, visible as 20 to the OS), running nsd with 20 worker processes and SO_REUSEPORT, it is able to answer 100% of the queries at 1.2 million q/s, and 97% of the queries at 1.6 million q/s.
> By this point, all 20 cpus are at 100% utilisation, so there's no way I can get anything better, and performance drops with increasing query rates. The only thing to do is to get faster cpus. Also, at this response rate, the outbound bandwidth is about 7.67 Gbit/s. So I could go with a somewhat faster cpu, and probably fill the 10G NIC outbound.
> 
> The other name servers don't use multiple processes, but instead use threads. Of these, knot seems to do the best. Knot 2.3.0 (which has SO_REUSEPORT), can keep up with about 1 million q/s, at which point cpu usage is 100%, and so performance drops after this point.
> 
> I've spent the weekend reading up on the intricacies of networking in linux, so I now have a much better understanding of how things work, especially with 10G NICs. However, I'm not going to tweak any NIC settings yet, because I don't want to mess with what I don't know.
> 
> Once I have tortured the various DNS software packages as much as I can, I will pick one and then keep testing just with it, but then adjust network settings to see if I can improve anything. However, I think that once cpu utilisation is at 100%, there's probably not much more I can do to the system, so my suspicion is that the default settings that my OS and hardware has are good enough for this purpose.
> 
> Regards,
> Anand Buddhdev
> RIPE NCC
> _______________________________________________
> dns-operations mailing list
> dns-operations at lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> dns-operations mailing list
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> 
> ________________________________
> 
> CONFIDENTIAL: This E-mail and any attachment are confidential and may contain reserved information. If you are not one of the named recipients, please notify the sender immediately. Moreover, you should not disclose the contents to any other person, or should the information contained be used for any purpose or stored or copied in any form.
> 
> _______________________________________________
> dns-operations mailing list
> dns-operations at lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> dns-operations mailing list
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Regards,
Kabindra Shrestha

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20160816/fd534b9a/attachment.sig>