[dns-operations] DNS server benchmarking sanity check

[Anand Buddhdev]

On 15/08/16 02:42, Mark Delany wrote:

Hello Mark and others,

> In general, for systems with many cores, I could not find a way to
> fully utilize the CPUs due to locking and serializing thru the kernel.
> 
> Increasing the number of threads to match the cores and even binding
> to CPUs and using unique SO_REUSEPORT sockets still showed a decline
> starting at around 8 threads/cores... as best I recall.

I'm still running my tests, and this should not yet be taken as a
conclusion, but of all the name servers I'm testing (bind, knot, nsd,
powerdns, yadifa), nsd is winning.

On my hardware (10 cpus, visible as 20 to the OS), running nsd with 20
worker processes and SO_REUSEPORT, it is able to answer 100% of the
queries at 1.2 million q/s, and 97% of the queries at 1.6 million q/s.
By this point, all 20 cpus are at 100% utilisation, so there's no way I
can get anything better, and performance drops with increasing query
rates. The only thing to do is to get faster cpus. Also, at this
response rate, the outbound bandwidth is about 7.67 Gbit/s. So I could
go with a somewhat faster cpu, and probably fill the 10G NIC outbound.

The other name servers don't use multiple processes, but instead use
threads. Of these, knot seems to do the best. Knot 2.3.0 (which has
SO_REUSEPORT), can keep up with about 1 million q/s, at which point cpu
usage is 100%, and so performance drops after this point.

I've spent the weekend reading up on the intricacies of networking in
linux, so I now have a much better understanding of how things work,
especially with 10G NICs. However, I'm not going to tweak any NIC
settings yet, because I don't want to mess with what I don't know.

Once I have tortured the various DNS software packages as much as I can,
I will pick one and then keep testing just with it, but then adjust
network settings to see if I can improve anything. However, I think that
once cpu utilisation is at 100%, there's probably not much more I can do
to the system, so my suspicion is that the default settings that my OS
and hardware has are good enough for this purpose.

Regards,
Anand Buddhdev
RIPE NCC