[dns-operations] DNS server benchmarking sanity check

[Marek Vavruša]

Anand,

first of all - thanks for doing this, it's a lot of repetitive work. With
benchmarking, there are many opinions and anecdotes, but not many are
backed with data. Your test setup looks sane to me - presuming the question
you're trying to answer "what sustained query rate can daemon X achieve in
our environment"? If you're interested in gauging your infrastructure
against attacks, you probably should use attack traffic.

There are many variables involved (hardware, drivers, os), and a lot of
things to tune. I think it's a good thing you reduced the benchmarking
scope to your system first. It's very useful to have baseline results first
(please do share them), instead of doing the "drunken man approach" tuning.
There are now many good tools - ftrace perf (as Florian mentioned),
systemtap, bcc ... to help you find out where the time is spent when you
get to actual tuning, no need for guessing.

Good luck, it'd be awesome if this somehow turned into a repeatable test
environment.

Marek


On 13 August 2016 at 03:43, Anand Buddhdev <anandb at ripe.net> wrote:

> Hi folks,
>
> I'm doing some DNS server benchmarking, and I think I've got a lab setup
> that looks right, but I'd like to run my config past you guys, the
> experts, to make sure I'm not overlooking anything obvious.
>
> I have 3 Dell R320 servers, each with a single Intel Xeon E5-2470
> 10-core processor at 2.40 GHz. The OS therefore sees 20 processors. The
> server has the usual on-board dual-port 1 Gbit ethernet ports, but I'm
> not using these for the testing. These are merely for logging in and OS
> management. The servers also have dual-port 10 GB NICs, with an intel
> chipset, and using the Linux "ixgbe" driver.
>
> One 10 GB NIC in each server is connected to a switch, so that the 3
> NICs are in the same LAN.
>
> The OS is an up-to-date CentOS 7 installation. The roles are thus:
>
> 1. server 1 is the source of packets. On it, I have compiled the latest
> tcpreplay with the --enable-quick-tx option, to allow it to play back
> packets at a high rate. I have a pcap file containing 6,882,162 real DNS
> queries (UDP-only) which I can send out to a DNS server.
>
> 2. server 2 is the DNS server. I have so far installed BIND 9.10.4-P1,
> NSD 4.1.10, Knot 2.3.0 and Yadifa 2.1.6 on it. They are all configured
> as root name servers, with the root, arpa and root-servers.net zones on
> them.
>
> I have done minimal tuning of this server. I have set these 4 sysctl
> settings:
>
> net.core.rmem_default=4194304
> net.core.rmem_max=4194304
> net.core.wmem_default=4194304
> net.core.wmem_max=4194304
>
> Additionally, I've configured the following iptables rules on it:
>
> -A PREROUTING -i p1p1 -p udp -m udp --dport 53 -j NOTRACK
> -A OUTPUT -o p1p1 -p udp -m udp --sport 53 -j NOTRACK
>
> With these rules, I ensure that no connection-tracking is done, and I
> can count the number of incoming packets (queries) and the number of
> outgoing packets (responses) easily.
>
> Finally, there is a default route installed on the server to send all
> the responses out to server 3.
>
> 3. Server 3 is just the sink. It received DNS responses, and there's a
> single iptables rules to drop them all (and count them):
>
> -A PREROUTING -s 193.0.14.129/32 -p udp -m udp --sport 53 -j DROP
>
>
> With this setup, I can emit a DNS query from the source, and it goes to
> the DNS server, which emits a response, and the response goes to the
> sink where it is discarded. On the source, tcpreplay is able to achieve
> a maximum packet rate of just over 3 million, so I can blast over 3
> million q/s towards the DNS server. At that rate, none of the DNS
> servers can keep up of course. However, I'm playing back packets are
> various rates to find the breaking points of the various servers.
>
> Now I know that there is a LOT more tweaking and tuning that can be
> done. For example the 10 GB NICs have lots of tunables. In addition,
> there are things like CPU affinity, IRQ affinity, etc that could be
> tweaked. But let's assume that we don't want to get into that level of
> tweaking and tuning, and just want to use the defaults of the OS with
> minimal adjustment, is this setup okay? Are there any other Linux kernel
> adjustments, especially on the DNS server, that should be adjusted for
> high packet rates?
>
> Regards,
> Anand Buddhdev
> RIPE NCC
> _______________________________________________
> dns-operations mailing list
> dns-operations at lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> dns-operations mailing list
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20160815/fef0dde6/attachment.html>