[dns-operations] BIND performance difference between RHEL 6.4 and FreeBSD 7

[Shawn Zhou]

Our performance tests show that ISC BIND (authoritative only setup) doesn't perform well on RHEL 6.4 in comparison with FreeBSD 7: bind_perf.png
 
   bind_perf.png
Shared with Dropbox  
View on www.dropbox.com Preview by Yahoo  

The "Drop Rate" is the ratio of the amount response never received and the amount of requests were sent. BIND is configured with 24 worker threads and 24 UPD listener, same as the CPU threads we have. BIND process already has '-20' nice priority set on RHEL and '20' on the FreeBSD host.The test hosts running RHEL 6.4 and FreeBSD 7 are identical in term of hardware:
2 x Xeon E5-2430, 24GB DDR3 RAM, 1Gb/s NIC. 

We conducted the tests by having our load generators re-play BIND query logs using our custom scripts and send the queries to the test server at a given rate, say, 200,00 query per seconds. The queries are preloaded into the memory so that's no overhead for our load generators to read queries from disk while sending test queries.

What we've observed that socket receiving queue (Recv-Q from netstat) drained very fast on FreeBSD but got back up pretty fast on RHEL 6 when we ramped up the test traffic.  With net.core.rmem_default set to 40MB, it only helps RHEL to be able to handle 180,000qps before we start to see receive buffer overruns again and drop rate increases linearly.

We are using governor performance on RHEL because default governor ondemand really hurts performance.


vmstat output gathered from RHEL and FreeBSD hosts showed that FreeBSD is scheduling those BIND threads more efficiently than RHEL as the number of running processes is quite stable on FreeBSD and context switches/interrupts ratio is lower and user space CPU utilization is better than RHEL. The below links showed vmstat gathered with 1-second interval during the test run (200k qps) which lasted for about 60 seconds. 

vmstat_from_freebsd.txt

 
   vmstat_from_freebsd.txt
Shared with Dropbox  
View on www.dropbox.com Preview by Yahoo  

vmstat_from_rhel6.txt

 
   vmstat_from_rhel6.txt
Shared with Dropbox  
View on www.dropbox.com Preview by Yahoo  


Anyone has ideas on how to tune the scheduler on RHEL6 to work better for BIND?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20140422/c05509ae/attachment.html>