[dns-operations] Stateless-TCP performance testing tool [Was: DNS Performance Test Over TCP]

[Florian Maury]

On Wed, May 22, 2013 at 02:12:19PM -0400, Jared Mauch wrote:
> It's not *that* hard to write a socket based application to do these
> types of things, esp for a lab environment where you're not seeing
> weird variable latency and RTT values for your TCP session.  You're
> just testing the two TCP stacks and the application layer
> interaction with them.
>
> - Jared


Hi,

I gave a thought to Francis Dupont's presentation at the DNS-OARC
Workshops, in Dublin, on the subject.  I fear that this might be more
complicated than one thinks.

As you stated correctly "You're [...] testing the two TCP stacks", and
I think this might be a problem.
If I'm correct, Francis' tool (perftcpdns) was run from a single
computer hammering the tested DNS server. Specific kernel
customizations were required on both computers to achieve the rate of
30 kq/s.

I have several concerns about this methodology:

  * The client might be the bottleneck

  * The same client, with always the same source IP address is
    hammering the server.  In real-world operations, this only happens
    when the system is under-attack and this attack is easily thwarted
    by setting up a maximum number of concurrent TCP connections per
    source IP. This statement is of course only valid if you enable
    "TCP pipelining" instead of opening a new TCP connection per
    query. If implementations are not already doing this, I think this
    is the way to go.


A better performance testing methodology came to my mind, after I
recalled of this paper:
http://caia.swin.edu.au/reports/101022A/CAIA-TR-101022A.pdf "Improving
DNS performance using ``Stateless'' TCP in FreeBSD 9". This paper is
about some kind of DNS proxy that implements stateless-TCP on the
server-side.  Although the idea does not seduce me, and it raises a
lot of security concerns, the concept can be reused as follow in your
particular context:

  * The client IP is configured on the server as its gateway (to force
   it to send every packet to the client)

  * The client is listening in promiscuous mode

  * The client does NOT forward received packets and does not reject
    them either

  * The server is listening with the tested software on port 53

  * The client spoofs a random IP and sends a raw TCP packet to the
    server port 53 with the SYN flag and a meaningful[*] SN to
    initiate a TCP connection.
    No state is kept, but a counter is incremented to take note of the
    initiated connection. The counter is used to make statistics about
    the number of initiated handshakes.

  * If the client receives a SYN/ACK packet, it sends the associated
    ACK packet, with raw sockets. Source address and client's SN are
    retrieved from the SYN/ACK packet. The client also sends a spoofed
    query and increments a counter to take note of the number of
    successful handshakes.

  * If the client receives a PSH/ACK packet, it increments a counter
    to take note of the number of replied queries. It uses the ACK
    value sent by the server to send a FIN packet. Answer to the query
    must stay small to avoid multiple PSH/ACK packets.

  * If the client receives a FIN/ACK packet, it sends an ACK packet.

  * All other packets are ignored.

[*] The SN is not chosen at random but is the number of milliseconds
since the performance test started. This allows us to compute, upon
receival of the FIN/ACK packet, how fast the whole transaction was.

Using a stateless client allows us to get rid of the client's TCP
stack limitation, and only react to the server answers. Counters
allows us to keep trace of the actual number of initiated TCP
handshakes, successful handshakes, answered queries and the duration
of the query process.  The added advantage is that this tool would
only work on controlled local networks and it should not cause more
damage than what SYN floods attack tools can already do.

What do you think of such a testing strategy? Are my concerns not
justified?  Unfortunately, I do not have the time to implement such a
tool, but if somebody picks up on the idea... let me know :o)

Cheers,

Florian Maury