[dns-operations] DNS Traffic Archive Protocol

[Bedrich Kosata]

Dear colleagues,

this is a more detailed follow-up on the announcement email Ondrej sent 
on Tuesday. It should give you a more detailed overview of the DNS 
traffic storage format we are currently developing and the ideas and 
reasons behind its design. It is in an early stage of development, so 
nothing is fixed and any suggestions are welcome.

The main objective of the project is to develop a format that would be 
suitable for long term storage of DNS traffic, mainly from high load 
servers, such as the TLD authoritative servers, etc.
Because the data throughput on such servers is very high, we strive to 
make the format as compact as possible, while maintaining all the 
important data. We use the following strategies to reach this goal:

- save only relevant data, remove details of lower network layers (store 
all data in the same way regardless of IP version or transport protocol).
- store data as efficiently as possible (use single bits or portions of 
a byte where possible, such as for IP version, transport protocol, etc.).
- combine DNS queries and responses together to remove redundancy.
- store repetitive data only when they change (server IP address, server 
port and RR class are good candidates for this)
- store only data important for the task at hand, i.e., make certain 
data only an optional part of the output. This way each user could 
fine-tune the format to suit his needs. A good candidate for this is the 
content of responses, which makes for the largest part of the data 
(especially with the introduction of DNSSEC).

The reason for such optimizations lies within a simple calculation. When 
only one byte is stored for every packet in a traffic of 10,000 queries 
per second (qps), it amounts to ~860 MB of data per day and ~300 GB per 
year. Therefore, for CZ.NIC, each wasted byte in the format structure 
means 300 GB of useless data per year.
This does not mean that really every bit must be accounted for, because 
simplicity of the format is also important and generic compression would 
probably be used on the resulting data anyway. However, one of the most 
important design goals is the space efficiency of the format.

Another important aspect of the format is its simplicity. It should 
enable much faster processing in comparison to pcap by removing all the 
underlying complexity of the transport layer - such as IP versions, 
fragmented packets, TCP sessions, etc. - and by combining queries and 
answers into one record.

At present the following was implemented in a prototype:

1/ conversion from pcap format into an experimental format which 
conforms to the criteria described above, but which does not allow 
storage of response content (only the RETCODE, time and size of the 
answer are stored). It is written in C and supports both IPv4 and IPv6, 
packet defragmentation in both protocols and TCP session reassembly.
The resulting data take only about 8 % of the original pcap files size.
This is what is currently stored:

   - time of query
   - client and server IP addresses + IP version
   - protocol – TCP, UDP
   - port numbers
   - name
   - RR type, RR class
   - OPCODE, RETCODE
   - EDNS version + payload size
   - response size
   - response time
   - presence of DNSSEC
   - recursion request
   - Query x Response x both
   - DNS id if a pair is not complete

2/ a library for reading the experimental format written in C which is 
capable of reading data 50x faster than from the pcap file.

3/ Python bindings for the above C library

4/ native implementation of the reading library in Python - just for 
testing.

In future, we plan to implement optional storage of response content and 
test several possible ways of implementing further compression in the 
format. We also plan to create an extensive set of tools for 
manipulating and analyzing data in this format. In future we would also 
like to make the program capable of listening directly on a network 
interface, rather than parsing pcap files.
We would also like to implement an export option that would reconstruct 
as much as possible of the original pcap files, so that the stored 
content may be used for example for testing of DNS servers by replaying 
the stored queries.

As I wrote at the beginning of this email, we are in an early stage of 
development, but the results so far are very interesting. We would be 
happy for any input on this subject and hope to have some working code 
to show you soon.

With best regards

Beda Kosata
CZ.NIC