[dns-operations] DNS Traffic Archive Protocol
bedrich.kosata at nic.cz
Thu Dec 2 11:36:38 UTC 2010
this is a more detailed follow-up on the announcement email Ondrej sent
on Tuesday. It should give you a more detailed overview of the DNS
traffic storage format we are currently developing and the ideas and
reasons behind its design. It is in an early stage of development, so
nothing is fixed and any suggestions are welcome.
The main objective of the project is to develop a format that would be
suitable for long term storage of DNS traffic, mainly from high load
servers, such as the TLD authoritative servers, etc.
Because the data throughput on such servers is very high, we strive to
make the format as compact as possible, while maintaining all the
important data. We use the following strategies to reach this goal:
- save only relevant data, remove details of lower network layers (store
all data in the same way regardless of IP version or transport protocol).
- store data as efficiently as possible (use single bits or portions of
a byte where possible, such as for IP version, transport protocol, etc.).
- combine DNS queries and responses together to remove redundancy.
- store repetitive data only when they change (server IP address, server
port and RR class are good candidates for this)
- store only data important for the task at hand, i.e., make certain
data only an optional part of the output. This way each user could
fine-tune the format to suit his needs. A good candidate for this is the
content of responses, which makes for the largest part of the data
(especially with the introduction of DNSSEC).
The reason for such optimizations lies within a simple calculation. When
only one byte is stored for every packet in a traffic of 10,000 queries
per second (qps), it amounts to ~860 MB of data per day and ~300 GB per
year. Therefore, for CZ.NIC, each wasted byte in the format structure
means 300 GB of useless data per year.
This does not mean that really every bit must be accounted for, because
simplicity of the format is also important and generic compression would
probably be used on the resulting data anyway. However, one of the most
important design goals is the space efficiency of the format.
Another important aspect of the format is its simplicity. It should
enable much faster processing in comparison to pcap by removing all the
underlying complexity of the transport layer - such as IP versions,
fragmented packets, TCP sessions, etc. - and by combining queries and
answers into one record.
At present the following was implemented in a prototype:
1/ conversion from pcap format into an experimental format which
conforms to the criteria described above, but which does not allow
storage of response content (only the RETCODE, time and size of the
answer are stored). It is written in C and supports both IPv4 and IPv6,
packet defragmentation in both protocols and TCP session reassembly.
The resulting data take only about 8 % of the original pcap files size.
This is what is currently stored:
- time of query
- client and server IP addresses + IP version
- protocol – TCP, UDP
- port numbers
- RR type, RR class
- OPCODE, RETCODE
- EDNS version + payload size
- response size
- response time
- presence of DNSSEC
- recursion request
- Query x Response x both
- DNS id if a pair is not complete
2/ a library for reading the experimental format written in C which is
capable of reading data 50x faster than from the pcap file.
3/ Python bindings for the above C library
4/ native implementation of the reading library in Python - just for
In future, we plan to implement optional storage of response content and
test several possible ways of implementing further compression in the
format. We also plan to create an extensive set of tools for
manipulating and analyzing data in this format. In future we would also
like to make the program capable of listening directly on a network
interface, rather than parsing pcap files.
We would also like to implement an export option that would reconstruct
as much as possible of the original pcap files, so that the stored
content may be used for example for testing of DNS servers by replaying
the stored queries.
As I wrote at the beginning of this email, we are in an early stage of
development, but the results so far are very interesting. We would be
happy for any input on this subject and hope to have some working code
to show you soon.
With best regards
More information about the dns-operations