[dns-operations] DNS Traffic Archive Protocol
regnauld at nsrc.org
Thu Dec 2 11:47:24 UTC 2010
Thanks for the clarification.
A few comments:
Bedrich Kosata (bedrich.kosata) writes:
> - combine DNS queries and responses together to remove redundancy.
Does the new format preserve the timestamp for when both query
and response were originally received ?
> The reason for such optimizations lies within a simple calculation.
> When only one byte is stored for every packet in a traffic of 10,000
> queries per second (qps), it amounts to ~860 MB of data per day and
> ~300 GB per year. Therefore, for CZ.NIC, each wasted byte in the
> format structure means 300 GB of useless data per year.
Considering a real-world scenarion, do you have a comparison,
estimated, of what it would require in diskspace to store a whole
year of data for an auth server for .CZ, using a traditional format
like ncap/pcap, nmsg, and the new format you propose ?
You mention 8% of the original pcap file sizes below, that's a factor
12, which is definitely interesting, and probably makes a good
argument against just saying "but disk is cheap" - especially the
improved processing speeds. What about nmsg ?
> 2/ a library for reading the experimental format written in C which
> is capable of reading data 50x faster than from the pcap file.
> We would also like to implement an export option that would
> reconstruct as much as possible of the original pcap files, so that
> the stored content may be used for example for testing of DNS
> servers by replaying the stored queries.
A filter/import module for wireshark/tshark would be nice to
have as well.
> As I wrote at the beginning of this email, we are in an early stage
> of development, but the results so far are very interesting. We
> would be happy for any input on this subject and hope to have some
> working code to show you soon.
In general, I like the idea, but I can certainly see a benefit to
having a more general format that could handle other protocols
as well, without having to start from scratch.
Look forward to examining the code.
More information about the dns-operations