[dns-operations] DNS Traffic Archive Protocol

[Phil Regnauld]

	Hi Bedrich,

	Thanks for the clarification.

	A few comments:

Bedrich Kosata (bedrich.kosata) writes:
> - combine DNS queries and responses together to remove redundancy.

	Does the new format preserve the timestamp for when both query
	and response were originally received ?

> The reason for such optimizations lies within a simple calculation.
> When only one byte is stored for every packet in a traffic of 10,000
> queries per second (qps), it amounts to ~860 MB of data per day and
> ~300 GB per year. Therefore, for CZ.NIC, each wasted byte in the
> format structure means 300 GB of useless data per year.

	Considering a real-world scenarion, do you have a comparison,
	estimated, of what it would require in diskspace to store a whole
	year of data for an auth server for .CZ, using a traditional format
	like ncap/pcap, nmsg, and the new format you propose ?

	You mention 8% of the original pcap file sizes below, that's a factor
	12, which is definitely interesting, and probably makes a good
	argument against just saying "but disk is cheap" - especially the
	improved processing speeds.  What about nmsg ?

> 2/ a library for reading the experimental format written in C which
> is capable of reading data 50x faster than from the pcap file.

	Nice.

> We would also like to implement an export option that would
> reconstruct as much as possible of the original pcap files, so that
> the stored content may be used for example for testing of DNS
> servers by replaying the stored queries.

	A filter/import module for wireshark/tshark would be nice to
	have as well.

> As I wrote at the beginning of this email, we are in an early stage
> of development, but the results so far are very interesting. We
> would be happy for any input on this subject and hope to have some
> working code to show you soon.

	In general, I like the idea, but I can certainly see a benefit to
	having a more general format that could handle other protocols
	as well, without having to start from scratch.

	Look forward to examining the code.

	Cheers,
	Phil