[dns-operations] compressing DNS traffic data

[Ondřej Surý]

Jim,

On 8.12.2010 11:36, Jim Reid wrote:
> On 8 Dec 2010, at 09:16, Stephane Bortzmeyer wrote:
>
>> "RR class" unlike what I wrote earlier, I believe you can safely
>> save two bytes here :-)
>
> I fail to understand this angels-on-pinhead debate when storage costs
>  less than a tenth of a cent per gigabyte in mainstream retail
> outlets.
>
> The cost of everyone's time discussing what to delete or what/how to
>  compress must be orders of magnitude more than a few hundred 1 TB
> disks. Assume we have 100-200 engineers here paid rather more than
> the minimum wage, each or whom spends an hour or two reading or
> posting on this thread about archiving DNS traffic. Then do the
> arithmetic.
>
> Why not just agree to store everything in wire format (maybe with
> added timestamps if sub-millisecond precision is available) and be
> done with it?

I found this logic quite flawed.  Just because the units of something
are cheap it doesn't mean that:

1. cost per unit are the same when you start piling them up.  It's not
just cost of manhours needed to maintain the storage, it's also the
electricity, cooling, cost per rack space, interconnections.

2. we need to waste it even if it's cheap.  It's the same flawed logic
why the US cars had such low mpg ratio - because gas was cheap there was
no "need for efficency".  If the "compression" ratio is 1:5 or 1:10
compared to raw packets, it will pay of now and in the future.

On 8.12.2010 05:36, Robert Edmonds wrote:
> Beda Kosata wrote:
>> Are we the still talking about the original dnsqr schema or is
>> this something new, similar to what Paul proposed? To me it seems
>> that with so many adjustments, you have just laid out plans for a
>> completely new schema.
>
> i'm talking about modifying the existing dnsqr schema to be more
> flexible and accomodate applications like this, not the creation of
> a new schema.  what i've described doesn't exist (yet) but if it's
> the right approach i'll be happy to implement it.

Robert, I have a feeling that we are steering towards "one size fits 
all" and I don't think we can fit all requirements in one big protocal. 
  Obviously your requirements are different than ours (and that's not 
really a bad thing) and I don't mind developing different (not 
necessarily competing) protocols for different needs.

 > i think this is veering off-topic for dns-ops.  mail followups to
 > nmsg-dev at lists.isc.org, please.

I think we have discussed it enough for now and we quite like Paul's 
proposal, so we might go that way.  We also have a mailing list for our 
dnsflow at dnsflow at lists.nic.cz, so everybody is welcome to join, but 
we'll keep you posted (here and/or at nmsg-dev - depends on the topic) 
after we have some final proposal ready and tested.

Ondrej
-- 
  Ondřej Surý
  vedoucí výzkumu/Head of R&D department
  -------------------------------------------
  CZ.NIC, z.s.p.o.    --    Laboratoře CZ.NIC
  Americka 23, 120 00 Praha 2, Czech Republic
  mailto:ondrej.sury at nic.cz    http://nic.cz/
  tel:+420.222745110       fax:+420.222745112
  -------------------------------------------