[dns-operations] compressing DNS traffic data

[Joe Greco]

> On 8/12/2010, at 8:55 AM, Jim Reid wrote:
> >> 2. we need to waste it even if it's cheap.  It's the same flawed logic
> >> why the US cars had such low mpg ratio - because gas was cheap there was
> >> no "need for efficency".
> > 
> > This is a flawed analogy Ondřej. The price of storage has been going in one direction for decades. So has disk capacity. The ability to store bits is almost infinite and the unit cost for doing so continues to fall. None of these things hold true for oil.
> 
> It is most certainly not infinite and that fact underlies this 
> protocol - the volume of DNS queries that many people see (ISPs, 
> TLDs, etc) is well beyond the point where the cost of storage is 
> insignificant.  Furthermore it is reasonable to expect that 
> volume of traffic to increase just as much as disk capacity 
> increases, so this is not going to go away.

It's worth noting here that the number of DNS requests has skyrocketed
over the years.

Back in the infancy of the Web, relative links were king, and some web
pages only generated one DNS lookup to enable the page to load, and many
other pages only had at most a handful.

By today's count, and I did just count, pulling up www.ebay.com results
in *204* lookups, www.cnn.com results in 92 lookups, Facebook 24, etc.
That's actual requests happening on the wire per tcpdump.  (I'm actually
a bit shocked.)

Further, today's DNS traffic is bigger; sites have more redundancy in
nameservers, may have more answers per question, may have DNSSEC signing, 
may have both IPv4 and IPv6 answers, etc.

Storing a terabyte of DNS requests on a single SATA disk probably isn't
that horribly hard, but what happens when the requirements are more
significant?  It makes sense to at least consider compression. 

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.