[dns-operations] 'dnstap' (Re: Prevalence of query/response logging?)

[Roland Dobbins]

On Jul 8, 2014, at 9:33 AM, Robert Edmonds <edmonds at mycre.ws> wrote:

> It looks like maintaining the #2/#3 "transport/encoding" split with IPFIX is impossible

It's an abstract goal which isn't an operational consideration, IMHO.

>  If IPFIX is well-suited for applications other than representing IP flows, it is awfully hard to tell from the outside without plowing through a ton of specifications.  This is itself a downside; we have to convince not just ourselves, but DNS software vendors to import this code and DNS software users that they might want to use this code.

It's actually pretty well-known in the operational community that IPFIX is extensive and well-suited to this sort of task.

In terms of collection/analysis tools, putting this sort of data into a standardized format would go a long ways towards ensuring that tools are available which can use it, and offer easy combinatorial analysis with data such as more classical flow records from routers/layer-3 switches, policy and even data from things like firewalls and load-balancers, etc.  Adoption by DNS vendors/coders is only half the battle, and since this is Something New to DNS vendors/coders, but not to vendors/coders of telemetry analysis systems, it's a consideration.

> to represent dnstap payloads with maximally sized DNS messages in a single frame, which I wanted to make a hard requirement for dnstap.

Is this intended to help with performance, or . . . ?

> (Possibly IPFIX can split payloads across multiple messages,

It can.

> What if you want to send payloads over an AF_UNIX socket, or via an HTTP(S) GET/POST, WebSockets connection, some new technology that hasn't been invented yet, etc.?

These are corner-cases, IMHO.

> I say "appears" above because my next complaint is that there are too many specifications documents for IPFIX.

One can say the same of DNS.

;>

All it would've taken to get the relevant information was asking folks plugged into the flow telemetry community about these various issues, rather than wading through lots of docs.

> Also, I found the following blog post rather interesting:

Without wasting a lot of space here dissecting it, there's a great deal he says which I don't agree with, FWIW.v

> It just doesn't look like a good fit for what we want to make possible, and there are a lot of general purpose technologies out there that I would consider first before considering IPFIX for a particular application.


The barrier to adoption, as I see it, is that this is now a one-off telemetry format, which generally (not always) tends to lead to low-to-no adoption.  I fear that's the case here (hopefully, I'm wrong!).

I personally think going the one-off route (pardon the pun, heh), no matter the perceived benefits, is self-defeating; I'll look into what's necessary to somehow transcode this into IPFIX, but it would've been much simpler (and much more likely to be widely adopted, IMHO) if IPFIX had simply been adopted in the first place.

I'm really grateful for the detailed explanation and insight into your thinking; even though I don't agree with the goal prioritization (a primary goal of any form of telemetry export should be relatively easy compatibility with existing collection/analysis systems and the use of formats with which there's likely going to be some degree of familiarity and experience with same, in order to maximize the chances of broad adoption) - it's educational and much appreciated!

----------------------------------------------------------------------
Roland Dobbins <rdobbins at arbor.net> // <http://www.arbornetworks.com>

                   Equo ne credite, Teucri.

    		   	  -- Laocoön