[dns-operations] ncap Perl Bindings

[Robert Edmonds]

Sidney Faber wrote:
> We're beating the ncap drum throughout .gov and .mil, preaching for
> its adoption and for more DNS research using it--studying DNS for more
> than just "this name resolves to this address."  Shortly you'll see
> some code contributions including a ncaptool module to search ncap
> files for hundreds or thousands of query names/domains at the same
> time, and a tool that lets you glob ncap files together as you pipe
> them into ncaptool.  More to follow (our interns have been busy this
> summer!).

the key feature that the NCAP file format has over pcap is that it
stores reassembled IP datagrams, which makes it good at storing UDP DNS
messages since reassembly doesn't have to occur in the analysis stage.

however, the NCAP format also destroys information: the original IP
header is reduced to a network layer pseudoheader, and fragmentation is
glossed over.  (briefly, the IP reassembly routines in libncap turn
complete fragment sequences into reassembled IP datagrams; you lose
fragment ordering, boundary, and arrival time information, and
incomplete fragments are discarded.) libncap also tries to recover DNS
information from ICMP payloads, and does not implement TCP at all.

NMSG is much more flexible since it's designed to be a generic container
format.  one of the formats it supports is an NCAP-analogue, but
unfortunately it uses the exact same IP reassembly code as libncap.
we're planning to replace the capture reassembly code with collation
code (and a corresponding format change) so that the original stream of
fragments can be preserved at message write time.  at message read time,
we'll still support an interface that returns reassembled datagrams as
well as a lower level interface that extracts the original packets.
this change will also make it easier to finally support TCP.

but these are core implementation details and shouldn't really affect
analysis being done out at the periphery, where all you need is the
pointer to the start of a DNS message and its length.  porting such
filter code to a new framework shouldn't take much time at all.

> Which makes me extremely disappointed to hear that ncap is going into
> maintenance mode.  Is there _any_ chance that the nmesg stuff can be
> rolled into the ncap suite, and essentially become another input pipe
> into ncaptool?

none at all; in fact it will probably be the other way around.  the
libncap/ncaptool API simply isn't expressive or granular enough to
support what nmsg can support.  on the drawing board (once the libnmsg
core is "finalized") is a purpose-built DNS capture and filtering and
analysis tool to replace ncaptool and dnscap and pcapdump and tshark.
(perhaps it should have its own mailing list?  since we're veering
off-topic for dns-operations.) it will support reading from pcap, ncap,
or nmsg sources.  it will have its own filtering language with a proper
grammar rather than the filter hacks that we have in ncaptool and dnscap
right now, and it will be embeddable as a library.  it will support
boolean predicates and be able to load filter code from external
modules.  (in contrast, ncaptool's filter modules are just chained
together.)

rolling anything into the "ncap suite" is fraught with peril; there's
only one person who can really understand ncap_create.c.  libncap can't
really be embedded in an application; you have to use the dlopen() hack
in ncaptool to interface with external code.  IMHO much of the code in
ncaptool should have been in the library instead, and in nmsg in fact
nmsgtool is simply a driver for libnmsg.

i can't stress enough the fact that ncap's modularity came about as my
first assignment at ISC, which was to take the original 2007-2008
version of ncap and an internal fork and unfork them, which i did by
inserting the minimum number of hooks into ncaptool and making them
accessible to dlopen()ed hunks of code.

> I _really_ want to see ncap grow into an open source pluggable dns
> analysis tool suite, I'd hate to have to branch off and build stuff on
> our own.

so do we, but we're going to rewrite rather than extend.  NCAP only
allows us to represent complete DNS messages.  with NMSG we're planning
on defining new message formats that allow us to store e.g. individual
RRsets along with metadata.  for instance, imagine a recursive DNS
server that has been modified to send out "diffs" of its RRset cache
(insertions, expirations, replacements, and perhaps discards) in
real-time.  you'd be able to build an audit trail of the internal state
of a DNS cache rather than trying to infer its state based on observing
the network layer as you have to do with ncap/pcap.

> Whatever you decide, thanks for what's been done already, everyone's
> very excited about using the ncap suite.

well, they should be getting excited about being able to process more
than just intact DNS messages :)

-- 
Robert Edmonds
edmonds at isc.org