[dns-operations] Analyze responses with drool and respdiff

[Jerry Lundström]

Hi all,

With the release of drool version 1.99.2 (and dnsjit v0.9.5) and a tool-
chain called respdiff it is now possible to replay a PCAP and do analysis
of the responses found in the PCAP with those received from the replay.
TL;DR? Check below for example!

[ also in HTML: https://medium.com/@dnsoarc/analyze-responses-with-drool-an
d-respdiff-168a2b9ffb49 ]


dnsjit

  https://www.dns-oarc.net/tools/dnsjit

dnsjit (developed by DNS-OARC) is a combination of parts taken from dsc,
dnscap, drool (when it was in C), and put together around Lua to create a
script-based engine for easy capturing, parsing and statistics gathering of
DNS messages while also providing facilities for replaying DNS traffic.


drool

  https://www.dns-oarc.net/tools/drool

drool (DNS Replay Tool, developed by DNS-OARC) can replay DNS traffic from
packet capture (PCAP) files and send it to a specified server, with options
such as to manipulate the timing between packets, as well as loop packets
infinitely or for a set number of iterations.

drool is now a Lua script that uses dnsjit and this work has been sponsored
by Comcast Innovation Fund.


respdiff

  https://gitlab.labs.nic.cz/knot/respdiff

Respdiff (developed by CZ.NIC, part of the Knot project) is an abbreviation
from “response differences” used as name for set of tools to gather answers
to DNS queries from DNS servers and compare them based on specified
criteria.


drool + respdiff

We (DNS-OARC and CZ.NIC) started a collaborative effort a few months back
to use each others tool. The aim is to have drool replay traffic and gather
the responses for respdiff to analyze.


Example (yay!)

drool comes packaged for most Linux distributions and is also compiled and
tested on FreeBSD and OpenBSD but respdiff was just moved out from being an
internal testing tool at CZ.NIC so it may be more tricky to get to run
currently. Here is two install examples, first for Debian 9 and later on
for CentOS 7.


Debian 9 installation

  sudo apt-get install -y build-essential wget python3-pip lmdb-utils
  sudo pip3 install --upgrade pip
  sudo git clone --depth=1 https://gitlab.labs.nic.cz/knot/respdiff.git /var/opt/respdiff
  sudo pip3 install -r /var/opt/respdiff/requirements.txt
  wget -O - https://pkg.dns-oarc.net/dns-oarc.distribution.key.gpg | sudo apt-key add -
  echo "deb http://pkg.dns-oarc.net/deb-pr stretch main" | sudo tee /etc/apt/sources.list.d/dns-oarc-pr.list
  sudo apt-get update
  sudo apt-get install -y drool


CentOS 7 installation

  sudo yum group install -y "Development Tools"
  sudo yum install -y yum-plugin-copr epel-release wget centos-release-scl
  sudo yum copr -y enable @dnsoarc/drool-pr
  sudo yum install -y drool rh-python36
  sudo scl enable rh-python36 'pip3 install --upgrade pip'
  sudo git clone --depth=1 https://gitlab.labs.nic.cz/knot/respdiff.git /var/opt/respdiff
  sudo scl enable rh-python36 'pip3 install -r /var/opt/respdiff/requirements.txt'


Replay and analyze

Because respdiff needs newer Python then what is available on CentOS 7 you
will need to enable it into a new shell first (if your testing on CentOS 7
of course).

  scl enable rh-python36 bash

Now get a DNS PCAP, replay it and run an analysis on the results.

  wget https://github.com/DNS-OARC/drool/raw/develop/src/test/dns.pcap
  drool respdiff /tmp/respdiff-results pcap dns.pcap google 8.8.8.8 53
  wget https://gist.github.com/jelu/219f5e4976121397158c4153393e54d8/raw/8d1f4691bc371e5cfc3dfef05b342a60619f86f0/respdiff.cfg
  /var/opt/respdiff/msgdiff.py -c ./respdiff.cfg /tmp/respdiff-results
  /var/opt/respdiff/diffsum.py -c ./respdiff.cfg /tmp/respdiff-results

Example output

  $ drool respdiff /tmp/respdiff-results pcap dns.pcap google 8.8.8.8 53
  << dnsjit v0.9.5 https://github.com/DNS-OARC/dnsjit/issues >>
  runtime 1.551086741     run     1.526865703     finish  0.024221038
          total   /sec
  packets 133     87.106547575651
  queries 41      26.852394365426
  sent    41      26.852394365426
  received        41      26.852394365426
  responses       41      26.852394365426
  timeouts        0
  errors  0

  $ /var/opt/respdiff/msgdiff.py -c ./respdiff.cfg /tmp/respdiff-results

  $ /var/opt/respdiff/diffsum.py -c ./respdiff.cfg /tmp/respdiff-results
  == Global statistics
  duration                       2           seconds
  queries                       41
  answers                       41  100.00 % of queries

  == Differences statistics
  upstream unstable              0    0.00 % of answers (ignoring)
  not 100% reproducible          0    0.00 % of answers (ignoring)
  target disagrees               0    0.00 % of not ignored answers


Replay your own PCAP or analyze other things

To replay and analyze a different PCAP you need to know a few things:
- The second argument to drool respdiff is the server name for the
responses found in the PCAP and needs to exist in respdiff.cfg
- The fourth argument to drool respdiff is the server name for the
responses received when replaying and needs to exist in respdiff.cfg
- drool respdiff can currently only replay against one host
- The criteria in the diff section of respdiff.cfg controls what fields are
analyzed, see respdiff.cfg in it’s main repository for comments on each
section
  https://gitlab.labs.nic.cz/knot/respdiff/blob/master/respdiff.cfg
- Re-run both msgdiff.py and diffsum.py to redo or do a new analysis on an
existing result


Still early development…

Please note that this is still in early development and things are missing
or possibly broke, but we would none the less be very happy if you can find
the time to test this a bit and report any issues you discover or any other
feedback you would like to give!

    drool: https://github.com/DNS-OARC/drool/issues
    respdiff: https://gitlab.labs.nic.cz/knot/respdiff/issues


Cheers,
Jerry Lundström (DNS-OARC) & Petr Špaček (CZ.NIC)