[dsc] dsp multi node setup: no data to display at this time
Mächler Philippe
P.Maechler at glattwerk.ch
Wed Apr 24 09:09:08 UTC 2019
Hello dsc/dsp users (repost with the right mail address)
I’m having a hard time setting up a multi node dsc/dsp setup.
First I had problems building the dsc Port on FreeBSD 11.2 and 12.0. In the meantime I built it in Poudriere and installed the package.
The collector is running find and I get the xml files in /var/lib/dsc
# less 1556094840.dscdata.xml
<dscdata>
<array name="pcap_stats" dimensions="2" start_time="1556094780" stop_time="1556094840">
<dimension number="1" type="ifname"/>
<dimension number="2" type="pcap_stat"/>
<data>
<ifname val="bge0">
<pcap_stat val="pkts_captured" count="4575"/>
<pcap_stat val="filter_received" count="4575"/>
</ifname>
<ifname val="bge1">
<pcap_stat val="pkts_captured" count="1239"/>
<pcap_stat val="filter_received" count="1239"/>
</ifname>
<ifname val="lo0">
<pcap_stat val="pkts_captured" count="4"/>
<pcap_stat val="filter_received" count="4"/>
</ifname>
<ifname val="lo1">
</ifname>
</data>
</array>
This looks like dsc is seeing and writing the data to its xml files. The transfer from dsc to dsp is done with rsync and I have those files on the presenter
# ll
total 43
-rw-rw-r-- 1 root wheel 42624 Apr 24 10:34 1556094840.dscdata.xml
-rw-rw-r-- 1 root wheel 40244 Apr 24 10:35 1556094900.dscdata.xml
drwxr-xr-x 2 root wheel 30 Apr 24 10:36 20190424/
drwxr-xr-x 3 root wheel 3 Apr 24 10:29 done/
-rw-r--r-- 1 root www 15 Apr 24 10:37 dsc-xml-extractor.out
The xml Files are processed by refile-and-grok.sh
# tail dsc-xml-extractor.out
wrote 17 lines to 20190424/client_port_range.dat
wrote 17 lines to 20190424/ipv6_rsn_abusers_count.dat
wrote 0 lines to 20190424/ipv6_rsn_abusers_accum.dat
wrote 17 lines to 20190424/qr_aa_bits.dat
wrote 955 lines to 20190424/client_addr_vs_rcode_accum.dat
wrote 2 lines to 20190424/chaos_types_and_names.dat
wrote 17 lines to 20190424/transport_vs_qtype.dat
wrote 17 lines to 20190424/pcap_stats.dat
wrote 17 lines to 20190424/rcode.dat
wrote 17 lines to 20190424/direction_vs_ipproto.dat
The files are “correctly” processed (I guess):
# less 20190424/client_addr_vs_rcode_accum.dat
X.Y.Z.214 0 3
X.Y.Z.9 3 3
X.Y.Z.80 3 1
The directory layout on the presenter is pretty simple:
pmaechler at DSP:/ # ll /var/lib/dsc/
total 2
drwxr-xr-x 3 root www 3 Apr 24 10:27 dns2/
drwxr-xr-x 3 root www 3 Apr 23 17:27 ns2/
-rw-r--r-- 1 root www 80 Apr 24 10:40 refile-and-grok.sh.stdout
pmaechler at DSP:/ # ll /var/lib/dsc/dns2/dns2/
total 26
drwxr-xr-x 2 root wheel 30 Apr 24 10:38 20190424/
drwxr-xr-x 3 root wheel 3 Apr 24 10:29 done/
-rw-r--r-- 1 root www 15 Apr 24 10:40 dsc-xml-extractor.out
pmaechler at DSP:/ #
I have a two servers (ns2 and dns2) each server has a node with the same name as the server
The whole setup is done by https://github.com/DNS-OARC/dsp/wiki/Multi-Node-Setup-Guide
pmaechler at DSP:/ # tail /var/log/nginx/error.log
2019/04/23 15:14:38 [crit] 8813#100452: *1 connect() to unix:/var/run/fcgiwrap.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.2.51, server: localhost, request: "GET /dsc-grapher.pl HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "192.168.3.25"
2019/04/23 15:14:38 [error] 8813#100452: *1 open() "/usr/local/www/nginx/favicon.ico" failed (2: No such file or directory), client: 192.168.2.51, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.3.25"
2019/04/23 15:15:53 [crit] 8855#100494: *1 connect() to unix:/var/run/fcgiwrap/fcgiwrap.sock failed (13: Permission denied) while connecting to upstream, client: 192.168.2.51, server: localhost, request: "GET /dsc-grapher.pl HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap/fcgiwrap.sock:", host: "192.168.3.25"
2019/04/23 15:16:59 [error] 8855#100494: *3 open() "/usr/local/www/nginx/favicon.ico" failed (2: No such file or directory), client: 192.168.2.51, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.3.25"
2019/04/23 15:55:38 [error] 8988#100557: *12 open() "/usr/local/www/nginx/favicon.ico" failed (2: No such file or directory), client: 192.168.2.51, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.3.25"
2019/04/23 17:13:54 [error] 8988#100557: *43 open() "/usr/local/www/nginx/favicon.ico" failed (2: No such file or directory), client: 192.168.2.51, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.3.25"
2019/04/24 08:04:36 [error] 8988#100557: *74 open() "/usr/local/www/nginx/favicon.ico" failed (2: No such file or directory), client: 192.168.2.51, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.3.25"
pmaechler at DSP:/ #
The permission problem on the socket is solved
And finally my dsp.config
pmaechler at DSP:/ # cat /usr/local/etc/dsp/dsc-grapher.cfg
server dns1 dns1
server dns2 dns2
server ns1 ns1
server ns2 ns2
trace_windows 1hour 4hour 1day 1week
accum_windows 1day 2days 3days 1week
embargo 0
#anonymize_ip
#no_http_header
#hide_nodes
debug_level 9
debug_file /var/log/dsc.log
#
# To update domain_list all_tlds:
# dig @f.root-servers.net . axfr \
# | awk '$3=="IN" && $4=="NS" {print $1}' \
# | tr A-Z a-z \
# | sed -e 's/\.$//' \
# | grep . \
# | sort \
# | uniq \
# | fmt 50 \
# | sed -e 's/^/domain_list all_tlds /'
domain_list all_tlds ac ad ae aero af ag ai al am an ao aq ar arpa as
domain_list all_tlds asia at au aw ax az ba bb bd be bf bg bh bi biz bj
domain_list all_tlds bm bn bo br bs bt bv bw by bz ca cat cc cd cf cg ch
domain_list all_tlds ci ck cl cm cn co com coop cr cu cv cx cy cz de dj
domain_list all_tlds dk dm do dz ec edu ee eg er es et eu fi fj fk fm fo
domain_list all_tlds fr ga gb gd ge gf gg gh gi gl gm gn gov gp gq gr gs
domain_list all_tlds gt gu gw gy hk hm hn hr ht hu id ie il im in info
domain_list all_tlds int io iq ir is it je jm jo jobs jp ke kg kh ki km
domain_list all_tlds kn kp kr kw ky kz la lb lc li lk lr ls lt lu lv ly
domain_list all_tlds ma mc md me mg mh mil mk ml mm mn mo mobi mp mq mr
domain_list all_tlds ms mt mu museum mv mw mx my mz na name nc ne net nf
domain_list all_tlds ng ni nl no np nr nu nz om org pa pe pf pg ph pk pl
domain_list all_tlds pm pn pr pro ps pt pw py qa re ro rs ru rw sa sb sc
domain_list all_tlds sd se sg sh si sj sk sl sm sn so sr st su sv sy sz
domain_list all_tlds tc td tel tf tg th tj tk tl tm tn to tp tr travel
domain_list all_tlds tt tv tw tz ua ug uk us uy uz va vc ve vg vi vn vu
domain_list all_tlds wf ws xn--0zwm56d xn--11b5bs3a9aj6g xn--80akhbyknj4f
domain_list all_tlds xn--9t4b11yi5a xn--deba0ad xn--g6w251d xn--hgbk6aj7f53bba
domain_list all_tlds xn--hlcj6aya9esc7a xn--jxalpdlp xn--kgbechtv xn--zckzah
domain_list all_tlds ye yt yu za zm zw
valid_domains some.system all_tlds
The debug_log is not really usefull for me ☹
# less /var/log/dsc.log
[14317] CFG=$VAR1 = {
'debug_level' => '9',
'nodemap' => {
'ns1' => {
'ns1' => [
'ns1'
]
},
'ns2' => {
'ns2' => [
'ns2'
]
},
'dns1' => {
'dns1' => [
'dns1'
]
},
'dns2' => {
'dns2' => [
'dns2'
]
}
},
'debug_fh' => bless( \*Symbol::GEN0, 'IO::File' ),
'domain_list' => {
'all_tlds' => [
'ac',
'ad',
'ae',
'aero',
[many more]
'xn--jxalpdlp',
'xn--kgbechtv',
'xn--zckzah',
'ye',
'yt',
'yu',
'za',
'zm',
'zw'
]
},
'trace_windows' => [
'1hour',
'4hour',
'1day',
'1week'
],
'serverlist' => [
'dns1',
'dns2',
'ns1',
'ns2'
],
'embargo' => 0,
'servers' => {
'dns1' => [
'dns1'
],
'ns2' => [
'ns2'
],
'dns2' => [
'dns2'
],
'ns1' => [
'ns1'
]
},
'accum_windows' => [
'1day',
'2days',
'3days',
'1week'
],
'valid_domains' => {
some.system' => 'all_tlds'
}
};
[14317] PLOT=$VAR1 = {
'data_reader' => sub { "DUMMY" },
'data_summer' => sub { "DUMMY" },
'yaxes' => {
'percent' => {
'label' => 'Percent of Queries',
'default' => 0,
'divideflag' => 0
},
'rate' => {
'divideflag' => 1,
'default' => 1,
'label' => 'Query Rate (q/s)'
}
},
'colors' => [
'xrgb(ff0000)',
…
'xrgb(e60066)'
],
'datafile' => 'qtype',
'plot_type' => 'trace',
'names' => [
'a',
…
'o'
],
'dataset' => 'qtype',
'plottitle' => 'Queries by Node',
'keys' => [
'a',
…
'o'
]
};
[14317] ===> starting at Tue Apr 23 17:25:40 CEST 2019
[14317] ARGS=$VAR1 = {
'content' => 'png',
'window' => 14400,
'plot' => 'bynode',
'end' => 1556033140,
'binsize' => 60,
'server' => 'none',
'yaxis' => 'rate',
'node' => 'all',
'mini' => 0
};
[14317] Plotting none all bynode 1556033140 14400 60
[14317] reading datafile took 0 seconds, 0 lines
[14317] loaded data: $VAR1 = {};
[14317] munged data: $VAR1 = {};
[14317] writing trace tmpfile took 0 seconds, 0 lines
[14317] graph took 0 seconds
[14317] <=== finished at Tue Apr 23 17:25:40 CEST 2019
The only thing that looks like an error is “writing trace tmpfile took 0 seconds, 0 lines” otherwise the setup looks ok for me, except that I don’t see the fancy graphs 😊
The .dat Files on /var/lib/dsc/$SERVER/$NODE belong to root:wheel with read permissions for everyone, when I changed them to www:www the effect is the same ☹
Does anybody has an idea whats wrong or where I should have a closer look?
TIA
Philippe
More information about the dsc
mailing list