[dns-operations] GPS time glitch last night

[Mukund Sivaraman]

Hi Bert

On Wed, Jan 01, 2020 at 10:32:48AM +0100, bert hubert wrote:
> On Wed, Jan 01, 2020 at 01:26:27PM +0530, Mukund Sivaraman wrote:
> > Some people are reporting a GPS time glitch list night, which seems to
> > have affected our ntpd instance too. From the logs, it appears to have
> > stopped trusting GPS clocks due to a greater than expected change in
> > $GPRMC time. A restart of the ntpd process fixed it.
> 
> Hi Muks,
> 
> Do you have some further details on the $GPRMC log lines?

During last night, the ntpd process suddenly dropped its GPS reference
clock. This is the first time we've observed this happening.  ntpq -np
showed an 'x' in the remote field where an 'o' should be. 'x' means "Out
of tolerance, do not use (discarded by intersection algorithm);"
(https://nlug.ml1.co.uk/2012/01/ntpq-p-output/831)

It was noticed in the morning. A drift is observed in
/var/log/peerstats.20191231. Corresponding to the log times of where the
drift starts, the $GPRMC sentences from /var/log/clockstats.20191231
show the following:

58848 73710.974 127.127.20.0 $GPRMC,202830,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73726.974 127.127.20.0 $GPRMC,202846,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73742.974 127.127.20.0 $GPRMC,202902,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73758.974 127.127.20.0 $GPRMC,202918,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73774.974 127.127.20.0 $GPRMC,202934,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73790.974 127.127.20.0 $GPRMC,202950,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73806.974 127.127.20.0 $GPRMC,203006,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73822.974 127.127.20.0 $GPRMC,203025,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73838.973 127.127.20.0 $GPRMC,203041,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73854.973 127.127.20.0 $GPRMC,203057,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73870.973 127.127.20.0 $GPRMC,203113,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73886.974 127.127.20.0 $GPRMC,203129,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__
58848 73902.973 127.127.20.0 $GPRMC,203145,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73918.973 127.127.20.0 $GPRMC,203201,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73934.973 127.127.20.0 $GPRMC,203217,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73950.973 127.127.20.0 $GPRMC,203233,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73966.973 127.127.20.0 $GPRMC,203249,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73982.973 127.127.20.0 $GPRMC,203305,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 73998.973 127.127.20.0 $GPRMC,203321,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74014.973 127.127.20.0 $GPRMC,203337,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74030.973 127.127.20.0 $GPRMC,203350,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74046.973 127.127.20.0 $GPRMC,203406,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74062.973 127.127.20.0 $GPRMC,203422,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74078.973 127.127.20.0 $GPRMC,203438,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74094.973 127.127.20.0 $GPRMC,203454,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74110.973 127.127.20.0 $GPRMC,203510,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__
58848 74126.973 127.127.20.0 $GPRMC,203526,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 74142.974 127.127.20.0 $GPRMC,203542,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__
58848 74158.973 127.127.20.0 $GPRMC,203558,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__

These messages are logged once every 16 seconds. The $GPRMC time (UTC)
jumped from 203006 to 203025 (delta 19 seconds), and then corrected back
from 203337 to 203350 (delta 13 seconds). Note that these $GPRMC lines
are the (derived) output of the GPS receiver.

The GPS receiver is a Garmin GPS 18x LVC running the latest firmware.