[dns-operations] GPS time glitch last night

Wed Jan 1 09:32:48 UTC 2020

On Wed, Jan 01, 2020 at 01:26:27PM +0530, Mukund Sivaraman wrote:
> Some people are reporting a GPS time glitch list night, which seems to
> have affected our ntpd instance too. From the logs, it appears to have
> stopped trusting GPS clocks due to a greater than expected change in
> $GPRMC time. A restart of the ntpd process fixed it.

Hi Muks,

Do you have some further details on the $GPRMC log lines?

Here in any case is what I found with my other project:

From: bert hubert <bert at hubertnet.nl>
To: nanog at nanog.org
Cc: Forrest Christian <lists at packetflux.com>, mattlists at rivervalleyinternet.net
Subject: GPS Sync Outage

Greetings,

This email is a response to https://mailman.nanog.org/pipermail/nanog/2020-January/105058.html
in which Forrest discussed GPS time sync issues today.

The galmon.eu project also monitors GPS, and one thing we found is that at
20:39:47 UTC today, which is near 2PM MST, GPS PRN 29 emitted a clock update
that is pretty weird. 

The 't0c' is the t=0 of the clock correction parameters and it went
backwards by 16 seconds, and it introduced a 5.9 nanosecond clock jump.

This was recorded by 18 of our stations, mostly in Europe and the US east
coast: 

Tue, 31 Dec 2019 20:39:47.0000 +0000 src 33 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749  Timejump: -5.89898 after -16 seconds, old t0c 15750
Tue, 31 Dec 2019 20:39:47.0005 +0000 src 13 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9999 +0000 src 3 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9997 +0000 src 15 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9998 +0000 src 101 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9996 +0000 src 32 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9996 +0000 src 57 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9999 +0000 src 37 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.0000 +0000 src 19 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.0004 +0000 src 49 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0000 +0000 src 27 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0005 +0000 src 56 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0004 +0000 src 41 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.4453 +0000 src 14 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0001 +0000 src 141 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0002 +0000 src 40 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0003 +0000 src 102 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0002 +0000 src 58 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749

This was the previous t0c:

Tue, 31 Dec 2019 20:39:18.0000 +0000 src 27 imptow 247173 GPS 29 at 0: 247176 frame 1 gpshealth = 0, wn 2086 t0c 15750

And this is the next one:

Tue, 31 Dec 2019 21:59:46.9996 +0000 src 33 imptow 252003 GPS 29 at 0: 252006 frame 1 gpshealth = 0, wn 2086 t0c 16199  Timejump: -0.192813 after 7200 seconds, old t0c 15749

We spend more of our time looking at Galileo, but we have all the GPS data.
I haven't looked enough at GPS to see if "t0c going backwards" is a common
thing, but it does not look common.

Perhaps this could be related. If anyone wants a copy of GPS traffic from
today, let me know.

        Bert

> 
> https://mailman.nanog.org/pipermail/nanog/2019-December/105053.html
> https://twitter.com/GalileoSats/status/1212190538750996480
> 
> If you are running a validating resolver or signer with local GPS based
> time sources, you may want to check synchronization.
> _______________________________________________
> dns-operations mailing list
> dns-operations at lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations