Disable Time Sync NOW—Ugly GPSd Bug Brings Sunday FAILs
On Sunday, you might find some equipment thinks it’s 2002. That’s because of a weird bug in gpsd—the code on which a bunch of Network Time Protocol servers rely.
If you can stop your devices updating the time for a few days until the dust settles, that might be a good idea. Let’s see where we are next week, but be prepared to wait until more patches get installed.
It’s yet another case of critical open source code being maintained by a single unpaid volunteer. In today’s SB Blogwatch, we batten down the hatches.
Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: Henge.
GPSD PTSD?
What’s the craic? Steven J. Vaughan-Nichols reports a dirty little secret—“Thanks to a nasty GPSD bug, real-life time travel trouble arrives this weekend”:
A world of trouble”
Security, identification, networks, everything that makes the internet go depends on accurate time-keeping. Some systems rely on … GPS appliances and the GPSD daemon to tell the exact time, and a nasty bug’s been uncovered in GPSD that’s going to pop up on October 24, 2021.
…
NTP is used to keep all internet-connected devices in sync with each other. How does NTP know what time it is? By synchronizing NTP servers with atomic clocks. … Many of them use GPSD.
…
[But] a bug in the … ”GPS Week Rollover” sanity checking code scheduled for November 2038 will instead trigger this Sunday, and cause … a lot of computers [to make] a quick, sharp visit to March 2002. … This will be ugly.
…
The fix is in August 2021’s GPSD 3.23 release. [But] just because your operating system is up to date does not mean that it will have the necessary GPSD fix. … Remember all that chatter about how awful Y2K was going to be? … I can easily see many companies ending up in a world of trouble if they don’t make sure their time-keeping is properly patched.
And Thomas Claburn adds—“If your apps or gadgets break down on Sunday, this may be why”:
It could get ugly”
A bug in gpsd that rolls clocks back to March 2002 is set to strike. … Gpsd is a service daemon that translates data from … Global Navigation Satellite Systems (GNSS), and Automatic Identification System (AIS) transmission sources … to sync a device’s system clock.
…
GPS satellites rely on multiple atomic clocks so their time data is highly accurate. [Gpsd] shows up in mobile embedded systems like UAVs, robot submarines, driverless cars, and in applications used in marine navigation and military IFF (Identification Friend or Foe) systems.
…
Gary Miller, maintainer of gpsd, acknowledged making the error, a simple miscalculation. … Miller, who is retired [said] he’d welcome support for the project. … “I know for a fact that a lot of military stuff uses it,” he said. … “I know it’s in at least one rocket system. I’m told it’s in tanks and delivery trucks and divers’ watches.” … “I suspect we’ll find versions of NTP appliances that got updated two years ago and they all fall flat on the 24th.”
…
Miller suggested financial firms might also run into compliance problems. … It could get ugly, he said.
What’s the nature of the bug? Bill Toulas explainifies—“GPS software bug may cause unexpected behavior this Sunday”:
Be prepared for the unexpected”
The implications are somewhat unpredictable. … It’s somewhat of a Y2K bug, so nobody can be sure about whether or not the devices will actually encounter functional or service reliability issues.
…
On October 24, 2021, all Network Time Protocol (NTP) servers using GPSD versions 3.20 through 3.22 are going to jump back 1024 weeks in time. … Every 1024 weeks (almost 20 years) a week number rollover phenomenon takes place in the [GPS] system due to an integer overflow on the broadcasted ten-digit binary, causing the internal value of the week count to drop to zero.
…
The last time it happened was on April 6, 2019, and it caused flight cancellations, wireless network crashes, and functional problems on older smartphones. … Be prepared for the unexpected this Sunday.
Couldn’t it be expanded to more than just 10 bits? robbak has another go unpicking the bug:
The logic turned out to be wrong”
There is an updated GPS spec that increases this, but that needs new satellites and receivers. This was a mistake in the code that allows GPSd to guess how many times it has rolled over. It relies on comparing UNIX time, which skips leap seconds, with GPS time, which does not.
Their code assumed that there would be at least one leap second in the last 4 years, which seemed like a safe bet – the planet had been spinning predictably requiring regular leap seconds every couple of years. But the planet has sped up a tad in the last few years, and no leap seconds have been needed, and UNIX time is a second behind where they though it would be. The logic turned out to be wrong and it will break this weekend.
Anything else lurking, ready to bite us? ClearCreek read the patch:
Weird consequences”
Can’t say that I followed all the code, but did note something interesting: In addition to the issues cause by using a 10 bit number in “weeks,” the GPS designers also assumed that the earth’s rotation would only slow down — never speed up.
Yet, it looks like that might happen soon, and that would result in a “negative” offset, with weird consequences. The reason? Global warming!
Did someone say IFF and missiles? The Man Who Fell To Earth is sarcastically “SHOCKED”:
… that there are mission critical military systems built on software systems the military contractors don’t understand.
This should be—interesting. This Anonymous Coward is breaking out the popcorn:
Sit back and watch the fun”
Any guesses what the results would be of having set up an entire company’s computers to only use a single … GPS dongle as their primary source of NTP? I’m mainly asking because I know someone who’s done that, and as I’m not very fond of them, so I’m just going to sit back and watch the fun from a safe distance.
This is yet another hidden FLOSS project on which we all rely. Here’s Dave559’s suggestion:
Mutual co-operation”
All of these “Nebraska Problems” (waves at OpenSSL) underpin so much of modern computer systems and the internet. Open source and volunteer effort is a great development model that has pulled us all up by our bootstraps to where we are today.
But when (effectively) the whole world is now entirely relying on these libraries to be robust … is it really not time to effectively mandate something like a 0.1% tax/tithe on software/hardware businesses to give solid permanent financial backing to these essential systems that they all rely on, and to ensure some paid development and testing/QA help so that these are not all entirely dependent on volunteer labour?
…
It is after all through mutual co-operation that most of humanity’s great developments have been made.
Meanwhile, the last shall be first—here’s toadlife:
First post! Thanks to GPSD transporting it back in time!
And Finally:
2002? That’s nothing: Henge takes us back to the 1990s
Hat tip: Happosai
You have been reading SB Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or [email protected]. Ask your doctor before reading. Your mileage may vary. E&OE. 30.
Image sauce: Ethan Medrano (via Unsplash)
