In Australia today there was a nationwide outage of computers operating our major supplier’s mobile network. Apparently it halted all commerce and banking, stopped train serices, blocked phone calls, and even led to failure of the emergency call service. It is not the first time, but todays event may be the largest ever.
It is reported that there was a failure of time syncing between nodes, thought to be caused by a software update.
That is a little more dramatic than the average home computer update failure , but it is in principle the same thing.
I feel sorry for the tech specialists who must have been under immense pressure to diagnose and fix such an event,.
I wonder, does this sort of major outage occur overseas , or is it a unique Australian problem… due probably to our “she’ll be right” approach .
Couple years back - that massive Crowdstrike flaw caused catastrophic outages across the planet… I’m still astounded that vast companies ran critical infrastructure on MS platforms - like flight schedules, passenger bookings… it all fell over…
Still not sure how a Telstra outage would affect train services - but apparently they factored in a dependancy on that into their transport systems… Probably something some shonky dodgy pezzo di merda consultant from the likes of Price Waterouse Coopers, or Ernst and Young or Deloitte Haskins insisted was mission critical…
I blame capitalism and rampant privatisation… they have to show improved “efficiency” EVERY year! Or improved profit margin - year after year! IT’s not sustainable in a finite reality… If they can’t increase the profit margin - they shave stuff off the side, year by year - every year… it’s relentless..
I wonder whether the PMG Department. ( that is the former government department that controlled phones , telegrams and postage) would have done any better.? I remember they were reliable but expensive.
National services always expensive, in the uk we had post office and British telecoms, in france we still have la poste but the service is getting worse, deliveries every 2 days now instead of every day, and removal of local post boxes. France Télécom were bought out by Orange mobile phone operator and there were 3 choices for phones now the market is so diverse I cannot keep up.
Same with electric companies.
The item I read about your problem suggested it was a network update due to time variations across the country not sure if that is true. .?
To answer Neville’s question (do these happen in other countries?)—yes, they do. In the US a year or two ago there was a nationwide outage of the Verizon cell system, the largest carrier in the country. I don’t know for certain if their data services were affected, but I think they were. Several large airlines in the US have suffered network breakdowns that grounded their planes for a while. The Federal Aviation Administration’s air traffic system has had local outages that required air traffic halts, although I’m not sure if the entire system ever crashed.
We are way out at the end of a long technological limb.
That seems to be the technological cause.
There will be a human cause somewhere.
and there is a strategic failure, as @don.karon said
“We are way out at the end of a long technological limb.”
and there is a political cause, as @daniel.m.tripp said
“I blame capitalism and rampant privatisation”
and Aristotle would have multiplied that by 4
There are sections of track with no way of knowing where a train is except by phoning the driver. They had to halt trains for safety reasons.
Once upon a time they had reliable systems - e.g. a “station master” would report the 5:44pm out of Sydney Central was on time - or 35 seconds delayed…
But nah… they don’t want to pay a station master, or his underling, or their tea lady…
The journey of our world along the digital technology path, is a bit like throwing the ladder away after you have climbed it… there is no way back. …older systems no longer exist.
I found this comment
"Andrew Dempster, a professor with UNSW’s Australian Centre for Space Engineering Research, said he could not recall another timing-related failure causing disruption on this scale in Australia.
Professor Dempster said while the outage appeared to show a failure rather than manipulation of GPS, it still demonstrated how vulnerable GPS-based timing systems could be."
I thought timing in computers was based on the internet NTP service.
Sure GPS positioning could determine your time zone, but is that all it means?
I wonder do these real time computer systems that run our phones have some different concept of time that involves time continuously varying with position instead of in zones? But even with that , the computers dont move… so why do they need GPS?
Does anyone understand this?
Update:
I found this
"How GPS Provides Time Synchronization
GPS plays a crucial role in maintaining accurate time synchronization for computer systems, including phone systems. This is achieved through the following mechanisms:
Atomic Clocks: GPS satellites are equipped with atomic clocks that provide highly precise time signals.
Broadcast Signals: These satellites continuously broadcast their time and location data, allowing GPS receivers to capture this information.
Importance of Accurate Timing
Accurate timing is essential for various applications in phone systems and other computer networks:"
It seems GPS satellites carry atomic clocks
Next question … why is that better than using NTP?
I had a former customer who hired some consultancy firm to provide a service to register and partially track, bluetooth devices in vehicles along some major thoroughfares… Basically to estimate in “semi” realtime if there were traffic bottlenecks, where they were, the extent of the bottleneck etc…
e.g. a BlueTooth adaptor in a car’s mac address pings a sensor at Point A, then it gets pinged again at Point B… and probably Point C and D too… and so on…
From those pings you can infer a few things…
But not if your times are all bogus!
The consultancy firm deployed hundreds of Raspberry Pi systems - guess what? The RPI does NOT HAVE A REAL-TIME CLOCK!
So - sometimes it would register a BT MAC address at Point C at a time BEFORE it traversed Points A and B!
I think they resolved it by setting up the ethernet switches the Pis were plugged into (next hop) as NTP servers - but it took months (maybe years?) - when the “consultancy firm” should have thought about this when deploying a sensor that DOES NOT HAVE A REALTIME CLOCK!
And really - when you want realtime results, you can’t really rely on NTP - you need a REALTIME CLOCK!
So why doesnt Telstra buy its own atomic clock.?.. relying on GPS is nuts and cheapskate. Someone could take our whole country out by disabling one satellite.
and
are phones so,so realtime that they need this degree of precision? We had no time issues with the old analog network
Surely all the Telstra computers have a real timeclock? Every PC has one.
It must have an ordinary computer clock?
Why is that not sufficient? What does a REAL-TIME CLOCK provide other than keeping time when the power is off?
I thought NTP was to prevent drift?
Why cant NTP be the hard clock you referred to?
I still dont see why Telstra need GPS for time syncing… they seem to be using it a s a cheap hard clock. Why not get their own hard clock ?.. it is a matter of national security that they be independent.
