Massive Rogers outage cut off 25 percent of Canada’s internet traffic for nearly all of Friday

Canadian telecom Rogers is suffering a major outage affecting landline phones, cellular connections, and internet connectivity throughout Canada that started early this morning. Downdetector listed thousands of reports for the issues that flooded in as people started to get up around 5AM ET and couldn’t get online.

Rogers first addressed the outage in a tweet from its official support account just before 9AM ET and then went silent for a couple of hours. On Friday afternoon, the company tweeted that its technical teams are working to restore services “alongside our global technology partners, and are making progress.”

Late Friday evening, Rogers CEO & President Tony Staffieri posted a letter to Canadians on the company website.

ear Canadians,

We know you count on Rogers to connect you to emergency services, make payments, serve your customers, connect with work and keep in touch with friends and family. We take that responsibility very seriously and today we let you down. We can and will do better.

As you know, we experienced a network outage across both wireless and wireline service that began early this morning.

We have made meaningful progress towards bringing our networks back online and many of our wireless customers are starting to see services return. We don’t yet have an ETA on when our networks will be fully restored but will we continue to share information with our customers as we restore full service.

We know going a full day without connectivity has real impacts on our customers, and all Canadians. On behalf of all of us here at Rogers, Rogers for Business, Fido, chatr and cityfone, I want to sincerely apologize for this service interruption and the impact it is having on people from coast to coast to coast.

As our teams continue working to resolve the situation, I want to make two commitments to you:

First, we are working to fully understand the root cause of this outage and we will make all the changes necessary to ensure that in the future we meet and exceed your expectations for our networks.

Second, we will make this right for our valued customers. We will proactively apply a credit to all our customers impacted by the outage and will share more details shortly.

I take full responsibility for ensuring we at Rogers earn back your full trust, and are once again there to connect you to what matters.

Sincerely,

Tony Staffieri

President and CEO

There is still no ETA for full restoration. In another tweet, the company promised that it would be “proactively crediting all customers,” and said that it had “every technical resource and partner” working on getting its network back up.

The CBC reported Friday evening that people are starting to see their home internet and wireless services return, while Cloudflare Radar data shows traffic is beginning to flow to and from the Rogers network again, although still not anywhere near the usual levels.

Image: Cloudflare

The data from Cloudflare, a content delivery network and DDoS mitigation company that helps many highly-trafficked sites manage their operations, shows communication between its load-sharing network and Rogers in Canada exhibiting its normal nightly pattern before dropping to essentially zero at around 3–4AM ET and registering a flat line ever since. At that same time, there was a massive spike in BGP (Border Gateway Protocol) announcements for the network that indicate changes in routing.

In a blog post published Friday afternoon, Cloudflare personnel speculated that the outage is the result of an internal error at Rogers, as opposed to a cyberattack or other cause. They cited that pattern of BGP announcements, noting that with the Rogers network now failing to announce its presence, the rest of the internet can’t find it. BGP is a fundamental part of the technology that helps move information from one place on the internet to another, and a problem with BGP routing information was the cause of a massive Facebook network outage last fall.

The outage has also disconnected Interac, the network Canadian banks use, disabling debit cards, ATMs, and the e-Transfer services that recently notched over 1 billion transactions in one year.

Passport offices and Canada’s tax-collecting Revenue Agency are among the government services that are unavailable today due to the outage. Both agencies also warned users the outage is cutting off multifactor authentication codes sent by voice or text message, so people who are logged out may not be able to log in at this time.

Even the Canadian Radio-television and Telecommunications Commission (CRTC), which regulates broadcasting and telecoms, says its phones aren’t working due to the outage.

Internet network disruption tracker NetBlocks shared real-time data showing Canada’s national connectivity dropped to just 75 percent of its normal levels.

A report by CBC News noted Rogers subsidiary Fido is having problems, too. The article says problems have affected card payment processing and ATMs across Canada. A CBC radio station in Kitchener, Ontario, is also offline, and some telephone services for Ottawa’s transit agency have been knocked out. Pictures from Toronto show people crammed into coffee shops or heading to the library to access Wi-Fi now that their other options are offline.

TekSavvy customer service has told customers the outage extends to “Ontario, Quebec and the Eastern Provinces,” with no ETA for resolution, while the CBC reports that Thunder Bay operator Tbaytel says the Rogers outage is nationwide.

Toronto police tweeted early in the morning after some people in the city experienced connection issues calling 911. “The Rogers Network is experiencing some technical difficulties,” said the Toronto Police Operations Twitter account. “We are working to resolve these issues.” In a follow-up tweet, the police confirmed their 911 center is operational and advised anyone who has to call to stay on the line if they connect and to retry calls if they don’t.

Update 9:52AM ET: Added Bank of Montreal information

Update 11:20AM ET: Noted ongoing outage, added information from Cloudflare Radar, TekSavvy, and Tbaytel

Update 11:33AM ET: Added statement from Rogers

Update 12:40PM ET: Added latest update from NetBlocks

Update 1:31PM ET: Added additional information about Interac banking network, — thanks, mbarriault.

Update 2:13PM ET: Added information about CRTC, passport offices, and tax services outage.

Update 3:39PM ET: Added Cloudflare notes about BGP as a possible cause of the outage, as well as Rogers’ statement that it is “making progress” in fixing the issue.

Update 11:47PM ET: Added statement from Rogers CEO and note that services are coming back online, slowly.

Source: TheVerge