Facebook blames 'faulty configuration change' for nearly six-hour outage
Facebook blamed a "faulty configuration change" for a nearly six-hour outage
that prevented the company's 3.5bn users from accessing its social media and
messaging services such as WhatsApp, Instagram and Messenger.
In a blog post, the company did not specify who executed the configuration
change and whether it was planned yesterday.
It said: "Our engineering teams have learned that configuration changes on
the backbone routers that coordinate network traffic between our data
centres caused issues that interrupted this communication.
"This disruption to network traffic had a cascading effect on the way our
data centres communicate, bringing our services to a halt.
"We want to make clear at this time we believe the root cause of this outage
was a faulty configuration change. We also have no evidence that user data
was compromised as a result of this downtime."
The platform added it was working to understand more about the outage in
order to "make our infrastructure more resilient".
Several Facebook employees who declined to be named had told Reuters earlier
that they believed that the outage was caused by an internal mistake in how
internet traffic is routed to its systems.
The failures of internal communication tools and other resources that depend
on that same network in order to work compounded the error, the employees
said.
Security experts have said an inadvertent mistake or sabotage by an insider
were both plausible.
The Facebook outage is the largest ever tracked by web monitoring group Down
Detector.
The outage was the second blow to the social media giant in as many days
after a whistleblower on Sunday accused the company of repeatedly
prioritising profit over clamping down on hate speech and misinformation.
As the world flocked to competing apps such as Twitter and TikTok, shares of
Facebook fell 4.9%, their biggest daily drop since last November, amid a
broader selloff in technology stocks on yesterday.
Shares rose about half a percent in after-hours trade following resumption
of service.
"To every small and large business, family, and individual who depends on
us, I'm sorry," Facebook Chief Technology Officer Mike Schroepfer tweeted,
adding that it "may take some time to get to 100%".
"Facebook basically locked its keys in its car," tweeted Jonathan Zittrain,
director of Harvard's Berkman Klein Center for Internet & Society.
Twitter yesterday reported higher-than-normal usage, which led to some
issues in people accessing posts and direct messages.
Twitter Support tweeted: "Sometimes more people than usual use Twitter. We
prepare for these moments, but today things didn't go exactly as planned.
"Some of you may have had an issue seeing replies and DMs as a result. This
has been fixed. Sorry about that!"
In one of the day's most popular tweets, video streaming company Netflix
shared a meme from its new hit show "Squid Game" captioned "When Instagram &
Facebook are down," that showed a person labeled "Twitter" holding up a
character on the verge of falling labeled "everyone".
Facebook, which is the world's largest seller of online ads after Google,
was losing about $545,000 in US ad revenue per hour during the outage,
according to estimates from ad measurement firm Standard Media Index.
Past downtime at internet companies has had little long-term affect on their
revenue growth, however.
Facebook's services, including apps such as Instagram, workplace tools it
sells to businesses and internal programs, went dark at 5pm Irish time.
Access started to return at around 10.45pm.
Soon after the outage started, Facebook acknowledged users were having
trouble accessing its apps but did not provide any specifics about the
nature of the problem or say how many users were affected.
The error message on Facebook's webpage suggested an error in the Domain
Name System (DNS), which allows web addresses to take users to their
destinations. A similar outage at cloud company Akamai Technologies took
down multiple websites in July.
On Sunday,
<https://www.rte.ie/news/world/2021/1004/1250549-facebook-whistleblower/>
Frances Haugen, who worked as a product manager on the civic misinformation
team at Facebook, revealed that she was the whistleblower who provided
documents underpinning a recent Wall Street Journal investigation and a US
Senate hearing last week on Instagram's harm to teen girls.
Ms Haugen is due to urge the same Senate subcommittee today to regulate the
company, which she plans to liken to tobacco companies that for decades
denied that smoking damaged health, according to prepared testimony seen by
Reuters.
What caused the Facebook outage?
Facebook, along with Instagram and WhatsApp, came back online after an
outage took the social media giants offline for several hours late
yesterday.
The <https://www.rte.ie/news/2021/1005/1250758-facebook/> company blamed
the issue on a "faulty configuration change" within its network
infrastructure which had a "cascading effect" that brought the firm's
platforms "to a halt".
Here is a closer look at the incident.
What happened?
Just before 5pm yesterday, people began noticing they could not access
Facebook, or other services it owns and runs like Instagram and WhatsApp.
It would be more than five hours before service began to return.
Service outages on major platforms are not uncommon, but ones of this length
are unusual, and it became clear Facebook was struggling to fix the problem.
In the meantime, other platforms such as Twitter and messaging app Signal
saw huge surges in traffic as people turned to them to get back online, with
some Twitter users even reporting issues at one point as the platform
strained under the weight of the sudden burst of additional users.
By late evening, access to Facebook and Instagram had returned for most
users, while WhatsApp said it was back up and running "at 100%" as of 3.30am
this morning.
What caused the issue?
In a statement, Facebook said the problem had been caused by a configuration
change to the "backbone routers" that coordinate traffic between the firm's
data centres. This caused the cascading effect which brought the company's
various services down.
The company has not yet offered any further insight on what specifically
caused the issue or how it was fixed.
But, web infrastructure and security firm Cloudflare has provided a detailed
breakdown of the incident as it saw it unfold, and said it revolved around
two key mechanisms which make the internet work - Domain Name System (DNS)
and Border Gateway Protocol (BGP).
In essence, DNS is the address book and BGP the roadmap for the internet,
helping people navigate the vast mesh of connected networks that make up the
internet to help them find the website they want and then the quickest route
to it.
Cloudflare said Facebook had, through a series of updates on Monday and
seemingly accidentally, told the BGP that the paths for everything Facebook
runs were no longer there - meaning people could no longer find a way to the
social network.
Experts have said this is most likely to have been caused by a software bug
in the updates or human error, although some have noted Facebook did not
rule out foul play being the cause of the incident in its statement -
however, there is currently no evidence to suggest that that is the case.
Why did it take so long to fix?
It appears that the problem not only took down the social media platforms,
but everything Facebook runs, including its own internal systems - with
reports that staff were locked out of offices as internet-connected keycard
entry systems went down, and were also unable to access their internal
communications platform.
As a result, it was hard for staff to initially diagnose and coordinate on
resolving the problem.
There were even reports in the US of Facebook having to send a team to one
of its data centres to reset the servers manually to fix the issue.
One expert also noted that ongoing social distancing measures because of the
pandemic and remote working may have also played a part.
Software testing expert, Adam Leon Smith of BCS, The Chartered Institute for
IT, said: "It is unlikely the issues were directly caused by people working
from home, however it is quite possible that it took so long to restore the
service because of reduced staffing within the data centre.
"This would compound the problem because the nature of the failure meant
that remote access to the data centre was also unavailable."
Can anything be done to prevent this from happening again?
This latest incident, after the major outages linked to Cloudflare in 2020
and Fastly earlier this year will again highlight the potential problems
with having large portions of the internet reliant on just a handful of
large companies and where one small issue can bring down huge segments of
online services.
There are currently no obvious solutions to this, but this latest outage is
likely to reignite the debate around internet infrastructure.
For many individuals and businesses too, the incident showed just how much
they depend on Facebook and its services not just to communicate, but also
to log in to other platforms.
In response, people have been encouraged to consider using other credentials
beyond their Facebook log-in details to access other online services.
RTÉ Business.
===========================================================
The fb-exchange mailing list
Manage account,
List Page: https://www.freelists.org/list/fb-exchange
Subscribe: mailto:fb-exchange-request@xxxxxxxxxxxxx?Subject=subscribe
Unsubscribe: mailto:fb-exchange-request@xxxxxxxxxxxxx?Subject=unsubscribe
Archive: https://www.freelists.org/archive/fb-exchange
Administrative contact: insight@xxxxxxxxxxxxxxxxxxxx
===========================================================
