The simple typo that brought down Amazon Cloud

Tuesday, 7 March, 2017

Human error was at fault when a web storage system powered by Amazon Cloud wobbled and fell over at the end of February, the internet giant has admitted.

Around 150,000 websites and top apps use Amazon Web Services, including filesharing on business messaging service Slack, the reader-led question and answer service Quora, and the GIF image search engine Giphy.

Considered (rightly or wrongly) to be more secure than traditional hosting services, it now appears the cloud is just as fragile and open to error as any other operation.

Amazon Cloud ‘unkillable?’

But it wasn’t a hack attack or malicious code that brought down Amazon’s £10bn-a-year service.

No, it was an “input” error by an Amazon employee.

In a series of tweets when the huge service started struggling with outages on 28 February 2017, Amazon detailed how S3 – their ‘Simple Storage Service’ – “was experiencing high error rates”.

The simple typo that brought down Amazon Cloud

The simple typo that brought down Amazon Cloud 1

The simple typo that brought down Amazon Cloud 2

In a statement published on their own website Amazon later admitted that an “authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process.”

“Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.”

With healthcare apps and interconnected home devices relying on the Amazon cloud network, outages like this could cause havoc if mistakes go unchecked.

Advice: Back up, back up and back up again

Amazon apologised for the error and said it would monitor its systems to make sure that commands entered incorrectly could not bring down web services.

The statement continued: “While we are proud of our long track record of availability with Amazon S3, we know how critical this service is to our customers, their applications and end users, and their businesses.

“We will do everything we can to learn from this event and use it to improve our availability even further.”

It’s yet another reminder that no system is infallible.

If you’ve got information or personal files that you just can’t bear to lose, you should be backing them up and making copies to store in a number of locations; whether that’s online, in the likes of cloud storage service Google Drive, or in a more traditional offline external hard drive.

Tom Rodgers author picture

By:

Tom is a tech journalist and former Editor at BroadbandDeals.co.uk.

News What's the story?

Keep up with the latest developments in UK broadband.

Why is the UK worried about Huawei and what does this mean for 5G?

The Huawei controversy continues as Boris Johnson says we ‘will not risk British security”.

Why is the UK worried about Huawei and what does this mean for 5G?Why is the UK worried about Huawei and what does this mean for 5G? Read more

Unmissable streaming TV shows for 2020

Telly has moved online - let's dive in to the most bingeworthy releases due this year!

Read more

The most complained about home broadband services

Who's failing to live up to expectations?

Read more

Brits spend up to 4.9 hours surfing the web at work!

Read more

Help Learn with us

Make the most of the internet with our broadband library.

What are the differences between HD, UHD, 4K and 8K?

What are the differences between HD, UHD, 4K and 8K?What are the differences between HD, UHD, 4K and 8K? Read more

How far can augmented reality take us?

Read more

Keeping the Internet of Things secure

Read more

The best broadband routers for 2020

Read more