The simple typo that brought down Amazon Cloud

Tuesday, 7 March, 2017

Human error was at fault when a web storage system powered by Amazon Cloud wobbled and fell over at the end of February, the internet giant has admitted.

Around 150,000 websites and top apps use Amazon Web Services, including filesharing on business messaging service Slack, the reader-led question and answer service Quora, and the GIF image search engine Giphy.

Considered (rightly or wrongly) to be more secure than traditional hosting services, it now appears the cloud is just as fragile and open to error as any other operation.

Amazon Cloud ‘unkillable?’

But it wasn’t a hack attack or malicious code that brought down Amazon’s £10bn-a-year service.

No, it was an “input” error by an Amazon employee.

In a series of tweets when the huge service started struggling with outages on 28 February 2017, Amazon detailed how S3 – their ‘Simple Storage Service’ – “was experiencing high error rates”.

The simple typo that brought down Amazon Cloud

The simple typo that brought down Amazon Cloud 1

The simple typo that brought down Amazon Cloud 2

In a statement published on their own website Amazon later admitted that an “authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process.”

“Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.”

With healthcare apps and interconnected home devices relying on the Amazon cloud network, outages like this could cause havoc if mistakes go unchecked.

Advice: Back up, back up and back up again

Amazon apologised for the error and said it would monitor its systems to make sure that commands entered incorrectly could not bring down web services.

The statement continued: “While we are proud of our long track record of availability with Amazon S3, we know how critical this service is to our customers, their applications and end users, and their businesses.

“We will do everything we can to learn from this event and use it to improve our availability even further.”

It’s yet another reminder that no system is infallible.

If you’ve got information or personal files that you just can’t bear to lose, you should be backing them up and making copies to store in a number of locations; whether that’s online, in the likes of cloud storage service Google Drive, or in a more traditional offline external hard drive.

Tom Rodgers author picture

By:

Tom is a tech journalist and former Editor at BroadbandDeals.co.uk.

News What's the story?

Keep up with the latest developments in UK broadband.

Hyperoptic welcomes new investment from KKR.

Leading global investment firm acquires the majority stake of UK ISP Hyperoptic after they announce plans to quadruple fibre coverage.

Hyperoptic welcomes new investment from KKR.Hyperoptic welcomes new investment from KKR. Read more

Post Office officially the UK’s cheapest broadband deal

The latest victory in the price wars goes to Post Office with an amazing introductory offer for just over £15 a month!

Read more

Plusnet price hike warning

Incoming broadband and phone price rises before Christmas

Read more

Openreach trials new full-fibre tech

New deployment methods could make rollouts cheaper and faster.

Read more

Help Learn with us

Make the most of the internet with our broadband library.

The top online scams affecting Millenials – and how to avoid them!

Cybercrime is becoming more sophisticated every year - here are the current most frequent scams affecting millennials!

The top online scams affecting Millenials – and how to avoid them!The top online scams affecting Millenials – and how to avoid them! Read more

Is your faster broadband fast enough?

Read more

The differences between uploads and downloads

Read more