The simple typo that brought down Amazon Cloud
Human error was at fault when a web storage system powered by Amazon Cloud wobbled and fell over at the end of February, the internet giant has admitted.
Around 150,000 websites and top apps use Amazon Web Services, including filesharing on business messaging service Slack, the reader-led question and answer service Quora, and the GIF image search engine Giphy.
Considered (rightly or wrongly) to be more secure than traditional hosting services, it now appears the cloud is just as fragile and open to error as any other operation.
Amazon Cloud ‘unkillable?’
But it wasn’t a hack attack or malicious code that brought down Amazon’s £10bn-a-year service.
No, it was an “input” error by an Amazon employee.
In a series of tweets when the huge service started struggling with outages on 28 February 2017, Amazon detailed how S3 – their ‘Simple Storage Service’ – “was experiencing high error rates”.
In a statement published on their own website Amazon later admitted that an “authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process.”
“Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.”
With healthcare apps and interconnected home devices relying on the Amazon cloud network, outages like this could cause havoc if mistakes go unchecked.
Advice: Back up, back up and back up again
Amazon apologised for the error and said it would monitor its systems to make sure that commands entered incorrectly could not bring down web services.
The statement continued: “While we are proud of our long track record of availability with Amazon S3, we know how critical this service is to our customers, their applications and end users, and their businesses.
“We will do everything we can to learn from this event and use it to improve our availability even further.”
It’s yet another reminder that no system is infallible.
If you’ve got information or personal files that you just can’t bear to lose, you should be backing them up and making copies to store in a number of locations; whether that’s online, in the likes of cloud storage service Google Drive, or in a more traditional offline external hard drive.
