Has Your Website Been Deleted? It May Not Be Hackers at Work… but a Blundering Web Host

It’s one of the more famous quotes about computer programming:

To err is human, but to really foul things up requires a computer.

 

Popular web-hosting firm 123-reg is regretting a catastrophic error it made this weekend, which saw it “effectively delete” an unspecified number of its customers’ websites during what should have been a routine maintenance operation on virtual private servers (VPS).

An email sent by the company to affected customers explained what went wrong:

“As part of a clean-up process on the 123-reg VPS platform, a script was run at 7am on 16.04.16. This script is run to show us the number of machines active against the master database. An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'Failure' scenario - showing no VM's and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore.”

In all, 67 of 123-reg’s servers were affected by the chaos-inducing “clean-up process” – hitting an unknown number of customer websites.

As BBC News reports, victims of the accidental website deletion included top Scottish football team Ross County, preventing fans from buying tickets and merchandise.

Days later, and things aren’t looking that much brighter for firms who were hosting their sites with 123-reg.

An update posted on 123-reg’s website reveals that the company is attempting to recover customer data with third-party data recovery software, a process which cannot always guarantee positive results:

“Our system engineers are testing multiple third party data recovery software and we are definitely seeing some results. As this is referring solely to data recovery, the next thing on our list is actually reimaging the hosts with the data we have retrieved (aka getting the VPS packages back online).”

“The reimaging process will take some time as well (we're provisioning new hosts, on new servers), however, things will get back on track for part of the customers pretty soon.”

Understandably, customers’ are expressing their frustration publicly on social media, and the web host’s reputation has been severely damaged.

Of course, it should go without saying that it was a human who wrote that computer script – so we shouldn’t completely blame our CPU-brained friends.

But the real problem here is not that a script was programmed to delete business websites without first suspending them into a 28-day holding bay or having a human eye doing a quick double-check before data was wiped, but instead that automated backups were not in place.

More expensive web hosting plans might have offered automatic backups of  a firm’s website data.

But even with that, you have to still trust that your web hosting provider is backing up your data properly.

All companies need to ask themselves if they are at risk of putting too much trust in their service providers and how they would recover if a catastrophe occurred that could effectively turn their online presence into a “Website not found” error message.

It’s easy to be wise with hindsight, but I wonder how many of those companies who have (for now at least) lost their web presence will fail to make their own backups in future.

In short, it may be prudent to do your own backups, so if the worst ever happens you have a chance of recovering.

Remember this – we are all on the lookout for threats from external attackers and malicious hackers, but sometimes data destruction doesn’t have to be the result of internet criminals or a ransomware attack, but instead with the very people you are paying to provide a professional service.