On May 31, Nicolas Beauvais, a French programmer, sparked an uproar by posting a series of tweets accusing a cloud provider of killing their company.

He is the CTO of Raisup, a two-person micro-AI start-up, against Digitalocean, an American cloud-hosting service with more than 300 employees. In 2015, DO was the second-largest cloud hosting service in the world, behind AWS.

1 automation script can kill a company?

What is this situation?

According to Nicolas's post, we have the following:

background

RAISUP bought virtual machines and spaces at DO, where web apps, workers, caches, databases, and backups are located.

Nicolas has written a Python automation script that is executed regularly every 2-3 months. To improve data processing efficiency, he executed scripts in parallel on multiple virtual machines, with 500,000 rows of data in about a second. This data pipeline is closed after execution.

1st blocked

On May 29, Raisup received an email from DO, who thought the Python automation script might be a malicious program and blocked all access to RAISUP's corporate account.

This also means that the RAISUP website is dead, the service it provides is dead, and the backup of the website is gone! Raisup is dead!

The first time unsealed

After communicating with DO many times, DO unblocked Raisup account.The Raisup website was down for 12 hours before normal access was restored.

Because all the virtual machines were shut down and the Redis memory to record progress was wiped, Nicolas had to restart the data pipeline,

Second Seal

However, the good times did not last long, and it was blocked again 4 hours later, most likely because of that script.

Over the next 30 hours, RAISUP sent four e-mails, but only received an automated response from DO, refusing to unseal them. If your company is inexplicably blocked, 30 hours is a long time to be anxiously waiting

无助的程序员，原本是到国外休假

Nicolas cries, “We're a two-person company, and I'm the only programmer. Today is the first day of my two-week vacation. When I arrived in Portugal and received the final reply from Do, I felt hurt and helpless. "

“We lost everything to our servers and, more importantly, a year of database backups. We now have to explain to our clients, Fortune 500 companies, why we can't restore their accounts. "

Nicolas thinks DO should at least allow him to download backups and save the company.

Do apologize

Response from DO official Twitter:

We hear and understand your concerns and apologize for the way this has been handled. We have restored the accounts and are conducting a thorough investigation into the incident. We will publish the survey results to provide full transparency to our customers and the community.

DO 的联合创始人在推特上回应：

The account should be reactivated and unblocked.We need to get to know how this is handled.It shouldn't take that long to recover the account, and it shouldn't be flagged twice.

The CTO of DO responded on Hacker News:

As CTO of DigitalOcean, I am very sorry for this situation and how it was handled.The account has now been fully restored and we are investigating the incident.We are planning to publish a public report that will provide full transparency to our customers and community.This happens due to false positives caused by our internal fraud and abuse system.While this is rare, it does happen and we make every effort to get our customers back online as quickly as possible.In this particular scenario, we were slow to react and made mistakes in handling false positives.This results in users being locked out for a long period of time.We apologize for our mistake and will share more details in the public report.

The tragedy triggered the support and sympathy of many colleagues

Nicolas' tragic experience, in addition to generating a lot of attention on Twitter on May 31.June 1 was also hotly debated on Hacker News.

Freemind:

That sounds dangerous. Can't DO come out and solve the problem? This is the first time I've heard of such a thing. We have been using DO for many years.

Lyndon Fawcett shared a similar experience:

The same thing happened to me a few years ago.DO cannot be trusted! I lost everything after entering my account from the admin panel.Their support team is dangerous and vulnerable to social engineering.

Some gave Nicolas directions to other cloud providers, while others said they would no longer use DO.

Netizen Davis suggested that if you want to store the backup with another service provider, you don't need to rely on only one service provider.

Don't put all your eggs in one basket!

Finally, to paraphrase programmer Jon Brown:

Even a startup with only one person should have some other offsite backup.Not afraid of what if, just ten thousand.Remember, never put all your eggs in one basket.

Here I believe that there are many small partners who want to learn Python, I myself am a Python old programmer who has been engaged in development for many years, resigned and is currently doing my own private customized courses, at the beginning of this year I spent a month to sort out a most suitable learning dry goods for 2019, from the most basic to, all kinds of frameworks have been sorted, to every Python partner, want to get can follow me and message me privately in the background: learn, you can get it for free.Life is short, I use Python!