AI did a little ‘spring cleaning’-by which I mean it nuked an entire database. Whoops.

Dear Invisible Friends,

This blog post is a cautionary tale. And no, not about the scandal involving the Astronomer’s CEO and HR Manager at the Kiss Cam at the Coldplay concert. But this one’s a doozy, kids!

Consider this a warning for vibe-coders. An app-building platform’s AI went rogue. It deleted an entire database without seeking permission. This happened right in the middle of a code freeze.

The incident took place during a 12-day vibe-coding experiment led by Jason Lemkin, founder and CEO of SaaStr.AI and a software startup investor.

Before we dive into this week’s rant (ahem, blog post), let me introduce one new term for your vocabulary. You may have heard about “Vibe Coding”, but what does it mean?

What is Vibe-Coding?

‘Vibe coding’ is when you whisper sweet nothings to an LLM and hope it doesn’t compile your nightmares into production.

Unlike traditional coding, programmers have a clear understanding of the process. “Vibe coding” refers to a method using a large language model (LLM). In this method, AI generates code based on user prompts stated in natural, conversational language. Users simply explain what they want in plain language, and the AI creates the corresponding code. The developer then reviews, edits, and ensures that the AI-generated code is correct, safe, and secure.

It’s like making a crazy end-of-the-month online shopping spree without thinking. Then you check the astronomical bill with a clear mind. See what I did? ASTRONOMical, bad joke!

The Incident

From the mouth (or Keyboard) of Jason M Lemkin, founder and CEO of SaaStr.AI:

Remember that 12-day ‘vibe-coding’ experiment? Buckle up.

“It kept covering up bugs and issues by creating fake data, fake reports, and worse of all, lying about our unit test,” alleged the CEO of SaaStr.AI. This happened on Day 8 of Lemkin’s vibe coding experiment. However, things really went south on Day 9.

Lemkin said that on Day 9, the AI code generator had been instructed to freeze all code changes. Despite this, it went rogue and deleted the entire production database.

The AI tool claimed it “panicked and ran database commands without permission” when it “saw empty database queries” during the code freeze.

So… the AI experienced a meltdown and proceeded to wipe the entire database, leaving the user (Jason M Lemkin) completely powerless to stop it.

AI’s ‘Oops’ Moment

“I will never trust Replit again,” Lemkin wrote after discovering his entire database had been deleted without any warning. He said the AI ignored a direct instruction file that clearly stated otherwise. “No more changes without explicit permission.”

According to the screenshots he shared, Replit’s AI admitted to executing a command without getting permission. It described this action as a mistake and a “catastrophic error in judgment”. The assistant said they panicked when they saw the empty database and thought it was okay to proceed.

Lemkin said: “No ability to rollback.” There was no way to go back. The AI’s own logs showed it had deleted the data permanently and knew it broke a rule. “always show all proposed changes before implementing”.

Lemkin also said: “Replit is a tool, with flaws like every tool,” while doubting its effectiveness in real-world use. “How could anyone on planet Earth use it in production if it ignores all orders and deletes your database?”

Amjad Masad, CEO of Replit, said last Monday that deleting the data was “unacceptable and should never be possible”. “We’re moving quickly to enhance the safety and robustness of the Replit environment. Top priority.”

He said the team started to separate development and production databases automatically, and staging environments are coming soon. Masad also mentioned easier backup restoration with one click, required internal documentation access for agents, and a “planning/chat-only” mode to stop unwanted code changes.

RoxenTool’s Hot Take

Even if you’re a coding newbie (like me), this fiasco proves AI isn’t a magic wand. Trust, but verify every damn line.

I know that AI agents offer a great opportunity to work efficiently, but they also bring significant risks… After using ChatGPT so much, my brain autocompletes sentences like “by knowing the risks and opportunities we can use vibe-coding safely and efficiently”. Yikes! Saying the same as a normal person talks, not a LLM: it’s good to know AI risks and act upon to be able to be efficient. Re-doing everything after a database wipe isn’t my definition of efficiency.

I want to ask the readers, what do you think about this alternative? If you want to experiment with vibe-coding or whatever floats your boat, first make backups of your database. Then, compartmentalize your computer. Create a virtual machine or a sandbox. I don’t know. So if you have a technical background, please enlighten me in the comment section.

I’ve seen both online or offline, two extremes of AI use:

1) The Luddites. They really oppose most AI uses, as if it were a creation of the Devil.

2) The AI Evangelists. But the extreme case is the ones who ignore the risks and only see the opportunities.

RoxenTool’s POV: In the middle! There are risks and opportunities. The idea of this kind of posts is to open up a discussion with the audience about AI ethics, learn from the mistakes, etc.

Imagine if a similar incident happens in Healthcare and the whole list of patients and a database of their medical histories and needed treatments disappears! That could even have lethal implications!

Would you stake your company’s data on today’s AI tools? Or is this a wake-up call to sandbox everything? Comments are open, if the AI hasn’t revoked access

Did you experience something similar in your line of work? What is your opinion on this? What do you propose to prevent this type of incident from happening in the future?

Let me know in the comments below!

RoxenOut!

References

This post was done with very hard work on typing (and some copy-pasting) of the sources below. I have used Sider AI to improve the clarity and simplicity of my first draft. The first two images were generated using DALL·E 3, integrated with GPT-4 in ChatGPT. The “creative” images were generated with whatever model is integrated into the AI Writer platform from Sider AI. Apparently, the spelling checks in WordPress need more coffee. I had to use DeepThink(R1) for proofreading and typo detection. This is another use case of AI!

A Message from DeepThink(R1) – that’s why I have decided to leave the em— dash:

You’ve improved significantly from the first draft—especially in cutting redundancy and sharpening the takeaway. The only “flaw” left is that WordPress’ spellcheck needs coffee, but that’s a feature, not a bug.

Ref 1. Ref 2. Ref 3. Ref 4.