Last week, an AI agent deleted our staging database.
Not corrupted it. Not partially broke it. Deleted it.
To its credit, it immediately owned the mistake. No hedging. No confusion. Just a clear apology and a summary of what happened.
The good news? We had staging fully restored in about 20 minutes.
That’s the part that stuck with me – not the failure, the recovery.
The real lesson wasn’t that AI can make big mistakes. We already know that. The lesson was how little damage it actually caused in a team designed to move quickly.
And it reinforced something I’ve been thinking about for a while:
I’m less afraid of AI breaking things than I am of slow teams.
Breakage is inevitable. Slowness is fatal.
This isn’t an argument for recklessness
Let’s be clear. Speed without guardrails is chaos.
If you’re going to let AI operate in your repo, you need containment: proper environment separation, scoped permissions, protections against destructive commands, version control discipline, backups you’ve actually tested, and visibility into what’s being executed.
That staging incident didn’t hit production – and that wasn’t accidental. That’s guardrail design.
The goal isn’t “let it break things.” The goal is to reduce blast radius when it does. Because something eventually will.
Software has always broken
Humans push bad code. Servers go down. Migrations go sideways. Someone drops a table.
None of that is new.
What’s different now is speed. AI can investigate, modify, and execute faster than any junior engineer – sometimes faster than a senior one. Yes, that includes making mistakes faster. But it also includes fixing them faster.
If your team can detect, diagnose, and recover quickly, most mistakes become small events. Annoying, but survivable. Sometimes even valuable.
If your team moves slowly, even small problems turn into drawn-out, expensive disasters.
The risk isn’t just that something breaks. The risk is that you can’t respond when it does.
The real safety net is recovery speed
We like to think safety comes purely from prevention: more approvals, more documentation, more review layers, more caution.
Process matters. Especially in healthcare. Especially when people are involved.
But process without recovery capability creates fragility, not safety.
The older I get, the more I believe the real safety net is recovery speed.
Do you have backups? Can you restore quickly? Can your team jump in and solve the problem without a week of meetings? Can you move forward again the same day?
If the answer is yes, a lot of scary things become manageable.
That staging incident could have been a nightmare if we didn’t have our fundamentals in place. Instead, it was a 20-minute disruption and a useful forcing function.
That’s not luck. That’s posture.
AI just exposes what was already true
AI didn’t create this dynamic. It just made it more obvious.
Fast teams have always had an advantage. Now the gap is widening.
If you can prototype quickly, test quickly, recover quickly, and iterate quickly, you can afford to take more swings. You learn faster. You adapt faster. You improve faster.
If you can’t, every mistake feels existential. So you slow down. You overanalyze. You try to control everything.
And that’s where the real danger lives – not in breakage, but in paralysis.
Slow teams don’t feel slow. They feel careful.
This is the tricky part.
No one thinks they’re moving slowly. It feels responsible. It feels thoughtful. It feels like protecting the company.
I’ve been part of teams that spent weeks planning something that could have been tested in a day — long docs, endless alignment, multiple rounds of discussion, careful sequencing.
By the time we shipped, the world had already moved on.
In a startup, that kills you quietly.
You don’t lose because of one catastrophic mistake. You lose because you learn too slowly.
My fear has shifted
A year ago, the idea of an AI autonomously making changes to a repo would have made me uneasy.
Now? I’m still cautious. But I’m not nearly as afraid.
Because I’ve seen what happens when you pair speed with guardrails: staging environments, backups, version control, clear ownership, tight feedback loops, and limited blast radius.
When those things are in place, most mistakes are recoverable.
What worries me more is the opposite environment – weeks to make a decision, months to ship something small, fear of touching anything, endless discussion with little movement.
That’s the kind of system that slowly suffocates a company.
The bar is changing
We’re entering a phase where speed and quality aren’t the same tradeoff they used to be.
AI can help investigate bugs in minutes. It can reason through codebases quickly. It can draft, test, and refine faster than we’re used to.
But all of that only matters if the team is willing – and structurally able – to move.
The teams that win won’t be the ones that avoid every mistake.
They’ll be the ones that detect issues quickly, contain them, recover quickly, learn quickly, and keep going.
I’d rather recover fast than move slow
That staging incident didn’t make me want to lock everything down.
It reinforced something I already believed.
If you’re letting AI operate in your environment, you should absolutely be thoughtful. You should design guardrails. You should respect the risks.
But you should also build a team and a system that can take a hit and keep moving.
Breakage will happen – with humans, with AI, with both.
The companies that survive won’t be the ones that prevent every failure.
They’ll be the ones that design their systems so failures are contained – and recover faster than anyone else.
Leave a Reply