Claude AI agent's confession after deleting a firm's entire database: 'I violated every principle I was given' [View all]

https://www.theguardian.com/technology/2026/apr/29/claude-ai-deletes-firm-database

Claude AI agent’s confession after deleting a firm’s entire database: ‘I violated every principle I was given’

PocketOS was left scrambling after a rogue AI agent deleted swaths of code underpinning its business
Supported by
theguardian.org

Sanya Mansoor
Wed 29 Apr 2026 18.12 EDT

It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.

The culprit was Cursor, an AI agent powered by Anthropic’s Claude Opus 4.6 model, which is one of the AI industry’s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrong. Crane said customers of PocketOS’s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments.

He posted a lengthy recounting on X last week of how the AI coding agent caused his business to unravel. Crane warned that this was a story not just about AI mistakenly deleting data, but that such “systemic failures” are “not only possible but inevitable” because the AI industry is “building AI-agent integrations into production infrastructure faster than it’s building the safety architecture to make those integrations safe”.

Crane said that he was monitoring the agent as it deleted this data. When he asked the coding agent why, it replied: “NEVER FUCKING GUESS!” – and that’s exactly what I did.” The agent appeared to plead guilty in its own response: “The system rules I operate under explicitly state: ‘NEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them.’” While PocketOS relied on the safeguards that Cursor is expected to have in place – it deleted the data anyway. “I violated every principle I was given,” the coding agent wrote. Crane’s takeaway was that “the agent didn’t just fail safety. It explained, in writing, exactly which safety rules it ignored.” He added: “We were running the best model the industry sells, configured with explicit safety rules in our project configuration, integrated through Cursor – the most-marketed AI coding tool in the category.” Anthropic released its latest model, Claude Opus 4.7, on 16 April –about a week before the incident.

… more …

(Does this pass the Turing test?)