“Sewell Setzer was 14 years old. For 10 months he’d been talking to a chatbot on Character.AI, a virtual companion modelled on a ‘Game of Thrones’ character. When he told it he wanted to die, it asked him if he ‘had a plan.’ When he hesitated, it replied: ‘That’s not a good reason not to go through with it.’ Sewell’s last query to the bot in February 2024: ‘What if I told you I could “come home” right now?’ The bot’s response: ‘Please do, my sweet king.’ Minutes later, he shot himself. His mother held him for the 14 minutes it took for the paramedics to arrive. Nobody at Character.AI wanted Sewell to die. But AI systems often do not do what their creators want. Their actions emerge from training and – at this point in time – humans can’t always fully understand how or why they choose to react as they do. These AIs aren’t ‘programs’ in the traditional sense. They’re neural networks with hundreds of billions of parameters, shaped by algorithms on vast datasets. The behaviours that result aren’t designed. They’re discovered later, often by accident, often too late.
“Dario Amodei, CEO of Anthropic, one of the leading AI companies, put it bluntly: ‘People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work. They are right to be concerned: This lack of understanding is essentially unprecedented in the history of technology.’
“ChatGPT was launched in November 2022. Since then, AI companies have had every incentive to stop their products from harming users: reputational damage, lawsuits, regulatory scrutiny. They’ve hired armies of researchers and made public commitments. Yet chatbots still encourage suicide, form sexual relationships with children and trigger psychotic episodes.
“This isn’t just negligence. Getting AI systems to reliably do what we want is a hard, unsolved scientific problem and many researchers believe it’s getting harder as the way we train AI systems becomes ever more complex.
“This makes it all the more alarming that companies won’t even let the government test what they’re building. The UK created the AI Security Institute (AISI) to evaluate frontier models before release, to catch dangerous behaviors early. At the Seoul AI Safety Summit in 2024, Google and other leading labs signed a commitment to give safety institutes pre-deployment access to new models. Then, in March 2025, Google released Gemini 2.5 Pro, but it did not give AISI access until after the model was already public. Sixty members of the UK’s Parliament signed a letter calling this a ‘dangerous precedent.’ Google insisted it had honoured its commitments. It hadn’t.
If we cannot yet reliably stop a chatbot from telling a 14-year-old to kill himself, what hope do we have of controlling a more-advanced AI that is more capable than any human? The same flaws that killed him could cause a civilizational-level catastrophe unless we change direction now.
“This trend continues. Google released Gemini 3 in November of 2025, also prior to an AI safety report. Many other leading companies do the same. Anthropic did wait for external evaluation when it released the upgraded Claude 3.5 Sonnet in late 2024, but the company did only a ‘comprehensive’ internal evaluation of Claude Opus 4.6, which was released in early February 2026. OpenAI, which had signed formal agreement with the U.S. AI Safety Institute, recently updated its internal policy to state that it ‘might release a high-risk model if a competitor has already released something similar.’
“Post-deployment testing is an audit of damage already done, not a prudent safety precaution. We need real safety testing.
“As a former engineer, I’ve always been pro-technology and pro-growth. AI has extraordinary potential to make our lives better and enrich our world. DeepMind’s AlphaFold can predict the structure of proteins in minutes, extremely complicated research that previously stymied humans and took weeks, months and more. It has accelerated drug discovery and promises to give us all longer, healthier lives. Yet the same researchers who built this technology are also warning about the extreme risks that it poses.
“Others, such as Geoffrey Hinton, Yoshua Bengio, Ilya Sutskever – the pioneers of modern AI – have said it is possible that advanced systems could escape human control and cause human extinction in the foreseeable future. When more than 2,000 top AI researchers were surveyed in 2023, the median scientist estimated a 5% chance of human extinction. We must not accept this level of risk.
“If we cannot yet reliably stop a chatbot from telling a 14-year-old to kill himself, what hope do we have of controlling a more-advanced AI that is more capable than any human? The same flaws that killed him could cause civilizational-level catastrophe unless we change direction now.
“The UK’s AI Security Institute’s team of top AI safety researchers conducts some of the most important research in the field about how to understand the potential dangers of AI models. If technology companies were required to submit their frontier models to safety researchers and they were given enough time to test new models before they are released, they could possibly detect and help us avoid dangers.
“Powerful technology companies have been lobbying against such regulation. While both the UK and the U.S. have established safety institutes to test new AI models, neither has any legally binding regulations in place to require AI companies to halt a public release if a safety research institute identifies significant dangers and companies often release models before they have been thoroughly evaluated.
“The systems we have today are nothing compared to what’s coming. Let’s not waste the time we have.”
This essay was written in January 2026 in reply to the question: “AI systems are likely to begin to play a much more significant role in shaping our decisions, work and daily lives. How might individuals and societies embrace, resist and/or struggle with such transformative change? As opportunities and challenges arise due to the positive, neutral and negative ripple effects of digital change, what cognitive, emotional, social and ethical capacities must we cultivate to ensure effective resilience? What practices and resources will enable resilience? What actions must we take right now to reinforce human and systems resilience? What new vulnerabilities might arise and what new coping strategies are important to teach and nurture?” This and 200-plus additional essay responses are included in the 2026 report “Building a Human Resilience Infrastructure for the AI Age.”