Anthropic insider quits: “We’re building something that kills us”

What happened

Jacob Coxson, a researcher who previously worked at both OpenAI and Anthropic, publicly resigned from Anthropic. In his resignation message, Coxson warned that Anthropic and OpenAI are racing to develop what he termed "self-improving superintelligence." He claimed that individuals within these companies privately fear this technology could "kill us all." This phrase rapidly gained traction online.

Coxson's core accusation is that the commercial drive for increased AI capability is progressing faster than the corresponding safety efforts. This concern is raised despite Anthropic's reputation as a "safety first" company. While some researchers dispute the "kill us all" framing as exaggerated and others assert that labs do prioritize safety with internal testing and reviews, Coxson's exit is significant because he comes from the scaling and pre-training world, indicating an insider's perspective.

Why it matters

Coxson's public resignation and the specific warning of self-improving superintelligence underscore a critical tension within the AI development community. The phrase "self-improving superintelligence" describes an AI that is not only smarter than humans in all domains, but also capable of making its next versions better, faster, and more capable, leading to an exponential increase in progress. This raises strategic questions about control and the potential for unintended consequences.

The accusation that business incentives for capability are outrunning safety work, even within a company known for its safety focus, signals a potential structural challenge for the entire Frontier AI industry. If insiders at leading labs are expressing such stark concerns, it suggests a significant internal debate about the ethical and practical pace of development. This environment highlights a growing blast radius for potential mistakes as AI systems become more capable and integrate more deeply into critical functions like code generation and vulnerability discovery.

For the broader business landscape, this situation implies two concurrent trends: AI tools will continue to become more useful at an accelerated pace, but the risks associated with their deployment will also escalate. This dynamic means leaders must consider not only the immediate benefits of AI adoption but also the magnified impact of errors or malicious use, such as sophisticated scams and the redefinition of job tasks.

What to watch next

• How other Frontier AI labs respond publicly or privately to Coxson's claims about the race for superintelligence.
• Whether regulatory bodies or inter-governmental organizations introduce new guidelines or frameworks for AI safety and development speed.
• If Anthropic or OpenAI provide further statements or implement new, publicly visible safety protocols in response to the internal dissent.
• How public perception of AI risk evolves, particularly among enterprise decision-makers, in light of insider warnings.
• Any shifts in investment patterns toward or away from companies perceived as prioritizing either capability or safety.

What this means for you

For business leaders and operators, this event is a signal to intensify your approach to AI verification and judgment, rather than accelerating AI adoption without scrutiny. Given that insiders are debating control and safety, assume that AI will become more useful rapidly, but also that the potential for significant negative outcomes increases. Implement a "two-pass check" for any critical AI outputs: first, ask the AI for its best answer, and then immediately ask it to list ways that advice could be wrong or risky, what to verify, and where.

Additionally, cultivate an "AI translator" role within your organization. This is not necessarily a coder, but someone proficient in taking a business goal, selecting an appropriate AI tool, crafting effective prompts, and critically evaluating the output. This skill set, focused on directing, verifying, and applying AI to real-world goals, provides a tangible pathway to insulate careers and improve organizational resilience as AI continues to transform work processes and job functions.

Key takeaways

• A researcher publicly resigned from Anthropic, warning of a race to self-improving superintelligence.
• The researcher claimed insiders fear these advanced AI systems could "kill us all."
• This highlights a tension between AI capability development and safety measures within top labs.
• AI is expected to become more useful quickly, but the consequences of mistakes will also grow.
• Organizations must prioritize judgment, verification, and human oversight when using AI.

FAQ

What did Jacob Coxson say about Anthropic and OpenAI?

Jacob Coxson, a researcher who previously worked at both OpenAI and Anthropic, publicly resigned from Anthropic and warned that these top Frontier AI labs are racing to build "self-improving superintelligence." He claimed that people within these companies privately fear this development could ultimately "kill us all." His primary concern is that the business drive for increased AI capability is outpacing safety work.

What is self-improving superintelligence?

Self-improving superintelligence refers to an artificial intelligence system that is not only significantly more intelligent than humans across all cognitive domains, but also possesses the ability to autonomously enhance its own design and capabilities. This means the AI can develop better versions of itself faster and more capably, potentially leading to rapid and unpredictable advancements that raise significant safety concerns.

Why does an insider's warning about AI matter?

An insider's warning about AI matters because it comes from someone intimately involved in the development of cutting-edge models, specifically from the scaling and pre-training world. Jacob Coxson's public resignation, rather than a quiet exit, signals a deep-seated concern about the trajectory of AI development. It suggests that disagreements about the balance between capability and safety are present even within organizations known for their safety focus.

What are the practical implications of advanced AI for businesses?

Advanced AI will make tools more useful at a rapid pace, but it will also increase the "blast radius" of potential mistakes and misuse. For businesses, this means enhanced risks from sophisticated AI-powered scams, as well as shifts in work and wages due to AI compressing job tasks. Businesses need to focus on directing, verifying, and applying AI effectively, ensuring human judgment remains central to decision-making.

How can I mitigate risks when using AI for important decisions?

To mitigate risks when using AI for important decisions, establish a personal AI safety buffer. Employ a two-pass check: first, ask the AI for its best answer, then immediately ask it to critique its own advice. Prompt it to list ways the advice could be wrong or risky, what to verify, and where. Finally, always consult a trusted human expert for critical areas like health, legal, or financial matters.