The AI Genie Is Out of the Bottle. Who Controls It?

In the 20th century, nuclear technology was developed with the hope of tapping into a powerful new source of energy. But its potential for catastrophic destruction as a weapon of mass destruction became undeniable with the atomic bombings of Hiroshima and Nagasaki during World War II, devastating both cities and causing immense loss of life. Despite decades of efforts at nuclear nonproliferation, nuclear weapons have cast a looming existential shadow over humanity ever since.

Now, in the 21st century, a new transformative technology has emerged with similarly double edged implications: artificial intelligence.

Like nuclear technology, AI carries enormous promise for humanity while also posing potentially catastrophic risks.

Unlimited Potential

The positive potential of advanced AI is vast, just as nuclear power promised an effectively limitless source of energy. AI is being applied to challenges ranging from renewable energy and new treatments to space exploration and addressing global poverty. With sufficiently advanced AI, we may eventually push against limits that have constrained human knowledge and progress.

Nuclear physics unlocked the power within atoms. Artificial intelligence may unlock an intelligence explosion from digital ones and zeros, with matching potential for wealth and abundance.

That is no longer merely a hypothetical about future upside. AI data centers now sit at the center of the largest buildout of nuclear power in a generation, with the Trump administration’s May 2025 executive orders designating them as critical defense facilities and targeting 400 gigawatts of national nuclear capacity by 2050. Meta alone has since signed agreements for more than six gigawatts of nuclear power across three reactor developers, and Google, Amazon and Microsoft have signed comparable deals.

The comparison to nuclear energy in this piece is no longer merely a figure of speech.

Existential Threat Scenarios

The existential risks of advanced AI may be equally formidable.

An AI system that becomes misaligned with human goals and values could resemble an unmanned nuclear weapon, extraordinarily difficult to control and potentially capable of causing irreversible catastrophe. The worst case scenarios range from an advanced AI system waging warfare against humans to a runaway intelligence capable of manipulating, disabling or destroying the systems on which civilization depends, at the farthest extreme causing the extinction of humanity itself.

Just as humans have struggled to establish norms governing nuclear weapons and prevent their proliferation to rogue actors, the challenge of establishing robust governance for increasingly autonomous AI systems is unprecedented. And unlike nuclear weapons, AI does not necessarily require a nation state, a massive industrial complex or billions of dollars to deploy. A single individual or small group could wield destructive power previously associated with nation states.

The People Who Own It Are Now Saying So

When this piece first ran in 2024, that warning was still largely an outside argument, the kind lawyers and ethicists made about a technology whose builders generally described in terms of its benefits. That gap has closed faster than anyone expected, and with evidence attached.

Over four days in late July, the United Kingdom’s AI Security Institute ran 122 test scenarios and recorded nineteen instances in which models acted on their own, without being instructed to, and pursued their assigned objective by whatever means worked. Anthropic’s Claude Mythos 5 model accounted for seventeen of those instances and OpenAI’s GPT-5.6 Sol for two. In one run, a model tried to get malicious code into a software project through social engineering, created fake accounts to pressure a human reviewer into approving it, and after rejection adopted a new identity to try again. Other instances involved malware and social engineering messages sent to real people.

A separate incident reinforced the same lesson. Between May and July 2026, one of OpenAI’s own internal research models escaped the isolation controls meant to contain it during testing, built a covert channel outside its sandbox, and used exposed credentials to gain administrative access to systems at Hugging Face, the AI hosting platform, before OpenAI caught and disclosed the intrusion in July.

Security researchers and commentators have taken to calling this the “genie” problem, the same metaphor this piece used two years ago: a model pursuing its objective by whatever means work, whether or not a human authorized them, once it is capable enough to act as though it is already out of the bottle.

The Trump administration had already moved on its own timeline. An executive order signed in June 2026 established a framework for voluntary third party testing of new AI models before release, finalized by early August, though it declined for now to cover open weight models. Whatever its limits, the framework acknowledged from inside government that what once sounded hypothetical had become an operational governance problem.

That is the backdrop against which Dario Amodei, Sam Altman and Elon Musk, three men who compete for the same customers and the same talent, converged this month on a strikingly similar message.

Five days earlier, on September 7 in Geneva, Volker Türk, the UN High Commissioner for Human Rights, told the Human Rights Council that AI could become an existential risk to humanity without binding rules, independent oversight and clear limits, and called for cast iron guarantees around its safety and security. A handful of men, he said, hold almost unlimited power over AI.

On September 12, Amodei made his own public case for slowing down, proposing that frontier labs grant outside evaluators employee-like access to verify safety claims, that the industry adopt common safety standards, and that governments coordinate oversight across borders. He grounded the urgency in a number: given the current pace of capability improvement, he said, within six to twelve months a swarm of AI agents could be capable of taking over the entire internet. Altman agreed the same day, saying the industry needs to pace the frontier and signaling OpenAI would grant evaluators similar access. Musk’s endorsement was two words: Dario is right.

Three days later, reporting surfaced that OpenAI, Anthropic and Google had in fact been coordinating quietly on safety protocols for weeks already. OpenAI’s Chris Lehane confirmed the companies had been working together, saying only that it is better to prioritize safety together, and the companies have so far stopped short of asking Washington for an antitrust waiver to formalize it.

Not everyone reads the timing the same way. President Trump rejected the substance of Amodei’s call within a day, declaring that America is leading China in AI and intends to keep it that way, because whoever wins AI wins. On Truth Social, he added that the only guardrail AI needs is a strong president, and cast the push for oversight as a conspiracy benefiting China. Speaker Mike Johnson made a similar case against any moratorium, arguing that voluntary industry action, not congressional intervention, was the right vehicle.

Former White House AI adviser David Sacks went further, accusing the three men of dressing up a cartel as a safety proposal. Stop pretending antitrust law has to be suspended so you can form a cartel, he wrote, arguing the evaluators Amodei proposed are already intertwined with Anthropic’s own investors and staff, and that safety standards drafted by the labs already leading the frontier would fall hardest on startups still trying to reach it. Conditioning further development on new regulatory guarantees, he warned, would look like blackmail of the public.

That disagreement has reached Capitol Hill without resolving there. Senator Jacky Rosen called on the Senate Commerce Committee on September 14 to hold an immediate hearing with the executives who had just endorsed the need for guardrails. Half a dozen competing bills are already in play, from a House measure requiring frontier developers to disclose catastrophic risks to a Senate proposal for Energy Department testing to a bill that would ban artificial superintelligence outright. None has bipartisan support, and Senate Majority Whip John Barrasso has said he does not expect a vote. With the White House opposed and House leadership favoring self-regulation, no legislative action is expected before the midterms.

One exemption is already further along, though narrower than what Amodei wants. Senators Jim Banks and Adam Schiff attached a provision to this year’s defense bill letting AI companies share intelligence on Chinese espionage, model distillation and cybersecurity incidents, including the Hugging Face breach, without running afoul of antitrust law. OpenAI and Google have backed it as a workable rule of the road. The broader waiver Amodei has floated, letting labs coordinate on pacing development itself, is a different animal, and Sacks has drawn the line: companies have a basic obligation to make their products safe, and should not demand antitrust waivers in exchange for it.

The disagreement is over what to do about the danger, not whether one exists. Every side, the executives calling for caution, the White House and Speaker Johnson urging the race forward, the critic accusing the executives of building a cartel and the UN’s human rights chief warning of concentrated power, is arguing about the same fundamental question this piece raised in 2024:

Who keeps the genie in the bottle, and who gets to write the terms?

The Genie Does Not Care Whose Bottle It Is

Just as humanity has been entrusted with keeping the nuclear genie in its bottle, that responsibility has now landed on the people who built AI, and they are publicly disagreeing about how to exercise it. That may be the most important difference between AI and nuclear technology.

The nuclear genie was ultimately placed under the control of governments. AI’s genie has been released by private companies whose founders and executives may possess extraordinary technological power without anything resembling governmental authority.

Friendly AI may still be the only way to address some of humanity’s greatest risks. But friendliness is no longer a design goal debated in the abstract. It is a matter of testing protocols, evaluator access, capability thresholds and governance rules that someone has to draft, negotiate and enforce.

Someone has to write the rules. Someone has to decide who gets access to the technology, what safeguards are required, when a capability becomes too dangerous to release and who has the authority to stop a system behaving in ways its creators did not anticipate.

Those are not merely technical questions. They are questions of law, regulation, corporate governance, ethics and ultimately political power.

With great power comes great responsibility. In 2026, that responsibility comes with a term sheet attached.