King Charles Warns Tech Bosses of ‘Existential Dangers’ of AI

By Elizabeth

King Charles III has warned the world’s leading artificial intelligence executives that the technology poses “existential dangers” if it falls into the wrong hands, urging them to find ways of controlling AI “before it is all too late.”

The monarch made the comments on Thursday while opening a summit he convened at Dumfries House in Ayrshire, Scotland. The gathering brought together senior figures from AI companies, government, and charities to discuss how the technology can be developed and deployed in ways that benefit society.

The King’s decision to convene the summit reflects his long-standing interest in existential risks facing humanity, most notably climate change and the environment. His entry into the AI debate is perhaps unsurprising given that pattern, though it represents a notable foray into one of the biggest current issues in technology and public policy.

The summit was organized with the assistance of the Ditchley Foundation, a charity that convenes discussions on major global issues. About 30 technology executives, policymakers, and ethics researchers attended the private gathering, according to Buckingham Palace.

A Warning of ‘Darker Capacities’

In his opening address, King Charles told the audience that the development of AI is “both intriguing and deeply concerning in equal measure.”

“AI is already showing its immense capacity to improve and save life—for example, in the field of life sciences and medicine,” he said. “Yet, those who have created these technologies are now increasingly warning that AI risks developing darker capacities—perhaps even to take life.”

The King said there seemed to be an “urgency” in “adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways”.

“Surely, we need sufficient means of control before it is all too late?” he asked.

His remarks were imbued with Christian symbolism, with the King speaking of the soul and humanity’s moral component. This echoed recent interventions from religious leaders, including Pope Leo XIV, who said in an encyclical published in May that AI must not remain in the hands “of a few”.

‘Our Humanity Must Remain Sacred’

The monarch, who has rarely spoken about AI before, called on the executives to ensure that as they develop the technology, “our humanity remains sacred”.

“Those in our world who value deeply our humanity and its vital moral component are anxiously seeking your reassurance that we will not lose control of our destiny, or our souls,” he said.

He added that “the decisions taken at this formative time will shape the world inherited by future generations,” and that “the task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world”.

Who Attended the Summit

The summit at Dumfries House, the Scottish headquarters of the King’s Foundation charity, was attended by bosses from Nvidia, Google DeepMind, OpenAI, and Anthropic. Nvidia chief executive Jensen Huang was among those present, along with the UK’s AI Minister Kanishka Narayan and the chief adviser to the Vatican on the ethics of AI.

Google DeepMind co-founder Demis Hassabis, OpenAI chief financial officer Sarah Friar, and senior representatives from Anthropic were also reported to be in attendance.

The King only attended the summit for about 20 minutes before leaving to allow for what he said he hoped would be “productive” discussions. British royals are constitutionally bound to remain above politics, meaning Charles can spotlight problems but not necessarily suggest ways to solve them. His role at the summit was that of a convener rather than an actor.

A Growing Debate

The King’s intervention comes amid an intensifying debate about the pace and risks of AI development. In recent weeks, a series of warnings have emerged from within the industry itself about the rapidly evolving capabilities of advanced or “frontier” AI models and the regulation needed to keep them in check.

On Thursday, the chief executive of Microsoft AI, Mustafa Suleyman, told the BBC that if AI systems are created that can act autonomously, define their own objectives, and run businesses, “we’re essentially seeding a new silicon species which will no doubt compete with us for resources”. He suggested that this approach, which he attributed to Anthropic, was “mistaken and misguided”.

Anthropic researcher Evan Hubinger recently warned that he believed there was a greater than 10% chance AI could pose a species-ending risk within 10 years. In a post on X, he wrote: “We do genuinely believe that AI could kill all humans! I personally think it’s >10% within the next 10 years. I believe Anthropic is trying its best, but we haven’t solved superintelligence alignment and aren’t clearly on track to.”

Hubinger’s comments came in response to the resignation of his colleague Jacob Cockx, who said leading companies in the industry were “gambling with our lives”.

Nvidia Chief Strikes Optimistic Note

Nvidia chief executive Jensen Huang struck a somewhat different tone at the summit, emphasising the potential benefits AI could bring to humanity.

Speaking to reporters, Huang said AI companies had a “responsibility” to test technology thoroughly before releasing it. “There have been incidents and, fortunately, those incidents did not cause harm,” he said. “But those incidents remind us that as we go from lab to product development, companies have to become much more rigorous about testing, about safeguards, about making sure that the product is ready before it’s released. If it’s not ready, hold it. You should go as fast as you can, but not faster than that.”

The Nvidia chief also emphasised the value of open models – systems that give users direct access to the software – as a way of “making sure that people and countries are not left behind”. He said such systems must be protected so that “researchers, universities, startups, and countries” can develop “applications that none of us could have imagined on our own”.

“Let’s lead with responsible optimism, build AI safely and securely, open it up to more people, expand our ambition, and solve problems that matter to the world,” Huang added.

A Consistent Voice of Caution

The King’s comments on Thursday were consistent with his previous public statements on AI. In a taped address at the UK’s AI Safety Summit in 2023, he called on attendees “to protect people’s privacy and livelihoods… to secure our democracies from harm and to ensure the benefits of new technology are shared by all”.

In 2024, at a gathering of Commonwealth heads of government, he referenced the importance of harnessing AI so it “works for our societies, not against them”.

The monarch has made a habit of referencing existential challenges facing humanity, most notably regarding climate change and the environment. His entry into the AI debate is perhaps unsurprising given that pattern, though it represents a notable foray into one of the biggest current issues in technology and public policy.

The Vatican’s Intervention

The Vatican has also weighed in on the AI ethics debate. In his first encyclical, Magnifica Humanitas, Pope Leo XIV directly addressed the power of big tech companies, warning that AI risks deepening inequality, weakening democracy, and eroding what it means to be human.

The 83-page document called for the “disarming” of AI – freeing it from military and economic interests – and for stronger national and international regulation of AI companies, as well as broad participation of individuals and communities in shaping the technology’s future.

“Disarming does not mean giving up on technology, but preventing technology from dominating man,” the Pope wrote. “It is not enough to regulate it; it must be disarmed, it must become welcoming and accessible.”

The Pope also specifically noted the growing danger of a few wealthy individuals influencing humanity’s future and livelihoods, widening “the gap between those who can participate in the digital revolution and those who remain on the margins”.

UK’s AI Governance Gap

Charles’ summit intervention comes as the UK faces criticism over what some observers describe as a governance gap in AI regulation. The government’s 2023 white paper made a deliberate choice not to immediately create new legislation, establish a single dedicated AI regulator, or set cross-sector binding rules.

Instead, it asked existing regulators – the Information Commissioner’s Office, Ofcom, the Financial Conduct Authority, the Competition and Markets Authority, and others – to apply five cross-sectoral principles: safety, transparency, fairness, accountability, and contestability. A central function was established to coordinate and monitor, but these arrangements were not put on a statutory footing.

The principles are not legally binding. A dedicated AI bill has been repeatedly hinted at but not delivered, and legislation is not expected before late 2026 at the earliest. A private member’s bill in the House of Lords in March 2025 proposed a central AI authority to coordinate oversight, but it did not have government support and stood little chance of becoming law.

The result is a regulatory structure that has been described by the Centre for Data Ethics and Innovation as having an “information gap” and a “governance gap”: no consistent, system-wide obligation to monitor AI deployment across the public sector, and no proactive regulation of public sector AI use.

Public Attitudes to AI Regulation

Academic research suggests there is significant public support for stricter safety oversight of AI. A study published in Scientific Reports found that a majority of respondents supported strict safety oversight. Interestingly, sociodemographic factors played only a minor role, while risk preference was one of the factors predicting support.

Another study examined how institutional trust shapes public preferences for AI regulation, finding that trust in government can increase support for both stronger and weaker forms of regulation, depending on the specific context.

What Happens Next

The summit was called to consider whether a “shared set of principles” can be developed to “help guide the future application” of AI. While the King did not stay for the discussions, his opening remarks set a tone of cautious urgency for the gathered executives and policymakers.

Buckingham Palace later shared a group photo of the AI bosses with King Charles, along with a message from the monarch expressing his gratitude for “the thoughtful stewardship demonstrated by those gathered here today as you consider the fundamental principles that should – and must – guide the development of Artificial Intelligence in the years ahead”.

The summit at Dumfries House is part of a broader effort by the King’s Foundation to convene conversations around technology and society. The charity has previously hosted gatherings on topics including sustainable agriculture, traditional crafts, and youth development.

Organisers said the decision to bring together people from different backgrounds was consistent with the King’s long-standing focus on addressing major societal challenges through cooperation and constructive dialogue.

For the AI executives present, the King’s remarks offered a moral framework for their work, even if the practical outcomes of the summit remain to be seen. For the wider public, the intervention placed one of the most consequential technological debates of the era within a royal context, adding a voice of caution to a conversation increasingly dominated by questions of speed, competition, and control.