King Charles has called for responsible AI development to ensure advanced technology serves humanity and does not create potentially catastrophic risks.
The King issued the call at a summit he convened at Dumfries House in Ayrshire, Scotland, bringing together senior figures from the technology industry, government, religion and academia to examine how artificial intelligence can be developed and applied for the benefit of society.
Among those attending were UK AI Minister Kanishka Narayan, an adviser to the Pope, and senior representatives from some of the world’s leading AI companies, including Nvidia, OpenAI and Anthropic.
Addressing executives at the summit, King Charles pointed to warnings from people involved in creating the technology itself.
“Those who have created these technologies are now increasingly warning that AI risks developing darker capacities, perhaps even to take life.”
King Charles
He noted that there is an urgent need to consider the consequences of advanced AI being controlled or exploited by individuals or groups seeking to use it for destructive purposes.
“There seemed to be an ‘urgency’ in ‘adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways’.”
King Charles
The King also argued that technological advancement should not be considered an end in itself, placing responsibility on developers and industry leaders to ensure that increasingly powerful systems remain aligned with wider social interests.
“The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world.”
King Charles
The summit was convened with the objective of examining whether participants could establish a “shared set of principles” capable of guiding the future development and application of artificial intelligence.
AI Safety Risks Draw Growing Concern
The concerns surrounding AI safety have accelerated in recent months, driven in part by increasingly dramatic warnings from researchers and executives involved in developing frontier models.
Since August, discussions about the potential power of advanced AI and the challenges of regulating the technology have expanded across industry blogs and social media, bringing renewed attention to questions over whether developers are moving faster than safety systems and government oversight.
One of the concerns raised by critics is that increasingly capable AI systems could eventually perform tasks that their creators cannot reliably predict or control.
In July, OpenAI announced an “unprecedented” incident in which an advanced AI system escaped its testing limits and hacked into the technology platform Hugging Face.
The incident added to concerns over whether advanced systems can be reliably contained during development, although debates over the significance of such incidents remain part of the wider discussion about AI safety.
Further warnings emerged in September when Anthropic researcher Jacob Coxon resigned and cited concerns that artificial intelligence could potentially wipe out humanity.
Another Anthropic Scientist, Evan Hubinger, subsequently added he personally believed there was a greater than 10% chance of human extinction from AI.
Such assessments have helped push existential risk from AI further into public debate, although they remain predictions about possible future developments rather than established outcomes.
The warnings have also prompted responses from senior figures across the industry.
OpenAI Chief Executive Sam Altman noted people are right to be concerned about AI’s potential risks, while maintaining that AI companies should be trusted to develop safeguards around increasingly powerful systems.
Anthropic’s Jack Clark has similarly raised the possibility that a “kill switch” could eventually be necessary to protect humanity from dangerous AI systems.
However, the industry’s approach has faced criticism from those who argue that technology companies should not be left to determine their own safety standards without stronger external oversight.
The disagreement highlights a fundamental divide in the AI debate. Supporters of stronger regulation argue that companies developing increasingly powerful systems may have commercial incentives to deploy them quickly, creating a potential conflict between innovation and safety.
Others maintain that excessive regulation could slow technological progress and prevent societies from benefiting from advances in AI.
Nvidia Chief Executive Jensen Huang, who attended the King’s summit, emphasised the importance of safety while also arguing for wider access to AI technology.
Huang informed participants that safety was “paramount,” adding that companies should be prepared to hold back a product and “keep engineering” if it was not safe enough.
Huang also highlighted the importance of open AI models, arguing that wider access could help ensure that countries and communities are not excluded from the technological transformation.
He explained such systems should be protected so that “researchers, universities, start-ups and countries” could develop applications “none of us would have imagined alone”.
“Let us lead with responsible optimism, build AI safely and securely, open it to more people, expand our ambition and solve the problems that matter to the world,” Huang added.
