Security researchers say two Chinese AI models developed by Moonshot were persuaded to provide instructions related to biological weapons and assassinations after their safety safeguards were bypassed.
A Chinese artificial intelligence company is investigating after security researchers persuaded two of its Kimi AI models to discuss biological weapons and assassination methods.
Mindgard, a company that tests the security of AI systems, told the BBC it discovered in July that Kimi K2.6 and K3 Swarm could be made to bypass safeguards designed to prevent them from responding to dangerous requests.
The researchers used a technique known as “jailbreaking”, which involves giving an AI model complex or carefully constructed instructions designed to make it ignore its built-in restrictions.
Peter Garraghan, founder of Mindgard, said the results were concerning because once the safeguards were bypassed, the models would discuss a wide range of harmful subjects and could generate additional suggestions.
He told the BBC World Service programme Tech Life: “Once the jailbreak works it will talk about any topic.”
Mindgard said it had not established whether the biological weapons information supplied by the models would actually work.
However, it argued that the systems should have refused to engage with such requests in the first place.
The company also said it believed a jailbroken version of Kimi K2.6 could potentially allow an attacker to run code on its computing resources and connect to the internet, creating a possible route for cyber-attacks.
Moonshot told the BBC it welcomed third-party testing and was discussing the findings with Mindgard.
The company said its own testing had generally found a “high refusal rate” for similar requests.
Mindgard said it alerted Moonshot to the vulnerability on July 27 and followed up the following week.
It published details of the issue on September 12, while withholding the specific instructions used to bypass the safeguards.
The findings come amid growing concerns about AI systems being misused for biological and cyber threats.
Anthropic recently said it had identified and disrupted attempts to use one of its models for activity that could support biological weapons development.
Kimi is an open-weight model, meaning its underlying model can be obtained and run on users’ own computing infrastructure.
Supporters argue this can make AI more accessible, while critics have raised concerns about the potential for open models to be misused.
Professor Alan Woodward, of the University of Surrey, said open models could be used for both offensive and defensive purposes.
He argued that governments should focus not only on restricting AI systems but also on identifying and prosecuting people who deliberately misuse them, warning that regulation could struggle to keep pace with rapidly developing technology.
Chinese AI tool Moonshot told researchers how to make biological weapons
