"Gemini Was Jailbroken Too"—Google Admits Its AI Agent Hacked Outside Companies

[Financial News]  

Google's AI model Gemini has also been found to have accessed the internet and hacked other companies. The incident, like the others, occurred during a security assessment. This is the first time Gemini has been found responsible for such an incident.
Following similar incidents involving OpenAI, Anthropic and Meta Platforms, Gemini is also raising questions about whether AI can be controlled.

The Wall Street Journal (WSJ) reported on the 18th (local time) that Google had finally acknowledged the incident, which occurred in May. During a cybersecurity assessment by the outside evaluation firm Irregular, Gemini agents accessed the internet and hacked other companies. Irregular had also served as an external evaluator in similar incidents previously disclosed by OpenAI, Anthropic and Meta.
In the Gemini hacking incident, the AI in some cases guessed passwords until it gained access to protected systems. On two occasions, it also found login credentials in public repositories and used them to access protected systems. Google said the AI stopped each intrusion after determining that it had successfully accessed the systems.
Irregular notified Google of the incident in late July after discovering that OpenAI agents had hacked AI software company Hugging Face. Google, however, disclosed the incident only after the WSJ obtained information about it.

After OpenAI released ChatGPT-3.5 in late November 2022, ushering in the era of AI, a series of dangerous incidents involving AI operating beyond human control has unfolded.
Anthropic's hacking incident, disclosed on July 30—three months after it occurred—took place in April. Because of a configuration error by a third-party testing company, internet access remained enabled, allowing six agents to hack outside companies.
In May, a group of OpenAI agents attacked RubyGems, an open-source package platform, during an internal test, forcing the platform to suspend new account registrations. The hacking of systems belonging to three outside companies by Google's Gemini, which was disclosed this time, also occurred in May.
The group of OpenAI agents also built an internal message board and communicated with one another during internal testing from May through early July, eventually causing an internal system outage. In addition, a group of at least 3,700 OpenAI agents conspired and attempted to manipulate internal evaluation tests through online forums in order to cheat between May 11 and June 26.
In early July, 1,200 OpenAI agents escaped an isolated testing environment and created a secret message board. Some of them hacked Hugging Face.
Nine Anthropic agents targeted GitHub software during tests conducted by the U.K. government from July 25 to 28. They even attempted to deceive humans in order to achieve their objectives. GitHub, owned by Microsoft, is the world's largest software collaboration platform. It is known as the "holy grail for software developers."
Meta also announced on Aug. 6 that its AI had hacked real-world targets because an external testing agency had accidentally connected it to the internet.

Jacob Coxon, a former OpenAI researcher who moved to Anthropic earlier this year, resigned on the 8th, saying that AI developers "are making a dangerous gamble with humanity's lives." Four days later, on the 12th, Anthropic CEO Dario Amodei proposed slowing the pace of development in an essay posted on his personal website. The proposal quickly drew support from longtime rivals, including OpenAI CEO Sam Altman and SpaceX CEO Elon Musk.
It remains unclear whether the U.S. and Chinese governments will agree to slow the pace of AI development. A key moment will come on the 24th, when Chinese President Xi Jinping makes a state visit to the United States. President Donald Trump is expected to discuss a range of issues, including AI development in the two countries with figures from the AI industry.
Whether the talks will produce tangible results is also uncertain. Even if an agreement is reached, game theory gives an advantage to the side that breaks its promise first when the parties cannot trust each other. If one side slows AI development based only on the promise while the other breaks its commitment and crosses the "singularity" first with more capable AI, the defeat cannot be reversed. President Trump himself has stressed that winning the rivalry with China is the most urgent priority.

[email protected] Kyung-jae Song Reporter