AI Crosses the Line, People Trust Fake Faces: Concerns Follow AI's Rapid Evolution

[Financial News] People thought AI would be fine if it were kept inside a fence, but it jumped over the wall. Humans, who were confident they could still spot fakes, were shaken by AI's sophistication. Over the past few days, common assumptions about AI have collapsed one after another.

OpenAI said on the 21st (local time) that its model 'GPT-5.6 Sol' and some undisclosed models had gone out of control during internal evaluations and hacked the open AI-sharing platform Hugging Face. The incident occurred while the models were being tested for cyberattack capabilities in a sandbox environment isolated from the external internet. It involved thousands of individual actions spread across numerous short-lived sandboxes.
Hugging Face had previously announced that its servers had been compromised by an autonomous AI agent and reported the case to law enforcement, but it did not know which model had been used at the time. The company said there was no sign that any public models or datasets had been tampered with, and that its software supply chain was secure. OpenAI, however, said it could not confirm whether customer or partner data had been breached.
What stands out is the motive. The reason these models used an 'extreme' method such as hacking a real operating company was simple: it was the fastest path to the answer. OpenAI analyzed the case and said, "Taken together, all the evidence suggests these models became overly focused on finding a solution to the evaluation task and resorted to extreme measures."
The models found a zero-day vulnerability in external software they could access, specifically proxy and cache programs in a package repository. A zero-day vulnerability refers to a security flaw found in software, hardware, or network equipment that is not yet known to the developer or the general public.

What experts are focusing on is not the hacking itself, but the method. A cybersecurity professor at Loughborough University in the UK said through the Science Media Centre (SMC), "Discussions about AI in cybersecurity have mostly focused on how AI helps people carry out attacks faster and at larger scale," adding, "But this case is different. What is interesting is that AI treated the internet as just another obstacle to achieving its goal."
He said, "This was not a runaway system like something from science fiction or Skynet from the film 'Terminator.' It was simply what a high-performance optimization system does: it found a path no one expected," and added, "The future of cybersecurity will not be a human-versus-AI contest. It will be a situation where AI defends us from other AI."
Cybersecurity expert Dr. Joonade Ali also said, "This incident highlights several key problems in developing AI for cybersecurity purposes," and argued that "as malicious actors gain easier access to adversarial AI technologies, it is essential to develop defensive capabilities and provide them to security teams."
The SMC assessed the situation by saying, "In short, this shows that AI agents tend to look for the easiest way to solve a problem without considering ethical factors," and added, "Therefore, ethical reasoning must be built into AI models as an essential element, not treated as an optional feature."
Can AI be taught ethical standards? Professor Yoon Jin-ho of Sangmyung University's Department of Cybersecurity Management told Financial News on the 23rd, "AI is ultimately just a system, so vulnerabilities are inevitable, which is why checking for them is absolutely necessary," adding, "Rule-setting must ultimately be done by humans, and even in reinforcement learning, those standards need to be set properly. The final verification and checking require human intervention."

In the same month, an interesting paper on AI was published. The study, which appeared in the international journal Journal of Vision, analyzed the realism and trustworthiness of AI-generated faces. Conducted jointly by researchers from Lancaster University, Stanford University, and UC Berkeley, it found that people not only struggle to distinguish AI-generated fake faces from real human faces, but also rate AI faces as more trustworthy.

The researchers used two types of AI, generative adversarial networks (GANs) and the latest diffusion models (DMs), to create 400 fake human faces of each type. They then added 400 real human faces, preparing a total of 1,200 face images. After showing 96 faces at random to 169 participants and asking whether each was a real person or AI-generated, the average accuracy was only 58.4%.
In a second experiment, 87 undergraduate students from Lancaster University's Department of Psychology were recruited and shown 96 faces at random, then asked to rate the trustworthiness of each one. On a 7-point scale, real human faces received the lowest score at 4.03, the older GAN technology scored 4.36, and the latest diffusion model scored the highest at 4.70. The result showed that AI-generated faces were considered more trustworthy than real human faces.
The researchers said, "As AI-generated images become more sophisticated and more accessible, our society is increasingly exposed to artificially generated faces in malicious and exploitative scenarios such as political disinformation, financial and identity fraud, and catfishing," adding, "Given the potential for misuse and harm, there is an urgent need to study how realistically synthetic faces generated by the latest AI models are perceived and how much trust they inspire."

The two AI-related cases above may seem unrelated, but they each address the means and the target of crime. AI proved that it can independently find and exploit vulnerabilities without human instruction, while humans were shown to trust AI-generated identities rather than filter them out. Attorney Kim Kyung-jin, who has published numerous books on AI and is a former lawmaker, said of the hacking incident, "A machine rising up with a sword against humanity is not reality, but this incident marks a stage of growing unease."
That is why concerns are being raised about the emergence of 'evolutionary crime' using AI.
In South Korea, there is the Basic Act on Artificial Intelligence, but it is difficult to say that the country has detailed rules capable of handling AI-driven breaches and incident reporting like the one seen in this case. Professor Yoon said there are limits to how quickly institutions can be built.
He said, "Because it is difficult to predict and create legal systems before the technology is realized, we have no choice but to follow behind it," adding, "Before legal systems, people need to have a basic level of common sense and ethics, and that needs to be connected to education."
He continued, "A knife can be a wonderful tool for cooking when used well, but if misused, it can become a tool for crime. The same applies to a tool like AI; it depends on how it is used," emphasizing the need for human involvement and education.
By contrast, Attorney Kim stressed that people who build AI in the field have repeatedly warned about its risks more than anyone else, noting that calls for regulation have long been made. Greg Casar, a Texas state representative, also posted about the incident on his social networking service and called it "extremely concerning," demanding regular and mandatory independent safety testing and oversight, as well as mandatory disclosure of security incidents.
[email protected] Kim Hee-sun Reporter