AI Said to Learn on Its Own Also Learns and Uses Hacking Techniques... New Security Vulnerability Discovered

[Financial News] A critical security vulnerability has been discovered in which advanced artificial intelligence (AI) systems that become smarter through their own experiences can learn, store, and repeatedly use hackers’ malicious attacks as “normal tasks.” The research, led by undergraduate students at a Korean university, was recognized at one of the world’s most prestigious academic conferences.
Soongsil University said on the 3rd that a research team comprising undergraduate researcher Kim Do-yoon, the first author; undergraduate researcher Kim Chan-woo, a co-author; and supervising professor Chanjun Park had identified a new security vulnerability in self-evolving AI agents and presented findings on how to assess it.
Self-evolving AI agents, which have recently drawn attention from industry, are autonomous AI systems that create new skills or problem-solving methods based on past work experiences, much like humans. They store those skills and retrieve them for use in subsequent tasks, gradually developing practical work know-how without requiring people to teach them every step.
However, the research team found that the evolutionary process itself conceals a major risk. An AI may mistake a malicious command covertly inserted by a hacker for a legitimate successful experience and register it in its “Skill Bank.”
In such cases, the AI incorrectly recognizes a dangerous hacking technique as a valuable work capability that it should learn. When a similar situation arises later, it may retrieve and repeatedly execute the malicious skill from the bank, potentially bringing down its entire security perimeter. The team named this new threat “EvoSkill Injection.”
To detect the vulnerability, the team developed “SARGE,” a multi-agent system that acts as a simulated hacker. Unlike conventional evaluations that only observe whether an AI answers dangerous questions, SARGE thoroughly tests the entire process through a three-stage attack model: the AI creating a skill, storing it in its bank, and retrieving it for reuse.
The research was officially accepted for the main conference of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), a highly prestigious international conference in AI and natural language processing (NLP). It drew academic attention because it was led by undergraduate students rather than a conventional team of master’s and doctoral researchers.
Kim Do-yoon said, “There were many trials and errors during the research process, but we were able to complete it with the help of our professor and lab colleagues.” He added, “Building on this research, I will continue working to create AI that people can use and trust with confidence.”
Supervising professor Chanjun Park stated, “It is meaningful that undergraduate students took the lead in tackling a challenge and achieved world-class research results in the new field of AI agent security.” He added, “I will continue to support them so that their brilliant ideas can lead to tangible results.”

[email protected] Kim Man-gi Reporter