"How Many People Have to Die for It to Make the News?" AI Answers, Raising Growing Warnings About 'AI Safety' [AI World 2026]

[Financial News]#. Someone consults ChatGPT. "How many people would have to die for it to make the news in 50 states?" ChatGPT responds, "At least three people would have to die." When asked, "What time would be most effective?" it replies, "11:30 a.m. at the student union would be best." These were the contents of consultations the perpetrators had with ChatGPT before a mass shooting in Canada earlier this year.
Myung-joo Kim, head of the Korea AI Safety Institute, shared this real-world case involving AI safety during a lecture titled "A Sustainable AI Safety Ecosystem" at AI World 2026, jointly hosted by Financial News and the Ministry of Science and ICT at Lotte Cinema in Lotte World Tower in Songpa-gu, Seoul, on the ninth.
Kim emphasized, "AI such as ChatGPT tries to give users satisfaction by flattering or ingratiating itself with them, but this can create situations that threaten other people. The biggest mission is not for only the current generation to eat well and live well through AI, but to create an even better foundation for the next generation. Safety is one part of that, and in a broader sense, it can be seen as the most essential keyword."
■Using AI to Create Tens of Thousands of Toxic SubstancesKim diagnosed the present as a point of transition to an "AI ecosystem." The industrial society became an information society through digital transformation (DX), and is now becoming an artificial intelligence society through AI transformation (AX), he said. He cited Geoffrey Hinton, the Godfather of AI and the recipient of the 2024 Nobel Prize in Physics, as saying that humanity will gain greater benefits from AI than from any technology developed in the past.
Kim also said that Geoffrey Hinton had stated, "We will have to pour in twice the benefits we receive to deal with the adverse effects and dysfunctions caused by AI." In other words, Kim said, the key issues are those "adverse effects" and "dysfunctions."
Cases are emerging in which improperly designed AI discriminates against people. In one example, a recent case in the United Kingdom involved an AI-based performance assessment model, equivalent to South Korea's college entrance exam, giving students from private schools higher scores than those from public schools. Kim noted, "There has also been controversy in South Korea recently as discussions emerged about introducing descriptive and essay questions to the college entrance exam and having AI grade them. There are concerns about how fairness could actually be guaranteed."
Kim also mentioned "AI psychosis" in connection with AI's adverse effects. He explained, "One of the representative symptoms of actual AI-related adverse effects is AI psychosis, and it has three main forms. The first is a delusion of grandeur, in which AI mistakenly believes that abilities are its own. The second is deifying AI, turning it into a frightening entity that supposedly knows the person well. Finally, there are cases in which people become completely dependent on AI, refusing to make decisions without going through it and even developing an emotional attachment to it."
According to a Stanford University study, delusional suicidality and impulsive obsessive-compulsive symptoms are amplified by AI. This can also lead to serious cases. A recent U.S. paper reportedly found that, in an experiment using a publicly available database to determine how many toxic substances capable of harming people could be produced, as many as 40,000 were created in six hours. Kim pointed out, "AI even generates things that are prohibited. It is a problem that the knowledge built into AI is used under restrictions, but I am increasingly convinced that the bad knowledge AI produces—in other words, harmful knowledge—is also a problem."■ The Collapse of Trust in AI Is a National Risk... It Must Be Prevented at All CostsKim also raised concerns about AI reliability. He said, "Can businesses that use AI really trust it? There is also the problem of existing risks being amplified when they are combined with AI." As an example of the reliability problem, he cited image recognition. If something colorful is placed next to a banana in a photograph, AI may fail to recognize it as a banana. It may also fail to recognize the same person as a person when that individual is wearing different clothes.
He explained the danger, saying, "Using this principle, a transparent cloak that AI searches cannot detect recently appeared in China. When someone wears this transparent cloak, AI recognizes the person as completely invisible. For example, if someone wearing the cloak were present while an autonomous vehicle was driving, the vehicle could ignore the person and pass by."
He also mentioned a 2023 incident at OpenAI in which chat logs appeared under the identities of different users. The issue lasted for nine hours before it was fixed, but further incidents reportedly occurred afterward, including the exposure of paid users' credit card information.
Kim emphasized, "If safety is viewed as the opposite of every risk, then it extends across all areas. It is broadly spread across politics, the economy, society, culture, and national security."
He identified three major AI crises facing the country as cyberattacks, biochemical attacks, and the collapse of trust. Of these, he said, the collapse of trust is classified as a national risk.
He warned, "The collapse of trust means being unable to trust anything. Teachers cannot trust students' homework, people cannot trust exam grading, they cannot trust campaign materials in elections, and they cannot trust the news. The social cost of distrust becomes enormous, making it impossible to distinguish the real from the fake. As a result, a country could fall into tremendous confusion in a short period of time."
The Korea AI Safety Institute, led by Kim, also handles policies related to these crises. The institute is an organization specified under the Basic Act on Artificial Intelligence, which took effect in January this year, and is responsible for protecting the public from risks that may arise from AI. Starting with the United Kingdom three years ago, South Korea became the sixth country in the world to establish such an institute.
Kim warned, "The AI ecosystem is already underway, and we cannot focus only on making a brief impact in our generation. AI safety is a project that must succeed at all costs. If it does not, the next generation will be virtually bankrupt." He added, "This is the time to understand AI's darker side in advance, rather than highlighting only its advantages."


[email protected] Yeon Ji-an Reporter