MSIT Releases Detailed Scores in K-LLM Evaluation Controversy... SK Telecom Takes First Place

[Financial News] The Ministry of Science and ICT (MSIT) has released the detailed scores from the second evaluation of its Independent AI Foundation Model Project (K-LLM). The move came after Motif Technologies, which was eliminated as the lowest-ranked of the four elite teams, formally objected and asked for the detailed evaluation results to be made public. Motif received the highest score in the Artificial Analysis Intelligence Index (AAII), but posted the lowest score in the benchmark assessment based on the National Information Society Agency (NIA) criteria. It also ranked last in both the expert review and the general public review.
On the 27th, MSIT released the detailed results of the second-stage evaluation for the K-LLM project. The ministry had already disclosed the main results twice, on the 18th and 20th, but appears to have decided to make the detailed scores public as well in an effort to ease growing controversy after Motif's objection.
Earlier in the day, Motif said in a statement that it had filed an objection with MSIT over the second-round K-LLM evaluation results. In its appeal, Motif requested that the criteria and itemized results for the expert and user evaluations be disclosed and that the results be reviewed again. The concern appears to be that, after MSIT explained Motif's elimination by saying it scored highly only in the benchmark assessment while receiving low marks from experts and users, the company could be misunderstood as having usability problems in future market competition.
The second-round K-LLM evaluation consisted of 40 points for benchmark testing, including 25 points for AAII and 15 points for NIA, 35 points for expert evaluation, and 25 points for user evaluation. Motif scored 47 points in AAII, the global AI model performance indicator, the highest among the participating models. Upstage scored 37 points, SK Telecom 35 points, and LG AI Research 31 points. According to Motif, Motif 3 ranks 10th globally among model companies and first outside the United States and China.
However, the detailed scores released by MSIT show that Motif ranked last in every major category.
In the NIA benchmark assessment, the SK Telecom elite team ranked first with 13.4 points, followed by the Upstage elite team with 13.3 points, the LG AI Research elite team with 12.8 points, and the Motif elite team with 12.7 points.
Motif's elite team also came in fourth in the expert evaluation with 27.1 points. The LG AI Research elite team ranked first with 29.5 points, followed by the SK Telecom elite team with 29.3 points and the Upstage elite team with 29.1 points.
In the AI expert user panel evaluation, which involved 49 AI startup CEOs, the SK Telecom elite team ranked first with 11.6 points. The LG AI Research elite team came in second with 11.3 points, the Upstage elite team ranked third with 10.8 points, and the Motif elite team ranked fourth with 8.4 points.
In the general public evaluation, which involved 185 participants, LG AI Research ranked first with 7.6 points. The SK Telecom elite team followed with 7.5 points, the Upstage elite team ranked third with 7.3 points, and the Motif elite team placed fourth with 5.7 points.
In the overall score, the SK Telecom elite team ranked first with 70.6 points, followed by the Upstage elite team with 69.9 points, the LG AI Research elite team with 69.0 points, and the Motif elite team with 65.8 points.
[email protected] Jang Min-kwon Reporter