"AI Performance Metrics Gain Importance: Customer Satisfaction Over Automation" [Preview of AI World]

[Financial News] "Good performance metrics are not numbers showing how much automation has been achieved. They show whether operations were conducted responsibly while delivering better results to customers."
Oh Soon-young, a senior solutions architect at Amazon Web Services (AWS) who will deliver a keynote speech at AI World 2026, jointly hosted by Financial News and the Ministry of Science and ICT on the 9th, told Financial News in an interview on the 7th that the importance of performance metrics for evaluating AI will grow as increasingly advanced AI agents are used across organizations. Rather than simply measuring whether work has been automated through AI, companies will focus on how well customer requests are handled, using criteria such as completion rates, processing times, errors and exceptions, time spent on human rework, customer satisfaction, revenue contribution and regulatory compliance. Oh emphasized, "Just as you would ask an employee what result they produced, you should examine whether an AI agent actually completed the work properly."
Oh also said organizations need to transform the way they work to make effective use of AI agents. AI agents may take over tasks in fields with abundant digital information and relatively clear procedures, while building relationships and assuming responsibility will remain human roles. Oh said, "A company's competitiveness will come not from the number of AIs it uses, but from how it manages the safe connection and adaptation of the capabilities it needs."
The following is a Q&A with Oh.
—As AI agents become more advanced, in which areas do you expect the greatest changes in human roles?
△AI agents will take on a growing share of tasks such as finding and organizing information, preparing drafts and processing work according to established procedures. Setting goals, judging unexpected situations, building relationships with people and taking responsibility for outcomes will become more important, and these roles are likely to remain with humans. Changes in human roles are likely to emerge first in areas with abundant digital information and relatively clear procedures, such as customer service, software development, finance and insurance, marketing, and finance and legal affairs. AI is less a technology that replaces people than a catalyst for redistributing roles between people and technology.
—How can companies introduce new AI-related performance metrics into their organizations or business models?
△It is no longer enough to evaluate AI solely by how natural its answers sound. Just as you would ask an employee what result they produced, you should examine whether an AI agent actually completed the work properly. Possible criteria include completion rates, processing times, errors and exceptions, time spent on human rework, customer satisfaction, revenue contribution and regulatory compliance. Many organizations still do not have sufficiently clear standards for determining whether AI has succeeded. Amazon uses a method called 'Working Backwards,' which starts with the customer and works backward. AI performance metrics should be developed in the same way. First, select a task that begins with a customer request and ends with its resolution, then measure its current time, cost and quality. Next, determine how much of the task AI will handle and when it should hand the work over to a person, while managing technical performance and business outcomes together. For example, for a customer-service AI, the rate of inquiries resolved on the first interaction, the repeat-inquiry rate and the total cost per inquiry are as important as answer accuracy. Rather than hiding failures, companies should use them for improvement. Good performance metrics are not numbers showing how much automation has been achieved. They show whether operations were conducted responsibly while delivering better results to customers.
—Do you expect AI to create an opportunity to strengthen the competitiveness of startups and small and medium-sized enterprises by making services that once had small markets, or highly personalized services, economically viable?
△That is entirely possible. AI agents reduce not only labor costs but also the time and expense involved in finding information, coordinating multiple people's schedules, conducting analysis and carrying out tasks. Services that were previously difficult to offer because there were too few customers to justify dedicated staff, or too expensive to design for individuals, could become new businesses. Startups and small and medium-sized enterprises can conduct market research, development, customer support, sales-material preparation and overseas expansion planning simultaneously with small teams, allowing them to test ideas more quickly. However, if everyone uses similar AI, it will be difficult to differentiate a business through the model alone. It may be more advantageous to solve a small but costly problem in a specific industry deeply than to offer a general-purpose service modeled on those of large companies. AI will not eliminate all differences in company size, but it will clearly expand opportunities for small organizations to challenge larger markets through expertise and speed.
—How does parallel task processing differ from conventional methods, and why can it improve work efficiency? Please explain with a specific example.
△With AI agents, one goal can be divided into smaller tasks so that research, development, testing and documentation proceed simultaneously, while people review the interim results. Simply increasing the number of AIs does not necessarily make work faster. The goal and completion criteria must be clear, the necessary materials and business context must be prepared, and there must be a way to verify the results. The Amazon Bedrock team completed a rebuild of a core AI system in 76 days with six people, even though the project had been expected to take 30 people 12 to 18 months. Separately, in a pilot project at Amazon's store, the 25 teams that adopted AI tools and new ways of working among more than 50 teams increased software deployment speed by a median of 4.5 times, after accounting for factors such as team size; some teams achieved increases of more than 10 times. This was not simply the result of writing code faster. It resulted from reorganizing work around goals, providing AI with sufficient context, and moving testing and verification to the front of the process. The key to a new way of working is not using more AI, but dividing work effectively, helping AI understand it and verifying the results.
—Standard technologies that enable AIs with different functions to communicate and collaborate are now being adopted in earnest. How will the work environment change at companies with these technical standards in place?
△The advantage of technical standards is that they make it easier to connect different systems. Once standards become established, work will be organized around the goal to be solved rather than around departmental or program boundaries. For example, a sales AI could organize customer information, a finance AI could review pricing terms and a legal AI could check contractual risks. As more AIs become interconnected, identity verification, access management and activity logs will become increasingly important. I believe corporate competitiveness will come not from the number of AIs, but from management systems that can safely connect and change the capabilities a company needs.
—'AI human resources management,' or AI HR, which involves assigning and managing the permissions of numerous AI agents, is also expected to emerge as an important capability for executives. How should companies prepare?
△What executives need is not the ability to introduce a large number of AIs, but the ability to design which goals to delegate, how far to delegate them and how to verify the results. The key is to clearly define what each AI is responsible for, what information and tools it may use, and how far it may make decisions independently. Companies should compile in one place the AIs they use, the tools connected to them and the departments responsible for them. Each AI should have a document, similar to an employee's job description, specifying its purpose, the materials and functions it may use, spending limits, prohibited actions and situations in which it must hand work over to a person. Amazon's operating principle is to clearly define each service's scope of responsibility and operating standards. AI management should be no different. Companies should grant only the minimum necessary permissions and manage the entire lifecycle, from development and testing through approval, operation and decommissioning. Performance should be evaluated not by the number of times an AI is used, but by the rate at which it properly completes tasks, cost, causes of failure, time spent on human rework and regulatory compliance. Final responsibility for the outcome always rests with people.
—If using the smartest and most expensive model for every task is not always the answer, how can companies use AI models cost-effectively?
△AI cost-effectiveness is not determined simply by choosing the model with the lowest usage fee. Companies must consider the total cost of completing a task while maintaining the desired quality. High-performance models can be used for complex judgments, while smaller, faster models can handle simple tasks such as classification, summarization and format conversion. Frequently repeated lengthy content can be processed once, stored and reused. Costs can also be reduced by managing unchanging work instructions separately from customer information that changes each time.
—Giving AI agents full authority can lead to unforeseen incidents. How should companies establish standards for human intervention?
△If people must approve every action individually, it is difficult to preserve AI's speed and efficiency. But if all authority is delegated, responsibility and trust may be undermined when an incident occurs. The standards should be based not on how intelligent the AI is, but on how risky the action is and whether it can be reversed if something goes wrong. For tasks between those extremes, people should monitor the process, intervene when they detect an anomaly and review the records afterward. What matters is not just an approval button. Companies must also prepare identity verification for users and AI, least-privilege access, limits on amounts, frequency and functions used, abnormal-behavior detection, activity logs and a mechanism to stop operations immediately. To increase AI autonomy safely, companies should not eliminate controls; they should make the standards more explicit.
[email protected] Jang Min-kwon Reporter