Friday, November 20, 2026

Moreh Boosts AI Inference Throughput by Up to 1.66x... Unveils Optimization Technology

Input
2026-11-20 11:52:55
Updated
2026-11-20 11:52:55
Moreh's TIDE is on display at Supercomputing 2026, held in Chicago, U.S. Photo courtesy of Moreh
[Financial News] Moreh, an artificial intelligence (AI) infrastructure solutions company, has unveiled technology that boosts large language model (LLM) inference throughput by up to 1.66x. The approach continuously optimizes the inference process using data generated by actual AI services. It is expected to improve the efficiency of existing computing resources and reduce the computational burden of operating AI services.
According to industry sources on the 20th, Moreh recently made its self-improving inference framework TIDE public for the first time at the high-performance computing (HPC) conference Supercomputing 2026 (SC26) in Chicago, U.S.
TIDE is a technology that continuously improves an auxiliary model using data generated as large AI models produce responses. It is designed to optimize the inference process accordingly, even as users' question types and work environments change.
The key is “speculative decoding.” A relatively small auxiliary model first predicts multiple tokens, which the large model then verifies all at once, reducing the time needed to generate a response.
With existing methods, prediction accuracy could decline when actual users' question types differed from the auxiliary model's training patterns. TIDE reuses internal information generated during the large model's inference process to train the auxiliary model incrementally. Because there is no need to run the large model again to create separate training data, this can reduce additional computational burden.
The gains in inference performance and the auxiliary model's prediction accuracy are also monitored in real time. When performance gains are low, unnecessary computation is reduced; when changes in task types cause prediction accuracy to fall, new data is collected and the auxiliary model is optimized again.
According to research results announced by Moreh, TIDE delivered up to 1.66 times the AI inference throughput of existing inference methods that do not use speculative decoding. Compared with existing online learning methods, it shortened the auxiliary model's training time by up to a factor of 3.02.
Performance gains were also confirmed in environments that use graphics processing units (GPUs) from different manufacturers together. Using an NVIDIA H100 GPU for large-model inference and an AMD Instinct MI250 GPU to train the auxiliary model increased overall throughput to as much as 1.22 times that of an approach that devotes the same GPU resources entirely to inference.
The research was based on technology developed during work on Moreh's AI inference platform, the MoAI Inference Framework.
Moreh is also focusing on developing Token Factory OS, infrastructure software that efficiently allocates workloads in data centers where different computing devices—including NVIDIA and AMD GPUs and Tenstorrent AI chips—are used together.
Moreh CEO Jo Gang-won said, “The competitiveness of AI infrastructure depends not simply on how many high-performance GPUs can be secured, but on how efficiently the computing resources available can be used to handle more AI workloads.”
[email protected] Jang Min-kwon Reporter