Kog Enables Processing of 3,000 Tokens Per Second with Standard GPUs

French startup Kog has announced a strategy to enhance AI inference speeds using standard data center GPUs. Kog has adopted an approach that involves designing software and model architecture together to maximize the utilization of NVIDIA and AMD GPUs. The company stated that it has integrated runtime, GPU kernels, and model architecture into an optimized pipeline to improve response speeds to requests from AI agents. According to a technology preview of the Kog inference engine released on May 28, it can process 3,000 output tokens per second per request using eight AMD MI300X GPUs, while eight NVIDIA H200 GPUs under the same conditions recorded 2,100 output tokens per second. Kog is primarily targeting AI coding agents and agent-based workflows. The Laneformer 2B model, released on Hugging Face on June 24, has 2.3 billion parameters and demonstrated performance of 45.1% on HumanEval+ and 51.6% on MBPP+. Kog reported in an August LinkedIn post that it achieved 2,857 tokens per second in a live demo. However, there is controversy regarding the interpretation of speed figures, and it has been pointed out that comparisons are difficult due to variations in model size and hardware. Kog's goal is to achieve low latency on existing GPU infrastructure.