SparseEngine: Efficient Inference for Large Language Models

Paper: SparseEngine: Sparse-First Inference Engine

Authors: Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu

Executive Summary

Recent advancements in large language models (LLMs) have highlighted challenges related to memory management and processing efficiency, especially when these models accumulate extensive interaction histories. The research introduces SparseEngine, a new inference engine designed to address these challenges by optimizing how memory is used and how computations are handled.

SparseEngine is built from the ground up with a focus on sparse attention mechanisms, which allow the system to reduce the memory strain typically caused by key-value (KV) caches. Traditional methods often struggle with integrating sparse attention into existing frameworks due to varying cache representations and workflows. SparseEngine tackles this by establishing a unified lifecycle contract that enables different methods to manage their KV representations and computations effectively while coordinating transitions with common infrastructure.

One of the key features of SparseEngine is its support for 15 different methods across four categories. This versatility allows for greater flexibility in how the engine can be deployed and utilized in various applications. Additionally, SparseEngine introduces two notable memory management innovations: Chain Cache and Prefix-Cache Pruning. Chain Cache facilitates cross-request state management by allowing the engine to resume KV eviction methods using retained history. This means that the engine can more efficiently handle memory by recalling previously used information rather than starting from scratch. On the other hand, Prefix-Cache Pruning intelligently removes KV entries from specified regions of history, all while maintaining the logical connections necessary for effective processing.

The performance improvements with SparseEngine are significant. Benchmarks indicate that it achieves over 10 times higher throughput with KV eviction compared to existing solutions. Moreover, it decodes data more than 2.5 times faster at equivalent concurrency levels when compared to the vLLM framework. Additionally, it offers over two times the speedup in end-to-end processing on agent benchmarks. These results, derived from simulations and benchmark evaluations, suggest that organizations utilizing LLMs could benefit from improved efficiency and responsiveness in their applications.

The development of SparseEngine holds promise for various sectors that rely on AI-driven interactions, such as customer support, content generation, and data analysis. By leveraging the advancements offered by SparseEngine, companies may enhance their operational capabilities, reduce costs associated with memory management, and improve user experiences through faster response times. The code for SparseEngine is publicly available, allowing enterprises to explore its potential benefits directly.

Academic Abstract

Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.