# GAAC: Efficient KV Compression for LLMs on Personal Devices

> Recent research presents the Group-Adaptive Additive Codec (GAAC), a system designed to improve the efficiency of serving large language models (LLMs) on personal devices. While traditional approaches...

**Source**: arxiv.org | **Published**: 2026-10-30 | **Type**: research

## Key Facts

- Implement GAAC to optimize KV caching for enhanced long-context inference in LLMs.
- Reduce storage requirements significantly by adopting GAAC’s lossless compression techniques.
- Leverage adaptive code widths to improve efficiency in data handling on personal devices.
- Enhance model performance by utilizing redundancy in binary floating-point representations.
- Integrate GAAC into existing systems for a seamless upgrade to LLM deployment efficiency.

## Summary

**Paper:** [GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices](https://arxiv.org/abs/2610.38210)

**Authors:** Yipin Guo, Arun M George, Siddharth Joshi

\## Executive Summary

Recent research presents the Group-Adaptive Additive Codec (GAAC), a system designed to improve the efficiency of serving large language models (LLMs) on personal devices. While traditional approaches to quantization have successfully reduced storage and memory needs for these models, they have not adequately addressed the challenges posed by key-value (KV) caching, which remains a limiting factor for handling long-context inference.

GAAC introduces a novel lossless compression method specifically for KV caches that minimizes storage requirements without sacrificing the integrity of the data. By utilizing a technique that exploits redundancy in the exponent values of binary floating-point (BF16) representations, GAAC can compress KV data effectively. This is achieved through a process that encodes the exponents using a per-channel base-plus-offset system, which allows for adaptive code widths across small groups of data while ensuring that the KV values remain unchanged.

One of the key innovations of GAAC is its ability to reconstruct KV data on-the-fly through a fused attention kernel, which eliminates the need for full-cache decompression. This feature not only enhances processing speed but also supports incremental updates to the cache and allows for efficient paged serving. 

The performance of GAAC was evaluated across five different models. The results showed that GAAC achieved a lossless compression rate of approximately 1.405 to 1.407 times for BF16 KV data. Additionally, it demonstrated positive compression performance on low-precision tensor caches, yielding compression rates of 1.09 to 1.14 times for FP8 E4M3 caches and 1.25 to 1.30 times for E5M2 caches.

In practical terms, deploying GAAC on NVIDIA DGX Spark systems resulted in a significant acceleration of decode attention processes, achieving speed improvements of 1.19 to 1.27 times for handling 64K-token contexts. The end-to-end decode steps also showed enhancements, with speed increases between 1.04 and 1.15 times. Furthermore, GAAC reduced the resident KV memory footprint by about 26%, while simultaneously boosting overall throughput by 13% to 29%, all under similar KV-memory budgets.

The research also indicates that GAAC is not limited to NVIDIA hardware; it has been tested on Apple M4 chips, where it achieved attention speedups ranging from 1.02 to 1.25 times. This suggests that GAAC could be a versatile solution for improving the efficiency of LLMs across various hardware platforms.

In summary, GAAC represents a promising advancement in the deployment of large language models, particularly for environments where storage and memory constraints are critical. Its ability to enhance processing speed and reduce memory usage could enable more effective applications of LLMs on personal devices, potentially broadening their accessibility and usability.

\## Academic Abstract

Weight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices. We present Group-Adaptive Additive Codec (GAAC), a lossless KV compression and serving system that exploits exponent redundancy to reduce storage and accelerate attention without introducing KV approximation. GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges. GAAC adapts code widths across small groups while preserving KV values bit-for-bit. A fused attention kernel reconstructs KV tiles on the fly, avoiding full-cache decompression. Block-wise compression and fixed-capacity slots support incremental cache updates and paged serving. Across five tested models, GAAC achieves 1.405-1.407$\times$ lossless BF16 KV compression. We also study compression performance on low-precision tensors FP8 E4M3 caches ( 1.09-1.14$\times$) and E5M2 caches (1.25-1.30$\times$). On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27$\times$ and end-to-end W4A16 decode steps by 1.04-1.15$\times$. In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets. On Apple M4, GAAC achieves 1.02-1.25$\times$ attention speedups, demonstrating portability beyond NVIDIA GPUs.

## Frequently Asked Questions

**What business problems does GAAC aim to solve?**

GAAC aims to solve the inefficiencies associated with serving large language models (LLMs) on personal devices, particularly by addressing the limitations of key-value (KV) caching which hinders long-context inference.

**Which industries could benefit most from the implementation of GAAC?**

Industries that heavily rely on natural language processing and large language models, such as technology, customer service, education, and healthcare, could benefit most from GAAC's improved efficiency in handling LLMs on personal devices.

**What are the practical implementation considerations for businesses looking to adopt GAAC?**

Businesses would need to consider the integration of GAAC with their existing systems for LLM inference, ensuring compatibility with current infrastructure and evaluating the potential impacts on processing speeds and data integrity during implementation.

**What resources or expertise are needed to implement GAAC effectively?**

Implementing GAAC effectively would require expertise in machine learning, particularly in the areas of model optimization and data compression, as well as resources for adapting and testing the new codec within existing applications.

**What competitive advantages could businesses gain by using GAAC?**

By using GAAC, businesses could gain competitive advantages through enhanced efficiency in processing large language models, leading to faster responses and improved user experiences on personal devices, which may result in higher customer satisfaction and retention.

## Links

- [Read on Welcome.AI](https://welcome.ai/content/gaac-efficient-kv-compression-for-llms-on-personal-devices)
- [Original source](https://arxiv.org/abs/2610.38210)

---

Source: Welcome.AI | https://welcome.ai/content/gaac-efficient-kv-compression-for-llms-on-personal-devices