KDDI's Buffmee App Cuts Response Latency and Boosts Trustworthiness
Buffmee redefines the consumer AI experience by blending information accuracy with rapid response times, drawing from a wide range of credible sources. Discover how KDDI tackled significant challenges to bring this innovative app to market.
Key Facts
- KDDI's Buffmee reduced response latency by 38%, enhancing user experience and retention.
- Automated evaluation improved groundedness scores by 25%, boosting content reliability and trust.
- Transitioning to binary evaluation minimized variance, streamlining performance assessment processes.
- Strategic content sampling cut evaluation workload by 75%, optimizing resource allocation and efficiency.
- Real-time bottleneck identification allowed KDDI to maintain accuracy while improving response speeds.
Summary
Summary
KDDI, a leading telecommunications carrier in Japan, faced challenges in developing Buffmee, a consumer Retrieval-Augmented Generation (RAG) app that delivers high-quality responses quickly. By implementing automated evaluation frameworks and performance optimization techniques, KDDI reduced total application response latency by 38% and improved Time To First Token (TTFT) by nearly 18%.
Background
KDDI operates in the telecommunications industry and is one of Japan's major carriers. Before deploying Buffmee, the company aimed to create an interactive AI service that could help users search for information and explore their interests. However, they encountered significant latency issues that hindered their ability to meet performance targets.
Challenge
The primary challenge KDDI faced was ensuring fast response times while maintaining high generation quality across diverse media types. They needed a reliable method to ground responses in a wide variety of proprietary content and eliminate hallucination in the results.
Solution
KDDI developed Buffmee by implementing a systematic AI evaluation process using the Gemini Enterprise Agent Platform Evaluation Service. They replaced manual testing with automated evaluation frameworks, such as LLM-as-a-Judge and the Rule of Hundreds, to assess the reliability of answers. Additionally, they utilized BigQuery Agent Analytics and the Agent Development Kit (ADK) to identify bottlenecks and optimize performance in real time.
Results
KDDI achieved a 38% reduction in total application response latency and an almost 18% improvement in TTFT. The implementation of automated evaluation frameworks led to a 25% increase in groundedness scores, ensuring that users received highly accurate and reliable outputs.
Key Insights
KDDI's experience highlights the importance of automated evaluation and performance analytics in developing scalable generative AI applications. Transitioning to binary evaluation, strategic content sampling, and modularizing prompts are effective strategies for improving response times and maintaining quality.
Customer Testimonial
"Our vision hinged on a platform where content, once ingested, would instantly function as a working RAG system. Google's careful, hands-on guidance made that a reality — we're sincerely grateful for their support." — Shunya Onoda, AI Product Department, KDDI.
Entities Mentioned
Companies
Products
Technologies
People
Key Concepts
Definitions
- Retrieval-Augmented Generation (RAG)
- A method that combines retrieval of information from various sources with generative AI to produce accurate and contextually relevant responses.
- TTFT
- Time To First Token, a metric that measures the time taken for a system to begin generating a response.
- Automated evaluation frameworks
- Systems designed to assess AI performance using data-driven methods, reducing the need for manual testing.
- Agent Development Kit (ADK)
- A toolkit used to develop and optimize AI agents, allowing for modular design and improved performance.
- Bottleneck identification
- The process of detecting and resolving performance issues that slow down system response times.
Use Cases
- →Interactive Q&A for users
- →Personalized learning experiences
- →Content summarization
- →Information searching
- →Deep-dive analysis of media
- →Automated testing of AI applications
Frequently Asked Questions
What is Buffmee?
Buffmee is a consumer-facing Retrieval-Augmented Generation app developed by KDDI that helps users search for information and explore personalized learning interests through interactive AI.
How did KDDI improve response times for Buffmee?
KDDI implemented automated evaluation frameworks and performance optimization techniques, which included analyzing production logs and optimizing system prompts to reduce latency.
What role did Google Cloud play in the development of Buffmee?
Google Cloud provided hands-on guidance and support to KDDI, helping them realize their vision for a reliable generative AI application.
What are the core principles for reliable evaluation mentioned in the article?
The core principles include transitioning to binary evaluation, strategic content sampling, thresholds grounded in product judgment, and modular splitting of massive prompts.
What challenges did KDDI face during the development of Buffmee?
KDDI faced challenges with latency issues and ensuring hallucination-free results while trying to ground a large variety of proprietary content.