# Digital Bank's AI Testing Reveals Insights on Customer Support Efficiency > This groundbreaking study explores how a major digital bank tests AI agents in customer service, uncovering key insights into AI performance and customer satisfaction. Discover how improvements in AI-driven support are shaping the future of customer interactions. **Source**: hackernoon.com | **Published**: 2026-10-30 | **Type**: research ## Key Facts - AI agents improved customer satisfaction but still lagged behind human benchmarks, indicating room for growth. - Card delivery saw a 29-point self-service gain, highlighting effective automation potential in simpler tasks. - Initial AI grader accuracy was only 68.88%, revealing vulnerabilities in evaluation methods before optimization. - New agent versions start with 1-5% exposure, suggesting a cautious approach to scaling AI capabilities in production. - Increased handle time may indicate successful AI automation of simpler tasks, shifting workload to human agents. ## Summary A digital bank serving over 100 million customers is testing agentic AI in its customer support operations, revealing critical insights into the deployment and evaluation of AI systems in high-volume environments. A recent paper accepted for presentation at the ACM KDD 2026 conference details the bank’s methodology for assessing the effectiveness of five AI agents across various functions, including card delivery and debt management. This research is significant as it provides a framework for other organizations considering similar AI implementations, highlighting the complexities of measuring AI performance against human benchmarks. The study's findings indicate that while all five AI agents improved customer satisfaction metrics, they still fell short of the performance levels achieved by human agents. The transactional Net Promoter Score (tNPS) for these AI agents showed notable gains, particularly in card delivery, which saw a 37-point increase in satisfaction. However, the scores remained below those of human agents, underscoring the ongoing challenge of achieving parity between AI and human performance in customer service. This gap signals to industry leaders that while AI can enhance efficiency, it may not yet be capable of fully replicating the nuanced understanding and empathy that human agents provide. The bank's approach to testing involved a rigorous evaluation process, including the use of an automated grading system to assess the AI agents’ responses. Initially, this grading system performed poorly, scoring below a simple baseline. However, through optimization efforts, the accuracy of the grading system improved significantly, reflecting the importance of continuous refinement in AI evaluations. This iterative process is critical for companies looking to implement agentic AI, as it emphasizes the need for robust validation methods before scaling AI capabilities. The deployment strategy for new AI versions is also noteworthy. The bank starts with a limited exposure of 1% to 5% to monitor performance metrics like self-service rates and escalation patterns. This cautious rollout allows for real-time adjustments based on customer interactions, which is essential in mitigating potential issues before they affect a larger user base. The ability to fine-tune AI agents based on live data not only enhances performance but also aligns with the growing demand for responsive and adaptable customer service solutions. However, the study highlights a nuanced understanding of performance metrics. For instance, while average handle time may increase as AI resolves simpler requests, the remaining cases for human agents may become more complex and time-consuming. This shift necessitates a reevaluation of what constitutes effective AI performance. Companies must consider the broader context of customer interactions and the nature of escalations to accurately assess the impact of AI on service quality. As organizations increasingly adopt agentic AI, the findings from this bank’s research signal a critical shift in customer service dynamics. Companies must recognize that AI is not a one-size-fits-all solution and that its integration requires careful consideration of operational workflows and customer expectations. The implications are clear: businesses that invest in understanding the interplay between AI and human agents will be better positioned to leverage technology for enhanced customer experiences. Looking ahead, the evolving landscape of customer service will likely see a greater emphasis on hybrid models that combine AI efficiencies with human expertise. As companies refine their AI strategies, those that prioritize ongoing evaluation and adaptability will gain a competitive edge, enabling them to meet the diverse needs of their customer base while optimizing operational efficiency. This dual focus on technology and human insight will be crucial as businesses navigate the complexities of a rapidly changing market environment. ## Entities - **Companies**: TELUS Digital - **Products**: agentic AI, automated grader - **Technologies**: language model, prompt-optimization method - **People**: Erin Walker - **Organizations**: ACM, KDD ## Key Concepts customer support AI agents, automated grading, Net Promoter Score (tNPS), self-service rate, human-agent benchmarks, exposure testing, customer interaction automation, AI performance metrics ## Definitions - **agentic AI**: A type of AI designed to handle customer interactions autonomously, improving efficiency and customer satisfaction. - **Net Promoter Score (tNPS)**: A metric used to gauge customer satisfaction and loyalty based on their likelihood to recommend a service. - **automated grader**: An AI system that evaluates the performance of other AI agents based on predefined criteria. - **self-service rate**: The percentage of customer requests resolved by AI without human intervention. - **exposure testing**: A method of gradually introducing new AI versions to users to monitor performance and satisfaction. ## Use Cases - card delivery - debt management - credit limits - card management - product explanation - customer interaction automation ## Frequently Asked Questions **What is agentic AI?** Agentic AI refers to artificial intelligence systems that autonomously manage customer interactions, aiming to enhance efficiency and customer satisfaction. These systems are designed to handle various tasks without human intervention. **How is customer satisfaction measured in AI interactions?** Customer satisfaction in AI interactions is often measured using the Net Promoter Score (tNPS), which assesses the likelihood of customers recommending the service based on their experience. This score helps evaluate the effectiveness of AI agents. **What role does the automated grader play?** The automated grader evaluates the performance of AI agents by comparing their responses to expert judgments. It helps ensure that the AI meets quality standards before deployment in customer-facing roles. **What is the significance of self-service rates?** Self-service rates indicate how effectively AI agents can resolve customer requests without human assistance. A higher self-service rate suggests that the AI is functioning well and improving operational efficiency. **How does exposure testing work?** Exposure testing involves gradually introducing new versions of AI agents to a small percentage of users. This allows teams to monitor performance metrics and customer satisfaction before a full rollout. ## Links - [Read on Welcome.AI](https://welcome.ai/content/digital-banks-ai-testing-reveals-insights-on-customer-support-efficiency) - [Original source](https://hackernoon.com/how-a-bank-serving-100-million-users-tests-agentic-ai-in-production) --- Source: Welcome.AI | https://welcome.ai/content/digital-banks-ai-testing-reveals-insights-on-customer-support-efficiency