Using GPT-4 to deliver a new customer service standard

The industry-standard metric for customer service, containment rate, measures the number of customer inquiries that are handled entirely by a chatbot and never reach a human agent. If you increase the containment rate, you’ll theoretically reduce operational expenses in customer service. But the team at Ada saw a problem with this metric. 

“Many solutions, including Ada, could easily deliver 80–100% containment rates, but if you actually opened up those conversation transcripts and read the experiences customers had, they were quite poor,” Gozzo said. 

By focusing their efforts on how well conversations were actually resolved, the team at Ada knew they could set a new standard in the customer service industry.

Initially using GPT‑4, combined with their historical data, Ada built a new evaluation framework capable of assessing conversations for how well they were resolved automatically. Ada’s system rates each conversation by how well customers receive relevant, accurate, and safe replies—without human intervention. “In our testing, our system achieved 80–90% agreement with a human reading the conversation,” Gozzo said.

Today, Ada’s AI Agent uses a multi-agent configuration, with a central planning agent and a collection of subagents, all powered by OpenAI’s API. “Any time you communicate with a business that’s using Ada, your question is going to be fed through multiple turns of OpenAI's models, understanding it, reflecting on it, invoking tools, and bringing in knowledge until the answer is generated,” Gozzo explained. 

The previous version of Ada’s product had a 70% containment rate, but a 30% resolution rate. Of the customers Ada has brought to the new system, they have the similar containment results, but typically see a resolution rate of up to 60%, with the highest-performing customers north of 80%. 

“From one technology to the next, we’ve doubled the amount of conversations we can automatically resolve with a great experience,” Gozzo said. For Ada’s customers, this metric has a massive downstream impact on ROI, including full time equivalent (FTE) savings, customer satisfaction, retention, and new signups.