ServiceNow Enhances Customer Success with LangSmith’s Intelligent Multi-Agent System

ServiceNow, a leading digital workflow platform, is advancing its customer success operations by utilizing LangSmith and LangGraph, according to a LangChain report. The company aims to streamline its internal sales and customer success processes by developing a comprehensive multi-agent system that manages the entire customer journey, from initial lead identification to post-sales growth.

Tackling Agent Fragmentation

The initiative addresses the challenge of agent fragmentation, where agents were previously scattered across different platform areas without a unified orchestration layer. This fragmentation complicated the coordination of complex workflows necessary for managing the complete customer lifecycle. ServiceNow’s solution involves building a multi-agent system capable of managing processes from lead qualification and deal closure to post-sales adoption, renewal, and customer advocacy.

Comprehensive Multi-Agent System

The intelligent agent system being developed by ServiceNow encompasses both pre-sales and post-sales workflows. Key stages in this system include lead qualification, opportunity discovery, economic buyer identification, onboarding, adoption tracking, usage realization, renewal, and customer advocacy. Each stage employs specialized agents to guide Account Executives and Customer Success Managers in fulfilling customer needs effectively.

Complex Agent Orchestration with LangGraph

ServiceNow utilizes LangGraph for sophisticated multi-agent coordination, employing map-reduce style graphs and subgraph calling to build a modular system. This architecture allows for efficient orchestration, as a supervisor agent activates specific subagents based on customer signals and lifecycle stages. The integration of LangGraph’s tools has enabled ServiceNow to create a robust technology stack for agent orchestration across its platform.

LangSmith’s Tracing Capabilities

LangSmith provides detailed tracing capabilities that enhance agent development by offering insights into input/output, context, and latency at every step of agent orchestration. These features facilitate debugging by structuring trace data into inputs and outputs for each node, making it easier to identify and rectify issues.

Evaluation Strategy with Custom Metrics

ServiceNow has implemented a rigorous evaluation framework using LangSmith, tailored to their multi-agent system. Custom scorers are defined for each agent’s specific tasks, with LLM-as-a-judge evaluators assessing agent responses. This strategy includes automated dataset creation, human feedback integration, and regression prevention, ensuring continuous improvement and effectiveness of the agents.

Future Developments and Testing

Currently in the testing phase, ServiceNow is evaluating agent performance in a controlled environment to refine their datasets and evaluation framework. The company plans to continue collecting real user data and leveraging LangSmith for ongoing monitoring of live agent performance. Future developments include using multi-turn evaluation to assess agent performance across end-to-end user interactions.

Through the integration of LangSmith and LangGraph, ServiceNow is poised to enhance its customer success operations significantly, creating a streamlined and intelligent system that supports the entire customer journey.

Image source: Shutterstock