Hi Community,
At our company, we were planning to use triage - however, the model’s precision hasn’t been ideal, partly due to data quality issues.
We’ve been exploring ServiceNow Predictive Intelligence to retrieve the top 10 results using either classification or similarity models on our incident data.
Our proposed improvement is to introduce an LLM (Large Language Model) to refine the prediction. The idea is for the LLM to evaluate the top 10 results from the ML mode - along with the description of each categor - to determine the most accurate classification.
We believe this could significantly increase the precision of both classification and similarity models, and in some cases, even replace the need for manual agent routing.
Assuming triage currently uses OpenAI embeddings for indexing, implementation would be straightforward:
-
Continue indexing all existing incident data, category descriptions, and service information.
-
Use a RAG approach to retrieve the top 10 results based on the user query.
-
Have the LLM select the best match.
Alternatively, using the classification model, we could:
-
Retrieve the top 10 classification results based on the user query.
-
Let the LLM determine the final category.
-
Apply a confidence threshold (e.g., 70%). If the model’s precision is 60%, only high-confidence results go to the LLM, since the correct category is usually within the top few predictions.
If results fall below the threshold, the ticket could be routed to an agent for review.
The same logic applies when using similarity model - retrieve the most similar incidents, then have the LLM choose the best one.
Additionally, since most of our agents follow knowledge articles linked to categories, we could extend this approach:
-
Retrieve the top 5 relevant knowledge articles based on user issue.
-
Have the LLM analyze them and select the one with the best category match.
This could effectively automate L1 routing.
Just some thoughts I’m planning to implement 🙂
It might be worth considering this approach in genera - especially when ML alone isn’t sufficient due to data quality limitations.