Hi,
I’d like to share my experience with the Triage 2.0 setup.
What model threshold are you using?
We set the model threshold to approximately 0.2.
What precision and coverage are you seeing at that threshold?
At this threshold, our target was:
- Coverage: 80% or higher
- Precision: 70–80%
Our actual results were:
- Coverage: 85% +
- Precision: 75% +
Coverage and precision are inversely proportional: improving one may reduce the other.
Why did you choose that threshold?
The threshold was selected based on our target precision and coverage. Moveworks engineers supported us throughout the setup process, which included:
- We Provided approximately 200 high-quality, correctly categorized tickets for model training.
- Reviewing the expected precision and coverage at different thresholds, Post Model training.
- Selecting the threshold that best matched our target results.
We did not independently test multiple thresholds before making the final selection because the Moveworks engineer confirmed that approximately 0.2 would meet our target. The production results were also consistent with the expected outcomes.
Now we(Customers) can review the Precision & Coverage by Threshold report by navigating to:
Moveworks Setup → Triage → Triage 2.0 → <Select the triage configuration> → Build Triage Index (Configure) → Fetch Report(Button)
How long did you monitor the results before keeping the threshold?
We enabled the triage for a day and monitored the performance, Since approximately 100 tickets within the triage scope are received daily, this provided an initial sample for validation.
What would cause you to change the threshold?
We would have updated the threshold if the expected precision and coverage is not met during testing.
Data source and reporting
The model was trained using latest production ticket data. The initial precision and coverage values were based on model testing
To monitor ongoing performance, we use:
- Moveworks Analytics
- A dashboard in our ticketing system for comparison and independent tracking
Note: Moveworks Analytics may take approximately one day to reflect the latest data. For example, today’s triaged ticket information may appear in the report tomorrow.
Additional consideration
Another important configuration is when triage occurs:
- Before ticket insertion, or
- After ticket insertion
The required configuration may vary depending on the ticketing platform and its customizations. Therefore, the triage configuration in the ticketing system may also need to be reviewed and updated.
I hope this information is helpful and up to date. Please let me know if any details are outdated or if I have missed anything important.
Thank you 😊
