Seeking Best Practices for Identifying Knowledge Gaps in Raw Interactions

Hi ​@hundleymf, 

If you haven’t already, I’d recommend checking out our AI Assistant Insights course in Moveworks Academy. It provides a full overview of the AI Assistant Insights area but two modules are especially helpful for this:

1. Knowledge Base Plugin Insights – This module highlights ways to identify knowledge gaps using the Knowledge Gaps table. It’s a great starting point to spot topics that may need additional knowledge. It also includes pre-filtered raw data for each topic, so you can dig deeper into the underlying interactions without a lot of manual filtering. A quick way to get started in platform:
   • Go to AI Assistant Insights
   • Go to the Plugin Insights dashboard
   • Select the Knowledge Base tab at the top of the dashboard
   • Scroll to the Knowledge Gaps table
   • Review topics with the highest number of interactions and select the interactions # for analysis on that topic
2. Raw Interactions –This module focuses on how to filter and analyze raw interaction data effectively.

Looking forward to hearing how others have approached scaling this analysis as well!

!-->

Thanks for your help ​@aadamson. I didn’t know that I could see the interactions from the Knowledge Gaps table, it’s not intuitive that if you click on Interactions that it shows you all the interactions. I’m trying to make this process more efficient—whether with Copilot, my company's GPT model, Python, or another tool—so it takes less time. 

The file I downloaded for September I used the Raw Interactions table and then filtered it. I removed unneeded columns and filtered two columns (Unsuccessful Plugins and Plugin Served). Even after that, it still has 29,912 rows. I cut it from about 54,000 rows to about 30,000, but there’s still a lot of work to get the result I want.

My goal is to find the top 10 knowledge gap clusters and list the top 5-10 actual user questions under each cluster. For example:

Cluster 1: IT Ticketing & Support Requests

Can you create a ticket to retire a Windows server KMCI1CAM20?

I want to issue a Power BI installation request ticket

Request a Power BI installation

I need to enter a general problem ticket
 

Cluster 2: Access & Permissions

\sgci0fil2001\C$ permission request

Good day AI Assistant, can you help me request access for CTRAC

How to set up access in visibility dashboard

How to apply IES permission
 

When I try this in Copilot or my company's GPT model, they summarize or generate sample questions instead of pulling the actual questions from the file, so I’m not sure what I’m doing wrong. Also, GPT has a 10 MB limit; my file is about 8 MB and takes a few minutes to load. Someone suggested copying and pasting rows, but I could only do 250 at a time, which is too slow. My goal is to find all the times my bot didn't answer a user, which indicates a knowledge gap. I need to see the users questions to see what was asked so I can get content into my bot. So maybe there is an easier way than the clusters.

I’d appreciate any ideas and guidance. Let me know if you have any questions. Thanks again!

By the way this is what I see for the Knowledge Gaps table, I wasn’t aware that I could click on “Total interactions” and that would show me the interactions.

Hey ​@hundleymf 

If I understand your requirement correctly, you want top clusters of knowledge gaps and sample questions that remained unanswered per cluster. 

Here’s an alternative approach :
1. Take CSV export of interactions and apply thee filters :

• Unsuccessful plugin = Knowledge base > These are all the cases where KB plugin was considered but not served to the user, one of the major reasons is lack of available resources.
• Used plugin = Nil OR Handoff > These filters your rows even further to scenarios where nothing was served to the user’s query OR a ticket was filed
• Interactions type = Free form text > These will only give the text based user questions that were given to the AI assistant.

2. Then, leverage the topic column to see which topics have maximum rows (one way that I use is to insert an aggregate bar chart for topic column, to see which topics have highest KB gaps)

Hi ​@hundleymf,

Thanks for starting a discussion on such a key requirement within the Knowledge space. We are trying to do the same and agree with your feedback that creating an org/domain specific cluster is the desired outcome but possibly the most challenging part. 

As you said, it is still a very manual process and additionally in our case, the in-built topic classification hasn’t been very actionable for us. We have started experimenting using the following steps:

1. Use the filters suggested here (unsuccessful plugin, used plugin and interaction type). 
2. Post this, we identify the top ticket driver “themes” from the tickets data (and not the interactions data). 
3. Then run the subset of data (step 1) in an internal AI tool to categorize the data by those themes and even sub-themes. Not all AI tools take Excel/CSV files as inputs so that’s a constraint. And we need to do a bit of a trial and error to instruct the tool about these “themes” and “sub-themes”. Note that we haven’t yet tried combining themes to perform step 3. 
4. Run the categorized data again in the internal AI tool to identify whether the query is suitable/viable for a Knowledge Article or not. Yet again, we need to do a few rounds of trial and error to define what is “Knowledge viability” for a particular theme. 
5. Once we have this data, a pivot table allows us to see the cluster post which we need the relevant teams to actually review, confirm if it is a genuine knowledge gap and close the gap. 

It will be great if this entire process can be achieved in an automated manner within the Moveworks capability itself (with greater accuracy in topic modelling as well as a more actionable clustering). And these enhancements can then also be applied to the Feedback data. 

I’m here for the knowledge management questions all day every day!

The first thing I’m going to say is you’re going to want to do that more frequently. A month’s worth of interaction is going to be too much. We try to do ours weekly so that we’re dealing with a much more limited set of prompts to sift through.

Then we do what this (putting it in an ordered list like everyone else):

1. Run it through a model to get the high level topics from the model
2. Look at the Knowledge Plugin to get what gaps are shown there
3. Look at the Knowledge Plugin to see what KBs have been shown to be helpful and unhelpful

Then when I’m done that I go back to the list of interactions and I search for the language that people are using when they ask for the things that I feel I want to make KBs on.

You can write as many KBs as you want, but you’ll want to make sure you’re reflecting at least a little but how your users are talking to make sure that your bot returns the answers you want.

We’ve done that when we’ve seen less than ideal answers and then after the next ingestion cycle the answer has been improved when we test with the same wording. We’ve done it where we’ve seen no answers and helped get the response we wanted when people were using the prompt.

Are any of you using Knowledge Writer or are you doing all the knowledge management yourself? 

We started using Knowledge Studio, and if you starting on your knowledge journey I would recommend it.

We moved away from it because, we already had a robust KM process and about 1500 articles referenced, and the way we use SNOW is a little … excessive … so there was a lot of KB articles created that were picking up our alerts and trying to make KB articles from that. However, if you’re using your ticket system responsibly and you need help with it I would recommend it to help get you information that you may not have known you needed through your ticketing history.