Internal helpdesk knowledge retrieval for a support team
This is a methodology scenario, not a client engagement. Chokmah has not delivered this engagement for a named client. The workflow, constraints and sequence describe how we would run it. Any figures shown are illustrative targets, not measured results.
Internal helpdesk knowledge retrieval uses a retrieval-augmented assistant to answer repetitive employee questions from policy documents. The common mistake is optimising deflection rate, which rewards confident wrong answers. This page describes how Chokmah would build it to cite sources and abstain.
- Deflection rate is gameable; optimising it directly produces confident wrong answers.
- Policy corpora conflict and supersede; retrieval must find the in-force version.
- The assistant cites its source and abstains to a human when confidence is low.
- Confident-wrong rate, not deflection, is the primary safety metric and release gate.
- Requests for a human, not for information, are escalated rather than answered.
The situation
An internal helpdesk answers repetitive questions from employees (IT access, HR policy, finance process, tooling) by finding the relevant policy or procedure and replying. Much of the answer already exists in a document somewhere; the work is finding the right one, reading the version that applies, and phrasing a reply.
The obvious framing is deflection: put a retrieval-augmented assistant in front of the queue and let it answer. That framing is where these projects usually go wrong. Deflection rate is easy to move and easy to game: an assistant that answers confidently from a stale or conflicting policy deflects the ticket and creates a worse problem downstream, because the employee acted on a wrong answer.
The real difficulty is that internal policy documents conflict, supersede each other, and vary by region or entity. This scenario is a composite of internal-helpdesk knowledge retrieval as it appears across support functions. It describes no specific client and reports no achieved result.
Constraints
- Policy documents conflict and supersede each other. Retrieving a plausible document is not the same as retrieving the one currently in force.
- Answers vary by region, entity and employee group. A correct answer for one population is a wrong answer for another.
- A wrong internal answer has real consequences: an employee acts on it. Deflection is not free if the deflected answer is wrong.
- Deflection rate is a gameable metric. Optimising it directly produces confident wrong answers.
- Some questions are requests for a human, not for information: distress, disputes, exceptions. These must be recognised and escalated, not answered.
How we would run it
- 1Week 1
Classify a quarter of tickets by whether a correct answer exists in the corpus
Sample resolved tickets and classify each: answerable from a current, unambiguous document; answerable only after resolving a conflict between documents; or not an information request at all. Only the first category is safely retrievable at the start. The size of that category, not the total ticket volume, is the realistic scope, and it is usually smaller than the deflection framing assumes.
- 2Week 1
Map the conflict and supersession structure of the policy corpus
Before building retrieval, understand what it will retrieve. Identify where documents conflict, where newer versions supersede older ones without the old ones being removed, and where answers legitimately differ by region or entity. This map determines whether retrieval can be trusted at all, and it frequently reveals that fixing the corpus is the higher-value first step.
- 3Week 2
Baseline resolution quality, not just resolution speed
Deflection and speed are the easy numbers and the wrong ones to optimise alone. Baseline first-contact resolution, reopen rate, and (most importantly) a sampled correctness rate on resolved tickets, so there is a measure of how often the current answer was actually right. Agree these with the support lead in writing. A build that raises speed while lowering correctness is a regression dressed as a win.
- 4Weeks 3-4
Build a retrieval assistant that abstains and cites
The assistant retrieves the current in-force document for the employee's region and group, answers only when it has a confident single source, cites that source in the reply, and abstains to a human when sources conflict or confidence is low. Abstention is a feature, not a failure: a system that hands off the hard questions cleanly is more valuable than one that answers all of them unreliably. It draws on the retrieval-quality-versus-generation-quality split as its core design principle.
- 5Weeks 4-5
Evaluate retrieval correctness and abstention behaviour separately
Build a labelled set of questions with the correct in-force source and the correct population, and score two things: did it retrieve the right source, and did it abstain when it should have. Track the confident-wrong rate (answers given confidently from the wrong or superseded source) as the primary safety metric. That number, not deflection rate, decides whether the assistant is safe to expose.
- 6Week 6
Hand over with the corpus-maintenance dependency stated
The team keeps the repository, the evaluation set and a clear statement that the assistant's correctness is bounded by the corpus's correctness. If policies conflict, the assistant will either abstain or occasionally answer from the wrong one; the durable fix is maintaining the source of truth, which the handover names as ongoing work the tool does not eliminate.
What we would not automate
Answering when policy sources conflict or are ambiguous
When two documents disagree or the in-force version is unclear, a confident automated answer is worse than no answer, because the employee acts on it. The correct behaviour is to abstain and route to a human. Building the system to answer anyway in order to lift the deflection number is optimising the metric at the expense of the outcome, and we would refuse to do it.
Questions that are requests for a human, not for information
Distress, disputes, harassment reports, and genuine exceptions are not retrieval problems. A system that treats them as questions to answer causes real harm. These must be recognised and escalated to a person immediately, and that classification is built in from the start rather than added after an incident.
Optimising deflection rate as the primary target
Deflection is the metric everyone asks for and the one most likely to produce confident wrong answers, because it goes up whether the answer was right or not. We would measure resolution correctness and confident-wrong rate instead, and we would tell the sponsor plainly that a lower, correct deflection rate beats a higher, unreliable one.
Illustrative targets
These are targets used to frame the engagement, not measured results from a delivered client project.
Illustrative target: Confident-wrong rate on the evaluation set: Primary safety metric; release gate
The rate at which the assistant answers confidently from a wrong or superseded source is measured on a labelled set with known in-force sources. It is treated as a gate on exposure because a confident wrong internal answer causes an employee to act incorrectly.
Illustrative target: Share of tickets safely answerable at first release: Single-source questions only
Scope is set to questions answerable from one current, unambiguous document for the employee's population, as measured in week one. Conflicted and population-dependent questions are routed to a person until the corpus supports them, so the covered share is usually smaller than a raw deflection target implies.
Illustrative target: Correct-abstention rate: Tracked alongside correctness
How often the assistant abstains when it should is measured directly, because abstention is the safety valve. A system pushed to abstain less without improving retrieval simply converts safe hand-offs into confident wrong answers, so the two metrics are always read together.
Frequently asked questions
Because it goes up whether the answer was right or not. An assistant that answers confidently from a stale or conflicting policy deflects the ticket and creates a worse problem downstream, since the employee acts on the wrong answer. We measure resolution correctness and the confident-wrong rate instead, and treat a lower but correct deflection rate as better than a higher, unreliable one.
The corpus. Internal policy documents conflict, supersede each other without the old versions being removed, and vary legitimately by region and entity. Retrieving a plausible document is not the same as retrieving the one currently in force for that employee. That is why the engagement starts by mapping the corpus's conflict and supersession structure, and why fixing the corpus is often the higher-value first step.
It answers only when it has a confident single in-force source for the employee's region and group, cites that source in the reply, and abstains to a human when sources conflict or confidence is low. Abstention is treated as a feature. On a labelled evaluation set we measure both whether it retrieved the right source and whether it abstained when it should have.
It is a Workflow Sprint: four to six weeks, built with three to five of your own people, who keep the code, the evaluation set and a corpus-maintenance runbook. The assistant's correctness is bounded by the corpus's correctness, so maintaining the source of truth is the ongoing work it does not eliminate, and the handover states that plainly.
No. It is a methodology scenario, not a client engagement. Chokmah has not delivered this for a named client. The workflow is a composite of how internal helpdesk retrieval appears across support functions, and every figure on the page is an illustrative target measured against a baseline, not an achieved result.
Have a workflow like this?
We name the workflow before we start. Book a free AI Reality Check and build one live.