HR policy helpdesk with retrieval and an evaluation set

Northgate's HR shared-services team answers about 1,800 employee questions a month. Roughly 40% are about leave, 25% about reimbursements and 15% about notice periods and transfers. Every answer is already written down in one of 40 policy PDFs, but employees can't find it, so they raise a ticket and wait two days.

Last year a vendor demo "chatbot" gave a confident, wrong answer about maternity leave in a town hall. The HR head now has one rule: no assistant goes live unless the team can show, on a fixed test set, how often it is right and that it cites the clause it used.

Your job is to build that assistant and the proof: - a retrieval pipeline over the policy documents (you will write a realistic synthetic set, because real policies are confidential); - an answer step that always cites document and section, and says "I don't know, please raise a ticket" when the policies don't cover the question; - an evaluation set of at least 60 questions with expected answers, and a script that scores the system on it.

Done looks like a small FastAPI service, a one-page evaluation report with retrieval hit rate, answer accuracy and refusal accuracy, and a short demo. HR will sign off if accuracy on the test set is at least 85% and the assistant never invents a policy.