Out-of-Scope Refusal, Citations & PII

Your assistant can retrieve and ground. But a real support bot in front of customers needs guardrails, and this is the lesson that turns a demo into something you would actually deploy. Three of them, and a hiring manager will look for all three:

• Refuse out-of-scope questions. If nothing in the knowledge base is close enough to the question, do not improvise -- return the refusal sentinel. This is the single most important safety behavior in RAG.
• Cite sources. When you do answer, say which article ids backed the answer, so a human can verify it.
• Redact PII. If you ever echo the user's text (into a log, a ticket, a prompt), strip out emails and phone numbers first.

Build a retriever again (re-use your TF-IDF from lesson 1) and add the guardrail layer:

answer_or_refuse(query, index)   # refuse, or answer WITH citations
redact_pii(text)                 # mask emails and phone numbers

answer_or_refuse(query, index) scores the query against the knowledge base. If the top similarity is below a threshold, the question is out of scope -- return the exact sentinel I do not know. Otherwise, return a non-empty answer string that cites the source ids it used. The grader checks two things about a real answer: it is not the sentinel, and it names the correct article id (for example the citation for a password question includes 0). A clean way to format a citation is a trailing (sources: [0]), but any form that contains the id works.

How to set the threshold: with stopwords removed, a truly off-topic question ("what is the capital of France?") shares no content words with any article, so its top cosine similarity is 0.0. An on-topic question scores well above zero. A small positive threshold like 0.05 cleanly separates them.

redact_pii(text) replaces any email with [EMAIL] and any phone number with [PHONE], leaving everything else untouched. Two regexes do it. An email is roughly name@host.tld; a phone number is a run of 7 or more digits possibly with + - ( ) space in between. Benign text -- including a short number like "30 days" -- must come back unchanged. Press Run to see an on-topic answer with a citation, an off-topic refusal, and PII masked.