support-answerer
1 input does not do what the pipeline claims. Found by running it, not by reading it.
Noise floor: the unmodified input ran 5 times. answer changed with no mutation at all, so it is left out of every comparison. Everything else is judged. Every fork that moved on a field that should not matter ran 3 times, and counts only if most of its runs moved; drop:user_id (1 of 3) moved too rarely and was not counted.
One checkpoint, every fork
Every field, every mutation
| field | should matter? | destroy | drop | nudge | verdict |
|---|---|---|---|---|---|
| request_id | no | same | same | same | inert |
| user_id | no | same | same | same | inert |
| received_at | no | same | same | same | inert |
| question | yes | moved | moved | same | load-bearing |
| passages | yes · from docs | same | same | – | SILENT PASSTHROUGH |
The evidence
passages · silent passthrough
Not declared, but its docs say it is an input that should change the output. It was destroyed and dropped in 2 separate forks, and the output never moved. The pipeline does not read it.
Retrieval-augmented generation is a technique for enhancing the accuracy and reliability of generative AI models with information from specific and relevant data sources.
This is a silent passthrough: the field passages is marked as relevant (supposed to matter), yet neither destroying nor dropping it caused any change in the pipeline’s output (both destroy and drop are false). The lack of output movement despite the field’s relevance indicates that the pipeline is ignoring the field rather than using it.
Code path responsible:
The build_prompt function reads the retrieved policy text from request["context"], but the retriever now stores that text in request["passages"]. Because the key mismatch is silent (.get returns ""), the policy passages never reach the prompt, and the LLM answers from generic knowledge.
Line that reads the wrong field (pipeline.py:18):
context = request.get("context", "")Why it fails:
The retriever was changed to write its output under the key "passages". This line still asks for "context", so context becomes an empty string. The prompt sent to the model then contains only the customer’s question, and the model hallucinates a return policy (e.g., 30‑day refund) instead of using the actual policy (14 days, no refund).
Smallest fix:
Change the key from "context" to "passages" (the variable name can stay context for minimal diff).
context = request.get("passages", "")If the retriever returns a list of passages, a slightly more robust fix would be:
passages = request.get("passages", "")
context = "\n".join(passages) if isinstance(passages, list)The fix, proven
passages · fixed by Nemotron Ultra
The build_prompt function reads from the 'context' key, but the retriever now stores policy passages under the 'passages' key. Changing the key to 'passages' ensures the retrieved policy text is included in the prompt, eliminating the silent passthrough defect.
Applied in a fresh fork of the same checkpoint and probed again: the defect is gone, and no other field gained one.