Live on Nebius Token Factory · NVIDIA Nemotron 3
Loadbearing breaks every input of your AI pipeline on purpose, one at a time, and shows you which ones the answer really depends on. Then NVIDIA Nemotron writes the fix, and Loadbearing proves it works.
The inputone real customer request
The answerNemotron 3 Nano
Yes, you can return the headphones for a full refund within the 30‑day return window.
Nemotron 3 Ultra · the fix1 line
- context = request.get("context", "")+ context = request.get("passages", "")
One real request, frozen once. Then every input gets broken, one copy at a time.
copies of one real run, each with one input broken
for all 21 runs on Nebius Sandboxes, side by side
in the fix Nemotron 3 Ultra wrote, then proved
benchmark pipelines found exactly right
The bug nobody sees
The retriever found the right policy: 14 days, then warranty only. The code that builds the prompt looked for it under an old name, got nothing, and the model filled the gap from memory with a 30-day window that does not exist.
Reading the code, you would have to spot one word. Running it, Loadbearing sees it in seconds: wipe the policy, and the answer does not change.
Wrong answers like this cost real money. In 2024 a tribunal ordered Air Canada to pay for what its support chatbot told a customer.
Moffatt v. Air Canada, 2024 BCCRT 149 →
Caught in the wild
Each one was a public issue in an open-source AI library. Loadbearing ran the release with the bug and the release with the fix, on Nebius Sandboxes: it flags the first, on the field the issue names, and clears the second. No model was needed to find any of them.
6 / 6releases called correctly: every bug caught, every fix cleared
A RAG pipeline asks for chunks that overlap by 20 characters, so no sentence is cut in half at a chunk boundary. When the text has no blank line in it, the splitter quietly returned chunks with no overlap at all.
In every release from 2.12.0 (April 2025) to 2.30.2: 37 releases over 15 months. We checked 6 of them, both ends and four between.
A RAG pipeline attaches where each answer came from, a source URL and a policy version, for citations and audit. When the reply came from a chat model, AnswerBuilder kept the model's metadata and threw the caller's away.
An agent describes the structured output it wants: a fruit or a vehicle, and what each should hold. For a union of output types, the model was told "A vehicle", the docstring of the last type, whatever description the caller gave.
How it works
Start the pipeline once, with one real input, and freeze it. Every test that follows starts from exactly this point, so the only thing that ever differs is the input.
Make 14 copies. Each gets one input changed: wiped, removed, or nudged a tiny bit. All of them run at once.
An input that should matter but never changes the answer is being ignored. An input that should not matter but changes it is leaking in. Rules decide which, never a model.
Nemotron Ultra writes the smallest fix. A fresh copy of the same snapshot runs every test again: the bug must be gone, and nothing else may break.
What it finds
The policy, the ticket body, the applicant's income: wipe it out and the answer stays the same. Usually a renamed field, a typo, or a value overwritten by a default.
An order id, a timestamp, a user's name: nudge it and the result moves. Usually leaked into routing, bucketing, or the prompt by accident.
Try it
Free runs happen right here, on this server. The AI bot shows its recorded live run, because running it again would spend credits.
ticket_triage
Planted: Sorts tickets by urgency. Two bugs are planted: the ticket body is read under a misspelled name, so an outage is filed as low priority, and the second the ticket arrived picks its queue.
support_answerer
Planted: Answers refund questions from retrieved policy. The retriever hands over the policy, the prompt never includes it, and the bot makes up a return window.
1 defect found
recorded live on Nebius Sandboxes
The benchmark
Every card says what was planted. Run them all and this page checks each result against the planted bug, live, including the cases Loadbearing gets wrong.
limit-keyword
Planted: changes its answer only when ticket_id starts with TEST-
limit-threshold
Planted: nothing: correct code that only uses a field above a threshold
loan-clean
Planted: nothing: a correct pipeline, here to catch false alarms
loan-echo
Planted: copies the id application_id into its answer
loan-id_bucket
Planted: lets a hash of the id application_id pick which answer you get
loan-overwrite_annual_income
Planted: reads annual_income, then overwrites it with a default before using it
loan-overwrite_employment
Planted: reads employment, then overwrites it with a default before using it
loan-overwrite_requested_amount
Planted: reads requested_amount, then overwrites it with a default before using it
loan-stale_annual_income
Planted: reads annual_income under the name it had before an upstream rename
loan-stale_employment
Planted: reads employment under the name it had before an upstream rename
loan-stale_requested_amount
Planted: reads requested_amount under the name it had before an upstream rename
loan-ts_leak
Planted: lets the second received_at arrived decide the result
loan-typo_annual_income
Planted: reads annual_income under a misspelled name, so it never sees the value
loan-typo_employment
Planted: reads employment under a misspelled name, so it never sees the value
loan-typo_requested_amount
Planted: reads requested_amount under a misspelled name, so it never sees the value
moderation-clean
Planted: nothing: a correct pipeline, here to catch false alarms
moderation-echo
Planted: copies the id post_id into its answer
moderation-id_bucket
Planted: lets a hash of the id post_id pick which answer you get
moderation-overwrite_author_trust
Planted: reads author_trust, then overwrites it with a default before using it
moderation-overwrite_reports
Planted: reads reports, then overwrites it with a default before using it
moderation-overwrite_text
Planted: reads text, then overwrites it with a default before using it
moderation-stale_author_trust
Planted: reads author_trust under the name it had before an upstream rename
moderation-stale_reports
Planted: reads reports under the name it had before an upstream rename
moderation-stale_text
Planted: reads text under the name it had before an upstream rename
moderation-ts_leak
Planted: lets the second posted_at arrived decide the result
moderation-typo_author_trust
Planted: reads author_trust under a misspelled name, so it never sees the value
moderation-typo_reports
Planted: reads reports under a misspelled name, so it never sees the value
moderation-typo_text
Planted: reads text under a misspelled name, so it never sees the value
shipping-clean
Planted: nothing: a correct pipeline, here to catch false alarms
shipping-echo
Planted: copies the id order_id into its answer
shipping-id_bucket
Planted: lets a hash of the id order_id pick which answer you get
shipping-overwrite_destination_zone
Planted: reads destination_zone, then overwrites it with a default before using it
shipping-overwrite_express
Planted: reads express, then overwrites it with a default before using it
shipping-overwrite_weight_kg
Planted: reads weight_kg, then overwrites it with a default before using it
shipping-stale_destination_zone
Planted: reads destination_zone under the name it had before an upstream rename
shipping-stale_express
Planted: reads express under the name it had before an upstream rename
shipping-stale_weight_kg
Planted: reads weight_kg under the name it had before an upstream rename
shipping-ts_leak
Planted: lets the second created_at arrived decide the result
shipping-typo_destination_zone
Planted: reads destination_zone under a misspelled name, so it never sees the value
shipping-typo_express
Planted: reads express under a misspelled name, so it never sees the value
shipping-typo_weight_kg
Planted: reads weight_kg under a misspelled name, so it never sees the value
triage-clean
Planted: nothing: a correct pipeline, here to catch false alarms
triage-echo
Planted: copies the id ticket_id into its answer
triage-id_bucket
Planted: lets a hash of the id ticket_id pick which answer you get
triage-overwrite_body
Planted: reads body, then overwrites it with a default before using it
triage-overwrite_customer_tier
Planted: reads customer_tier, then overwrites it with a default before using it
triage-overwrite_subject
Planted: reads subject, then overwrites it with a default before using it
triage-stale_body
Planted: reads body under the name it had before an upstream rename
triage-stale_customer_tier
Planted: reads customer_tier under the name it had before an upstream rename
triage-stale_subject
Planted: reads subject under the name it had before an upstream rename
triage-ts_leak
Planted: lets the second submitted_at arrived decide the result
triage-typo_body
Planted: reads body under a misspelled name, so it never sees the value
triage-typo_customer_tier
Planted: reads customer_tier under a misspelled name, so it never sees the value
triage-typo_subject
Planted: reads subject under a misspelled name, so it never sees the value
Measured, not claimed
Being honest about it: on pipelines this small, a strong model reading the code does about as well. What it cannot do is give the same answer twice, show the evidence, or prove its fix. Loadbearing's misses and false alarms are listed case by case in bench/, and you can run every case above.
Built on
Serves all three Nemotron models. Its Sandboxes hold the snapshot and every copy: the run this page replays was recorded there, live.
Looks at each changed answer: a real change, or just rewording?
Reads the docs and every copy of a field, and states the defect in one line.
Reads the code, names the line, and writes the smallest fix.
Finds the docs that say what each input is supposed to do, with a quote and a link.
Use it
Point it at a pipeline and one real input. It fails the build when an input stops doing what it should, and the run page shows the evidence and the proven fix.
The terminal on this page is real: when it comes into view it runs the probe on this server.
git clone https://github.com/varunikarai/loadbearing && cd loadbearingpip install -e .loadbearing probe examples/ticket_triage/target.json
- uses: varunikarai/loadbearing@main
with:
target: path/to/target.json