n8n RAG Evaluation: Blind A/B Test Two Models

"We shipped the new retrieval strategy because it felt better on the examples we tried." That sentence has shipped more silent regressions than any bug. It's also the exact gap that n8n RAG evaluation exists to fill — replacing developer instinct with a repeatable, blind, scored comparison.

n8n's own evaluation tooling covers LLM-as-a-judge, built-in correctness and helpfulness metrics, and document-relevance scoring. Solid foundations. What the official framework doesn't hand you is a head-to-head test rig: run two models or two RAG configs across the same test set, hide which is which, score them blind, and report a win rate you'd defend to your team. That's the build worth doing.

The reason to do it in n8n rather than a Python notebook is durability. A workflow runs on a scheduleTrigger v1.2 every time you change a prompt, logs results to a sheet you can diff over weeks, and doesn't rot in a forgotten Colab tab.

What you can automate

• Run any two OpenAI models head-to-head on a fixed test suite
• Anonymize both responses so the judge scores without model-name bias
• Score each answer on accuracy, completeness, clarity, and conciseness (1-10)
• Aggregate per-dimension averages, win rates, and an overall recommendation
• Generate a markdown findings report ready for a standup or stakeholder
• Log every run summary to Google Sheets for trend tracking and audit history
• Re-run the whole suite automatically whenever a prompt or retrieval config changes

The blind A/B evaluation pipeline

Manual / Schedule Trigger
  → Code (parse test cases from Config)
  → Split (one item per question)
  → OpenAI (baseline model)   ┐
  → OpenAI (candidate model)  ┘ → both responses captured
  → Code (assign A/B slots randomly, build judge prompt)
  → OpenAI (judge scores A and B, 1-10 per dimension)
  → Code (parse scores, validate, tally winner)
  → Code (aggregate averages + win rates → recommendation)
  → OpenAI (write markdown report)
  → Google Sheets (append run summary)

The whole design rests on one idea: the judge must not know which response is yours. Everything else is plumbing.

1. Define the test set

Keep your test cases in a Config node (n8n-nodes-base.set v3.4) or a Data Table. n8n's evaluation guide makes a point worth repeating: stock the set with real edge cases, past failure points, and adversarial inputs, not random questions. A test set of softballs makes every model look great and tells you nothing.

Parse the cases out with a n8n-nodes-base.code v2 node and split them so each question flows through as its own item.

2. Run both models

Send each question to two @n8n/n8n-nodes-langchain.openAi v2.1 nodes, one baseline and one candidate. Capture both responses. Each node returns its text at $json.output[0].content[0].text. If you wired this from an old tutorial and the responses come back empty, that's the path: the deprecated n8n-nodes-base.openai node read $json.choices[0].message.content, which the current node doesn't populate.

3. Anonymize and assemble the judge prompt

Here's the step that makes the whole thing honest. In a n8n-nodes-base.code v2 node, randomly assign each model's answer to slot "Response A" or "Response B" per question, and remember the mapping. Build a judge prompt that presents both anonymously with a clear rubric.

Randomize the slot, don't just hardcode baseline-as-A. Judges have a measurable position bias toward the first option. If your candidate always sits in slot B, you'll under-credit it for reasons that have nothing to do with quality.

4. Score blind

Send the assembled prompt to a judge model, a strong one, since it's grading, not generating. Ask for strict JSON: a 1-10 score per dimension for both A and B, plus a one-line justification. Parse it in a n8n-nodes-base.code v2 node, validate that every score is a number in range, and map A/B back to baseline/candidate using the mapping you stored in step 3.

A blunt take that saves money: don't use your cheapest model as the judge. The judge's reliability caps the reliability of the entire eval. A weak judge produces confident, wrong verdicts, and you'll trust them because the workflow ran clean. Spend on the judge, save on the candidates.

5. Aggregate and report

A final n8n-nodes-base.code v2 node tallies per-dimension averages, computes the win rate (how often candidate beat baseline across the suite), and emits an overall recommendation. A synthesis LLM call turns those numbers into a readable markdown report, and a Google Sheets append stores the run so you can watch quality drift over time.

n8n RAG evaluation patterns

The blind-scoring spine generalizes well beyond a single model comparison.

Pattern 1: Prompt regression guard

Wire the eval to a scheduleTrigger v1.2 or run it before any prompt change ships. The baseline is your current production prompt; the candidate is the edit. If the win rate doesn't clear a threshold, the change doesn't ship. This turns "it felt better" into a gate.

Pattern 2: RAG retrieval comparison

Instead of two models, hold the model fixed and vary the retrieval config — chunk size, top-k, reranking on versus off. Same blind-scoring spine, but now you're measuring whether the new retrieval actually surfaces more relevant context. Pair it with n8n's built-in document-relevance metric for a second read.

Question → Retrieval Config A → Answer A ┐
Question → Retrieval Config B → Answer B ┘ → blind judge → win rate

Pattern 3: Groundedness check

Add a second judge pass that scores only whether each answer is grounded in the retrieved documents, catching hallucinations a quality score alone would miss. An answer can read beautifully and still be invented.

n8n nodes you'll use most

Getting started with n8n RAG evaluation

1. Add an OpenAI credential and a Config node holding your test questions and the two model names.
2. Build the parse-and-split step so each question flows as its own item.
3. Wire the baseline and candidate @n8n/n8n-nodes-langchain.openAi v2.1 nodes.
4. Write the n8n-nodes-base.code v2 anonymizer — random A/B slot plus stored mapping.
5. Add the judge call with a strict-JSON rubric and the score parser.
6. Build the aggregation node and the markdown report step.
7. Append to Google Sheets, then run the suite twice to confirm the win rate is stable.

If you'd rather not hand-build the anonymizer and the score aggregation, the RAG Eval: Blind A/B Agent Tester ships the whole rig ready to point at your test set.

n8n RAG evaluation is downstream of building the AI workflow in the first place. The walkthroughs in n8n LLM agent workflows and n8n OpenAI workflows cover the build side, including the parse-step discipline this eval depends on. To keep the test set's source documents fresh, the Onboarding Knowledge Retention template handles the knowledge-base side.