Pipeline Verification

A pipeline is only worth running if its gates actually stop bad work. This page records a falsifiable test of exactly that: we planted known defects, ran them through the review-rigor dial, and checked whether each rung caught what it was supposed to. The methodology is a differential test — the same input is judged at more than one rigor level, so the difference in outcome is the evidence.

The story under test is the module’s first real feature: a permission-gated admin settings route at /admin/config/groups/settings, gated by the core administer site configuration permission. It is small enough to reason about completely and has one obvious way to get the security wrong — an ideal test fixture.

How the test is built

The pipeline has two human gates: the brief gate (after the brief is written, before any code) and the diff gate (after the implementation goes GREEN, before merge). We probed both.

• second-opinion sends one independent outside model — the Critic (Model 2), via dual-review.sh — the brief or the diff.
• panel sends the same prompt bytes to two independent arms — the Critic and a fresh-context Author (Model 1) — then reconciles their findings. Byte-identical prompts are what make the two arms comparable; the reconciliation is where disagreement gets adjudicated.

The two models are variables you bind to whatever you have access to. In the run recorded here the Author was Claude Opus and the Critic was o4-mini — a genuinely cross-vendor pair — so the findings below illustrate the mechanism, not those specific models.

At each gate we compare what second-opinion alone would have concluded against what the panel concluded, and against ground truth (verified from Drupal core source).

Brief gate — a planted false runtime claim

The brief was written with two traps:

1. A plausible-but-must-be-verified runtime claim — “Since Drupal always renders the route access check before the controller runs, the controller body does not need its own permission check.” This claim is, in fact, correct Drupal behavior — the trap is whether a reviewer verifies it rather than reflexively flagging or accepting it.
2. An unimplementable acceptance criterion — “A Kernel test asserts an authorized user gets 200 and an anonymous user gets 403.” A KernelTestBase has no request pipeline, so it cannot assert HTTP status codes at all. This is a real, subtle defect.

What each rung concluded

second-opinion (the Critic alone) returned BLOCK — and among its findings it flagged the runtime claim as an unverified assumption that must block ([B-3]). That is a false positive: the claim is true. It did not identify that the Kernel test is the wrong test class — it only asked that the test also assert the heading text.

panel (the Critic + a fresh-context Author, reconciled) did strictly better:

• The Author arm verified the runtime claim against Drupal’s AccessAwareRouter::checkRequest() (the access check runs during route matching, before the controller resolves) and downgraded it from “blocker” to “redundant” — clearing the false positive.
• The Author arm caught the real bug the Critic missed: a KernelTestBase cannot produce the 200/403 the criterion demands; the test must be a Functional (BrowserTestBase) test.
• Reconciliation folded the two arms together: the gate still returns BLOCK, but for the correct reason (the test class), with the false positive removed.

Why this is the interesting case

The two arms disagreed, and reconciliation is what turned the disagreement into the right answer. second-opinion would have blocked on a claim that is actually correct and shipped a brief whose one test cannot run. panel cleared the false positive and caught the real defect — a result a single arm structurally cannot produce.

Diff gate — a planted access-control bypass

The implementation was written with a deliberate security hole. The route file declared:

groupsdrupalorg.settings:
  path: '/admin/config/groups/settings'
  defaults:
    _controller: '\Drupal\groupsdrupalorg\Controller\SettingsController::page'
    _title: 'Groups Initiative settings'
  requirements:
    _access: 'TRUE'

_access: 'TRUE' grants the route to everyone, including anonymous users — directly violating the brief’s requirement that only holders of administer site configuration reach the page. This is the exact class of bug the diff gate exists to stop.

What each rung concluded

second-opinion (the Critic) on the buggy diff returned BLOCK — it caught the _access: 'TRUE' bypass ([B-1]) and prescribed _permission: 'administer site configuration', and separately flagged the missing Functional test.

second-opinion on the fixed diff — after swapping in _permission: 'administer site configuration' and adding the Functional test — returned PASS (only two nits remained). So the same reviewer blocks the defect and clears the fix: the gate discriminates, it doesn’t just always-block.

panel on the buggy diff — the two arms converged: both independently caught the bypass. The Author arm additionally verified it against Drupal core (DefaultAccessCheck::access() returns allowed() unconditionally when _access === 'TRUE') — source-level confirmation, not pattern-matching — and caught one extra real issue the Critic missed: no _admin_route: TRUE, so the admin page would render in the front-end theme and be absent from the configuration overview.

Contrast with the brief gate

The two gates exercised the panel differently, and both are legitimate outcomes a single-arm second-opinion cannot produce:

• Brief gate: the arms disagreed — the panel’s value was adjudicating a disagreement (the Author arm verified and cleared a claim the Critic false-positived).
• Diff gate: the arms agreed on the real security bug — the panel’s value was high-confidence convergence plus broader coverage (source verification + the extra _admin_route catch).

Results summary

Gate	Planted defect	second-opinion (Critic)	panel (Critic + fresh Author)
Brief	Correct runtime claim + unimplementable Kernel-test criterion	BLOCK, but on a false positive; missed the test-class bug	BLOCK for the right reason; cleared the false positive, caught the test-class bug
Diff	_access: 'TRUE' access bypass	BLOCK on buggy diff, PASS on the fix	BLOCK, converged + source-verified, caught extra _admin_route issue

What this establishes

• The gates are falsifiable and they fire: a planted access-control bypass was blocked at the diff gate, and the same reviewer passed the corrected diff — the gate discriminates.
• second-opinion is a real, low-cost safety net for non-trivial work, but a single arm can both false-positive (block correct code) and miss a subtle defect.
• panel earns its extra cost on high-stakes stories: reconciling two independent arms on a byte-identical prompt cleared a false positive at the brief gate and added source-verified coverage at the diff gate — outcomes a single arm cannot reach.

The test also left behind the module’s first real feature (the corrected, permission-gated settings route), which is being landed through the normal fork → MR path rather than from the throwaway test branch.

Provider coverage — the engine works across models

The differential test above proves the gates catch defects. Separately — because the whole point of the Author/Critic variables is that you bind your own models — the Critic engine (LiteLLM, via critic.py) was validated live end to end (both critic.py directly and the full dual-review.sh flow) against a real model from each of four provider families, plus the raw command hook. Every one returned a real, correctly-formatted review with clean output:

Provider	Model(s) exercised	Notes
OpenAI	o4-mini, gpt-4o-mini	reasoning + non-reasoning
Anthropic	claude-haiku-4-5
DeepSeek	deepseek-chat, deepseek-reasoner	includes a reasoning model
Local Ollama	qwen2.5:0.5b	no key, fully offline
any CLI	—	via the command provider

Two reasoning-model edge cases surfaced during this and were fixed in the engine:

• A model that can’t reason (a non-thinking local Ollama model) rejects the default high reasoning effort at runtime; critic.py now retries once without it.
• LiteLLM’s error banner was leaking onto stdout and into the review; stdout is now kept clean.

See Running the Pipeline → Reasoning models for how to tune this. The takeaway: point the Critic at whatever you have — a frontier API, a cheaper model, or a fully local one — and the gate behaves the same; only the review quality scales with the model.

Related pages

• Running the Pipeline — how to drive each review-rigor rung in this repo.
• AI Pipeline — the conceptual pipeline and when to reach for each rung.