---
name: experiment-validity-checker
description: Systematic experimental validity audit for empirical AI research. Checks internal validity, construct validity, statistical conclusion validity, and AI-specific threats (self-evaluation bias, condition awareness, benchmark contamination). Produces a structured validity report with PASS/FAIL per threat category and required fixes before data collection. Use before running experiments (Node 4→5 gate) or to audit existing experiment designs.
---

# Experiment Validity Checker

## Purpose

Catches experimental design flaws that technical correctness checks miss. Specifically addresses threats that cause results to be uninterpretable even when the data is numerically clean.

Grounded in:
- Shadish, Cook & Campbell (2002) — *Experimental and Quasi-Experimental Designs*
- Wohlin et al. (2012) — *Experimentation in Software Engineering*
- Chang et al. (2023) — *A Survey on Evaluation of LLMs* (NeurIPS 2023)
- AI-specific: Ribeiro et al. checklist, NeurIPS 2023 evaluation guidelines

---

## Five Validity Threat Categories

### Category 1: Measurement Independence (MI)
*Can the measurement be trusted as objective?*

| Check | Question | Why it matters |
|-------|----------|---------------|
| MI-1 | Is the evaluator/judge a **different agent** from the generator? | Same agent self-scoring inflates scores for the condition it "knows" is supposed to win |
| MI-2 | Is the evaluator **blind to condition assignment**? | Knowing "this is Condition D (the proposed method)" creates demand characteristics |
| MI-3 | Is the **same model family** used for generation AND evaluation? | Self-serving bias: model recognizes and prefers its own outputs |
| MI-4 | Are evaluations run in **randomized order**, not condition order? | Order effects: later scores drift toward earlier anchors |
| MI-5 | For multi-dimension rubrics: are dimension scores assigned **independently**, not holistically? | Halo effect: high accuracy makes everything else score high |

**Our experiment failure:** MI-1, MI-2, MI-3 all violated. Same subagent generated AND scored, with full knowledge of condition.

**Symptom to watch for:** SD=0 or near-zero variance within high-performing conditions → evaluator anchored on condition label, not content.

---

### Category 2: Internal Validity (IV)
*Does the manipulation (architecture) actually cause the outcome (quality)?*

| Check | Question |
|-------|----------|
| IV-1 | Are **all confounding variables controlled**? (token budget, model version, temperature, system prompt, task prompt wording) |
| IV-2 | Are domains/tasks **randomly assigned** to condition order, not systematically? |
| IV-3 | Is there a **no-treatment baseline** (single agent, zero loops)? |
| IV-4 | Is the **manipulation check** present? (verify that Condition B actually ran more inner iterations than A, etc.) |
| IV-5 | Could **history effects** explain results? (later runs benefit from warmed-up context or cached patterns) |

---

### Category 3: Construct Validity (CV)
*Does the metric measure what you claim it measures?*

| Check | Question |
|-------|----------|
| CV-1 | Is the quality metric (e.g., OQS) **face-valid** for the construct "quality of knowledge work output"? |
| CV-2 | **Mono-method bias**: is there only one measurement instrument? (If yes: require ≥2 independent measures) |
| CV-3 | **Mono-operation bias**: is there only one task type? (If yes: require ≥2 task types or note as threat) |
| CV-4 | Is the metric **sensitive** to the expected manipulation? (e.g., if architecture changes depth but metric only measures length, it's insensitive) |
| CV-5 | Is the metric **pre-registered** before data collection, or selected post-hoc? |

---

### Category 4: Statistical Conclusion Validity (SV)
*Are the statistical claims warranted?*

| Check | Question |
|-------|----------|
| SV-1 | **Power**: is n per condition sufficient for power ≥ 0.80 at expected effect size? |
| SV-2 | **Assumption verification**: are ANOVA/t-test assumptions checked? (normality, homogeneity of variance — Levene's test) |
| SV-3 | **Multiple comparisons**: if testing H1+H2+H3 simultaneously, is family-wise error rate controlled? (Bonferroni or FDR) |
| SV-4 | **Effect size** reported alongside p-values? (Cohen's d, η², ω²) |
| SV-5 | **SD=0 check**: if any condition has near-zero variance, flag as "degenerate — statistical tests invalid for that condition" |

---

### Category 5: AI-Specific Threats (AI)
*Threats unique to LLM/agent evaluation*

| Check | Question |
|-------|----------|
| AI-1 | **Self-evaluation bias**: does the model evaluate outputs it generated? → CRITICAL FAIL if yes without mitigation |
| AI-2 | **Condition awareness**: does the evaluator have access to metadata revealing which condition produced the output? |
| AI-3 | **Benchmark contamination**: is the task similar to content in the model's training data? (reduces external validity) |
| AI-4 | **Prompt sensitivity**: would minor rewording of the task prompt change results substantially? (recommend sensitivity test) |
| AI-5 | **Length bias**: does the metric correlate with output length independent of quality? (LLM judges often prefer longer outputs) |
| AI-6 | **Self-enhancement bias**: does the model give higher scores to outputs from the same model family? |

---

## How to Run This Checker

### Pre-experiment (recommended — run between Node 4 and Node 5)

Read the experiment design document and score each check above.

Output format: `validity_report.json`

```json
{
  "experiment": "...",
  "checked_at": "Node 4→5 gate",
  "categories": {
    "MI": {
      "checks": {
        "MI-1": {"pass": true/false, "finding": "...", "severity": "critical/major/minor"},
        "MI-2": {...},
        ...
      },
      "category_pass": true/false
    },
    "IV": {...},
    "CV": {...},
    "SV": {...},
    "AI": {...}
  },
  "critical_fails": ["MI-1: ...", "AI-1: ..."],
  "major_fails": [...],
  "overall_verdict": "PROCEED" | "FIX_REQUIRED" | "HALT",
  "required_fixes": [
    {
      "check": "MI-1",
      "problem": "Generator and evaluator are the same agent",
      "fix": "Spawn a separate Evaluator subagent with ONLY the report text and rubric — no condition metadata, no task context",
      "implementation": "After generating report, write ONLY the text to a temp file. Spawn fresh subagent reading only that file + rubric."
    }
  ]
}
```

**Verdict rules:**
- Any `critical` fail → `HALT` (do not proceed to data collection)
- Any `major` fail → `FIX_REQUIRED` (fix before proceeding)
- Only `minor` fails → `PROCEED` with disclosed limitations

**Critical severity:** MI-1, MI-2, AI-1, AI-2 (destroy result interpretability)
**Major severity:** SV-1, SV-5, IV-1, CV-2 (serious but survivable with disclosure)
**Minor severity:** IV-2, IV-5, CV-3, AI-3, AI-4, AI-5 (disclose as limitations)

---

### Post-experiment audit (use when data is already collected)

Run the same checks, but shift output to:
- Which threats are present in the **already collected data**
- Which findings can still be salvaged with appropriate caveats
- Which claims must be **retracted or reframed** in the paper

---

## Required Fix for Self-Evaluation Bias (MI-1 / AI-1)

This is the most common and most damaging failure in LLM evaluation experiments.

**Pattern that causes it:**
```
Single subagent call:
  [SYSTEM] You are running experiment condition D...
  [USER]   Generate the report, then score it.
  → Agent generates report, scores it with full condition awareness
```

**Fixed pattern:**
```
Step 1 — Generate (subagent A, has condition context):
  [SYSTEM] You are writing a Technology Assessment Report.
  [USER]   Topic: {domain}. Produce the report. Condition D protocol: [...]
  → Writes report, saves to reports/D3_D_text_only.txt (TEXT ONLY, no score)

Step 2 — Score (subagent B, BLINDED):
  [SYSTEM] You are a quality evaluator. Score the following report on 7 dimensions (1-7 each).
           Do NOT infer anything about how the report was produced.
  [USER]   <report text only, no run_id, no condition label, no metadata>
           Rubric: [...]
  → Returns dimension scores only
  → Orchestrator logs run_id + condition + scores together
```

Key requirements for blinded evaluator:
1. No `run_id` in the prompt (contains condition letter)
2. No process log in the prompt (D3_D shows inner loop iterations)
3. No system prompt mentioning the experiment
4. Ideally: different model family from generator

---

## Integration with dual-loop-research

This skill runs as **Node 4.5** — a gate between experiment design (Node 4) and implementation (Node 5).

In `dual-loop-research`, Node 4.5 works like Node 3 (Judgment Gate): binary verdict.
- `PROCEED` → continue to Node 5
- `FIX_REQUIRED` → feedback to Node 4 Generator to revise design
- `HALT` → escalate to Planner / trigger Outer Loop

Node 5 Generator must implement the **Required Fixes** listed in `validity_report.json` before running any experiments.

---

## Invoke

```
/experiment-validity-checker
Design: [path to exp_design.md]
Mode: pre-experiment | post-audit
```