--- name: phd-agent description: End-to-end academic research paper writer that mirrors a real PhD workflow — Foundation → Pilot → Full Experiment → Writing — with a substantive advisor review at every step. The advisor (main conversation) gives dialectical critique rather than checklist scoring, may BACKTRACK to earlier phases, and (optionally) pauses for user confirmation at three named gates. Generators are spawned as subagents; pilot results must show mechanism activation before full experiment runs. Venue (CHI / NeurIPS / CSCW / ACL / etc), pilot sample size, and gate-confirmation mode are all configurable per-invocation. Produces a venue-ready paper PDF + conference PPT, with a live web dashboard at /dashboard.html. Use when writing a full research paper from scratch with literature, hypotheses, pilot, experiments, analysis, and writing. --- # PhD Agent — End-to-End Research Paper Harness The shape of the workflow: > Literature → hypotheses → design → **pilot** → reflect → full study → analyze → write — with an **advisor conversation** at every step, and the freedom to **backtrack** when something earlier looks wrong. The advisor is the main conversation (you). Generators are subagents. The advisor produces structured critique, not a 0–100 score, and is allowed to send the project back to a previous phase. The pilot phase is mandatory — its job is to detect mechanism failures and design bugs before any compute is spent on the full run. --- ## Architecture Overview ``` PHASE 1 — FOUNDATION Step 1.1 Literature survey (Generator subagent → Advisor critique) Step 1.2 Hypotheses (Generator subagent → Advisor critique) Step 1.3 Experiment design (Generator subagent → Advisor critique) ── Phase Gate 1 ───────────────────────────────────────────────────── Advisor writes a brief; user confirmation only if `gates_mode = manual` PHASE 2 — PILOT Step 2.1 Pilot run (Generator subagent: code + execute pilot N) Step 2.2 Pilot analysis (Generator subagent → Advisor critique) Step 2.3 Pilot verdict (Advisor + Adversarial subagent) ── Phase Gate 2 ───────────────────────────────────────────────────── Verdict ∈ {PROCEED, REVISE_DESIGN→1.3, REVISE_HYPOTHESIS→1.2, REVISE_FOUNDATION→1.1, ABORT} PHASE 3 — FULL EXPERIMENT Step 3.1 Full run (Generator subagent: code + execute full N) Step 3.2 Full analysis (Generator subagent → Advisor critique) Step 3.3 Significance verdict (Advisor + Adversarial subagent) ── Phase Gate 3 ───────────────────────────────────────────────────── Verdict ∈ {PROCEED, ADD_RUNS→3.1, REVISE_ANALYSIS→3.2, BACKTRACK→Phase 2, ABORT} PHASE 4 — WRITING Step 4.1 Discussion / framing (Generator subagent → Advisor critique) Step 4.2 Paper (LaTeX → PDF) (Generator subagent → Advisor critique) Step 4.3 Slides (PPTX) (Generator subagent → Advisor critique) CROSS-CUTTING: state.json is the single source of truth (rewritten on every step). advisor_log.jsonl — every advisor verdict appended. backtrack_log.jsonl — every backtrack event with reason. dashboard.html — auto-regenerated after every state write. ``` ### Core invariants - **Pilot phase is mandatory** with a manipulation-check verdict before any full run. - Advisor is dialectical, not a score-out-of-100. - **Backtracking is first-class** — a verdict can route the project to any earlier step. - **Phase Gates are opt-in** (`gates_mode = manual` to require user confirmation; default is `auto`). - **Adversarial subagent** is invoked at the two pilot/full verdicts to catch self-flattering interpretations. - **Web dashboard** auto-updates so the user never has to read JSON to know where things stand. - **Venue is parametric** — no hardcoded venue name anywhere in this skill or in the prompts it issues. Wherever a prompt needs the venue, it contains the literal string `` and you replace it with `state.target_venue` when you actually call the tool. --- ## Step 0: Initialize Project ### 0.1 Create directory structure ``` paper/ ← or whatever output dir the user passed state.json ← truth source, rewritten on every step advisor_log.jsonl ← one line per advisor verdict backtrack_log.jsonl ← one line per backtrack event orchestrator_log.md ← human-readable audit trail (append-only) dashboard.html ← auto-generated, open in browser update_dashboard.py ← regenerates dashboard.html from state + logs phase1_foundation/ step1_lit/ step2_hyp/ step3_design/ phase2_pilot/ step1_run/ step2_analysis/ step3_verdict/ phase3_full/ step1_run/ step2_analysis/ step3_verdict/ phase4_writing/ step1_discussion/ step2_paper/ step3_slides/ ``` ### 0.2 Initialize state.json ```json { "topic": "", "target_venue": "", "output_dir": "", "config": { "gates_mode": "auto", "pilot_n_per_condition": null, "pilot_n_rationale": null, "max_inner_iter": 5, "backtrack_limit_per_step": 3 }, "current_phase": 1, "current_step": "1.1", "phase_gate_status": { "1": "pending", "2": "pending", "3": "pending" }, "steps": { "1.1": {"name": "literature", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "1.2": {"name": "hypotheses", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "1.3": {"name": "exp_design", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "2.1": {"name": "pilot_run", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "2.2": {"name": "pilot_analysis", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "2.3": {"name": "pilot_verdict", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "3.1": {"name": "full_run", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "3.2": {"name": "full_analysis", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "3.3": {"name": "full_verdict", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "4.1": {"name": "discussion", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "4.2": {"name": "paper", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null}, "4.3": {"name": "slides", "status": "pending", "iterations": 0, "last_verdict": null, "artifact": null} }, "backtrack_count": {"1.1": 0, "1.2": 0, "1.3": 0, "2.1": 0, "2.2": 0, "2.3": 0, "3.1": 0, "3.2": 0, "3.3": 0}, "final_output": {"paper": null, "slides": null, "dashboard": "/dashboard.html"} } ``` ### 0.2.1 Configuration knobs | Field | Default | Meaning | |---|---|---| | `config.gates_mode` | `"auto"` | `auto` = advisor writes a gate brief and proceeds without asking the user; `manual` = call AskUserQuestion at every Phase Gate and block until reply | | `config.pilot_n_per_condition` | `null` | Set during Step 1.3 by the design generator, based on the actual experiment. **Never hardcode.** Rule of thumb: enough trials per condition to make the manipulation-check signal interpretable (typically 1–10% of full N, with a floor of 2). | | `config.pilot_n_rationale` | `null` | One-sentence justification for the chosen pilot N (advisor will challenge this). | | `config.max_inner_iter` | `5` | Max REVISE iterations per step before `max_iter_reached` is logged. | | `config.backtrack_limit_per_step` | `3` | Hard cap on how many times any single step may be backtracked into before escalation. | **Read these from state, not from memory.** Wherever a prompt template in this skill contains a literal `` or `` etc., replace it with the value from `state.target_venue` / `state.config.*` when you actually call the tool. ### 0.3 Write `update_dashboard.py` A static-HTML generator (no server needed). Reads `state.json`, `advisor_log.jsonl`, `backtrack_log.jsonl`, and any artifact summaries; writes `dashboard.html` with `` so it self-refreshes when opened in a browser. The dashboard MUST display: - **Header**: topic, venue, current phase + step, gates_mode, time of last update - **Phase progress bar**: 4 phases × 3 steps each (12 cells); colored by status (`pending` grey, `in_progress` yellow, `done` green, `backtracked` orange, `failed` red) - **Phase Gate banners**: for each of the 3 gates, show `pending / auto_passed / awaiting_user / passed` with the most recent advisor verdict and adversarial counterpoint - **Latest advisor verdict per step**: collapsible card with verdict, link to `advisor_review_iter{n}.md`, and a **Checks Run badge** (green = all required checks PASS, amber = any INCONCLUSIVE, red = any FAIL or required check missing). Expanding the card lists every check name → result → evidence-file link, and the anti-pattern self-test answers. **This is how the user verifies the advisor actually did the work.** - **Backtrack log table**: from / to / reason / round - **Key results panel** (visible from Phase 2 onwards): manipulation-check status, primary effect with CI, p-value, sample size — pulled from the latest analysis artifact - **Artifact links**: clickable file:// links to every produced artifact, including every `advisor_review_iter*.md` and `*_evidence.md` The script is invoked as `python paper/update_dashboard.py` after every state mutation. Implementation should be ≤ 250 lines, stdlib only (no external deps), and tolerant of missing optional fields. ### 0.4 Announce to user > "Project initialized at `/`. Open `/dashboard.html` in your browser for a live view (auto-refreshes every 10s). Gates mode = `` (in `auto` I proceed through Phase Gates without asking; in `manual` I block at each gate). Truth source is `/state.json`." --- ## The Advisor Role (main conversation) You — the main conversation — are the advisor. Do not delegate critique to a subagent. The point of being in the main conversation is that you carry the cumulative project context. ### Advisor critique template (use for every step) Write this to `/phaseX_*/stepY_*/advisor_review_iterN.md`. The §4 `CHECKS_RUN` block is the enforcement mechanism — it is what makes "did the advisor actually do the spot-check" auditable instead of trust-me. ``` ADVISOR REVIEW — Step {x.y} {step_name} (iteration {n}) 1. WHAT THE GENERATOR PRODUCED (one paragraph, factual) 2. STRENGTHS (concrete, evidence-cited — point to file:line) 3. CONCERNS (most-important first; each must cite evidence from the artifact, prior steps, or the literature — not vibes) 4. CHECKS_RUN (mandatory; every check required for this step must appear here with a concrete result OR the line "SKIPPED — ") - check: evidence: result: PASS | FAIL | INCONCLUSIVE note: <≤1 line, e.g. "all 5 papers found in Scholar; titles + URLs in advisor_review_iterN_evidence.md"> - check: ... 5. WHAT A SKEPTICAL REVIEWER WOULD SAY (steel-man the strongest objection; replace with state.target_venue when writing) 6. ANTI_PATTERN_SELF_TEST (every item below must be answered yes/no — see "Anti-pattern checklist" at the end of this skill for the canonical list) - read_artifact_directly: yes | no - recomputed_or_spotchecked: yes | no | n/a - consistent_with_prior_steps: yes | no - manipulation_checks_pass (Phase 2/3 verdicts only): yes | no | n/a - adversarial_objection_addressed (2.3 / 3.3 only): yes | no | n/a - would_user_regret_this_pass: yes | no 7. VERDICT (one of): - PASS — proceed to next step - REVISE — same step, another iteration; list specific fixes - BACKTRACK_TO_{step} — earlier step is broken; explain what changed our mind - ABORT — fundamental problem, no path forward; explain 8. SPECIFIC ASKS (numbered, actionable, each tied to a concern in §3) ``` ### Hard rule: no PASS without complete §4 and §6 PASS is only legal when, in this very review: - §4 contains an entry for **every** check in the step's `required_checks` list (see each step below) with `result ∈ {PASS, INCONCLUSIVE}`. Any `FAIL` ⇒ REVISE. - §6 has all `no_*` items answered `no` and all `yes_*` items answered `yes`. Any deviation ⇒ REVISE. - Each §4 evidence pointer resolves: file paths exist; cited line numbers contain what's claimed; subagent outputs (e.g., scholar verifications) are saved to a sibling `*_evidence.md` file so the user can re-open them. If you catch yourself wanting to PASS without filling §4 / §6: that is the failure mode this rule exists to stop. Fill them honestly or set verdict to REVISE. ### Advisor must AVOID - Numeric scoring as the primary judgment (use it only to anchor severity if helpful) - Praising the generator for completing the task (completion is table stakes) - Accepting "looks reasonable" without recomputing or re-reading - Verdict PASS without filling out §4 CHECKS_RUN and §6 ANTI_PATTERN_SELF_TEST honestly ### Append every verdict to `advisor_log.jsonl` The `checks_run` and `anti_pattern_self_test` arrays are required — they are the machine-readable mirror of §4 and §6 of the markdown review. The dashboard reads these directly and renders them per step. ```json { "ts": "", "step": "1.1", "iter": 1, "verdict": "REVISE", "summary": "<≤200 chars>", "review_md": "/phase1_foundation/step1_lit/advisor_review_iter1.md", "checks_run": [ {"check": "spot_check_5_papers", "result": "FAIL", "evidence_file": "/phase1_foundation/step1_lit/advisor_review_iter1_evidence.md", "note": "2 of 5 not found in Scholar"} ], "anti_pattern_self_test": { "read_artifact_directly": true, "recomputed_or_spotchecked": true, "consistent_with_prior_steps": true, "manipulation_checks_pass": null, "adversarial_objection_addressed": null, "would_user_regret_this_pass": true }, "specific_asks": ["remove papers X, Y from survey", "..."] } ``` Then run `python paper/update_dashboard.py`. The dashboard surfaces a per-step "Checks Run" badge (green = all PASS, amber = any INCONCLUSIVE, red = any FAIL or any required check missing). --- ## Per-step inner loop For each step (12 in total): ``` iter = 0 while True: iter += 1 UPDATE state.json: current_step = "x.y", steps["x.y"].status = "in_progress", iterations = iter LOG to orchestrator_log.md REGENERATE dashboard spawn Generator subagent: - subagent_type chosen for the step (see step definitions) - prompt = step spec + previous advisor asks (if iter > 1) + paths to all prior-step artifacts - subagent writes its artifact to paper/phaseX_*/stepY_*/ [main conversation = Advisor] Read the artifact directly. Spot-check claims. Run the critique template above. Append verdict to advisor_log.jsonl. UPDATE state.json: steps["x.y"].last_verdict = verdict, .artifact = path REGENERATE dashboard if verdict == PASS: steps["x.y"].status = "done"; break if verdict == REVISE: if iter >= state.config.max_inner_iter: mark "max_iter_reached", proceed with best effort, log warning; break continue if verdict.startswith("BACKTRACK_TO_"): target = parse target step backtrack_count[current_step] += 1 if backtrack_count[current_step] > state.config.backtrack_limit_per_step: ESCALATE to user (AskUserQuestion): cycle limit hit; options = [continue anyway, abort, re-spec by hand] else: mark current step "backtracked"; jump to target step (it re-enters its own inner loop) append to backtrack_log.jsonl break if verdict == ABORT: ESCALATE to user; do not silently continue. ``` --- ## Phase 1 — Foundation ### Step 1.1 Literature Survey - **Generator**: subagent_type=`general-purpose` (or Explore for thorough search), invoking `/meta-research` Phase 1 + `/google-scholar-search`. - **Output**: `/phase1_foundation/step1_lit/survey.md` — ≥30 papers, >50% from the last 24 months, explicit research-gap section. - **Advisor required_checks** (each must appear in §4 of the review): 1. `spot_check_5_papers` — pick 5 citations at random; verify via `/google-scholar-search`. Save the queries + results to `advisor_review_iter{n}_evidence.md`. Any unverifiable paper ⇒ FAIL ⇒ REVISE with that paper removed. 2. `gap_is_actual_gap` — search for the claimed gap; confirm no paper in the survey already addresses it. Cite the search query + top results in evidence. 3. `recency_threshold` — count papers from the last 24 months; record the count and the threshold. ### Step 1.2 Hypotheses - **Generator**: subagent_type=`general-purpose`, `/meta-research` Phase 2. - **Output**: `/phase1_foundation/step2_hyp/hypotheses.md` — 2–4 hypotheses, each with operationalization, predicted direction, and ≥2 supporting citations from Step 1.1. - **Advisor required_checks**: 1. `falsifiability_per_hypothesis` — for each H_i list the observation that would refute it. 2. `novelty_vs_survey` — for each H_i confirm no paper in `survey.md` already proves it; cite which papers came closest. 3. `venue_fit` — explain in 1–2 sentences why this contribution clears the bar for `state.target_venue`. If it doesn't, verdict is `BACKTRACK_TO_1.1` with the specific depth gap. ### Step 1.3 Experiment Design - **Generator**: subagent_type=`general-purpose`, `/meta-research` Phase 3 for power analysis (subagent uses `scipy.stats` / `statsmodels` directly). - **Output**: `/phase1_foundation/step3_design/exp_design.md` — conditions (≥3: proposed + ≥2 baselines), DV (primary = Pairwise Win Rate; secondaries declared), full-experiment sample size with power ≥ 0.80 at α = 0.05, **manipulation checks** (one per claimed mechanism: how will we know the treatment actually happened in each condition?), blinding protocol for any LLM-as-Judge use, feasibility check, **AND a pilot-N proposal**. - **Pilot-N proposal**: the design generator picks `pilot_n_per_condition` and writes it to state via the orchestrator, plus a one-sentence `pilot_n_rationale` (e.g. "1 trial / condition is enough to verify mechanism activation since the manipulation check is binary"; or "5 / condition because the manipulation signal is noisy"). The number is NOT fixed — it depends on the experiment. - **Advisor required_checks**: 1. `experiment_validity_checker_run` — invoke `/experiment-validity-checker` and paste verdict + critical-fail list into evidence. 2. `manipulation_check_per_mechanism` — list every mechanism a hypothesis depends on; confirm `exp_design.md` defines a manipulation check for each (FAIL if any missing). 3. `pilot_n_rationale_defensible` — quote the rationale; either accept it or REVISE. Hardcoded numbers without rationale ⇒ FAIL. 4. `power_analysis_present` — confirm sample-size calculation is shown with assumed effect size. ### Phase Gate 1 Before pilot, the advisor writes a one-screen "Phase 1 brief": - 3-bullet summary of literature gap - The hypotheses - The conditions and primary DV - Manipulation checks proposed - The chosen `pilot_n_per_condition` and its rationale - Risks the advisor is aware of The brief is written to `/phase_gates/gate_1.md` and surfaced on the dashboard. Then, depending on `config.gates_mode`: - `auto` (default) → set `phase_gate_status["1"] = "auto_passed"`, log advisor's recommended verdict, proceed. - `manual` → call AskUserQuestion with options `proceed` / `request revisions: ` / `abort`. Block until reply. Set `phase_gate_status["1"]` accordingly. Regenerate dashboard. --- ## Phase 2 — Pilot The point of pilot is **not** to prove anything — it is to detect mechanism failures and design bugs before spending compute on the full run. ### Step 2.1 Pilot Run - **Generator**: subagent_type=`general-purpose`. Writes Python under `/phase2_pilot/step1_run/run.py` and **executes** it for the pilot N **declared in `state.config.pilot_n_per_condition`** (set during Step 1.3 by the design generator, never hardcoded). Total pilot runs = `pilot_n_per_condition × number_of_conditions_from_exp_design`. - **Output**: `pilot_results.json` (raw, with timestamps), `process_log.jsonl` (per-trial process events — every iteration, every escalation, every tool call so manipulation checks can verify mechanism activation), `run.log`. - **Hard requirement**: `process_log.jsonl` must capture the events that the **manipulation checks** in §1.3 depend on. ### Step 2.2 Pilot Analysis - **Generator**: subagent_type=`general-purpose`, `/python-dataviz` (subagent uses `scipy.stats` / `statsmodels` directly). - **Output**: `paper/phase2_pilot/step2_analysis/pilot_analysis.md` covering, in this exact order: 1. **Manipulation-check report**: for each declared mechanism, did it activate? (e.g., "Condition D's outer loop fired in 0/2 runs → mechanism NOT activated"). This section is non-negotiable. 2. Descriptive statistics by condition (means, SD, pairwise wins). 3. Effect-size point estimates with CIs (small-N pilots: report effect sizes, do not run underpowered NHST). 4. Anomalies / surprises. ### Step 2.3 Pilot Verdict (Advisor + Adversarial subagent) The advisor first drafts a verdict using the critique template, then **spawns an adversarial subagent** with the prompt below. Wherever you see `` in the prompt, replace it with `state.target_venue` when you actually call Agent — do not pass the literal string `` to the subagent. > "You are a skeptical program-committee member. Read the attached `exp_design.md`, `pilot_analysis.md`, and `process_log.jsonl`. Your job is to find the strongest reason this pilot does NOT justify proceeding to a full run. Specifically check: did the claimed mechanisms activate? Are confounds visible in the process data? Is the effect-size direction what the hypothesis predicted? Return a structured `objection.md` with: (1) primary objection, (2) supporting evidence with file:line cites, (3) what would change your mind." **Advisor required_checks for 2.3** (in addition to the verdict): 1. `manipulation_check_pass` — quote the manipulation-check section of `pilot_analysis.md`; mark each mechanism as activated / not activated. Any "not activated" ⇒ you may NOT verdict PROCEED — choose a REVISE_* target instead. 2. `adversarial_objection_addressed` — for each objection in `objection.md`, write a direct response (not "noted"). Save responses inline in the review. 3. `process_log_supports_claims` — for each numerical claim in `pilot_analysis.md` you intend to rely on, point to the specific lines in `process_log.jsonl` that back it. The advisor reads `objection.md`, integrates it, and writes the final verdict — one of: - `PROCEED` → Phase 3 - `REVISE_DESIGN → 1.3` (e.g., manipulation checks impossible to verify; add instrumentation) - `REVISE_HYPOTHESIS → 1.2` (e.g., predicted mechanism doesn't activate even when conditions are met → hypothesis was wrong about the mechanism) - `REVISE_FOUNDATION → 1.1` (e.g., literature missed a confound) - `ABORT` (no path forward) ### Phase Gate 2 Advisor writes `/phase_gates/gate_2.md` with: - Manipulation-check table (which mechanisms fired, which didn't) - Effect-size summary with CIs - Adversarial subagent's primary objection - Advisor's recommended verdict Then per `config.gates_mode`: - `auto` → log advisor's recommended verdict, set `phase_gate_status["2"] = "auto_passed"` (or `"auto_backtracked"` if verdict ≠ PROCEED), proceed accordingly. - `manual` → call AskUserQuestion with options `proceed` / `force backtrack to: ` / `abort` / `request revisions: `. Block until reply. Update `phase_gate_status["2"]`. Regenerate dashboard. --- ## Phase 3 — Full Experiment ### Step 3.1 Full Run - **Generator**: subagent_type=`general-purpose`. Re-uses `pilot/run.py` if unchanged, else regenerates. Executes the full N from `exp_design.md`. Writes `full_results.json`, `process_log.jsonl`, `run.log` to `paper/phase3_full/step1_run/`. - **Blinding**: enforced exactly as specified in `exp_design.md`. Scoring subagent (if LLM-as-Judge) sees only artifact text + rubric; no condition labels, run IDs, or process logs. - **Hard requirement**: every number that will appear in the paper must exist in `full_results.json` or `process_log.jsonl`. No exceptions. ### Step 3.2 Full Analysis - **Generator**: subagent_type=`general-purpose`, `/python-dataviz` (subagent uses `scipy.stats` / `statsmodels` directly). - **Output**: `/phase3_full/step2_analysis/stats.md` + publication-grade figures (PNG + PDF, 300 DPI, colorblind-friendly). - **Provenance rule**: every reported statistic cites its source line, e.g. `[src: full_results.json:L142]`. - **Required content**: manipulation-check report (again, on the full N), pre-registered analyses, exploratory analyses (clearly labeled), effect sizes with CIs, robustness checks. - **Advisor required_checks**: 1. `recompute_3_random_stats` — pick 3 numbers from `stats.md`; recompute from `full_results.json`; record reported_value, recomputed_value, and pass/fail in evidence. Any mismatch ⇒ FAIL. 2. `provenance_complete` — sample 5 statistics; confirm each has a `[src: ...]` cite and the cited line exists. 3. `manipulation_check_pass_full_n` — same gate as 2.3 but on full data. If a mechanism didn't activate at full N, the verdict cannot be PROCEED. 4. `figure_legibility_spotcheck` — open one PNG; confirm axes labeled, fonts legible, no clipping. ### Step 3.3 Significance Verdict (Advisor + Adversarial subagent) Same pattern as 2.3. Adversarial subagent prompt — replace `` with `state.target_venue` and `` with the advisor's draft when calling Agent: > "You are a skeptical program-committee member. The authors claim . Read `stats.md` and `full_results.json`. Identify: (1) any selective reporting (analyses run but not shown), (2) whether the manipulation checks support a causal interpretation, (3) any way the effect could be explained by a confound, (4) whether the effect size matters in practice. Write `objection.md`." **Advisor required_checks for 3.3**: 1. `selective_reporting_audit` — list every analysis mentioned anywhere in Phase 3 artifacts; confirm each appears in `stats.md` (or is explicitly marked as exploratory / dropped, with reason). 2. `manipulation_check_pass_full_n` — quote and gate on the full-N manipulation check. 3. `adversarial_objection_addressed` — direct response per objection. 4. `effect_size_practical` — state the effect size in plain English; decide whether it matters; record the judgment. Advisor verdict ∈ {`PROCEED`, `ADD_RUNS → 3.1`, `REVISE_ANALYSIS → 3.2`, `BACKTRACK → Phase 2`, `ABORT`}. ### Phase Gate 3 Advisor writes `/phase_gates/gate_3.md` with: - Primary effect with CI and effect size - Manipulation-check summary - Adversarial objection - Whether the result is publishable as-is (advisor's honest opinion, including "publishable as a null result" if applicable) Then per `config.gates_mode`: `auto` proceeds with the recommended verdict; `manual` calls AskUserQuestion and blocks. Update `phase_gate_status["3"]`. --- ## Phase 4 — Writing ### Step 4.1 Discussion / Framing - **Generator**: subagent_type=`general-purpose`. Drafts `discussion.md` — what the result means, how it relates to literature, limitations, future work. Must use only facts established in Phases 1–3. - **Advisor required_checks**: 1. `every_claim_traceable` — every factual claim has a citation (paper from 1.1) or a result line (from 3.2). Sample 5 at random; FAIL if any unsourced. 2. `limitations_honest` — confirm limitations include the actual weaknesses surfaced in the adversarial objections, not boilerplate. 3. `no_retrofitted_hypotheses` — diff the hypotheses in `discussion.md` against `hypotheses.md`. They must match. Surprise findings must be framed as such. ### Step 4.2 Paper (LaTeX → PDF) - **Generator**: subagent_type=`general-purpose`, invokes `/research-paper-writer`. Writes `paper.tex` and compiles to `paper.pdf` using a local LaTeX toolchain — `tectonic` preferred, else `xelatex`/`pdflatex`. If no local toolchain is available, fall back to `/typetex` (hosted compile API). - **Output**: `/phase4_writing/step2_paper/paper.tex` + `paper.pdf`. - **Format**: read `state.target_venue` and pick the appropriate template (e.g. ACM sigconf for CHI/CSCW, NeurIPS template for NeurIPS, ACL template for ACL/EMNLP). Page limit follows the venue's own rules. - **Advisor required_checks**: 1. `pdf_compiles` — confirm `paper.pdf` exists and is non-empty. 2. `5_numbers_traced` — pick 5 numerical claims; each must trace to `stats.md`/`full_results.json` line (record both). 3. `5_citations_traced` — pick 5 citations; each must appear in `survey.md`. Save the lookup in evidence. 4. `no_new_facts` — any fact not anchored in Phases 1–3 ⇒ FAIL. ### Step 4.3 Slides (PPTX) - **Generator**: subagent_type=`general-purpose`, invokes `/powerpoint-pptx`. Outputs `slides.pptx`. - All data slides reuse Phase 3 figures (no redrawing from memory). - **Advisor required_checks**: 1. `slides_match_paper` — every result on a slide matches a number in the paper. 2. `figures_reused_not_redrawn` — confirm data figures come from Phase 3 PNG/PDF assets, not regenerated by memory. When 4.3 passes, set `final_output.paper` and `final_output.slides`, regenerate dashboard one last time, and tell the user the paths. --- ## Backtracking rules (concrete) - A backtrack jumps back to the named step. That step re-enters its inner loop with iter=0. - All downstream steps are reset to `pending` (their iterations counter is preserved for telemetry but `last_verdict` and `artifact` are nulled). - `backtrack_count[from_step]` increments. When any step's count exceeds `state.config.backtrack_limit_per_step` (default 3), escalate to user. - Append to `backtrack_log.jsonl`: ```json {"ts":"","from":"2.3","to":"1.2","reason":"","round":2} ``` - Regenerate dashboard. --- ## Subagent strategy summary | Step | Generator subagent | Advisor | Adversarial subagent | |---|---|---|---| | 1.1 Literature | yes (general-purpose) | main | — | | 1.2 Hypotheses | yes | main | — | | 1.3 Design | yes | main | — | | 2.1 Pilot run | yes (executes code) | main | — | | 2.2 Pilot analy | yes | main | — | | 2.3 Pilot verdict | — | main | **yes** | | 3.1 Full run | yes | main | — | | 3.2 Full analy | yes | main | — | | 3.3 Full verdict | — | main | **yes** | | 4.1 Discussion | yes | main | — | | 4.2 Paper | yes (writes + compiles) | main | — | | 4.3 Slides | yes | main | — | Why this split: - **Generator = subagent**: protects main context from large artifact bodies and long tool-call traces. - **Advisor = main conversation**: needs cumulative project memory to spot inconsistencies across phases. - **Adversarial subagent = fresh subagent**: a subagent with no exposure to the advisor's draft is less likely to mirror its rationalizations. --- ## State.json invariants - Rewrite atomically (write to `state.json.tmp`, then rename). - After every rewrite, also call `python paper/update_dashboard.py`. - Append-only logs (`advisor_log.jsonl`, `backtrack_log.jsonl`, `orchestrator_log.md`) — never rewrite history. - When backtracking, do not delete past artifacts; future analysts (and the user) want to see what was abandoned and why. --- ## How to invoke ``` /phd-agent Topic: Venue: Output: Gates: # optional, default = auto ``` The skill will: 1. Ask for missing required arguments (Topic, Venue, Output). `Gates` defaults to `auto`. 2. Initialize the output directory, state.json (with the venue and gates_mode written in), dashboard. 3. Tell you the dashboard URL (file://...). 4. Run the 4 phases. At each Phase Gate, write the gate brief to disk; in `auto` mode proceed with the advisor's recommended verdict, in `manual` mode block on AskUserQuestion. 5. Deliver `paper.pdf` + `slides.pptx` and leave the dashboard live. Switching gate mode mid-run: the user can edit `state.config.gates_mode` directly between phases, or say "switch gates to manual" — the orchestrator re-reads the config before each gate. --- ## Verification commands the user can ask at any time - "show me the dashboard" → I print the file:// path to `dashboard.html` and remind it auto-refreshes - "show me state.json" → I read and display the current state - "show me the latest advisor verdict" → I print the last entry from `advisor_log.jsonl` - "show me the checks for step x.y" → I print the `checks_run` array + `anti_pattern_self_test` for that step's most recent advisor_log entry, and link the `advisor_review_iter*.md` + `*_evidence.md` files - "show me the spot-check evidence for x.y" → I open the `*_evidence.md` next to the matching review (e.g., the actual Scholar queries + URLs for the 5-paper check) - "show me the backtrack history" → I print `backtrack_log.jsonl` - "what step are we on" → I report from state.json (not memory) - "show me the verdict for step x.y" → I read `steps["x.y"].last_verdict` and the corresponding advisor_log entry - "force backtrack to step x.y" → I obey, append to backtrack_log with reason="user-forced", jump to that step --- ## Anti-pattern checklist — the §6 ANTI_PATTERN_SELF_TEST canonical list These are the same items §6 of every advisor review must answer. Answers go into both the markdown review (`§6`) and the JSONL log entry (`anti_pattern_self_test`). The dashboard renders them. | Key | Question | Required answer to PASS | |---|---|---| | `read_artifact_directly` | Did I open the artifact file myself, not just trust the generator's summary? | yes | | `recomputed_or_spotchecked` | Did I recompute or spot-check ≥1 numerical/citation claim where applicable? | yes (or n/a if step has no numbers/cites) | | `consistent_with_prior_steps` | Did I diff against earlier artifacts to catch contradictions? | yes | | `manipulation_checks_pass` | (Phase 2/3 verdicts only) Do manipulation checks confirm the mechanism activated? | yes (or n/a) | | `adversarial_objection_addressed` | (2.3 / 3.3 only) Did I write a direct response to each objection — not a dismissal? | yes (or n/a) | | `would_user_regret_this_pass` | If the user opened the dashboard in 30 minutes, would they regret this PASS? | no | Any deviation from the required answer ⇒ verdict is REVISE, not PASS. The dashboard's per-step Checks Run badge goes red as soon as the JSONL log shows a deviation, so the user spots it without re-reading the markdown.