v1.0.0

idea-quality

microsoft@microsoft⭐ 3.6k stars· last commit 1mo ago· 1 open issues

Score the QUALITY of a research idea at the idea stage — given a Markdown file with Title / Motivation / Method sections (no experiments needed), produce a per-axis quality assessment with cited evidence and an overall 0–100 score plus a verdict; or, given two such idea files, a blind head-to-head comparison. TRIGGER when the user asks "how good is this idea", "rate / score / grade this research idea or proposal", "review my idea .md before I write it up", "is this contribution strong enough to pursue", "which of these two idea files is stronger", or hands over an idea Markdown file (Title/Motivation/Method) and wants a quality judgment. Use this even when the user does not say the word "score" but clearly wants an assessment of how strong an idea is. DO NOT trigger for prior-art / overlap / "is this novel vs existing work" checks (that is a literature-collision task), for full reviews of finished papers that already have results, or for generating the idea itself.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

7.8/10
Verified
Oct 12, 2026

// RATINGS

⭐GitHub Stars
⭐⭐⭐⭐⭐ 3.6k on GitHubGitHub ↗

Very popular

🟢ProSkills ScoreAI Verified
7.8/10
📍

Not yet listed on ClawHub or SkillsMP

// README

Our AI co-author, from research problem to final publication.

Code arXiv-Idea arXiv-Reel License: MIT Status: Active

ResearchStudio is a collection of skills covering the entire research lifecycle — from an under-specified research direction to a published paper. The skills run on Claude Code and Codex.


: one research paper, all deliverables (editable poster, video, blog, and reel)

📄 Paper  ·  💻 Code  ·  🎬 Demo

: one research direction, one reviewer-defensible idea card

📄 Paper  ·  💻 Code  ·  🎬 Demo

News 🔥🔥🔥

  • [2026-09-15] release pptx2video is now open source — turn editable PowerPoint slides into narrated videos with animations and captions. Use it as a skill or run the CLI without an LLM.
  • [2026-08-25] release Trending Paper is released — meet the trending research through ResearchStudio Reels.
  • [2026-08-13] feature IdeaSpark keeps the whole user query — the user query is persisted verbatim and read by every phase that reasons about intent; papers named in it without a link are resolved and deep-read; and a stated solution direction reaches idea selection, which must record whether it followed or departed from that direction, and why.
  • [2026-07-08] release ResearchStudio-Reel is released — the post-paper half: turn a finished paper PDF into the artifacts a publication needs — a print-ready poster, a narrated walkthrough video, a bilingual blog post, and an interactive reel viewer.
  • [2026-07-03] release ResearchStudio-Idea is released — the pre-paper half: turn an under-specified research direction into a reviewer-defensible, implementable idea, grounded in an empirical taxonomy induced from a large-scale corpus of ICLR / ICML / NeurIPS submissions.

Quick Start

We recommend creating an isolated environment first, using either conda or uv:

Option 1 — conda:

conda create -n researchstudio "python>=3.10" -y
conda activate researchstudio

Option 2 — uv:

uv venv --python 3.10
source .venv/bin/activate
# For Windows PowerShell:
# .venv\Scripts\Activate.ps1

Then pick one of the two install paths below:

Option 1 — clone and run install.sh (installs native tools, Python deps, symlinks the skills from a local clone and scaffolds .env):

git clone https://github.com/microsoft/ResearchStudio.git && cd ResearchStudio
bash install.sh

Option 2 — interactive npx installer (choose which plugins to install, enter your API keys when prompted, and it writes .env):

npx github:microsoft/ResearchStudio

For usage, see Idea and Reel. We recommend using these skills with model versions $\ge$ claude-opus-4.6 or gpt-5.5.

License

MIT

Citation

If ResearchStudio helps your research workflow, please cite:

@article{xiao2026researchstudioreel,
  title   = {ResearchStudio-Reel: Automate the Last Mile of Research from Paper to Poster, Video, and Blog},
  author  = {Lingao Xiao and Yalun Dai and Yangyu Huang and Qihao Zhao and Wenshan Wu and Hugo He and Ruishuo Chen and Jin Jiang and Qianli Ma and Jiahuan Zhang and Xin Zhang and Ying Xin and Yang Ou and Yan Xia and Scarlett Li and Longbo Huang and Zhipeng Zhang and Yang He and Yap Kim Hui and Yan Lu},
  journal = {arXiv preprint arXiv:2607.04438},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.04438}
}

@article{zhao2026researchstudioidea,
  title   = {ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes},
  author  = {Qihao Zhao and Yangyu Huang and Yalun Dai and Lingao Xiao and Jianjun Gao and Xin Zhang and Wenshan Wu and Scarlett Li and Yang He and Yan Lu and Yap Kim Hui},
  journal = {arXiv preprint arXiv:2607.04439},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.04439}
}

// HOW IT'S BUILT

// FILES

2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • ResearchStudio-Idea/evaluation/idea_quality/SKILL.md7.9 KBupstream ↗
    ---
    name: idea-quality
    description: Score the QUALITY of a research idea at the idea stage — given a Markdown file with Title / Motivation / Method sections (no experiments needed), produce a per-axis quality assessment with cited evidence and an overall 0–100 score plus a verdict; or, given two such idea files, a blind head-to-head comparison. TRIGGER when the user asks "how good is this idea", "rate / score / grade this research idea or proposal", "review my idea .md before I write it up", "is this contribution strong enough to pursue", "which of these two idea files is stronger", or hands over an idea Markdown file (Title/Motivation/Method) and wants a quality judgment. Use this even when the user does not say the word "score" but clearly wants an assessment of how strong an idea is. DO NOT trigger for prior-art / overlap / "is this novel vs existing work" checks (that is a literature-collision task), for full reviews of finished papers that already have results, or for generating the idea itself.
    ---
    
    # Idea Quality
    
    Judge how strong a research idea is **at the idea stage** — before any experiments exist — and return a quality score with reasons a reviewer would recognize. This skill is self-contained: it judges from first principles on the three axes below. It consults no external corpus, dataset, or other skill, so its judgment is reproducible from the idea text alone.
    
    ## Input: an idea Markdown file
    
    The idea is a `.md` file with three sections. Read the file, then map each section to what it feeds:
    
    ```markdown
    # Title
    <≤ ~15 words: the idea's handle>
    
    ## Motivation
    <the bottleneck / gap the idea attacks, why it matters, and why it is still open>
    
    ## Method
    <the proposed contribution as concrete numbered steps>
    ```
    
    - **Title** → the handle.
    - **Motivation** → feeds **Axis A** (is the problem worth attacking) and gives **Axis C** the "problem" half.
    - **Method** → feeds **Axis B** (is the method good) and gives **Axis C** the "method" half.
    
    If a section is missing or thin, infer the most reasonable reading from the rest and note the assumption in the report — do not stall asking for clarification. If the user passes two files (or one file with two ideas) and wants a comparison, run the **pairwise** track.
    
    ## Scope: what this judges, and what it deliberately does not
    
    - **No experiments exist.** Every axis is a *reasoning-level* judgment — the kind a reviewer makes from an abstract before seeing results. **Never invent or assume experimental results to score with.** If a claim's truth would need an experiment, judge the *plausibility of its argument*, not an imagined outcome.
    - **Not a prior-art check.** "Has someone already done this?" is a separate literature-collision task. Judge whether the idea is *good* (real gap, deep method, sound + on-target), NOT whether a near-duplicate exists. Judge the contribution's intrinsic ambition, not its novelty against a literature search.
    
    A strong score means *the idea is strong*; it does not predict acceptance, which also turns on execution this skill cannot see.
    
    ## The three axes
    
    Score on exactly these three. Each is (a) assessable from the idea alone without experiments, (b) able to discriminate strong from weak, and (c) not a proxy for writing polish.
    
    **A — Problem position quality.** Is the gap the Motivation identifies *real, important, non-obvious, and genuinely open* — or shallow, already-solved, or a conveniently easy target?
    - Strong: a gap that matters and that the field has not closed; naming it is itself insightful.
    - Weak: an obvious / already-handled problem, or a soft target chosen so it can be "solved" cheaply.
    - *Why it exists:* when the author picks their own bottleneck, the cheapest way to look successful is to pick a soft target. This axis is what stops a soft-bottleneck-plus-trivial-fix from scoring well.
    
    **B — Method quality.** Is the proposed method good in itself? Score it as ONE number, but in the Reason you MUST decompose into three named sub-judgments, because a single number otherwise hides *which* of them is weak:
    - **depth** — a genuinely new mechanism / construction / reframing vs an incremental tweak (a new schedule, one extra loss term, a hyperparameter).
    - **soundness** — does the "why it should work" argument hold up internally? Are assumptions justified, or is there hand-waving / an unstated condition / a logical gap? *Judge the method's own logic, regardless of which problem it aims at — fit is Axis C.*
    - **feasibility** — is it buildable in principle, or does it require resources / oracles / data that don't plausibly exist? *This means "buildable", NOT "will get good numbers" — do not hallucinate results.* This is the softest sub-judgment; let depth dominate B and treat feasibility as a tie-breaker, not an equal third.
    - *Note:* B is about the **magnitude and integrity of the method itself**, judged intrinsically — NOT whether a duplicate exists in the literature (separate prior-art task).
    
    **C — Problem-fit.** Does the Method actually target and *plausibly resolve the specific gap in the Motivation* — rather than an adjacent, easier, or different problem?
    - Strong: the method's mechanism clearly bears on the identified gap; if it works, that gap closes.
    - Weak: a method aimed at a different problem than the one claimed, or one whose connection to the gap is asserted, not shown.
    - *Boundary vs B-soundness:* B-soundness asks "is the method's own logic coherent?"; C asks "does that logic connect to THIS problem?" A method can be internally sound (B high) yet solve the wrong problem (C low); or on-target (C high) yet hand-wavy (B low). Keep them separate.
    
    ## Scoring: dual-track
    
    Always run the **absolute** track. When comparing two ideas, ALSO run the **pairwise** track — and treat pairwise as the trustworthy signal, because relative judgments are far less noisy than absolute numbers (which bunch around 3/5). The absolute 0–100 is a readable secondary readout, not a calibrated truth.
    
    Per axis, score **1–5** (integer). There are deliberately no fixed level descriptors — instead **every score quotes the specific phrase / element of the idea that justifies it.** A score with no quoted evidence is not allowed; the evidence requirement, not a rubric table, is what keeps scores honest. Judge **substance, not length or fluency** — a short crisp idea can score 5; a long polished one can score 2.
    
    ### Track 1 — Absolute (per idea)
    
    Overall score (equal weight, each axis 1→0 and 5→full):
    ```
    overall = round(100 * (A + B + C - 3) / 12)
    ```
    Verdict band: `strong` ≥ 67 · `borderline` 34–66 · `weak` < 34.
    
    **A/C gate (overrides the band).** A and C are near-necessary, B is not — so:
    - If **A ≤ 2 OR C ≤ 2**, the verdict cannot be `strong` (cap at `borderline` at best), regardless of the number. Reason: a great method on a trivial problem (low A), or a great problem with a method that doesn't address it (low C), is not a strong idea — and the equal-weight mean would otherwise mislead. State the cap and why in one line.
    - A low **B alone does NOT cap**: a simple method that genuinely addresses an important problem (counter-intuitive minimalism) can still be strong.
    
    ### Track 2 — Pairwise (comparing two ideas)
    
    For each axis state which idea is stronger (**Idea 1 / Idea 2 / tie**) and why, in one line; for axis B, note per-sub-judgment (depth / soundness / feasibility) which arm wins so the diagnostic isn't lost. Then give an overall winner **holistically** — but problem-fit (C) is near-necessary: **a decisive C loss usually decides the overall, even if that idea wins A and B**, because a method that doesn't fit its problem isn't the better idea. Decide each axis independently; do not let one axis sway the others.
    
    **Judge blind to source.** If two ideas are compared as part of an evaluation, do NOT factor in which system/model produced which — provenance is not evidence of quality. (For this to be fair, the two inputs must share the same Title/Motivation/Method format; flag
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/microsoft-researchstudio-researchstudio-idea-evaluation-idea-quality.md

// REPO STATS

3.6k stars
1 open issues
Last commit: 1mo ago