⏳ This skill is pending AI review.
Scores will appear once the review pipeline completes.
deeppapernote
Generate a high-quality deep-reading note for a single paper and write it into an Obsidian-style vault. Use when the user gives a paper title, DOI, URL, arXiv ID, Zotero item, or local PDF and wants a polished Markdown note with strong structure, evidence-based analysis, and figure placeholders.
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
Not yet listed on ClawHub or SkillsMP
// README
DeepPaperNote
Turn one complex paper into an Obsidian note you will actually want to keep.
When you study an important paper, the hard part is rarely the reading. It is turning what you understood into a note you can still use later. Finding the source, matching figures to the text, copying numbers over, and tracing the method often take the better part of a day. And the papers you have read end up scattered, with nothing connecting one to another.
DeepPaperNote is an agent skill that runs in Claude Code, Codex, and any other agent that supports skills. Hand it a paper, and it reads the full text, writes an illustrated deep-reading note, saves it into your Obsidian vault, and links it to the other papers you have read. Every paper you read grows your academic wiki a little.
- 🚀 One sentence, one note. No prompts to learn, no templates to set up. Say "Generate a deep-reading note for this paper" with a title, DOI, arXiv ID, or PDF, and it finds the paper, reads it in full, extracts the figures, and saves the note into your vault.
- 🎓 Reads like a domain expert. From CS and AI to medicine, psychology, and the humanities, from new methods and benchmarks to surveys and clinical trials, your papers can all go to DeepPaperNote. Each of 10 paper types gets its own way of reading, drawn from more than 50 reviewer guidelines and reporting standards such as the NeurIPS review criteria and CONSORT, and tested on 34 real papers.
- 🎯 Accurate and complete. Every conclusion traces back to the paper, numbers come straight from its text and tables, and every main experiment is covered. When the paper does not report something, the note says so instead of guessing. Before saving, every note goes through an independent review and a strict quality check, and when the full text cannot be read, it never makes one up.
- 🖼️ Figures, tables, and formulas in the text. Figures and tables are cropped from the PDF and placed right where the note discusses them, and key formulas render in Obsidian. No more flipping back to the PDF.
- 🧠 A vault that knows you better the more you use it. Each note links to related papers in your vault and goes into the domain folders you already use. Record your research directions, and each note's Takeaways tell you what the paper means for your own work.
📰 News
- [v3.0.0] 🎓 Notes read each paper by its type: 10 paper types each get their own way of reading and note structure, every figure is visually reviewed before it enters the note, and an independent quality review runs before saving. Release notes
- [v3.0.0] 🧠 Every note joins your vault: notes gain Takeaways and Connections sections and carry structured properties, and two new skills,
research-memoryandorganize-wiki, tailor the takeaways to your research directions and keep your domain folders in order. - [v2.3.1] 🔌 Added support for DeepPaperNote Connector: papers you save from your browser can get their notes in the same paper directory. Release notes
News lists only the three most recent user-facing milestones. See the changelog and GitHub Releases for the full history.
🚀 Quick Start
1. Install
Claude Code: install from the plugin marketplace
claude plugin marketplace add 917Dhj/DeepPaperNote
claude plugin install deeppapernote@deeppapernote
Codex and other agents that support skills: install with the skills CLI
npx skills add 917Dhj/DeepPaperNote
The plugin installs all four skills at once. The skills CLI lets you choose which skills to install and which agents receive them. To try it out, deeppapernote alone is enough; see below for what the other three do.
2. Install the PDF dependency
python3 -m pip install PyMuPDF
Python 3.10 or newer is required. If you use uv, run uv pip install PyMuPDF in the Python environment DeepPaperNote will use.
3. Read a paper
Tell your agent in one sentence, with the paper's title, DOI, URL, arXiv ID, or local PDF:
Generate a deep-reading note for this paper: <paper>
Turn this paper into an Obsidian note: <paper>
You can also call it with the slash command /deeppapernote <paper>.
The first time, it asks you two things: whether notes should be in English or Chinese, and whether to save them in the current workspace or your Obsidian vault. It remembers your answers and does not ask again. To use a different language or location for one paper, just say so in the request. For environment variables and other settings, see User Configuration.
4. Update to a new version
Claude Code:
claude plugin marketplace update deeppapernote
claude plugin update deeppapernote@deeppapernote
Restart Claude Code to apply the update.
skills CLI:
npx skills update
👀 What a Note Looks Like

Every note has three parts: the opening and closing are the same for every paper, and the body in between follows the paper's type.
| Part | Sections |
|---|---|
| Opening | Core Information · Abstract · Contributions · One-Sentence Summary |
| Body | Method paper: Background and Problem → Limitations of Prior Work → Key Idea and Challenges → Method → Experimental Evidence → Deep AnalysisBenchmark: Evaluation Gap → Task and Design Goals → Construction and Quality Control → Evaluation Framework → Findings → What It Really MeasuresSurvey: Scope and Search → Taxonomy → Directions → Consensus, Disagreement, and Gaps…10 paper types in all |
| Closing | Limitations · Takeaways · Connections · My Notes |
The closing sections are where the note meets your vault:
- Takeaways: Transferable Ideas and Next Steps, covering which of the paper's ideas carry over to other problems and what to do after reading it. Once you set up research memory, this part is written against your own research directions.
- Connections: links to related paper notes in your vault, and the key prior work you need to understand this paper.
- My Notes: yours to write in, and kept exactly as it is when the note is regenerated.
Each paper gets its own folder, and figures and tables are named after their labels in the paper:
<your papers folder>/
└── NLP/ ← domain folder, reusing the ones you already have
└── Attention_Is_All_You_Need/
├── Attention_Is_All_You_Need.en.md
└── images/
├── figure_1.png
└── table_2.png
Each note opens with structured properties you can query and filter with Obsidi
// HOW IT'S BUILT
// FILES
60 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.
skills/deeppapernote/SKILL.md22 KB2216b61e66fdupstream ↗
--- name: deeppapernote description: Generate a high-quality deep-reading note for a single paper and write it into an Obsidian-style vault. Use when the user gives a paper title, DOI, URL, arXiv ID, Zotero item, or local PDF and wants a polished Markdown note with strong structure, evidence-based analysis, and figure placeholders. --- # DeepPaperNote Turn one paper into a replication-grade deep-reading note and save it to the workspace or Obsidian target of the resolved configuration. Each run handles one paper. A rewrite of the abstract is never a finished note. ## Standard The note reconstructs the paper's argument as one story that runs through its middle sections: the problem it takes on, how the work answers it, what the evidence shows, what the paper does not prove, and why it is worth keeping. The paper type sets the middle sections and the questions they must answer (`references/paper-types.md`). Write as a senior colleague explaining the paper to a smart researcher from an adjacent field: someone who reads papers but does not know this paper's subfield, its jargon, or its baselines. Lead each paragraph with its point, explain a term the first time it appears, and keep the technical depth the paper type calls for: equations and implementation detail for method, system, and theory papers; design, measurement, and argument for the others. These hold at every step: - **Evidence-first:** `*_raw_sections.jsonl` and `*_source_manifest.json` are the canonical reading material. Finish only from inspected full-text evidence, never from the title, abstract, or headings alone. - **Model-first:** scripts gather and structure evidence. You choose the paper type, emphasis, contributions, mechanism, limitations, and prose; script suggestions are hints. - **Fail closed:** when the evidence cannot support a deep note, stop as Stop Conditions describes. - **Placeholder-first figures:** every major figure and table starts as a placeholder. It becomes an image only after Visual Review accepts it and Formal Save materializes it. Image extraction is optional and never reduces textual coverage. ## Language Integrity Contract After Configuration Readiness, resolve one `output_language` (`zh-CN` or `en`) and pass it as `--language` to every script. `source_manifest.language_hint` describes the source text only and never selects the note profile. The value travels through Save Target Admission → Figure Plan → Figure/Table Decisions → Synthesis Bundle → Note Plan → Grounding Lint → Final Note Lint → Final Quality Review → Final Readability Review → Formal Save. Every artifact records it as `output_language`, and each script stops on a missing or mismatched value. Final Note Lint records `note_sha256`, so any edit after lint means rerunning it. Formal Save checks every binding before any save side effect. `references/output-language.md` describes the two profiles. ## Workflow Run the steps in order. A step is complete at its **done when**, never because a partial artifact exists. At any point, `scripts/run_state.py status --workdir <run_directory> --prefix <run_prefix>` reports verified stages, stale artifacts, and the `next_action`; `references/run-state.md` explains how to bind the artifacts you write. 1. **Configuration Readiness.** Resolve Run Overrides from the explicit request, CLI, and process environment. Read User Configuration only for fields they leave unset; `references/user-configuration.md` covers admission, migration, repair, and Preference Changes. **Done when** the save mode, its target, and the output language are resolved and valid. 2. **Acquire.** `scripts/run_pipeline.py acquire --input <paper> --prefix <run_prefix>` resolves the identity, collects metadata, fetches the PDF, and extracts the raw source text, following Sources below. Without `--workdir` it picks a directory under `~/.deeppapernote/runs/` and prints `workdir=`; reuse that value in every later step, or pass a writable `--workdir` when a sandbox blocks the default. **Done when** the source manifest exists and is not `blocked`. 3. **Save Target Admission.** In Obsidian mode, run `scripts/write_obsidian_note.py --preflight` with the resolved title, `--language`, Vault, `--source-manifest`, and `--output <run_directory>/<run_prefix>_admission.json`. Its result is authoritative: | Result | Action | |---|---| | reuse of an existing directory | Use that directory and its domain. A verified PDF-only directory needs no confirmation. | | `same_language_note_exists` | Ask whether to overwrite the reported note, and say what its `user_notes` result means: `present` keeps the user's My Notes block (about `chars` characters) and regenerates everything else; `none` found no notes of the user's; `heading_missing` found no My Notes heading, so nothing can be kept. When `legacy_format` is true, add that the kept block predates the current note format and may be old model-written text. Edits elsewhere in the note are replaced. Carry its `existing_read_date`, when non-empty, into the new note's `read_date`. On yes, rerun with `--overwrite-existing-note --expected-existing-note-sha256 <reported_sha256>` and carry it to Formal Save. On no, stop without writing. | | several matching directories or sources | Ask for the selection as `references/paper-archive.md` describes, and carry it through Formal Save. | | `domain_required`, `new_domain_reason_required`, or `domain_name_invalid` | Choose the domain from the receipt's `domain_overview` as `references/domain-routing.md` describes, then rerun with `--domain`, adding `--new-domain-reason` for a new domain. Carry both to Formal Save. | | any other conflict | Report it and stop. | Workspace mode runs the same preflight with `--save-mode workspace` only to choose the domain. **Done when** an admitted result with its `domain` is saved. 4. **Prepare.** `scripts/run_pipeline.py prepare --workdir <run_directory> --prefix <run_prefix> --admission <admission.json>` (workspace mode omits `--admission`) extracts structure and PDF assets, plans figures, builds the complete figure/table decision table, and writes the synthesis bundle, all offline. It refuses to replace decisions that hold Visual Review work unless the user accepts repeating those reviews and you pass `--discard-reviews`. Rerun a failed phase whole. **Done when** the bundle exists. 5. **Visual Review and Figure/Table Decision Freeze.** Record a verdict for every `review_pending` item with `scripts/plan_figure_table_decisions.py review`, send any Figure/Table the note needs into review with `promote`, and drop a usable candidate the note has no use for with a recorded reason, as `references/figure-placement.md` describes. Every decision must satisfy `writing_contract.figure_table_contract`. **Done when** `scripts/plan_figure_table_decisions.py summary --decisions <decisions.json>` lists no `review_pending_labels`; its table gives each insert's embed. 6. **Note plan.** Read the bundle and the raw sections, choose the paper type as `references/paper-types.md` describes, and fill every value of `writing_contract.note_plan_contract.template` to create a short JSON `note_plan` with the exact `output_language`. Set `reader_research_direction` to the research direction the request states, verbatim; when the request states none and `research_memory.status` is `ok`, copy `research_memory.suggested_reader_research_direction`; otherwise leave it empty. Write `storyline` as three to five sentences that tell this paper's own story the way the authors build it, from the problem through the idea and the evidence to what it means; it is the outline the middle sections follow, not text for the note. Fill `key_prior_work` with the three to five works this paper builds on, and put in `connections` only those `connection_candidates` from the bundle that truly relate to this paper, each with a relation key and a one-sentence reason. `references/evidence-first.md` shows a complete plan. **Done when** Grounding passes. 7. **Grounding.** `scripts/lint_grounding.py --note-plan <note_plan> --source-manifest <manifest> --bundle-json <bundle> --figure-decisions <decisions.json> --language <language> --output <grounding.json>`. **Done when** it reports no issue; keep its JSON for Formal Save. 8. **Draft.** Write the note from the plan, following Note Shape, as one Markdown file in UTF-8 without BOM and with LF line endings. **Done when** the complete draft is on disk. 9. **Final Note Lint.** `scripts/lint_note.py --input <note> --plan-file <note_plan> --language <language> --output <lint.json>`. **Done when** every reported `passes_*` gate is `true`. A failed style gate is normal for several rounds: fix the reported text and rerun. Each `zh-CN` mixed-language issue names the English it counted in `english_run` or `english_words`; translate that wording, or wrap a retained proper noun in backticks, keeping formulas inline and proper nouns intact. Escalate to the user only when the same failures repeat unchanged across several rounds. 10. **Final Quality Review and Final Readability Review.** Follow `references/final-quality-review.md` for the reviewer prompt, the recorded `final_quality_review`, the two-revision limit, and the `final_readability_review` receipt. A readability edit repeats Final Note Lint and a fresh quality review. **Done when** both reviews pass for the current note bytes. 11. **Formal Save.** Run `scripts/write_obsidian_note.py` with `--content-file`, `--plan-file`, `--lint-json`, `--grounding-lint-json`, `--source-manifest`, `--figure-decisions`, `--final-quality-review`, `--final-readability-review`, the admitted `--domain` and `--new-domain-reason`, and any approved overwrite hash, each naming the exact file the linters and reviewers checked. It repeats admission, refuses missing, failed, or stale gates, refuses unresolvable `[[links]]`, and materializes each `insert` into the paper-local `images/` directory. **Done when** the save succeeds. Both lint artifacts bind the plan through `note_plan_sha256`, and Grounding Lint also binds the source manifest and decisions. These are exact-byte hashes, so reformatting JSON invalidates them: rerun both linters after a plan change, Grounding Lint after a source or decision change, and Final Note Lint after a note edit. | Save Target | Formal Save | |---|---| | `save_mode=obsidian` with a usable Vault | Save to that Vault. | | `save_mode=obsidian` and the save fails | Keep the Vault target and recover within scope; otherwise report `blocked`, and do not switch to workspace. | | `save_mode=workspace` | Save inside the workspace output root. | Running `scripts/run_pipeline.py` without a phase performs both phases in workspace mode only; Obsidian mode needs Save Target Admission between them. A normal request finishes in one run: text, figure decisions, image materialization, and save. Produce a text-only note only when the user asks for one. ## Sources Prefer the strongest source in this order: 1. local PDF path given by the user 2. local Zotero item and local Zotero attachment if available 3. verified local Obsidian PDF for the accepted work and requested version 4. DOI and publisher metadata 5. arXiv or open-access PDF sources 6. Semantic Scholar or OpenAlex for metadata backfill `acquire` checks the desktop library through the Zotero Local API first. Its `--zotero-mode` is `auto` by default and falls back to the web providers when Zotero has no match; `off` makes no Local API request, and `required` demands a Zotero match. An explicit Zotero key must resolve locally. A trusted JSON artifact or explicit local PDF stays authoritative and skips this lookup. A confident local-library hit outranks any title-only web match. When a library match exposes only an attachment key and filename, `scripts/locate_zotero_attachment.py` finds the PDF under Zotero storage. In Obsidian mode, a matching local archive is verified and reused before any download; `references/paper-archive.md` covers version and source selection and Connector handoff. ## Stop Conditions Stop, report the blocked stage and the remaining stages, and ask for what would unblock it when: - the paper identity is ambiguous, including several equally plausible local items: ask for a DOI, arXiv ID, or exact Zotero key - no PDF is available after every supported acquisition path: local PDF, Zotero attachment, metadata `pdf_url`, direct PDF URL, arXiv or open-access sources, an accessible publisher PDF, DOI enrichment, and any other implemented fetch path - the source manifest is `blocked` with `blocked_code: unreadable_text`: ask for an OCR version or another copy, since rerunning cannot fix it - the PDF or evidence is too weak for a real deep note: ask for a better PDF, OCR, or other source material When any other step fails, retry it or take a fallback this skill names. Otherwise stop as above. An abstract-only, provisional, or partial note is never the finished output. ## Note Shape - Start with an Obsidian YAML properties block above the `#` title heading. Include `tags` with exactly one `papers/<domain>` value, the admitted `domain`, and useful `aliases`; add `date`, `doi`, or `arxiv_id` when known, and omit unknown fields. Always include the fields of `writing_contract.frontmatter_contract`: `paper_type` matching the plan, `methods`, `datasets`, and `tasks` as lists of at most 5 central items, and `read_date` as `YYYY-MM-DD`. A list value that names a term in the bundle's `glossary_terms` is written as that term's `link`, copied verbatim; every other value is plain text. Formal Save refuses a frontmatter link that is not a glossary term note. - Use real `#`, `##`, and `###` headings. The top-level sections are exactly `writing_contract.section_order`: the fixed head, the middle sections of `contracts_by_paper_type[paper_type]`, then the fixed tail, in that order. `references/paper-types.md` explains how to choose the type, answer its core questions, and write the weak-evidence callout. A non-trivial paper gets several meaningful `###` subsections in its middle sections. - Core Information is a fixed metadata block holding only `writing_contract.core_info_fields`, in order. Analysis belongs in later sections. - Abstract renders the original abstract faithfully in the configured language, translated in `zh-CN` mode, with no later evidence or judgment added. - Contributions lists the paper's actual innovations and why each matters. - A paper type with a `workflow` gets that subsection in its host section: an overview chain, then a step table that mirrors `note_plan.workflow` (`references/paper-types.md`). - Write the middle sections in order, following `storyline`: each section picks up where the previous one left off, and a paragraph that serves no part of the story is cut. Within a section, write ordinary paragraphs, one point each, conclusion first and details after. Explain each term or acronym in one plain sentence the first time it appears, and keep an established English name rather than translating it word for word. - Keep the numbers that carry a conclusion in the prose and move the other results into tables, so every main experiment stays in the note. A comparison of three or more systems, settings, datasets, metrics, or ablation rows becomes a compact Markdown table with a sentence saying what it shows; tables are welcome wherever they make the comparison clearer. Formulas follow the type's `formula_rules`, as `$...$` or `$$...$$`. - Takeaways holds exactly the two subsections of `writing_contract.takeaways_contract`, one or two items each, and each item is one or two sentences: reusable ideas that name the paper's mechanism, where it transfers, and its precondition, then follow-up questions or experiments specific to this paper, at least one of which tests the paper's own claims or evidence. With a non-empty `reader_research_direction`, connect at least one transferable idea to that direction whenever a credible transfer exists, start every item that connects to it with `takeaways_contract.judgment_marker`, and say plainly when no credible transfer exists. - Connections follows `writing_contract.connections_contract`: the in-vault subsection lists exactly the planned `connections` as `[[target|Title]]` entries with their relation labels, or its fixed empty statement; the key-prior-work subsection lists the planned `key_prior_work` as plain text. Never write the bibliography, and never write a `[[link]]` whose target is not a bundle candidate. Prose outside Connections carries no `[[link]]`; paper-glossary links terms in the body after saving. Formal Save refuses links it cannot find in the vault, and any link in workspace mode. A recommended next paper must already appear here. `references/obsidian-format.md` shows the format. - My Notes is the last section and belongs to the user. The draft holds exactly `writing_contract.my_notes_contract.guidance` there. Formal Save keeps the user's own block from the note it replaces. - A short paper still gets a complete deep note: spend the space on protocol details, ablations, limitations, and replication implications. - The readability pass checks the story holds together, every term is explained on first use, and each paragraph leads with its point; it rewrites leftover language into natural prose in the configured language and keeps stable proper nouns. It adds no new facts or filler. `references/final-writing.md` and `references/obsidian-format.md` hold the full writing and Markdown rules. ## Reporting Say each phrase only when its condition holds: | Phrase | Condition | |---|---| | `笔记已完成` | The workflow is complete through Formal Save. | | `已生成草稿` | The draft exists, but lint, a review, or the save is pending. | | `已通过校验` | Final Note Lint ran and passed. | | `已保存到 Obsidian` | Formal Save succeeded. | Lint is a floor, not polish: while the readability review is pending, say the draft passed script lint but has not finished its final language review. When a run stops early, name the current stage and the remaining stages. Final user report: - After a successful Formal Save, report in the user's conversation language, and derive every claim from current-run artifacts; the `summary` of `scripts/run_state.py status` carries these fields. - Lead with the final note link or path, the save mode, and the actual saved domain, read from the note path under the configured papers root in Obsidian mode or the output root in workspace mode. When admission reused an existing directory, report that directory's existing domain. When the save receipt's `new_domain` is true, say that this domain folder was created, give its `new_domain_reason`, and suggest the organize-wiki skill to add the domain to the user's convention. - Then report, in order: 1. the paper title and strongest verified identifier 2. the Grounding Lint, Final Note Lint, Final Quality Review, and Final Readability Review results, plus the warning count 3. the materialized and retained-placeholder figure/table counts, plus each dropped item with its reason from `figure_drops` 4. whether the generated part of the saved note matches the Final Note Lint `note_sha256` - When `reader_research_direction` came from `research_memory`, say in one line that the transferable ideas followed the user's research memory. - When the save receipt's `glossary_followup.status` is `suggested`, suggest running paper-glossary on this paper and give its `source_manifest` and `note_path`; when `regenerated` is true, add that term links the glossary had added to the replaced note return only after that rerun. For any other status, say nothing about the glossary. - When the save replaced an existing note, report the save receipt's `user_notes.status`: `preserved` kept the user's My Notes; `untouched` found none to keep; `heading_missing` found no My Notes heading. Only when `user_notes.legacy_format` is true, add that the kept block predates the current note format and may be old model-written text the user can delete. - When a save receipt `replaced_images` entry lists other notes in `also_embedded_by`, say in one line that the new crop of that figure also shows in those notes. - Add overwrite actions, preference changes, or user-relevant warnings only when they occurred. ## References `SKILL.md` and the synthesis bundle carry a normal run. Read a reference only when its row applies: | Reference | When | |---|---| | `references/user-configuration.md` | Configuration Readiness | | `references/paper-archive.md` | existing Obsidian directories, local PDF versions, Connector handoff | | `references/domain-routing.md` | choosing the domain folder, the `papers/` tag, the domains file | | `references/figure-placement.md` | Visual Review, promotion, drops, and placement | | `references/run-state.md` | reading `run_state.py status` and binding artifacts | | `references/final-quality-review.md` | Final Quality Review and Final Readability Review | | `references/output-language.md` | language profiles, while drafting or debugging either language | | `references/evidence-first.md` | a complete note plan example | | `references/deep-analysis.md`, `references/final-writing.md` | a complex paper or a draft that is too shallow | | `references/obsidian-format.md` | Markdown, vault, frontmatter, and Connections formatting | | `references/paper-types.md` | choosing the paper type, core answers, workflow, weak-evidence callout | | `references/note-quality.md` | final review | | `references/metadata-sources.md` | incomplete metadata | | `references/architecture.md` | repository maintenance | ## Environment - The scripts need Python 3.10 or newer. When `python3` is older, use another interpreter of 3.10 or newer, such as `python3.12`, for the whole run; when none exists, stop and report the version found. - Run `scripts/check_environment.py` only when a concrete dependency or integration problem blocks a step, and explain the required dependencies, optional enhancements, and downgrade behavior directly.skills/deeppapernote/scripts/_domain_routing.py8.2 KB073e168aabe6upstream ↗
"""Domain routing: the overview the model reads and the checks on the domain it chooses. The model picks the domain folder for a new paper directory. This module only describes the existing domains and validates the choice; it never picks one. """ from __future__ import annotations import json import re from pathlib import Path from typing import Any from _frontmatter import FrontmatterError, frontmatter_block, parse_frontmatter from _note_paths import existing_domain_dirs, resolve_note_output_mode, safe_relative_path_parts DOMAINS_FILE = Path("DeepPaperNote") / "domains.md" SAMPLE_TITLES = 8 TAG_PREFIX = "papers/" FIELD_KEYS = { "定义": "definition", "definition": "definition", "收录": "includes", "includes": "includes", "不收录": "excludes", "excludes": "excludes", "代表论文": "representative_papers", "representative papers": "representative_papers", } DOMAIN_HEADING_RE = re.compile(r"^##[ \t]+(?!#)(.+?)[ \t]*#*[ \t]*$") FIELD_RE = re.compile(r"^[ \t]*[-*][ \t]+([^::]+?)[ \t]*[::][ \t]*(.*?)[ \t]*$") NEW_DOMAIN_NAME_RE = re.compile(r"[\w-]+") DOMAIN_CHOICE_CODES = frozenset( {"domain_required", "domain_name_invalid", "new_domain_reason_required"} ) class DomainChoiceError(Exception): def __init__(self, code: str, message: str): super().__init__(message) self.code = code self.message = message def domain_locations(config: dict[str, Any]) -> tuple[Path, Path | None]: """The folder that holds the domains, and the domains file (Obsidian mode only).""" output_mode, root = resolve_note_output_mode(config) if output_mode != "obsidian": return root, None papers_dir = str(config.get("papers_dir", "Research/Papers")).strip() or "Research/Papers" return root / Path(*safe_relative_path_parts(papers_dir, "papers_dir")), root / DOMAINS_FILE def parse_domains_file(text: str) -> dict[str, dict[str, str]]: """Each `##` heading names a domain; `- Label: text` lines under it fill its fields.""" text = text.removeprefix("").replace("\r\n", "\n") try: block = frontmatter_block(text) except FrontmatterError: block = None if block is not None: text = text[4 + len(block):].partition("\n")[2] domains: dict[str, dict[str, str]] = {} current: dict[str, str] | None = None for line in text.split("\n"): heading = DOMAIN_HEADING_RE.match(line) if heading: current = domains.setdefault(heading.group(1).strip(), {}) continue if line.startswith("#"): current = None continue field = FIELD_RE.match(line) if current is not None else None if field: key = FIELD_KEYS.get(field.group(1).strip().lower()) if key and field.group(2) and key not in current: current[key] = field.group(2) return domains def read_domains_file(path: Path | None) -> dict[str, dict[str, str]] | None: if path is None or not path.is_file(): return None try: return parse_domains_file(path.read_text(encoding="utf-8", errors="replace")) except OSError: return None def paper_folders(domain_dir: Path) -> list[Path]: if not domain_dir.is_dir(): return [] return [ child for child in sorted(domain_dir.iterdir()) if child.is_dir() and not child.name.startswith(".") ] def paper_title(folder: Path) -> str: """The sidecar title, else a note's `title` property or H1, else the folder name.""" try: title = json.loads((folder / ".deeppapernote.json").read_text(encoding="utf-8"))["title"] if isinstance(title, str) and title.strip(): return title.strip() except (OSError, ValueError, KeyError, TypeError): pass for note in sorted(folder.glob("*.md")): try: text = note.read_text(encoding="utf-8", errors="replace")[:4000] except OSError: continue try: fields = parse_frontmatter(text) or {} except FrontmatterError: fields = {} title = fields.get("title") if isinstance(title, str) and title.strip(): return title.strip() heading = re.search(r"^# (.+)$", text, flags=re.MULTILINE) if heading: return heading.group(1).strip() return folder.name.replace("_", " ") def domain_overview(config: dict[str, Any]) -> dict[str, Any]: """What the model reads before choosing: every domain with its definition or sample titles.""" base, domains_file = domain_locations(config) convention = read_domains_file(domains_file) on_disk = existing_domain_dirs(config) domains = [] for name in sorted(set(on_disk) | set(convention or {})): folders = paper_folders(base / name) if name in on_disk else [] entry: dict[str, Any] = { "name": name, "paper_count": len(folders), "on_disk": name in on_disk, "in_convention": name in (convention or {}), } if convention and name in convention: entry.update(convention[name]) else: entry["sample_titles"] = [paper_title(folder) for folder in folders[:SAMPLE_TITLES]] domains.append(entry) return { "source": "folders" if convention is None else "domains_file", "domains_file": None if convention is None else str(domains_file), "domains": domains, "guide": "references/domain-routing.md", } def resolve_domain_choice( config: dict[str, Any], domain: str, new_domain_reason: str = "" ) -> tuple[str, bool]: """Return (domain, is_new) for the model's choice, or raise DomainChoiceError.""" name = (domain or "").strip() if not name: raise DomainChoiceError( "domain_required", "Choose a domain from domain_overview and rerun with --domain.", ) if ( "/" in name or "\\" in name or name.startswith(".") or any(ord(char) < 32 for char in name) ): raise DomainChoiceError( "domain_name_invalid", f"A domain is one folder name inside the papers root: {name}" ) base, domains_file = domain_locations(config) known = set(existing_domain_dirs(config)) | set(read_domains_file(domains_file) or {}) for existing in sorted(known): if existing.casefold() == name.casefold(): return existing, False if (base / name).exists(): raise DomainChoiceError( "domain_name_invalid", f"{name} already exists and is not a domain folder." ) if not NEW_DOMAIN_NAME_RE.fullmatch(name) or name.isdigit(): raise DomainChoiceError( "domain_name_invalid", f"A new domain name must work as a tag: letters, digits, _ and - only: {name}", ) if not (new_domain_reason or "").strip(): raise DomainChoiceError( "new_domain_reason_required", f"{name} is a new domain; pass --new-domain-reason saying why no existing domain fits.", ) return name, True def saved_domain(config: dict[str, Any], note_path: Path) -> str: """The domain a note is saved under: its paper folder's parent below the papers root.""" base, _ = domain_locations(config) parent = Path(note_path).parent.parent try: parts = parent.resolve().relative_to(base.resolve()).parts except ValueError: parts = () return "/".join(parts) or parent.name def domain_tag(domain: str) -> str: return TAG_PREFIX + re.sub(r"\s+", "_", domain.strip()) def domain_tag_issue(text: str, domain: str) -> str: """Empty when the note carries exactly one papers/ tag and it names the saved domain.""" try: fields = parse_frontmatter(text) or {} except FrontmatterError: fields = {} tags = fields.get("tags") values = tags if isinstance(tags, list) else [tags] if isinstance(tags, str) else [] found = [tag.strip() for tag in values if tag.strip().startswith(TAG_PREFIX)] expected = domain_tag(domain) if found == [expected]: return "" return ( f"The note must carry exactly one {TAG_PREFIX} tag, {expected}, matching the saved " f"domain {domain}; found {', '.join(found) or 'none'}." )skills/deeppapernote/scripts/_evidence_heuristics.py8.1 KBa8fbfd8e4056upstream ↗
"""Keyword heuristics that pick candidate evidence sentences from paper text.""" from __future__ import annotations import re from _pdf_text import split_sentences from common import normalize_title, normalize_whitespace def infer_paper_type(title: str, abstract: str) -> tuple[str, str]: lower = f"{title} {abstract}".lower() survey_terms = [ "survey", "tutorial", "overview", "systematic review", "literature review", "scoping review", "meta-analysis", "meta analysis", ] if any(re.search(rf"(?<![a-z0-9]){re.escape(token)}(?![a-z0-9])", lower) for token in survey_terms): return "survey_or_review", "The paper is an explicit survey, review, tutorial, overview, or meta-analysis." if title.strip().lower().startswith("position:"): return "position", "The title declares a position paper." if any(token in lower for token in ["depression", "anxiety", "mental health", "clinical", "patient", "psychiatric", "psychological", "hospital"]): return "human_subjects_empirical", "The paper is closer to an empirical study of human participants." if any(token in lower for token in ["semi-structured interview", "focus group", "thematic analysis", "ethnograph", "grounded theory", "qualitative study"]): return "qualitative_empirical", "The paper reports qualitative fieldwork or thematic analysis." if any(token in lower for token in ["humanities", "theoretical framing", "interpretive", "conceptual analysis", "argument structure", "philosoph"]): return "humanities_argument", "The paper emphasizes interpretation or conceptual argument." if any(token in lower for token in ["benchmark", "leaderboard", "evaluation suite", "dataset", "corpus"]): return "benchmark_or_dataset", "The paper emphasizes benchmark, dataset, or evaluation design." if any(token in lower for token in ["we prove", "theorem", "lower bound", "upper bound", "sample complexity"]): return "theory", "The paper centers on proved results." return "method", "The paper is best treated as a method-focused technical paper." def extract_dataset_candidates(text: str) -> list[str]: found: list[str] = [] seen = set() for sentence in split_sentences(text): if not any(token in sentence.lower() for token in ["dataset", "benchmark", "corpus", "participants", "patients"]): continue candidates = re.findall(r"\b[A-Z][A-Za-z0-9+\-]{2,}(?:[ -][A-Z][A-Za-z0-9+\-]{2,})?\b", sentence) for candidate in candidates: norm = candidate.lower() if norm in seen: continue seen.add(norm) found.append(candidate) if len(found) >= 8: return found return found def extract_metric_claims(text: str) -> list[str]: claims: list[str] = [] seen = set() for sentence in split_sentences(text): lower = sentence.lower() if not re.search(r"\d", sentence): continue if not any(token in lower for token in ["accuracy", "f1", "auc", "auprc", "mae", "rmse", "score", "%", "outperform", "improv", "bac"]): continue normalized = normalize_whitespace(sentence) key = normalize_title(normalized) if key in seen: continue seen.add(key) claims.append(normalized) if len(claims) >= 8: break return claims def extract_negative_claims(text: str, *, limit: int = 6) -> list[str]: claims: list[str] = [] seen = set() explicit_negative_tokens = [ "worse", "degrade", "degraded", "drop", "dropped", "decrease", "decreased", "unstable", "instability", "fail", "failed", "fails", "collapse", "collapsed", "underperform", "underperformed", "sensitive", "sensitivity", "trade-off", "tradeoff", "hurt performance", "hurts performance", ] ablation_tokens = [ "without", "w/o", "remove", "removed", "removing", "omit", "omits", "omitted", "omitting", "ablation", ] performance_tokens = [ "accuracy", "f1", "auc", "auprc", "mae", "rmse", "score", "performance", "%", "result", "results", "training", "converge", "convergence", "stable", "stability", "baseline", ] positive_only_tokens = [ "outperform", "improv", "better than", "achieve", "state-of-the-art", "sota", ] for sentence in split_sentences(text): normalized = normalize_whitespace(sentence) if not normalized: continue lower = normalized.lower() has_explicit_negative = any(token in lower for token in explicit_negative_tokens) has_ablation_marker = any(token in lower for token in ablation_tokens) has_performance_context = any(token in lower for token in performance_tokens) or bool(re.search(r"\d", normalized)) has_positive_only = any(token in lower for token in positive_only_tokens) if not has_explicit_negative: if not (has_ablation_marker and has_performance_context): continue if has_positive_only and not any(token in lower for token in ["trade-off", "tradeoff", "sensitive", "stability"]): continue key = normalize_title(normalized) if key in seen: continue seen.add(key) claims.append(normalized) if len(claims) >= limit: break return claims def extract_mechanism_flow_sentences(text: str, *, limit: int = 8) -> list[str]: claims: list[str] = [] seen = set() action_tokens = [ "encode", "encoded", "encoding", "extract", "extracted", "project", "projected", "pool", "pooled", "fuse", "fused", "fusion", "concat", "concatenate", "query", "queried", "align", "aligned", "compress", "compressed", "send", "sent", "feed", "fed", "generate", "generated", "predict", "predicted", "decode", "decoded", "update", "updated", "freeze", "frozen", "fine-tune", "finetune", ] flow_tokens = [ "input", "inputs", "output", "outputs", "token", "tokens", "feature", "features", "representation", "representations", "encoder", "decoder", "attention", "module", "modules", "llm", "language model", "query token", "cross-attention", "projection", "state", "states", ] for sentence in split_sentences(text): normalized = normalize_whitespace(sentence) if not normalized: continue lower = normalized.lower() if not any(token in lower for token in action_tokens): continue if not any(token in lower for token in flow_tokens): continue key = normalize_title(normalized) if key in seen: continue seen.add(key) claims.append(normalized) if len(claims) >= limit: break return claims def pick_sentences_by_keywords(text: str, keywords: list[str], *, limit: int = 5) -> list[str]: picked: list[str] = [] seen = set() for sentence in split_sentences(text): lower = sentence.lower() if not any(keyword in lower for keyword in keywords): continue normalized = normalize_title(sentence) if normalized in seen: continue seen.add(normalized) picked.append(normalize_whitespace(sentence)) if len(picked) >= limit: break return pickedskills/deeppapernote/scripts/_final_quality.py12 KBa69bba2ab754upstream ↗
from __future__ import annotations import hashlib import json import math import re # Version 2 added the takeaways check; version 3 added the connections check; version 4 # added the core_questions and narrative checks. The readability receipt and the revision # journal kept their shape and stay at version 1. QUALITY_REVIEW_SCHEMA_VERSION = 4 RECEIPT_SCHEMA_VERSION = 1 QUALITY_CHECKS = ( "central_claims", "mechanism_result_map", "claim_boundaries", "takeaways", "connections", "core_questions", "narrative", ) # Plan lists each check reviews; a blocking finding must point at an item of one of them. QUALITY_CHECK_PLAN_FIELDS = { "central_claims": ("central_claims",), "mechanism_result_map": ("mechanism_result_map",), "claim_boundaries": ("claim_boundaries",), "takeaways": ("reuse_takeaways", "followup_questions"), "connections": ("connections", "key_prior_work"), } # The two v4 checks review plan fields that are not lists: core_answers is keyed by # question id, and storyline is one string. QUESTION_REF_RE = re.compile(r"/core_answers/([a-z_]+\.[a-z_]+)") STORYLINE_REF = "/storyline" # A vault may hold no related notes; the connections check then reviews key_prior_work alone. MAY_BE_EMPTY_PLAN_FIELDS = frozenset({"connections"}) SHA256_RE = re.compile(r"[0-9a-f]{64}") PLAN_REF_RE = re.compile( r"/(central_claims|mechanism_result_map|claim_boundaries|reuse_takeaways|followup_questions" r"|connections|key_prior_work)" r"/(0|[1-9][0-9]*)(/what_it_does_not_prove)?" ) def strict_json_loads(text: str) -> dict: def object_pairs(pairs: list[tuple[str, object]]) -> dict: result = {} for key, value in pairs: if key in result: raise ValueError(f"Duplicate JSON field: {key}") result[key] = value return result def reject_constant(value: str) -> None: raise ValueError(f"Nonfinite JSON value: {value}") def finite_float(value: str) -> float: result = float(value) if not math.isfinite(result): reject_constant(value) return result value = json.loads( text, object_pairs_hook=object_pairs, parse_constant=reject_constant, parse_float=finite_float, ) if not isinstance(value, dict): raise ValueError("Final review artifact must be a JSON object.") return value def canonical_note_text(raw: bytes) -> str: if raw.startswith(b"\xef\xbb\xbf") or b"\r" in raw: raise ValueError( "Final draft must be canonical UTF-8 without BOM and with LF line endings." ) try: return raw.decode("utf-8") except UnicodeDecodeError as exc: raise ValueError("Final draft must be canonical UTF-8.") from exc def _keys(value: object, keys: set[str], name: str) -> None: if not isinstance(value, dict) or set(value) != keys: raise ValueError(f"{name} requires exactly these fields: {', '.join(sorted(keys))}.") def _version_language( value: dict, language: str, name: str, *, version: int = RECEIPT_SCHEMA_VERSION, hint: str = "" ) -> None: if type(value["schema_version"]) is not int or value["schema_version"] != version: raise ValueError(f"{name} requires schema_version {version}.{hint}") if language not in {"en", "zh-CN"} or value["output_language"] != language: raise ValueError(f"{name} output_language does not match {language}.") def _sha(value: object, name: str, *, empty: bool = False) -> None: if empty and value == "": return if not isinstance(value, str) or SHA256_RE.fullmatch(value) is None: raise ValueError(f"{name} requires a lowercase SHA-256.") def _valid_plan_ref(reference: str, check: str, plan: dict) -> bool: if check == "core_questions": match = QUESTION_REF_RE.fullmatch(reference) return match is not None and match.group(1) in plan.get("core_answers", {}) if check == "narrative": return reference == STORYLINE_REF match = PLAN_REF_RE.fullmatch(reference) if match is None: return False group, index, suffix = match.groups() if suffix: if check != "claim_boundaries" or group != "central_claims": return False elif group not in QUALITY_CHECK_PLAN_FIELDS[check]: return False items = plan.get(group) if not isinstance(items, list) or int(index) >= len(items): return False return not suffix or ( isinstance(items[int(index)], dict) and "what_it_does_not_prove" in items[int(index)] ) def validate_quality( review: dict, *, note_sha256: str, plan_sha256: str, plan: dict, language: str, ) -> str: _keys( review, { "schema_version", "output_language", "verdict", "reviewed_note_sha256", "reviewed_plan_sha256", "checks", "blocking_findings", }, "Final Quality Review", ) _version_language( review, language, "Final Quality Review", version=QUALITY_REVIEW_SCHEMA_VERSION, hint=" A review written before the core_questions and narrative checks is stale; rerun the full review.", ) for key, expected in ( ("reviewed_note_sha256", note_sha256), ("reviewed_plan_sha256", plan_sha256), ): _sha(review[key], key) if review[key] != expected: raise ValueError(f"Final Quality Review {key} is stale; rerun the full review.") if ( not isinstance(plan, dict) or plan.get("output_language") != language or any( not isinstance(plan.get(field), list) or (not plan[field] and field not in MAY_BE_EMPTY_PLAN_FIELDS) for fields in QUALITY_CHECK_PLAN_FIELDS.values() for field in fields ) or not isinstance(plan.get("core_answers"), dict) or not plan["core_answers"] or not isinstance(plan.get("storyline"), str) or not plan["storyline"].strip() ): raise ValueError( "Note Plan requires the current language, nonempty quality-check lists, " "core_answers, and a storyline." ) _keys(review["checks"], set(QUALITY_CHECKS), "Final Quality Review checks") if any(type(value) is not bool for value in review["checks"].values()): raise ValueError("Final Quality Review checks must be literal booleans.") if review["verdict"] not in ("pass", "fail"): raise ValueError("Final Quality Review verdict must be pass or fail.") findings = review["blocking_findings"] if not isinstance(findings, list): raise ValueError("Final Quality Review blocking_findings must be an array.") false_checks = {key for key, value in review["checks"].items() if not value} covered = set() for finding in findings: _keys(finding, {"check", "plan_ref", "reason", "required_revision"}, "Blocking finding") if any(not isinstance(value, str) or not value.strip() for value in finding.values()): raise ValueError("Blocking finding fields must be nonempty strings.") check = finding["check"] if check not in false_checks or not _valid_plan_ref(finding["plan_ref"], check, plan): raise ValueError("Blocking finding must reference an existing item of a false check.") covered.add(check) if (review["verdict"] == "pass" and (false_checks or findings)) or ( review["verdict"] == "fail" and (not false_checks or covered != false_checks) ): raise ValueError("Final Quality Review verdict, checks, and findings are inconsistent.") return hashlib.sha256( json.dumps(review, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode( "utf-8" ) ).hexdigest() def validate_readability(receipt: dict, *, note_sha256: str, language: str) -> None: _keys( receipt, { "schema_version", "output_language", "verdict", "reviewed_note_sha256", }, "Final Readability Review", ) _version_language(receipt, language, "Final Readability Review") _sha(receipt["reviewed_note_sha256"], "reviewed_note_sha256") if receipt["verdict"] != "pass" or receipt["reviewed_note_sha256"] != note_sha256: raise ValueError("Final Readability Review must pass for the current final note bytes.") def validate_state(state: dict, *, language: str) -> None: _keys( state, { "schema_version", "output_language", "revisions_used", "last_review", "pending_note_sha256", "exhausted_note_sha256", "seen_review_ids", }, "Final quality state", ) _version_language(state, language, "Final quality state") if type(state["revisions_used"]) is not int or not 0 <= state["revisions_used"] <= 2: raise ValueError("Final quality state revision count must be between zero and two.") last = state["last_review"] _keys(last, {"review_id", "note_sha256", "plan_sha256", "verdict"}, "Recorded review") for key in ("review_id", "note_sha256", "plan_sha256"): _sha(last[key], key) if last["verdict"] not in ("pass", "fail"): raise ValueError("Recorded review verdict must be pass or fail.") seen = state["seen_review_ids"] if not isinstance(seen, list) or not seen: raise ValueError("Final quality state requires seen review identities.") for identity in seen: _sha(identity, "seen review identity") if len(set(seen)) != len(seen) or last["review_id"] not in seen: raise ValueError("Final quality state review identities are inconsistent.") for key in ("pending_note_sha256", "exhausted_note_sha256"): _sha(state[key], key, empty=True) pending, exhausted = state["pending_note_sha256"], state["exhausted_note_sha256"] if (pending and pending != last["note_sha256"]) or (last["verdict"] == "fail" and not pending): raise ValueError("Final quality state pending revision is inconsistent.") if exhausted and (state["revisions_used"] != 2 or exhausted != pending): raise ValueError("Final quality state exhaustion is inconsistent.") if last["verdict"] == "fail" and state["revisions_used"] == 2 and not exhausted: raise ValueError("Final quality state must retain its exhausted revision anchor.") def advance_state(state: dict | None, review: dict, review_id: str, *, language: str) -> dict: if state is None: state = { "schema_version": 1, "output_language": language, "revisions_used": 0, "pending_note_sha256": "", "exhausted_note_sha256": "", "seen_review_ids": [], } else: validate_state(state, language=language) if review_id in state["seen_review_ids"]: if review_id != state["last_review"]["review_id"]: raise ValueError("Old review replay cannot restore a previous passing observation.") return state result = {**state, "seen_review_ids": [*state["seen_review_ids"], review_id]} note = review["reviewed_note_sha256"] if result["exhausted_note_sha256"] and note != result["exhausted_note_sha256"]: raise ValueError("Final quality revision limit exhausted; a third revision is not allowed.") if result["pending_note_sha256"] and note != result["pending_note_sha256"]: if result["revisions_used"] == 2: raise ValueError("Final quality revision limit exhausted.") result["revisions_used"] += 1 result["pending_note_sha256"] = "" if review["verdict"] == "fail": result["pending_note_sha256"] = note if result["revisions_used"] == 2: result["exhausted_note_sha256"] = note result["last_review"] = { "review_id": review_id, "note_sha256": note, "plan_sha256": review["reviewed_plan_sha256"], "verdict": review["verdict"], } return result def require_recorded_pass(state: dict, review: dict, review_id: str, *, language: str) -> None: validate_state(state, language=language) expected = { "review_id": review_id, "note_sha256": review["reviewed_note_sha256"], "plan_sha256": review["reviewed_plan_sha256"], "verdict": "pass", } if review["verdict"] != "pass" or state["last_review"] != expected: raise ValueError("Formal Save requires a current recorded passing Final Quality Review.")skills/deeppapernote/scripts/_frontmatter.py3.3 KB7e82bcb4a88dupstream ↗
"""Read the small YAML subset DeepPaperNote notes use for Obsidian properties. Supported shapes, one key per line at the top level: key: scalar key: "quoted scalar" key: [a, "b, c", "[[Link]]"] key: - item - "[[Link]]" Anything else (nested maps, orphan list items, duplicate keys) raises FrontmatterError, so lint fails closed instead of guessing. """ from __future__ import annotations import re KEY_RE = re.compile(r"^([A-Za-z_][\w-]*)\s*:(?:\s+(.*?))?\s*$") ITEM_RE = re.compile(r"^\s+-\s+(.*?)\s*$|^\s+-\s*$") class FrontmatterError(ValueError): pass def frontmatter_block(text: str) -> str | None: """Return the text between the opening and closing `---` lines, or None.""" normalized = text.removeprefix("").replace("\r\n", "\n") if not normalized.startswith("---\n"): return None end = re.search(r"^---[ \t]*$", normalized[4:], flags=re.MULTILINE) if end is None: raise FrontmatterError("frontmatter is not closed") return normalized[4 : 4 + end.start()] def _unquote(value: str) -> str: value = value.strip() if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'": return value[1:-1] return value def _flow_list(value: str) -> list[str]: inner = value[1:-1] items: list[str] = [] current: list[str] = [] quote = "" for ch in inner: if quote: current.append(ch) if ch == quote: quote = "" elif ch in "\"'": quote = ch current.append(ch) elif ch == ",": items.append("".join(current)) current = [] else: current.append(ch) if quote: raise FrontmatterError("unterminated quote in list") items.append("".join(current)) parsed = [_unquote(item) for item in items] return [] if parsed == [""] else parsed def _scalar_or_list(value: str) -> str | list[str]: if value.startswith("["): if not value.endswith("]") or value.startswith("[["): if value.startswith("[[") and value.endswith("]]"): return value # an unquoted wikilink scalar raise FrontmatterError(f"unterminated list: {value}") return _flow_list(value) return _unquote(value) def parse_frontmatter(text: str) -> dict[str, str | list[str]] | None: block = frontmatter_block(text) if block is None: return None fields: dict[str, str | list[str]] = {} current_list: list[str] | None = None for line in block.split("\n"): if not line.strip() or line.lstrip().startswith("#"): continue if line[0].isspace(): item = ITEM_RE.match(line) if item is None or current_list is None: raise FrontmatterError(f"unsupported line: {line.strip()}") current_list.append(_unquote(item.group(1) or "")) continue match = KEY_RE.match(line) if match is None: raise FrontmatterError(f"unsupported line: {line.strip()}") key, value = match.group(1), match.group(2) or "" if key in fields: raise FrontmatterError(f"duplicate key: {key}") if value: fields[key] = _scalar_or_list(value) current_list = None else: current_list = [] fields[key] = current_list return fieldsskills/deeppapernote/scripts/_identity.py65 KBbb0f47360716upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/_local_pdf_title.py9.7 KB82ea224a7a7aupstream ↗
"""Read title and identifier hints from a local PDF.""" from __future__ import annotations import re from pathlib import Path from typing import Any from _identity import ( candidate_priority_score, choose_best_title_match, clean_local_pdf_stem, extract_arxiv_id, extract_doi, is_probable_local_pdf_artifact_title, normalize_identity_title, publication_quality_score, title_similarity, ) from _pdf_text import clean_pdf_line from common import ( fitz, normalize_whitespace, ) TITLE_MIN_LETTER_SHARE = 0.6 TITLE_MAX_GARBLE_SHARE = 0.05 TITLE_GARBLE_CHARS = set("!\"#$%*<=>@[\\]^_`{|}~") TITLE_LINE_SIZE_RATIO = 0.9 TITLE_LINE_SPACING = 1.6 TITLE_FOOTNOTE_MARKS_RE = re.compile(r"[\s*∗†‡§¶]+$") CJK_CHAR_RE = re.compile(r"[㐀-鿿]") def is_mostly_letters(text: str) -> bool: """Tell readable words from a garbled text layer, which decodes to stray symbols.""" visible = [char for char in TITLE_FOOTNOTE_MARKS_RE.sub("", text) if not char.isspace()] if not visible: return False letters = sum(1 for char in visible if char.isalpha()) garble = sum(1 for char in visible if char in TITLE_GARBLE_CHARS) return ( letters / len(visible) >= TITLE_MIN_LETTER_SHARE and garble / len(visible) <= TITLE_MAX_GARBLE_SHARE ) def is_plausible_pdf_title_line(line: str) -> bool: normalized = clean_pdf_line(line) lower = normalized.lower() # A Chinese title has no spaces between words, so only its length counts. single_word = len(normalized.split()) < 2 and not CJK_CHAR_RE.search(normalized) if len(normalized) < 8 or (len(normalized) < 20 and single_word): return False if not is_mostly_letters(normalized): return False if normalized.count(",") >= 3: return False if any(token in lower for token in ["doi.org/", "http://", "https://", "www.", "check for updates"]): return False if lower in {"abstract", "article", "preprint"}: return False if lower.startswith("npj |") or lower.startswith("arxiv:") or lower.startswith("submitted to"): return False if any(re.search(pattern, lower) for pattern in LOW_CONFIDENCE_LOCAL_TITLE_PATTERNS): return False if " doi:" in lower or lower.startswith("doi:"): return False return True def join_title_lines(lines: list[str]) -> str: title = "" for line in lines: if not title: title = line elif title.endswith("-") and line[:1].islower(): title = title[:-1] + line elif CJK_CHAR_RE.match(title[-1]) and CJK_CHAR_RE.match(line[0]): title += line else: title = f"{title} {line}" return TITLE_FOOTNOTE_MARKS_RE.sub("", title) def first_page_title_from_layout(page: Any) -> str: """Return the first page's title: its largest plausible line plus the adjacent lines set in the same type, so a journal name or volume line printed above it is passed over.""" try: blocks = page.get_text("dict").get("blocks", []) except Exception: return "" lines = [] for block in blocks: for line in block.get("lines", []): if tuple(line.get("dir", (1.0, 0.0))) != (1.0, 0.0): continue spans = line.get("spans", []) text = clean_pdf_line("".join(str(span.get("text", "")) for span in spans)) if text: sizes = [ float(span.get("size", 0)) for span in spans if str(span.get("text", "")).strip() ] size = max(sizes) lines.append({"text": text, "size": size, "top": float(line["bbox"][1])}) candidates = [line for line in lines if is_plausible_pdf_title_line(line["text"])] if not candidates: return "" anchor = max(candidates, key=lambda line: (line["size"], -line["top"])) ordered = sorted(lines, key=lambda line: line["top"]) start = end = ordered.index(anchor) def continues_title(edge: dict, neighbour: dict) -> bool: return ( neighbour["size"] >= anchor["size"] * TITLE_LINE_SIZE_RATIO and abs(neighbour["top"] - edge["top"]) <= anchor["size"] * TITLE_LINE_SPACING and is_mostly_letters(neighbour["text"]) ) while start > 0 and continues_title(ordered[start], ordered[start - 1]): start -= 1 while end + 1 < len(ordered) and continues_title(ordered[end], ordered[end + 1]): end += 1 return join_title_lines([line["text"] for line in ordered[start : end + 1]]) def first_page_title_candidate(first_page_text: str) -> str: for raw_line in (first_page_text or "").splitlines(): if is_plausible_pdf_title_line(raw_line): return clean_pdf_line(raw_line) return "" def extract_local_pdf_hints(pdf_path: Path) -> dict[str, Any]: raw_title = normalize_whitespace(pdf_path.stem.replace("_", " ")) cleaned_title = clean_local_pdf_stem(pdf_path.stem) hints: dict[str, Any] = { "title": cleaned_title or raw_title, "local_pdf_title_source": "local_pdf_stem_used", } if is_probable_local_pdf_artifact_title(raw_title): hints["local_pdf_artifact_title"] = True if fitz is None: return hints metadata_title = "" metadata_subject = "" first_page_text = "" layout_title = "" try: doc = fitz.open(pdf_path) except Exception: return hints try: metadata = doc.metadata or {} metadata_title = normalize_whitespace(str(metadata.get("title", ""))) metadata_subject = normalize_whitespace(str(metadata.get("subject", ""))) if len(doc): first_page_text = doc[0].get_text("text") layout_title = first_page_title_from_layout(doc[0]) except Exception: return hints finally: doc.close() if metadata_title: hints["title"] = metadata_title hints["local_pdf_title_source"] = "pdf_metadata_title_used" # Tools that write the metadata can drop part of the printed title, such as a # venue's "Position:" lead-in, so identity resolution also sees the printed one. if layout_title and normalize_identity_title(layout_title) != normalize_identity_title( metadata_title ): hints["local_pdf_first_page_title"] = layout_title else: page_title = layout_title or first_page_title_candidate(first_page_text) if page_title: hints["title"] = page_title hints["local_pdf_title_source"] = "first_page_title_used" searchable = "\n".join(part for part in [metadata_subject, metadata_title, first_page_text] if part) doi = extract_doi(searchable) if doi: hints["doi"] = doi arxiv_id = extract_arxiv_id(searchable) if arxiv_id: hints["arxiv_id"] = arxiv_id return hints def choose_local_pdf_corrected_title(base: dict[str, Any], candidates: list[dict[str, Any]]) -> str: current_title = normalize_whitespace(str(base.get("title", ""))) if not current_title: return "" titled_candidates = [candidate for candidate in candidates if normalize_whitespace(str(candidate.get("title", "")))] if local_pdf_title_needs_identity_correction(base): identity_matches = [ candidate for candidate in titled_candidates if external_identity_matches(base, candidate) ] if identity_matches: best_identity = sorted( identity_matches, key=lambda item: ( candidate_priority_score(item), publication_quality_score(item), len(normalize_whitespace(str(item.get("title", "")))), ), reverse=True, )[0] return normalize_whitespace(str(best_identity.get("title", ""))) if not is_probable_local_pdf_artifact_title(current_title): return "" best = choose_best_title_match(current_title, titled_candidates) if not best: return "" candidate_title = normalize_whitespace(str(best.get("title", ""))) if not candidate_title: return "" if title_similarity(current_title, candidate_title) < 0.55: return "" if not (best.get("doi") or best.get("arxiv_id") or publication_quality_score(best) >= 2): return "" return candidate_title LOW_CONFIDENCE_LOCAL_TITLE_PATTERNS = ( r"^(?:research|original|review)\s+article$", r"^provided proper attribution\b", r"^published as\b", r"^submitted to\b", r"^accepted (?:as|to|for)\b", r"^conference paper\b", r"^proceedings of\b", r"^arxiv\s*:", r"^doi\s*:", ) def local_pdf_title_needs_identity_correction(base: dict[str, Any]) -> bool: title = normalize_whitespace(str(base.get("title", ""))) if not title: return False if base.get("local_pdf_artifact_title") or is_probable_local_pdf_artifact_title(title): return True source = normalize_whitespace(str(base.get("local_pdf_title_source", ""))) if source not in {"first_page_title_used", "local_pdf_stem_used", "pdf_metadata_title_used"}: return False lowered = title.lower() return any(re.search(pattern, lowered) for pattern in LOW_CONFIDENCE_LOCAL_TITLE_PATTERNS) def external_identity_matches(base: dict[str, Any], candidate: dict[str, Any]) -> bool: base_doi = extract_doi(str(base.get("doi", ""))) or "" candidate_doi = extract_doi(str(candidate.get("doi", ""))) or "" if base_doi and candidate_doi and base_doi.lower() == candidate_doi.lower(): return True base_arxiv = normalize_whitespace(str(base.get("arxiv_id", ""))) candidate_arxiv = normalize_whitespace(str(candidate.get("arxiv_id", ""))) return bool(base_arxiv and candidate_arxiv and base_arxiv == candidate_arxiv)skills/deeppapernote/scripts/_metadata_sources.py23 KB571ffce223ccupstream ↗
"""Fetch and normalize paper metadata from arXiv, Crossref, Semantic Scholar, and OpenAlex.""" from __future__ import annotations import json import re import ssl import urllib.parse import urllib.request import xml.etree.ElementTree as ET from copy import deepcopy from pathlib import Path from typing import Any from _identity import ( _record_arxiv_id, _string_field, adjudicate_identity_observations, apply_identity_confidence, choose_best_title_match, clean_local_pdf_stem, extract_arxiv_id, extract_doi, infer_source_type, normalize_identity_title, paper_id_for_record, ) from _local_pdf_title import extract_local_pdf_hints from common import ( env_config_value, normalize_title, normalize_whitespace, strip_tags, ) try: import certifi # type: ignore except ImportError: # pragma: no cover certifi = None ARXIV_NS = { "atom": "http://www.w3.org/2005/Atom", "arxiv": "http://arxiv.org/schemas/atom", } SEMANTIC_SCHOLAR_SEARCH_URL = "https://api.semanticscholar.org/graph/v1/paper/search" OPENALEX_WORKS_URL = "https://api.openalex.org/works" CROSSREF_WORKS_URL = "https://api.crossref.org/works" DEFAULT_USER_AGENT = "DeepPaperNote/0.1" def fallback_arxiv_record(arxiv_id: str, source_type: str, source_url: str = "") -> dict[str, Any]: paper = { "status": "ok", "source_type": source_type, "source_url": source_url or f"https://arxiv.org/abs/{arxiv_id}", "arxiv_id": arxiv_id, "pdf_url": f"https://arxiv.org/pdf/{arxiv_id}.pdf", "metadata_sources": [source_type], } paper["paper_id"] = paper_id_for_record(paper) return apply_identity_confidence(paper) def http_get_text(url: str, *, timeout: int = 30, headers: dict[str, str] | None = None) -> str: request = urllib.request.Request(url, headers=headers or {"User-Agent": DEFAULT_USER_AGENT}) with urllib.request.urlopen(request, timeout=timeout, context=https_ssl_context(url)) as response: return response.read().decode("utf-8") def http_get_json(url: str, *, timeout: int = 30, headers: dict[str, str] | None = None) -> dict[str, Any]: return json.loads(http_get_text(url, timeout=timeout, headers=headers)) def http_get_bytes(url: str, *, timeout: int = 60, headers: dict[str, str] | None = None) -> bytes: request = urllib.request.Request(url, headers=headers or {"User-Agent": DEFAULT_USER_AGENT}) with urllib.request.urlopen(request, timeout=timeout, context=https_ssl_context(url)) as response: return response.read() def https_ssl_context(url: str) -> ssl.SSLContext | None: if not str(url).lower().startswith("https://") or certifi is None: return None try: return ssl.create_default_context(cafile=certifi.where()) except Exception: return None def semantic_scholar_headers() -> dict[str, str]: headers = {"User-Agent": DEFAULT_USER_AGENT} api_key = env_config_value("DEEPPAPERNOTE_SEMANTIC_SCHOLAR_API_KEY", "SEMANTIC_SCHOLAR_API_KEY") if api_key: headers["x-api-key"] = api_key return headers def parse_arxiv_xml(xml_content: str) -> list[dict[str, Any]]: papers: list[dict[str, Any]] = [] root = ET.fromstring(xml_content) for entry in root.findall("atom:entry", ARXIV_NS): paper: dict[str, Any] = { "source": "arxiv", "source_type": "arxiv", "metadata_sources": ["arxiv"], } id_elem = entry.find("atom:id", ARXIV_NS) if id_elem is not None and id_elem.text: paper["source_url"] = normalize_whitespace(id_elem.text) paper["url"] = paper["source_url"] arxiv_id = extract_arxiv_id(paper["source_url"]) if arxiv_id: paper["arxiv_id"] = arxiv_id title_elem = entry.find("atom:title", ARXIV_NS) paper["title"] = normalize_whitespace(title_elem.text if title_elem is not None else "") summary_elem = entry.find("atom:summary", ARXIV_NS) paper["abstract"] = normalize_whitespace(summary_elem.text if summary_elem is not None else "") journal_ref_elem = entry.find("arxiv:journal_ref", ARXIV_NS) journal_ref = normalize_whitespace(journal_ref_elem.text if journal_ref_elem is not None else "") if journal_ref: paper["venue"] = journal_ref doi_elem = entry.find("arxiv:doi", ARXIV_NS) if doi_elem is not None and doi_elem.text: paper["doi"] = normalize_whitespace(doi_elem.text) authors = [] for author in entry.findall("atom:author", ARXIV_NS): name_elem = author.find("atom:name", ARXIV_NS) if name_elem is not None and name_elem.text: authors.append(normalize_whitespace(name_elem.text)) paper["authors"] = authors published_elem = entry.find("atom:published", ARXIV_NS) if published_elem is not None and published_elem.text: paper["published"] = normalize_whitespace(published_elem.text) if re.match(r"^\d{4}", paper["published"]): paper["year"] = paper["published"][:4] for link in entry.findall("atom:link", ARXIV_NS): if link.get("title") == "pdf" and link.get("href"): paper["pdf_url"] = str(link.get("href")) break papers.append(paper) return papers def fetch_arxiv_entries(*, search_query: str = "", id_list: str = "", max_results: int = 10) -> list[dict[str, Any]]: params = urllib.parse.urlencode( { "search_query": search_query, "id_list": id_list, "start": 0, "max_results": max_results, } ) try: xml_content = http_get_text(f"https://export.arxiv.org/api/query?{params}") except Exception: return [] if not normalize_whitespace(xml_content): return [] try: return parse_arxiv_xml(xml_content) except Exception: return [] def safe_fetch_arxiv_entries(*, search_query: str = "", id_list: str = "", max_results: int = 10) -> list[dict[str, Any]]: try: return fetch_arxiv_entries(search_query=search_query, id_list=id_list, max_results=max_results) except Exception: return [] def normalize_crossref_work(item: dict[str, Any]) -> dict[str, Any]: title = normalize_whitespace(" ".join(item.get("title") or [])) # Crossref splits "LightGCN: Simplifying ..." into title and subtitle. subtitle = normalize_whitespace(" ".join(item.get("subtitle") or [])) if subtitle and subtitle.casefold() not in title.casefold(): title = f"{title}: {subtitle}" if title else subtitle authors = [] affiliations = [] for author in item.get("author", []) or []: given = normalize_whitespace(str(author.get("given", ""))) family = normalize_whitespace(str(author.get("family", ""))) name = normalize_whitespace(" ".join(part for part in [given, family] if part)) if name: authors.append(name) for aff in author.get("affiliation", []) or []: aff_name = normalize_whitespace(str(aff.get("name", ""))) if aff_name and aff_name not in affiliations: affiliations.append(aff_name) venue = normalize_whitespace(" ".join(item.get("container-title") or [])) published = ( item.get("published-print", {}).get("date-parts") or item.get("published-online", {}).get("date-parts") or item.get("issued", {}).get("date-parts") or [] ) year = "" if published and isinstance(published, list) and isinstance(published[0], list) and published[0]: year = str(published[0][0]) doi = normalize_whitespace(str(item.get("DOI", ""))) source_url = normalize_whitespace(str(item.get("URL", ""))) return { "title": title, "authors": authors, "affiliations": affiliations, "venue": venue, "doi": doi, "source": "crossref", "source_type": "crossref", "source_url": source_url, "year": year, "published": year, "abstract": strip_tags(str(item.get("abstract", ""))), "metadata_sources": ["crossref"], } def fetch_crossref_by_doi(doi: str) -> dict[str, Any] | None: url = f"{CROSSREF_WORKS_URL}/{urllib.parse.quote(doi)}" try: data = http_get_json(url, headers={"User-Agent": DEFAULT_USER_AGENT}) except Exception: return None message = data.get("message") or {} if not isinstance(message, dict): return None return normalize_crossref_work(message) def search_crossref_by_title(title: str, *, limit: int = 5) -> list[dict[str, Any]]: params = urllib.parse.urlencode({"query.title": title, "rows": limit}) try: data = http_get_json(f"{CROSSREF_WORKS_URL}?{params}", headers={"User-Agent": DEFAULT_USER_AGENT}) except Exception: return [] items = data.get("message", {}).get("items", []) or [] return [normalize_crossref_work(item) for item in items if isinstance(item, dict)] def normalize_semantic_scholar_paper(paper: dict[str, Any]) -> dict[str, Any]: ext_ids = paper.get("externalIds") or {} doi = normalize_whitespace(str(ext_ids.get("DOI", ""))) arxiv_id = normalize_whitespace(str(ext_ids.get("ArXiv", ""))) affiliations: list[str] = [] authors: list[str] = [] for author in paper.get("authors", []) or []: if not isinstance(author, dict): continue name = normalize_whitespace(str(author.get("name", ""))) if name: authors.append(name) raw_affs = author.get("affiliations", []) or [] if isinstance(raw_affs, str): raw_affs = [raw_affs] for aff in raw_affs: aff_name = normalize_whitespace(str(aff)) if aff_name and aff_name not in affiliations: affiliations.append(aff_name) result = { "title": normalize_whitespace(str(paper.get("title", ""))), "abstract": normalize_whitespace(str(paper.get("abstract", ""))), "authors": authors, "affiliations": affiliations, "venue": normalize_whitespace(str(paper.get("venue", ""))), "year": normalize_whitespace(str(paper.get("year", ""))), "doi": doi, "arxiv_id": arxiv_id, "source": "semantic_scholar", "source_type": "semantic_scholar", "source_url": normalize_whitespace(str(paper.get("url", ""))), "metadata_sources": ["semantic_scholar"], } if arxiv_id and not result.get("pdf_url"): result["pdf_url"] = f"https://arxiv.org/pdf/{arxiv_id}.pdf" return result def search_semantic_scholar(query: str, *, limit: int = 5) -> list[dict[str, Any]]: params = urllib.parse.urlencode( { "query": query, "limit": limit, "fields": "title,abstract,year,venue,url,externalIds,authors.name,authors.affiliations", } ) try: data = http_get_json( f"{SEMANTIC_SCHOLAR_SEARCH_URL}?{params}", headers=semantic_scholar_headers(), ) except Exception: return [] items = data.get("data", []) or [] return [normalize_semantic_scholar_paper(item) for item in items if isinstance(item, dict)] def normalize_openalex_work(item: dict[str, Any]) -> dict[str, Any]: title = normalize_whitespace(str(item.get("display_name", ""))) authors = [] affiliations = [] for authorship in item.get("authorships", []) or []: if not isinstance(authorship, dict): continue author = authorship.get("author", {}) or {} name = normalize_whitespace(str(author.get("display_name", ""))) if name: authors.append(name) for institution in authorship.get("institutions", []) or []: if not isinstance(institution, dict): continue inst_name = normalize_whitespace(str(institution.get("display_name", ""))) if inst_name and inst_name not in affiliations: affiliations.append(inst_name) ids = item.get("ids", {}) or {} doi_url = normalize_whitespace(str(ids.get("doi", ""))) doi = extract_doi(doi_url or normalize_whitespace(str(item.get("doi", "")))) or "" primary_location = item.get("primary_location", {}) or {} pdf_url = normalize_whitespace(str((primary_location.get("pdf_url") or ""))) landing_page_url = normalize_whitespace(str(primary_location.get("landing_page_url") or "")) best_oa = item.get("best_oa_location", {}) or {} if not pdf_url: pdf_url = normalize_whitespace(str(best_oa.get("pdf_url") or "")) if not landing_page_url: landing_page_url = normalize_whitespace(str(best_oa.get("landing_page_url") or "")) venue = normalize_whitespace(str((primary_location.get("source", {}) or {}).get("display_name", ""))) year = normalize_whitespace(str(item.get("publication_year", ""))) return { "title": title, "authors": authors, "affiliations": affiliations, "venue": venue, "year": year, "doi": doi, "source": "openalex", "source_type": "openalex", "source_url": landing_page_url or normalize_whitespace(str(item.get("id", ""))), "pdf_url": pdf_url, "abstract": "", "metadata_sources": ["openalex"], } def fetch_openalex_by_doi(doi: str) -> dict[str, Any] | None: url = f"{OPENALEX_WORKS_URL}/https://doi.org/{urllib.parse.quote(doi, safe='')}" try: data = http_get_json(url, headers={"User-Agent": DEFAULT_USER_AGENT}) except Exception: return None if not isinstance(data, dict): return None return normalize_openalex_work(data) def search_openalex_by_title(title: str, *, limit: int = 5) -> list[dict[str, Any]]: params = urllib.parse.urlencode({"search": title, "per-page": limit}) try: data = http_get_json(f"{OPENALEX_WORKS_URL}?{params}", headers={"User-Agent": DEFAULT_USER_AGENT}) except Exception: return [] items = data.get("results", []) or [] return [normalize_openalex_work(item) for item in items if isinstance(item, dict)] def merge_metadata_records(*records: dict[str, Any]) -> dict[str, Any]: merged: dict[str, Any] = {} metadata_sources: list[str] = [] additive_list_fields = {"affiliations", "metadata_sources", "identity_confidence_reasons"} for record in records: if not isinstance(record, dict): continue for key, value in record.items(): if value in ("", None, [], {}): continue if key == "authors": if not merged.get("authors"): values = value if isinstance(value, list) else [value] seen = set() deduped = [] for item in values: cleaned = normalize_whitespace(str(item)) marker = normalize_title(cleaned) if cleaned and marker and marker not in seen: deduped.append(cleaned) seen.add(marker) if deduped: merged["authors"] = deduped continue if key in additive_list_fields: current = merged.setdefault(key, []) if not isinstance(current, list): current = [] merged[key] = current values = value if isinstance(value, list) else [value] for item in values: cleaned = normalize_whitespace(str(item)) if cleaned and cleaned not in current: current.append(cleaned) continue if key not in merged or merged[key] in ("", None): merged[key] = value for record in records: if isinstance(record, dict): for source in record.get("metadata_sources", []) or []: source_name = normalize_whitespace(str(source)) if source_name and source_name not in metadata_sources: metadata_sources.append(source_name) if metadata_sources: merged["metadata_sources"] = metadata_sources merged["paper_id"] = paper_id_for_record(merged) return merged def _has_strong_work_identifier(record: dict[str, Any]) -> bool: paper_id = _string_field(record, "paper_id").lower() return bool( extract_doi(_string_field(record, "doi")) or extract_doi(_string_field(record, "source_url")) or _record_arxiv_id(record) or paper_id.startswith(("doi:", "arxiv:")) ) def resolve_reference(value: str) -> dict[str, Any]: source_type = infer_source_type(value) stripped = (value or "").strip() if source_type == "local_pdf": path = Path(stripped).expanduser().resolve() hints = extract_local_pdf_hints(path) paper = { "status": "ok", "source_type": "local_pdf", "source_url": str(path), "local_pdf_path": str(path), "title": normalize_whitespace(str(hints.get("title", ""))) or clean_local_pdf_stem(path.stem) or path.stem.replace("_", " "), "metadata_sources": ["local_pdf"], } if hints.get("local_pdf_title_source"): paper["local_pdf_title_source"] = hints["local_pdf_title_source"] if hints.get("local_pdf_artifact_title"): paper["local_pdf_artifact_title"] = True if hints.get("local_pdf_first_page_title"): paper["local_pdf_first_page_title"] = hints["local_pdf_first_page_title"] doi = normalize_whitespace(str(hints.get("doi", ""))) arxiv_id = normalize_whitespace(str(hints.get("arxiv_id", ""))) if doi: paper["doi"] = doi if arxiv_id: paper["arxiv_id"] = arxiv_id paper["paper_id"] = paper_id_for_record(paper) return apply_identity_confidence(paper) if source_type == "arxiv_id": arxiv_id = extract_arxiv_id(stripped) or "" if arxiv_id: return fallback_arxiv_record(arxiv_id, "arxiv_id") if source_type == "arxiv_url": arxiv_id = extract_arxiv_id(stripped) or "" if arxiv_id: return fallback_arxiv_record(arxiv_id, "arxiv_url", source_url=stripped) if source_type in {"doi", "doi_url"}: doi = extract_doi(stripped) or "" paper = { "status": "ok", "source_type": "doi", "source_url": f"https://doi.org/{doi}", "doi": doi, "metadata_sources": ["doi"], } paper["paper_id"] = paper_id_for_record(paper) return apply_identity_confidence(paper) if source_type == "pdf_url": filename = Path(urllib.parse.urlparse(stripped).path).stem or "paper" paper = { "status": "ok", "source_type": "pdf_url", "source_url": stripped, "pdf_url": stripped, "title": filename.replace("_", " "), "metadata_sources": ["pdf_url"], } paper["paper_id"] = paper_id_for_record(paper) return apply_identity_confidence(paper) if source_type == "url": doi = extract_doi(stripped) if doi: return resolve_reference(doi) paper = { "status": "ok", "source_type": "url", "source_url": stripped, "metadata_sources": ["url"], } paper["paper_id"] = paper_id_for_record(paper) return apply_identity_confidence(paper) if source_type == "zotero_key": paper = { "status": "ok", "source_type": "zotero_key", "zotero_key": stripped, "source_url": "", "metadata_sources": ["zotero_key"], } paper["paper_id"] = paper_id_for_record(paper) return apply_identity_confidence(paper) title = stripped paper = { "status": "ok", "source_type": "title_query", "title": title, "source_url": "", "metadata_sources": ["title_query"], } paper["paper_id"] = paper_id_for_record(paper) return apply_identity_confidence(paper) def collect_metadata_observations(record: dict[str, Any]) -> list[dict[str, Any]]: base = dict(record) observations = [ deepcopy(item) for item in base.get("identity_observations", []) or [] if isinstance(item, dict) ] doi = normalize_whitespace(str(base.get("doi", ""))) title = normalize_whitespace(str(base.get("title", ""))) arxiv_id = normalize_whitespace(str(base.get("arxiv_id", ""))) def append( provider: str, kind: str, value: str, candidate: dict[str, Any] | None, relation: dict[str, str] | None = None, ) -> None: if not candidate: return observation = { "provider": provider, "retrieved_by": {"kind": kind, "value": value}, "record": deepcopy(candidate), } if relation: observation["relation"] = relation if observation not in observations: observations.append(observation) if doi: crossref = fetch_crossref_by_doi(doi) append("crossref", "doi", doi, crossref) openalex = fetch_openalex_by_doi(doi) append("openalex", "doi", doi, openalex) sem = choose_best_title_match(title or doi, search_semantic_scholar(doi, limit=3)) append("semantic_scholar", "doi", doi, sem) if arxiv_id: arxiv = safe_fetch_arxiv_entries(id_list=arxiv_id, max_results=1) if arxiv: append("arxiv", "arxiv_id", arxiv_id, arxiv[0]) if title: sem = choose_best_title_match(title, search_semantic_scholar(title, limit=5)) append("semantic_scholar", "title", title, sem) oa = choose_best_title_match(title, search_openalex_by_title(title, limit=5)) append("openalex", "title", title, oa) cross = choose_best_title_match(title, search_crossref_by_title(title, limit=5)) append("crossref", "title", title, cross) arxiv_candidates = safe_fetch_arxiv_entries( search_query=f'ti:"{title}"', max_results=5 ) arxiv = choose_best_title_match(title, arxiv_candidates) exact_matches = [ candidate for candidate in arxiv_candidates if normalize_identity_title(_string_field(candidate, "title")) == normalize_identity_title(title) ] relation = ( { "kind": "arxiv_lookup", "match_kind": "title", "match_resolution": "unique_exact", } if arxiv is not None and len(exact_matches) == 1 and arxiv == exact_matches[0] else None ) append("arxiv", "title", title, arxiv, relation) return observations def enrich_metadata(record: dict[str, Any]) -> dict[str, Any]: base = dict(record) observations = collect_metadata_observations(base) merged = adjudicate_identity_observations(base, observations)["accepted_metadata"] if not merged.get("year") and merged.get("published") and re.match(r"^\d{4}", str(merged["published"])): merged["year"] = str(merged["published"])[:4] if merged.get("doi") and not merged.get("source_url"): merged["source_url"] = f"https://doi.org/{merged['doi']}" if merged.get("arxiv_id") and not merged.get("pdf_url"): merged["pdf_url"] = f"https://arxiv.org/pdf/{merged['arxiv_id']}.pdf" if merged.get("arxiv_id") and not merged.get("doi"): merged["doi"] = f"10.48550/arXiv.{merged['arxiv_id']}" merged["paper_id"] = paper_id_for_record(merged) return apply_identity_confidence(merged)skills/deeppapernote/scripts/_note_paths.py5.4 KBd2d73343b7e0upstream ↗
"""Resolve where a note is saved: the save target, the domain folders, and the note path.""" from __future__ import annotations import re from pathlib import Path, PureWindowsPath from typing import Any from common import slugify_filename def configured_obsidian_vault(config: dict[str, Any]) -> Path | None: if str(config.get("save_mode", "")).strip() == "workspace": return None vault = str(config.get("obsidian_vault", "")).strip() if not vault: return None vault_path = Path(vault).expanduser().resolve() if not vault_path.exists() or not vault_path.is_dir(): raise RuntimeError(f"Configured Obsidian vault does not exist: {vault_path}") return vault_path def require_obsidian_vault(config: dict[str, Any]) -> Path: vault_path = configured_obsidian_vault(config) if vault_path is None: raise RuntimeError("Missing Obsidian Vault in User Configuration or the current Run Override.") return vault_path def resolve_note_output_mode(config: dict[str, Any]) -> tuple[str, Path]: save_mode = str(config.get("save_mode", "")).strip() if save_mode == "obsidian": return ("obsidian", require_obsidian_vault(config)) if not save_mode: vault_path = configured_obsidian_vault(config) if vault_path is not None: return ("obsidian", vault_path) workspace_root = Path.cwd().resolve() output_dir = str(config.get("workspace_output_dir", "DeepPaperNote_output")).strip() or "DeepPaperNote_output" output_path = workspace_root / Path( *safe_relative_path_parts(output_dir, "workspace_output_dir") ) return ("workspace", require_path_within(workspace_root, output_path, "workspace_output_dir")) def safe_relative_path_parts(value: str | Path, field: str) -> tuple[str, ...]: raw = str(value).strip() if not raw: return () windows_path = PureWindowsPath(raw) if ( Path(raw).is_absolute() or windows_path.is_absolute() or windows_path.drive or windows_path.root ): raise RuntimeError( f"Unsafe {field}: expected a relative path inside the Save Target: {raw}" ) parts = tuple(part for part in re.split(r"[\\/]+", raw) if part and part != ".") if ".." in parts: raise RuntimeError( f"Unsafe {field}: expected a relative path inside the Save Target: {raw}" ) return parts def require_path_within(root: str | Path, path: str | Path, field: str) -> Path: root_path = Path(root).expanduser().resolve() candidate = Path(path).expanduser().resolve() try: candidate.relative_to(root_path) except ValueError as exc: raise RuntimeError(f"Unsafe {field}: path escapes the Save Target: {path}") from exc return Path(path) def resolve_note_asset_dir(note_path: str | Path, asset_subdir: str) -> Path: asset_subdir_parts = safe_relative_path_parts(asset_subdir, "asset_subdir") if not asset_subdir_parts: raise RuntimeError( "Unsafe asset_subdir: expected a relative directory inside the Save Target" ) note_dir = Path(note_path).parent return require_path_within( note_dir, note_dir / Path(*asset_subdir_parts), "asset_subdir", ) def is_probable_paper_folder(path: Path) -> bool: if not path.is_dir(): return False return any( (path / marker).exists() for marker in ( f"{path.name}.md", f"{path.name}.zh-CN.md", f"{path.name}.en.md", ".deeppapernote.json", ) ) def existing_domain_dirs(config: dict[str, Any]) -> list[str]: output_mode, root_path = resolve_note_output_mode(config) papers_dir = str(config.get("papers_dir", "Research/Papers")).strip() or "Research/Papers" base_dir = root_path / Path(papers_dir) if output_mode == "obsidian" else root_path if not base_dir.exists() or not base_dir.is_dir(): return [] names: list[str] = [] for child in sorted(base_dir.iterdir()): if not child.is_dir(): continue if is_probable_paper_folder(child): continue names.append(child.name) return names def resolve_obsidian_note_path( config: dict[str, Any], *, title: str, subdir: str = "", filename: str = "", ) -> Path: def path_from_parts(parts: tuple[str, ...]) -> Path: return Path(*parts) if parts else Path() output_mode, root_path = resolve_note_output_mode(config) papers_dir = str(config.get("papers_dir", "Research/Papers")).strip() or "Research/Papers" papers_dir_parts = safe_relative_path_parts(papers_dir, "papers_dir") relative_dir = path_from_parts(papers_dir_parts) if output_mode == "obsidian" else Path() if subdir: relative_dir = relative_dir / path_from_parts(safe_relative_path_parts(subdir, "subdir")) note_slug = slugify_filename(title) target_name = filename.strip() or f"{note_slug}.md" target_name_parts = safe_relative_path_parts(target_name, "filename") target_name_path = path_from_parts(target_name_parts) folder_name = target_name_path.stem or note_slug target_path = root_path / relative_dir / folder_name / target_name_path save_target = ( root_path / path_from_parts(papers_dir_parts) if output_mode == "obsidian" else root_path ) require_path_within(root_path, save_target, "papers_dir") return require_path_within(save_target, target_path, "note path")skills/deeppapernote/scripts/_pdf_text.py16 KBeb30da34f7c3upstream ↗
"""Extract sections, appendix pages, and caption lines from PDF text.""" from __future__ import annotations import re import unicodedata from pathlib import Path from typing import Any from common import fitz, normalize_title, normalize_whitespace PDF_LIGATURE_MAP = { "ß": "ss", "ff": "ff", "fi": "fi", "fl": "fl", "ffi": "ffi", "ffl": "ffl", } def normalize_pdf_text_artifacts(text: str) -> str: normalized = text or "" for original, replacement in PDF_LIGATURE_MAP.items(): normalized = normalized.replace(original, replacement) return normalized def split_sentences(text: str) -> list[str]: text = re.sub(r"\s+", " ", text or "").strip() if not text: return [] parts = re.split(r"(?<=[.!?。!?])\s+", text) return [part.strip() for part in parts if part.strip()] def clean_pdf_line(line: str) -> str: line = re.sub(r"\s+", " ", normalize_pdf_text_artifacts(line or "")).strip() if not line: return "" if re.fullmatch(r"\d+", line): return "" if re.fullmatch(r"page \d+", line.lower()): return "" if len(line) <= 2 and not re.search(r"[㐀-鿿]", line): return "" return line def normalize_heading(line: str) -> str: line = normalize_pdf_text_artifacts(line or "").strip().lower() line = re.sub(r"^\s*(?:section\s*)?\d+(\.\d+)*[\s.、.::-]*", "", line) line = re.sub(r"^\s*[一二三四五六七八九十百千]+[、..:\s-]*", "", line) line = re.sub(r"^\s*[ivxlcdm]+[.)\s]+", "", line) line = re.sub(r"[^a-z0-9㐀-鿿\s]", " ", line) line = re.sub(r"(?<=[㐀-鿿])\s+(?=[㐀-鿿])", "", line) return re.sub(r"\s+", " ", line).strip() SECTION_ALIASES = { "abstract": {"abstract", "摘要"}, "introduction": { "introduction", "background", "preliminaries", "preliminary", "related work", "literature review", "引言", "绪论", "背景", "相关工作", "文献综述", }, "method": { "method", "methods", "approach", "approaches", "methodology", "framework", "model", "models", "materials", "materials and methods", "study design", "方法", "方法学", "研究方法", "材料与方法", "实验方法", "研究设计", "模型", "框架", "系统设计", }, "data": { "data", "dataset", "datasets", "corpus", "data and materials", "数据", "数据集", "语料库", }, "experiment": { "experiment", "experiments", "evaluation", "evaluations", "results", "experimental results", "evaluation results", "analysis", "findings", "ablations", "ablation", "实验", "实验结果", "结果", "研究结果", "评价", "评估", "分析", "发现", "消融", }, "conclusion": { "conclusion", "conclusions", "discussion", "discussions", "future work", "limitations", "limitation", "结论", "总结", "讨论", "局限", "不足", "未来工作", }, } STOP_SECTION_ALIASES = { "references", "bibliography", "appendix", "appendices", "supplementary material", "acknowledgments", "acknowledgements", "参考文献", "附录", "补充材料", "致谢", } STOP_SECTION_REASONS = { "references": "references", "bibliography": "references", "参考文献": "references", "appendix": "appendix", "appendices": "appendix", "supplementary material": "appendix", "附录": "appendix", "补充材料": "appendix", "acknowledgments": "acknowledgments", "acknowledgements": "acknowledgments", "致谢": "acknowledgments", } def match_section_heading(line: str) -> str | None: normalized = normalize_heading(line) if not normalized: return None if normalized in STOP_SECTION_ALIASES: return "stop" for section, aliases in SECTION_ALIASES.items(): if normalized in aliases: return section return None def stop_section_reason(line: str, *, allow_prefix: bool = False) -> str: normalized = normalize_heading(line) if not normalized: return "" reason = STOP_SECTION_REASONS.get(normalized, "") if reason or not allow_prefix: return reason if normalized.startswith(("references ", "bibliography ")): return "references" if normalized.startswith(("appendix ", "appendices ", "supplementary material ")): return "appendix" if normalized.startswith(("acknowledgments ", "acknowledgements ")): return "acknowledgments" return "" def pdf_coverage_summary(pdf_path: Path, max_pages: int | None = None) -> dict[str, Any]: summary: dict[str, Any] = { "total_pages": None, "text_max_pages": max_pages, "text_pages_scanned": 0, "truncated_due_to_page_limit": False, "appendix_detected": False, "appendix_start_page": None, "references_start_page": None, "section_stop_reason": "", "section_stop_page": None, } if fitz is None or not pdf_path.is_file(): return summary try: doc = fitz.open(pdf_path) except Exception: return summary try: total_pages = len(doc) page_limit = total_pages if max_pages is None else min(total_pages, max_pages) summary["total_pages"] = total_pages summary["text_pages_scanned"] = page_limit summary["truncated_due_to_page_limit"] = max_pages is not None and total_pages > max_pages for page_index in range(total_pages): page_number = page_index + 1 text = doc[page_index].get_text("text") for raw_line in text.splitlines(): line = clean_pdf_line(raw_line) if not line: continue exact_reason = stop_section_reason(line) detected_reason = exact_reason or stop_section_reason(line, allow_prefix=True) if not detected_reason: continue if detected_reason == "references" and summary["references_start_page"] is None: summary["references_start_page"] = page_number if detected_reason == "appendix" and summary["appendix_start_page"] is None: summary["appendix_start_page"] = page_number summary["appendix_detected"] = True if exact_reason and not summary["section_stop_reason"]: summary["section_stop_reason"] = exact_reason summary["section_stop_page"] = page_number break finally: doc.close() return summary def extract_appendix_page_texts( pdf_path: Path, appendix_start_page: int | None, ) -> list[dict[str, Any]]: if fitz is None or not pdf_path.is_file() or not appendix_start_page: return [] try: doc = fitz.open(pdf_path) except Exception: return [] pages: list[dict[str, Any]] = [] try: start_index = max(int(appendix_start_page) - 1, 0) for page_index in range(start_index, len(doc)): text = doc[page_index].get_text("text") cleaned = normalize_whitespace(text) if cleaned: pages.append({"page": page_index + 1, "text": text}) finally: doc.close() return pages def appendix_section_title(line: str) -> str: cleaned = clean_pdf_line(line) if not cleaned: return "" lower = cleaned.lower() if lower in {"appendix", "appendices", "supplementary material"}: return "" if re.match(r"^fig(?:ure)?\.?\s*\d+", cleaned, re.IGNORECASE): return "" if re.match(r"^table\.?\s*\d+", cleaned, re.IGNORECASE): return "" if len(cleaned) > 120: return "" if re.match(r"^(?:appendix\s+)?[A-Z]\.?\s+.{3,}$", cleaned, re.IGNORECASE): return cleaned return "" def extract_appendix_index( pdf_path: Path, pdf_coverage: dict[str, Any] | None = None, *, max_sections: int = 20, max_captions: int = 24, ) -> dict[str, Any]: pdf_coverage = pdf_coverage or {} start_page = pdf_coverage.get("appendix_start_page") index: dict[str, Any] = { "appendix_detected": bool(pdf_coverage.get("appendix_detected")), "start_page": start_page, "sections": [], "figure_captions": [], "table_captions": [], } if not index["appendix_detected"] or not start_page: return index seen_sections = set() seen_figure_captions = set() seen_table_captions = set() for page in extract_appendix_page_texts(pdf_path, int(start_page)): page_number = int(page.get("page", 0) or 0) text = str(page.get("text", "")) for raw_line in text.splitlines(): title = appendix_section_title(raw_line) marker = normalize_title(title) if title and marker not in seen_sections and len(index["sections"]) < max_sections: seen_sections.add(marker) index["sections"].append({"title": title, "page": page_number}) for caption in extract_caption_lines(text, "figure"): marker = f"{caption.get('id', '').lower()}::{caption.get('caption', '').lower()}" if marker in seen_figure_captions or len(index["figure_captions"]) >= max_captions: continue seen_figure_captions.add(marker) index["figure_captions"].append({**caption, "page_hint": f"p.{page_number}"}) for caption in extract_caption_lines(text, "table"): marker = f"{caption.get('id', '').lower()}::{caption.get('caption', '').lower()}" if marker in seen_table_captions or len(index["table_captions"]) >= max_captions: continue seen_table_captions.add(marker) index["table_captions"].append({**caption, "page_hint": f"p.{page_number}"}) return index def extract_pdf_sections(pdf_path: Path, max_pages: int | None = None) -> dict[str, str]: if fitz is None: return {} sections: dict[str, list[str]] = {"preamble": []} current = "preamble" doc = fitz.open(pdf_path) try: page_limit = len(doc) if max_pages is None else min(len(doc), max_pages) for page_index in range(page_limit): text = doc[page_index].get_text("text") reached_stop = False for raw_line in text.splitlines(): line = clean_pdf_line(raw_line) if not line: continue heading = match_section_heading(line) if heading == "stop": reached_stop = True break if heading: current = heading sections.setdefault(current, []) continue sections.setdefault(current, []).append(line) if reached_stop: break finally: doc.close() collapsed = {} for key, value in sections.items(): if not value: continue text = re.sub(r"\s+", " ", " ".join(value)).strip() if text: collapsed[key] = text return collapsed def extract_pdf_text(pdf_path: Path, max_pages: int | None = None) -> str: if fitz is None: return "" doc = fitz.open(pdf_path) try: page_limit = len(doc) if max_pages is None else min(len(doc), max_pages) texts = [doc[i].get_text("text") for i in range(page_limit)] finally: doc.close() return "\n".join(texts) def normalize_caption_label(label: str) -> str: label = normalize_whitespace(unicodedata.normalize("NFKC", label)) chinese_match = re.match(r"^(图|表)\s*([A-Z]?\d+[a-z]?)$", label, re.IGNORECASE) if chinese_match: return f"{chinese_match.group(1)} {chinese_match.group(2)}" extended_figure_match = re.match( r"^extended\s+data\s+fig(?:ure)?\.?\s*(\d+[a-z]?)$", label, re.IGNORECASE, ) if extended_figure_match: return f"Extended Data Fig {extended_figure_match.group(1)}" extended_table_match = re.match( r"^extended\s+data\s+table\.?\s*(\d+[a-z]?)$", label, re.IGNORECASE, ) if extended_table_match: return f"Extended Data Table {extended_table_match.group(1)}" scheme_match = re.match(r"^(scheme|algorithm)\.?\s*(\d+[a-z]?)$", label, re.IGNORECASE) if scheme_match: return f"{scheme_match.group(1).capitalize()} {scheme_match.group(2)}" supplementary_match = re.match( r"^(supplementary)\s+(fig(?:ure)?|table)\.?\s*(\d+[a-z]?)$", label, re.IGNORECASE, ) if supplementary_match: return f"{supplementary_match.group(1)} {supplementary_match.group(2)} {supplementary_match.group(3)}" english_match = re.match( r"^(fig(?:ure)?|table)\.?\s*([AS]?\d+[a-z]?)$", label, re.IGNORECASE, ) if english_match: return f"{english_match.group(1)} {english_match.group(2)}" return label CAPTION_REFERENCE_VERBS = { "show", "shows", "illustrate", "illustrates", "plot", "plots", "present", "presents", "report", "reports", "depict", "depicts", "compare", "compares", "summarize", "summarizes", "demonstrate", "demonstrates", } def caption_label_key(label: str) -> str: normalized = normalize_caption_label(label).lower() normalized = re.sub(r"\bfigure\b", "fig", normalized) normalized = re.sub(r"\bfig\.\s*", "fig ", normalized) normalized = re.sub(r"\btable\.\s*", "table ", normalized) return normalize_whitespace(normalized) def caption_preference_score(label: str, caption: str) -> int: cleaned_caption = normalize_whitespace(caption) lowered_label = normalize_whitespace(label).lower() first_word_match = re.match(r"^([A-Za-z][A-Za-z-]*)\b", cleaned_caption) first_word = first_word_match.group(1).lower() if first_word_match else "" score = len(cleaned_caption) if lowered_label.startswith(("figure", "table")): score += 25 if first_word in CAPTION_REFERENCE_VERBS or cleaned_caption[:1].islower(): score -= 80 if len(cleaned_caption) < 12: score -= 20 return score def extract_caption_lines(pdf_text: str, kind: str) -> list[dict[str, str]]: grouped: dict[str, dict[str, str]] = {} scores: dict[str, int] = {} order: list[str] = [] lines = [clean_pdf_line(line) for line in pdf_text.splitlines()] if kind == "figure": pattern = re.compile( r"^((?:" r"supplementary\s+fig(?:ure)?\.?\s*\d+[a-z]?" r"|extended\s+data\s+fig(?:ure)?\.?\s*\d+[a-z]?" r"|scheme\.?\s*\d+[a-z]?" r"|algorithm\.?\s*\d+[a-z]?" r"|fig(?:ure)?\.?\s*[AS]?\d+[a-z]?" r"|图\.?\s*[A-Z]?\d+[a-z]?" r"))(?!\.\d)(?:[::.。,\s、|—–-]+|$)(.*)$", re.IGNORECASE, ) else: pattern = re.compile( r"^((?:" r"supplementary\s+table\.?\s*\d+[a-z]?" r"|extended\s+data\s+table\.?\s*\d+[a-z]?" r"|table\.?\s*[AS]?\d+[a-z]?" r"|表\.?\s*[A-Z]?\d+[a-z]?" r"))(?!\.\d)(?:[::.。,\s、|—–-]+|$)(.*)$", re.IGNORECASE, ) for idx, line in enumerate(lines): if not line: continue match = pattern.match(line) if not match: continue label = normalize_caption_label(match.group(1)) caption = normalize_whitespace(match.group(2)) if not caption and idx + 1 < len(lines): caption = normalize_whitespace(lines[idx + 1]) key = caption_label_key(label) if not key: continue candidate = {"id": label, "caption": caption} score = caption_preference_score(label, caption) if key not in grouped: grouped[key] = candidate scores[key] = score order.append(key) continue if score > scores[key]: grouped[key] = candidate scores[key] = score return [grouped[key] for key in order]skills/deeppapernote/scripts/_user_notes.py4.6 KB83e476d25b3bupstream ↗
"""Keep the user-owned My Notes section across note regeneration. The My Notes section is the last top-level section of a note. Everything from its heading line to the end of the file belongs to the user. A generated draft holds only the standard guidance there; Formal Save swaps in the user's block from the note it replaces. """ from __future__ import annotations import hashlib import re from localization import note_schema def _heading_re(language: str) -> re.Pattern[str]: heading = note_schema(language)["sections"]["my_notes"] return re.compile(rf"^##[ \t]+{re.escape(heading)}[ \t]*$", flags=re.MULTILINE) def split_user_notes(text: str, language: str) -> tuple[str, str] | None: """Return (generated part, user block), or None when the heading is absent.""" match = _heading_re(language).search(text) if match is None: return None return text[: match.start()], text[match.start():] def _is_legacy_format(generated: str, language: str) -> bool: """A note saved before the Takeaways section existed; its My Notes may be model text.""" heading = note_schema(language)["sections"]["takeaways"] return re.search(rf"^##[ \t]+{re.escape(heading)}[ \t]*$", generated, flags=re.MULTILINE) is None def _block_body(block: str) -> str: _, _, body = block.partition("\n") return body def is_default_block(block: str, language: str) -> bool: return _block_body(block).strip() == note_schema(language)["my_notes_guidance"] def _content_chars(block: str) -> int: return sum(1 for ch in _block_body(block) if not ch.isspace()) def _normalize(text: str) -> str: return text.removeprefix("").replace("\r\n", "\n").replace("\r", "\n") def inspect_existing_user_notes(text: str, language: str) -> dict: """Describe the user content of a note about to be replaced, for the overwrite prompt.""" parts = split_user_notes(_normalize(text), language) if parts is None: return {"status": "heading_missing", "chars": 0, "legacy_format": False} generated, block = parts if is_default_block(block, language): return {"status": "none", "chars": 0, "legacy_format": False} return { "status": "present", "chars": _content_chars(block), "legacy_format": _is_legacy_format(generated, language), } def merge_user_notes( draft_text: str, existing_text: str | None, language: str, *, recorded_saved_sha256: str = "", ) -> tuple[str, dict]: """Carry the user's My Notes block from the existing note into the reviewed draft. The block is treated as untouched, and the draft's guidance kept, when the existing note is byte-identical to what Formal Save last recorded or when the block is exactly the standard guidance. Any other block is kept as it is. """ if existing_text is None: return draft_text, {"status": "no_existing_note", "chars": 0, "user_notes_sha256": "", "legacy_format": False} if recorded_saved_sha256 and ( hashlib.sha256(existing_text.encode("utf-8")).hexdigest() == recorded_saved_sha256 ): return draft_text, {"status": "untouched", "chars": 0, "user_notes_sha256": "", "legacy_format": False} existing_parts = split_user_notes(_normalize(existing_text), language) if existing_parts is None: return draft_text, {"status": "heading_missing", "chars": 0, "user_notes_sha256": "", "legacy_format": False} existing_generated, block = existing_parts if is_default_block(block, language): return draft_text, {"status": "untouched", "chars": 0, "user_notes_sha256": "", "legacy_format": False} draft_parts = split_user_notes(draft_text, language) if draft_parts is None: raise ValueError( "The draft has no My Notes section, so the user's notes in the existing note " "cannot be kept; restore the section and rerun Final Note Lint." ) if not block.endswith("\n"): block += "\n" return draft_parts[0] + block, { "status": "preserved", "chars": _content_chars(block), "user_notes_sha256": hashlib.sha256(block.encode("utf-8")).hexdigest(), "legacy_format": _is_legacy_format(existing_generated, language), } def restore_generated_text(saved_text: str, draft_text: str, language: str) -> str: """Rebuild the reviewed draft from a saved note by swapping the user block back out.""" saved_parts = split_user_notes(saved_text, language) draft_parts = split_user_notes(draft_text, language) if saved_parts is None or draft_parts is None: return saved_text return saved_parts[0] + draft_parts[1]skills/deeppapernote/scripts/_zotero_local.py31 KB0fce5c431889upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/build_identity_contract.py2.5 KB4c929e04d378upstream ↗
#!/usr/bin/env python3 """Emit the accepted acquisition Paper Identity contract for trusted inputs.""" from __future__ import annotations import argparse from _identity import ( ACCEPTED_IDENTITY_VERDICTS, build_canonical_identity_artifact, build_identity_repair_trace, ) from common import ( emit, maybe_load_json_record, require_ok_input_artifact, ) def parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__ or "build identity contract") p.add_argument("--input", required=True, help="Metadata JSON path or JSON string.") p.add_argument("--resolve", required=True, help="Resolve artifact path for provenance.") p.add_argument("--trace-output", required=True, help="Identity repair trace JSON output path.") p.add_argument("--output", required=True, help="Canonical Identity Artifact JSON output path.") return p def load_required_record(value: str, consumer: str, *, producer: str) -> dict: record = maybe_load_json_record(value) if record is None: raise SystemExit(f"{consumer} requires a JSON acquisition artifact input.") accepted = dict(require_ok_input_artifact(record, consumer)) if str(accepted.get("script", "")) != producer: raise SystemExit(f"{consumer} requires a {producer} producer artifact.") return accepted def main() -> None: args = parser().parse_args() metadata = load_required_record( args.input, "build_identity_contract.py", producer="collect_metadata.py", ) if not isinstance(metadata.get("identity_observations"), list): raise SystemExit( "build_identity_contract.py requires metadata identity_observations." ) source_record = load_required_record( args.resolve, "build_identity_contract.py", producer="resolve_paper.py", ) trace = build_identity_repair_trace( metadata, source_record=source_record, resolve_artifact_path=args.resolve, metadata_artifact_path=args.input, ) emit(trace, args.trace_output) identity = build_canonical_identity_artifact( metadata, source_record=source_record, repair_trace_path=args.trace_output, resolve_artifact_path=args.resolve, metadata_artifact_path=args.input, ) emit(identity, args.output) verdict = str(identity.get("identity_verdict", "")).lower().replace("-", "_") if verdict not in ACCEPTED_IDENTITY_VERDICTS: raise SystemExit(1) if __name__ == "__main__": main()skills/deeppapernote/scripts/build_synthesis_bundle.py25 KB320f01607145upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/check_environment.py5.0 KB68f980e872dcupstream ↗
#!/usr/bin/env python3 """Inspect the local DeepPaperNote environment for maintenance and troubleshooting.""" from __future__ import annotations import argparse import importlib.util import shutil import sys from pathlib import Path from _zotero_local import probe_zotero_local_api from common import emit, env_config_value, runtime_config from user_configuration import inspect_configuration def parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__ or "check environment") p.add_argument("--output", default="", help="Optional JSON output path.") return p def import_available(module_name: str) -> bool: return importlib.util.find_spec(module_name) is not None def find_obsidian_candidates() -> list[str]: roots = [ Path.home() / "Documents", Path.home() / "Desktop", ] results: list[str] = [] seen: set[str] = set() for root in roots: if not root.exists(): continue try: for path in root.rglob("*"): if not path.is_dir(): continue if path == root: continue name = path.name.lower() if "obsidian" not in name and "vault" not in name: continue resolved = str(path.resolve()) if resolved in seen: continue seen.add(resolved) results.append(resolved) if len(results) >= 8: return results except Exception: continue return results def find_local_zotero_hints() -> list[str]: candidates = [ Path.home() / "Zotero", Path.home() / "Library" / "Application Support" / "Zotero", ] hits: list[str] = [] for path in candidates: if path.exists(): hits.append(str(path.resolve())) return hits def main() -> None: args = parser().parse_args() configuration = inspect_configuration() config = runtime_config() if configuration["state"] == "ready" else {} zotero_local_api = probe_zotero_local_api() obsidian_vault = str(config.get("obsidian_vault", "")).strip() obsidian_vault_exists = bool(obsidian_vault) and Path(obsidian_vault).expanduser().exists() tesseract_path = shutil.which("tesseract") or "" pdftoppm_path = shutil.which("pdftoppm") or "" payload = { "status": "ok", "script": "check_environment.py", "tool_role": "maintenance", "user_configuration": configuration, "python": { "executable": sys.executable, "version": sys.version.split()[0], "fitz_installed": import_available("fitz"), "pytesseract_installed": import_available("pytesseract"), "pillow_installed": import_available("PIL"), }, "obsidian": { "configured": bool(obsidian_vault), "vault_path": obsidian_vault, "vault_exists": obsidian_vault_exists, "papers_dir": str(config.get("papers_dir", "")), "output_dir": str(config.get("output_dir", "")), "candidate_vaults": find_obsidian_candidates(), }, "workspace_fallback": { "available": True, "current_working_directory": str(Path.cwd().resolve()), "workspace_output_dir": str(config.get("workspace_output_dir", "DeepPaperNote_output")), "note": ( "With save_mode=workspace, DeepPaperNote saves under the current working directory." ), }, "zotero": { "local_hints": find_local_zotero_hints(), "local_api": zotero_local_api, "local_api_available": bool(zotero_local_api.get("ready")), "local_api_status": str(zotero_local_api.get("status", "error")), "local_api_version": str(zotero_local_api.get("api_version", "")), "local_api_schema_version": str(zotero_local_api.get("schema_version", "")), "mcp_available_from_script": False, "session_integration_checked_by_script": False, "note": ( "The built-in read-only Local API check is reported here. Session-scoped " "library integrations must still be checked by the active agent at runtime." ), }, "ocr": { "tesseract_installed": bool(tesseract_path), "tesseract_path": tesseract_path, "pytesseract_installed": import_available("pytesseract"), "pillow_installed": import_available("PIL"), "pdftoppm_installed": bool(pdftoppm_path), "pdftoppm_path": pdftoppm_path, }, "metadata": { "maintenance_utility": True, "semantic_scholar_api_key_configured": bool( env_config_value( "DEEPPAPERNOTE_SEMANTIC_SCHOLAR_API_KEY", "SEMANTIC_SCHOLAR_API_KEY" ) ), }, } emit(payload, args.output) if __name__ == "__main__": main()skills/deeppapernote/scripts/citation_links.py11 KB3c43afa2984dupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/collect_metadata.py1.2 KBde837d1954b3upstream ↗
#!/usr/bin/env python3 """Collect and merge metadata from Zotero, DOI, Semantic Scholar, OpenAlex, arXiv, and publisher pages.""" from __future__ import annotations from _identity import paper_id_for_record from _metadata_sources import ( collect_metadata_observations, resolve_reference, ) from common import ( base_parser, emit, maybe_load_json_record, require_ok_input_artifact, ) def main() -> None: parser = base_parser(__doc__ or "collect metadata") args = parser.parse_args() if not args.input: raise SystemExit("collect_metadata.py requires --input.") input_record = maybe_load_json_record(args.input) if input_record is not None: record = dict(require_ok_input_artifact(input_record, "collect_metadata.py")) else: record = resolve_reference(args.input) metadata = dict(record) metadata["identity_observations"] = collect_metadata_observations(record) metadata["paper_id"] = args.paper_id or metadata.get("paper_id") or paper_id_for_record(metadata) metadata["status"] = "ok" metadata["script"] = "collect_metadata.py" emit(metadata, args.output) if __name__ == "__main__": main()skills/deeppapernote/scripts/common.py7.1 KB70d6933c65caupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/contracts.py29 KBcb82accd5e4fupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/create_input_record.py3.1 KBce5ba413ecdeupstream ↗
#!/usr/bin/env python3 """Create a deterministic paper input record from trusted metadata such as Zotero results.""" from __future__ import annotations import argparse import json from _identity import paper_id_for_record from common import emit, normalize_whitespace def parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__ or "create input record") p.add_argument("--title", required=True, help="Canonical paper title.") p.add_argument("--translated-title", default="", help="Optional translated title.") p.add_argument("--authors-json", default="[]", help="JSON array of author names.") p.add_argument("--affiliations-json", default="[]", help="JSON array of affiliations.") p.add_argument("--year", default="", help="Publication year.") p.add_argument("--venue", default="", help="Venue or journal.") p.add_argument("--doi", default="", help="DOI if known.") p.add_argument("--arxiv-id", default="", help="arXiv id if known.") p.add_argument("--source-url", default="", help="Canonical source URL.") p.add_argument("--pdf-url", default="", help="Preferred PDF URL if known.") p.add_argument("--local-pdf-path", default="", help="Local PDF path if already available.") p.add_argument("--zotero-key", default="", help="Local Zotero item key.") p.add_argument("--abstract", default="", help="Abstract text when available.") p.add_argument("--source-type", default="", help="Optional explicit source type.") p.add_argument("--output", default="", help="Output JSON path.") return p def parse_json_list(raw: str) -> list[str]: try: data = json.loads(raw) except Exception: return [] if not isinstance(data, list): return [] values: list[str] = [] for item in data: cleaned = normalize_whitespace(str(item)) if cleaned: values.append(cleaned) return values def main() -> None: args = parser().parse_args() record = { "status": "ok", "script": "create_input_record.py", "title": normalize_whitespace(args.title), "translated_title": normalize_whitespace(args.translated_title), "authors": parse_json_list(args.authors_json), "affiliations": parse_json_list(args.affiliations_json), "year": normalize_whitespace(args.year), "venue": normalize_whitespace(args.venue), "doi": normalize_whitespace(args.doi), "arxiv_id": normalize_whitespace(args.arxiv_id), "source_url": normalize_whitespace(args.source_url), "pdf_url": normalize_whitespace(args.pdf_url), "local_pdf_path": normalize_whitespace(args.local_pdf_path), "zotero_key": normalize_whitespace(args.zotero_key), "abstract": normalize_whitespace(args.abstract), "source_type": normalize_whitespace(args.source_type) or ("zotero_seed" if normalize_whitespace(args.zotero_key) else "seed_record"), "metadata_sources": ["zotero_seed"] if normalize_whitespace(args.zotero_key) else ["seed_record"], } record["paper_id"] = paper_id_for_record(record) emit(record, args.output) if __name__ == "__main__": main()skills/deeppapernote/scripts/extract_evidence.py30 KBade4465833a6upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/extract_pdf_assets.py63 KB8f65f2d397e0upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/extract_source_text.py16 KB01a31fb7285eupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/fetch_pdf.py12 KBa303edb6defbupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/find_archived_source.py1.7 KB8b37bf9ef558upstream ↗
#!/usr/bin/env python3 """Find a verified local source without changing the Vault.""" from __future__ import annotations import argparse import re from pathlib import Path from common import emit, maybe_load_json_record, runtime_config from paper_archive import ArchiveError, find_source def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--input", required=True, help="Metadata JSON or original paper reference.") parser.add_argument("--vault", default="") parser.add_argument( "--reference", default="", help="Original user reference; preserves explicit versions." ) parser.add_argument("--target-directory", default="") parser.add_argument("--source-sha256", default="") parser.add_argument("--output", default="") args = parser.parse_args() record = maybe_load_json_record(args.input) or {"source_url": args.input} if re.fullmatch(r"\d{4}\.\d{4,5}(?:v\d+)?", args.input): record["arxiv_id"] = args.input vault = args.vault or runtime_config().get("obsidian_vault", "") try: if not vault or not Path(vault).expanduser().is_dir(): raise ArchiveError("invalid_vault", [vault]) source = find_source( Path(vault).expanduser().resolve(), record, args.reference or args.input, args.target_directory, args.source_sha256, ) emit({"status": "ok" if source else "not_found", **(source or {})}, args.output) except ArchiveError as exc: emit({"status": "blocked", "conflict_code": exc.code, "candidates": exc.paths}, args.output) raise SystemExit(2) from exc if __name__ == "__main__": main()skills/deeppapernote/scripts/glossary_terms.py9.5 KB53e154898cb4upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/lint_grounding.py29 KB60a3b367f3f1upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/lint_note.py76 KBe1547cf21cdaupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/localization.py5.3 KB07d4d0b73ee9upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/locate_zotero_attachment.py3.6 KB5dd240340ac5upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/paper_archive.py21 KB5e6745c26644upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/pdf_captions.py15 KB048f9b866034upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/plan_figure_table_decisions.py34 KB42525b473ed6upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/plan_figures.py21 KB2f901df14916upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/research_memory.py3.3 KBf2965e6035e2upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/resolve_paper.py7.1 KB9b0fb12b9263upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/run_pipeline.py14 KB285d5abc3dc4upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/run_state.py52 KBfc08b85ad7e6upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/source_corpus.py17 KB63c6df8a543aupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/type_profiles.py44 KB3f44489ee4c6upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/user_configuration.py16 KB8281cfafb10cupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/vault_connections.py11 KBf48c1a817bc3upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/scripts/write_obsidian_note.py50 KBf7ce30b15793upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/architecture.md2.8 KB330fb58300b8upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/deep-analysis.md9.3 KB80041690dd7aupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/domain-routing.md3.2 KBb661e5016619upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/evidence-first.md12 KB98c9192ea82eupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/figure-placement.md11 KB66f15a29709eupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/final-quality-review.md9.6 KB28404fd1bf09upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/final-writing.md19 KB813c8e208411upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/metadata-sources.md1.4 KBb6ba84aa9b4cupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/note-quality.md5.0 KBbfafe50c57e0upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/obsidian-format.md13 KBca5fa88dd0b6upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/output-language.md4.9 KB7a6457d51534upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/paper-archive.md4.8 KB5da103a9b30cupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/paper-types.md6.0 KB8c8d611eb6f1upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/run-state.md7.5 KBb3740c659f21upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/references/user-configuration.md5.3 KB3ada4636da2fupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/deeppapernote/agents/openai.yaml283 B519b6b0f88c5upstream ↗
interface: display_name: "DeepPaperNote" short_description: "Generate a high-quality deep-reading note for one paper with a raw-source manifest workflow and Obsidian-oriented save semantics." default_prompt: "Use $deeppapernote to create the deep-reading note for this paper."README.mdupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
Plain-text copy, no login: /skill-bundles/917dhj-deeppapernote-skills-deeppapernote.md
