v3.0.1

datasheets

aklofas@aklofas⭐ 1.7k stars· last commit 9d ago· 6 open issues

Extract structured specifications from electronic component datasheet PDFs — pinouts, electrical characteristics, peripherals, topology, and features. Cache extractions per project for consumption by schematic and PCB analyzers. Primary consumer infrastructure for `kicad`, `emc`, `spice`, and `thermal` analyzers. Use this skill whenever the user asks to extract, verify, or read specs from a component datasheet; when analyzers need verified IC knowledge (EN pin thresholds, PG presence, USB periph

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

7.9/10
Verified
Oct 12, 2026

// RATINGS

⭐GitHub Stars
⭐⭐⭐⭐⭐ 1.7k on GitHubGitHub ↗

Very popular

🟢ProSkills ScoreAI Verified
7.9/10
📍

Not yet listed on ClawHub or SkillsMP

// README

⚡ kicad-happy

CI Python 3.10+ License: MIT Mentioned in Awesome KiCad

AI-powered design review for KiCad. Analyzes schematics, PCB layouts, and Gerbers. Catches real bugs before you order boards.

Works with Claude Code, OpenAI Codex, GitHub Copilot CLI, Google Antigravity, and opencode, as a GitHub Action for automated PR reviews, or as standalone Python scripts you can run anywhere.

These skills turn your AI coding agent into a full-fledged electronics design assistant that understands your KiCad projects at a deep level: parses schematics and PCB layouts into structured data, cross-references component values against datasheets, detects common design errors, and walks you through the full prototype-to-production workflow.

🔬 What it looks like in practice

Point your agent at a KiCad project and it does the rest — parses your schematic and PCB files, traces the nets, computes rail voltages from the feedback dividers it finds, and tells you what's wrong before you spend money on boards.

"Analyze my KiCad project at hardware/rev2/"

Here's a condensed example from an open-source robot controller board. The agent found all of this automatically:

It builds your power tree — tracing each regulator it recognises from input to load, computing output voltages from feedback dividers:

VBUS (USB-C / battery input, fused)
├── AP63357 buck (500kHz switching) → 5V
│   └── Feedback: R8/R9 ratio=0.155 → Vout=3.87V
│       Power dissipation: ~0.15W (85% efficiency assumed)
└── RT9080-3.3 LDO → 3.3V
    └── Decoupling: 16 caps, 10.8µF total

It identifies subcircuits — not just passives, but the functional blocks and how they connect:

SubcircuitDetails
Motor drive9x P-MOSFET switches (DMG2305UX), transistor-driven H-bridges
FiltersRC signal conditioning at 16Hz, 169Hz, and 1.03kHz (input filtering and debounce)
LightingWS2812B addressable LED chain on GPIO, 60mA estimated draw
SensorsOnboard sensor interface, crystal oscillator with load cap validation
ProtectionESD clamp on USB D+/D-, dual input fuses (0.75A signal, 2.5A motor)

It audits connectors for ESD protection — and flags the ones that are exposed:

ESD coverage: 19 connectors audited

  USB-C:     ESD clamp on D+/D-  ✓ (partial — 13 signal pins per ground ⚠️)
  Fuse F1:   2.5A motor input  ✓
  Fuse F2:   0.75A signal input  ✓
  ⚠️ 6-pin header:    no protection (exposed signals)
  ⚠️ Motor outputs:   no protection (exposed to back-EMF)
  ⚠️ Servo connectors: no protection (exposed signals)
  ⚠️ Sensor port:     no protection
  ... 19 of 19 connectors have coverage gaps

It validates your passive networks — computing actual circuit behavior from component values:

DetectionComponentsComputed ValueWhat It Means
RC filterR21/C31fc = 15.9 HzLow-pass for slow analog signal
RC filterR1/C13fc = 169 HzDebounce / noise rejection
RC filterR2/C14fc = 1.03 kHzSignal conditioning
FeedbackR8/R9ratio = 0.155Buck converter output voltage set
DividerR42/R43ratio = 0.500Voltage sensing (half)
CrystalY1CL = 14.0 pFLoad cap status: ok (target: 18 pF, -22%)

It suggests applicable certifications — based on what it detects in the design:

Suggested certifications:
  FCC Part 15 Subpart B (US) — unintentional radiator compliance
  CISPR 32 / CE EMC Directive (EU) — EMC compliance for EU market

It checks production readiness — BOM lock status, connector ground distribution, decoupling adequacy:

BOM lock: 0% — no MPNs assigned (prototype stage)
Decoupling: 5 rails, 34 caps total (132µF motor, 110µF logic, 10.8µF 3.3V)
Connector ground: USB-C has 13:1 signal-to-ground ratio (recommended ≤3:1)

For complete examples with all sections, see:

For the end-to-end walkthrough from S-expression parsing through signal detection and datasheet cross-referencing, see How It Works.

🚀 Install

[!TIP] For detailed installation, upgrade, and troubleshooting guidance across all platforms, have your AI agent read install-guidance.md. It covers platform-specific quirks, known bugs, workarounds, and OS-specific issues.

Claude Code:

/plugin marketplace add aklofas/kicad-happy
/plugin install kicad-happy@kicad-happy

[!NOTE] /plugin update may not detect new versions due to a known Claude Code issue. To get the latest version, clear the cache and reinstall:

rm -rf ~/.claude/plugins/cache/kicad-happy ~/.claude/plugins/marketplaces/kicad-happy
/plugin marketplace add aklofas/kicad-happy
/plugin install kicad-happy@kicad-happy

OpenAI Codex:

Use Codex's built-in skill installer first:

"Use $skill-installer to install the kicad-happy skills from https://github.com/aklofas/kicad-happy"

If you prefer a manual install, install the skills into ~/.codex/skills/.

Google Antigravity CLI (agy) / Gemini:

Install directly from GitHub as an Antigravity plugin:

agy plugin install https://github.com/aklofas/kicad-happy.git

Or from a local checkout:

git clone https://github.com/aklofas/kicad-happy.git
agy plugin install kicad-happy

Toggle when needed:

agy plugin disable kicad-happy   # Disable when not doing electronics review
agy plugin enable kicad-happy    # Enable when working on KiCad projects

See install-guidance.md for workspace-scope installs, slash commands, and upgrade notes.

opencode:

git clone https://github.com/aklofas/kicad-happy.git
cd kicad-happy
opencode

The repo ships .opencode/opencode.json, which opencode auto-discovers and uses to load all 11 skills from ./skills/. For global availability across all projects, see install-guidance.md.

Claude Code (macOS / Linux):

git clone https://github.com/aklofas/kicad-happy.git
cd kicad-happy
mkdir -p ~/.claude/skills
for skill in kicad spice emc datasheets bom digikey mouser lcsc element14 jlcpcb pcbway; do
  ln -sf "$(pwd)/skills/$skill" ~/.claude/skills/$skill
done

OpenAI Codex — global install (macOS / Linux):

git clone https://github.com/aklofas/kicad-happy.git
cd kicad-happy
mkdir -p ~/.codex/skills
for skill in kicad spice emc datasheets bom digikey mouser lcsc element14 jlcpcb pcbway; do
  ln -sf "$(pwd)/skills/$skill" ~/.codex/skills/$skill
done

Windows PowerShell (Codex):

git clone https://github.

// HOW IT'S BUILT

// FILES

66 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • skills/datasheets/SKILL.md11 KB898421396469upstream ↗
    ---
    name: datasheets
    description: Extract structured specifications from electronic component datasheet PDFs — pinouts, electrical characteristics, peripherals, topology, and features. Cache extractions per project for consumption by schematic and PCB analyzers. Primary consumer infrastructure for `kicad`, `emc`, `spice`, and `thermal` analyzers. Use this skill whenever the user asks to extract, verify, or read specs from a component datasheet; when analyzers need verified IC knowledge (EN pin thresholds, PG presence, USB peripheral speed); or when a review mentions datasheet coverage, extraction quality, or per-MPN specifications. Also triggers on "extract this datasheet", "what are the specs for MPN X", "verify datasheet extraction", or "check pin functions for part Y".
    ---
    
    # Datasheets Skill
    
    ## Related Skills
    
    | Skill | Relationship |
    |-------|--------------|
    | `digikey` / `mouser` / `lcsc` / `element14` | **Producers** — download the PDFs under `<project>/datasheets/` that this skill extracts from |
    | `kicad` | **Primary consumer** — VM-001/PU-001/FS-001/PP-001/LR-001/XT-001 + the datasheet-backed lookup detectors (AM-001/OV-001/TJ-001/FT-001/EX-001/PM-001) query extractions via `lookup(mpn)` for verified-IC knowledge |
    | `emc` | **Consumer** — switching-frequency, package-Rθ_JA, and operating-voltage data sharpen EMC heuristics |
    | `spice` | **Consumer (limited)** — the model resolver reads legacy `spice_specs` fields only; v1.4 typed facts are not used |
    | `thermal` | **Consumer** — package Rθ_JA + junction temperature limits drive Tj estimates (TS-001..TJ-001) |
    | `bom` | **Indirect** — coverage of structured extractions affects BOM verification confidence |
    
    **Handoff guidance:** This skill is consumer infrastructure. The typical flow is `distributor skill downloads PDF → datasheets skill extracts → analyzer skill queries`. Use this skill directly when (a) the user asks to extract or verify a specific MPN, (b) an analyzer reports `trust_level: low` and the gap is per-MPN extraction quality, or (c) a new MPN was added to the BOM and downstream detectors should pick up its verified specs. Don't run this skill in isolation if the user just wants a design review — call it from the kicad workflow at the "Sync datasheets" step instead.
    
    ## Purpose
    
    Extract structured, machine-readable specifications from component datasheet PDFs and make them available to analyzer skills. Works on whatever PDFs are downloaded under `<project>/datasheets/` (downloads are owned by distributor skills like `digikey`, `mouser`, `lcsc`, `element14`).
    
    ## Scope
    
    This skill owns:
    - **Extraction schemas** — canonical JSON structures for per-MPN specs. v1.4 ships 6 JSON Schema Draft 2020-12 schemas under `schemas/` (`base`, `pinout`, `spec_value`, `regulator`, `extraction`, `manifest`) plus 5 v1.4 category extensions (diode, transistor, opamp, mcu, crystal). v1.3 cache format (`EXTRACTION_VERSION` in `scripts/datasheet_extract_cache.py`) is still read for compat.
    - **Typed access layer (v1.4)** — `datasheet_types/` package exposes `DatasheetFacts`, `SpecValue`, `Pin`, `Pinout`, `lookup()`, `best()`, `trusted()`, `has_data()`. Recommended for all new consumers.
    - **PDF page selection** — heuristics to pick pages most likely to contain pinouts, e-chars, applications, SPICE models.
    - **Quality scoring** — v1.4 scores 0–100 on pinout completeness (20 %), base completeness (25 %) and category-extension completeness (25 %), plus a reserved 30 % awarded in full. v1.3 5-dimension weighted rubric still applies to legacy caches.
    - **Consumer APIs** — `datasheet_types.cache_reader` (`from datasheet_types import lookup`) for v1.4 typed access; `scripts/datasheet_features.py` for the v1.3 dict-shaped helpers (`get_regulator_features`, `get_mcu_features`, `get_pin_function`) — the v1.3 helpers dual-read v1.4 caches and translate to v1.3 dict shape for legacy detector code.
    - **Verification** — `datasheet_verify.py` (v1.3, schema-vs-usage cross-check) plus `verify_v14_extraction` (v1.4, power_domain references resolve, recommended ≤ absolute, regulator pin references exist).
    
    ## Non-goals
    
    - **No PDF downloading.** That is owned by distributor skills (`digikey`, `mouser`, `lcsc`, `element14`).
    - **No global library.** Each project's extractions live in `<project>/datasheets/extracted/`. There is no shared cross-project cache.
    
    ## Cache location
    
    ```
    <project>/
      design.kicad_sch
      datasheets/
        TPS61023DRLR.pdf        # downloaded by distributor skills
        extracted/
          manifest.json         # extraction manifest (legacy name: index.json)
          TPS61023DRLR.json     # structured extraction (this skill's output)
    ```
    
    ## Reference guides
    
    - `references/extraction-schema.md` — legacy v1.3 cache fields; the current v1.4 contracts are `schemas/extraction.schema.json`, `schemas/base.schema.json` and the category schemas, with extraction instructions under `prompts/`
    - `references/field-extraction-guide.md` — how to find each field in datasheets from common vendors (TI, ST, NXP, Espressif, Microchip)
    - `references/quality-scoring.md` — legacy v1.3 0–10 rubric; the current 0–100 score is summarised at its top
    - `references/consumer-api.md` — how kicad/emc/spice/thermal consume extractions
    - `references/cache-layout.md` — v1.4 cache directory convention (per-MPN files, `_families/` reservation, staleness rules)
    
    ## Entry-point scripts
    
    - `scripts/datasheet_extract_cache.py` — v1.3 cache manager, resolver, indexer
    - `scripts/datasheet_page_selector.py` — page selection heuristics (manual helper; no pipeline script imports it)
    - `scripts/datasheet_score.py` — v1.3 extraction quality scoring
    - `scripts/datasheet_verify.py` — cross-check extraction vs schematic usage (v1.3 + v1.4 `verify_v14_extraction` mode)
    - `scripts/datasheet_features.py` — v1.3 consumer helper API (dual-reads v1.4 caches via `_derive_*_v14` translators)
    - `scripts/plan_extraction.py` — **v1.4** orchestration plan generator (extraction pipeline)
    - `scripts/merge_results.py` — **v1.4** per-task result validator + merger
    - `datasheet_types/` — **v1.4** typed access layer package (`DatasheetFacts`, `SpecValue`, `Pin`, `Pinout`, `lookup`, `best`, `trusted`, `has_data`)
    - `distributor_core/` — shared sync/fetch plumbing for the four distributor skills (BOM projection, `manifest.json`, download, verification, the sync driver); not an entry point of this skill
    
    ## Extraction workflow
    
    Run the scout first: dispatch `prompts/scout.md` for the MPN and PDF and save its schema-valid, unwrapped output, with the PDF's `"pdf_sha256": "sha256:<hex>"` added, as `<cache_dir>/<sanitized_mpn>.scout.json` (the MPN with every character outside `[A-Za-z0-9_.-]` replaced by `_`: `LM317/NOPB` → `LM317_NOPB`) (`references/dispatch-claude-code.md` Step 1). Then run `python3 <skill-path>/scripts/plan_extraction.py <mpn> <pdf_path> --use-cached-scout --cache-dir <cache_dir>` (`<skill-path>` is this skill's directory) to write the plan, dispatch its tasks, and run `merge_results.py <mpn> --cache-dir <cache_dir>` to validate and merge their results. Without `--use-cached-scout` it exits 2 (it cannot run the scout itself); with the flag and no saved scout it stops with a FileNotFoundError. `merge_results.py` and `validate_extraction_result.py` need the `jsonschema` package (`pip install jsonschema`); everything else here is stdlib-only. Full scout→plan→dispatch→merge procedure: [`references/extraction-pipeline.md`](references/extraction-pipeline.md).
    
    ## Consuming extractions (v1.4 typed API)
    
    The recommended consumer surface is the typed `lookup(mpn, cache_dir=...)` facade plus the trust-gating helpers from `datasheet_types`. Import like:
    
    ```python
    import sys, pathlib
    sys.path.insert(0, "<datasheets-skill-path>")
    from datasheet_types import lookup, has_data, best, trusted
    
    # Returns Optional[DatasheetFacts]. None on cache miss or unreadable cache; quality is reported on the object, not gated.
    facts = lookup("TPS61023DRLR", cache_dir=pathlib.Path("datasheets/extracted"))
    if facts is None:
        return  # heuristic-only path; no datasheet evidence available
    
    # Field-level trust gating — every SpecValue list runs through has_data() / best() / trusted().
    vin = facts.base.recommended_operating.get("VIN")  # Optional[list[SpecValue]]; recommended_operating is dict[str, list[SpecValue]]
    if has_data(vin):
        # Most-trusted single value (first SpecValue meeting threshold, preserves extractor order).
        rec = best(vin, min_confidence="medium")  # Optional[SpecValue]
        if rec is not None and rec.min is not None:
            ...  # use rec.min, rec.max, rec.typ, rec.unit, rec.evidence.{page,section,confidence}
    
    # All SpecValues at threshold (for multi-value fields like absolute_max).
    hi_conf = trusted(facts.base.absolute_max.get("VDD", []), min_confidence="high")
    ```
    
    **Defensive patterns** (mirrors `kicad/SKILL.md` § "Probing Analyzer JSON"):
    
    - `lookup()` returns `None` on a cache miss or an unreadable cache (malformed, non-UTF-8, symlinked, not a regular file, oversized, not an object); a stale PDF or a low quality score is described on the facts (`facts.stale`, `facts.quality`), never a `None`. Always guard with `if facts is None: return`.
    - Category extensions are optional on `DatasheetFacts`. `facts.regulator` is `None` when the part isn't in the `regulator` category — check before dereferencing.
    - SpecValue lists can be `None` (field not extracted), `[]` (extracted but empty), or `list[SpecValue]`. `has_data()` collapses the first two to `False`; pair with `best()` / `trusted()` for confidence gating.
    - `SpecValue.min` / `.max` / `.typ` are each `Optional[float]`. A SpecValue carrying only `typ` (no range) makes `>` / `<` comparisons against `.min` / `.max` raise `TypeError` — guard with explicit `is not None` chains on every numeric access.
    - `confidence` is one of `"low"` / `"medium"` / `"high"`. Calling `best()` / `trusted()` with any other string raises `ValueError`.
    
    ## v1.3 compat shim
    
    Legacy detectors still call `get_regulator_features(mpn)` / `get_mcu_features(mpn)` / `get_pin_function(mpn, pin)` from `scripts/datasheet_features.py`. These dual-read v1.4 caches and translate to the v1.3 dict shape. New code should use `lookup()` directly.
    
    ## When to trigger this skill
    
    - **Immediately after downloading datasheets** via `sync_datasheets_digikey.py`, `sync_datasheets_lcsc.py`, or equivalent. Without extraction, IC-aware checks (VM-001 rail voltage, PS-001 power-good, PR-004 USB, DP-002 USB speed classification) fall back to heuristics on unknown ICs.
    - **Before running analyzers on a new project** where datasheets are present but `datasheets/extracted/` is empty — the analyzers won't produce the extractions themselves.
    - **When a review flags low trust level** due to missing manufacturer evidence: extracting the ICs referenced by power regulators, MCUs, and high-speed peripherals typically flips `trust_level: low` → `mixed` or `high`.
    - **When a user asks for pin verification** ("verify U1 pin names match datasheet") — this skill's cached extraction is the authoritative source.
    
  • skills/datasheets/scripts/_smoke_v14_roundtrip.py8.9 KB921095a8a8ccupstream ↗
    #!/usr/bin/env python3
    """v1.4/v2 cache round-trip smoke test.
    
    Builds a minimal synthetic analysis dict referencing an IC with MPN
    LM2596-ADJ, plants the canonical v2-format example extraction at
    <tmpdir>/datasheets/extracted/LM2596-ADJ.json, and calls
    run_datasheet_verification().
    
    Expected behavior:
    - No crash.
    - Returns a dict with findings[] and summary.
    - ics_with_extractions=1 (LM2596-ADJ passes the v1.4 trust gate).
    - Pin-voltage checks run via the v2→v1 adapter (domain-level limits).
      With VIN=12V (well within op_max=40V / abs_max=45V), no voltage
      violation findings are produced; on a 50V rail, VIN (pin 1) gets one
      abs-max error (45V).
    - One extraction_not_verifiable INFO finding is emitted because
      required-external and decoupling checks have no v2 equivalent data.
    
    Exit code 0 = pass, 1 = fail (any exception or wrong return shape).
    """
    
    import shutil
    import sys
    import tempfile
    from pathlib import Path
    
    DS_SCRIPTS = Path(__file__).resolve().parent
    EXAMPLE_FIXTURE = Path(__file__).resolve().parents[1] / "examples" / "lm2596-adj.json"
    
    sys.path.insert(0, str(DS_SCRIPTS))
    
    
    def build_synthetic_analysis() -> dict:
        """Minimal analysis dict referencing an IC with MPN LM2596-ADJ."""
        return {
            "file": "/tmp/fake.kicad_sch",
            "components": [
                {
                    "reference": "U1",
                    "type": "ic",
                    "value": "LM2596-ADJ",
                    "mpn": "LM2596-ADJ",
                    "pin_nets": {"1": "VIN", "2": "OUT", "3": "GND", "4": "FB", "5": "ON_OFF"},
                },
                {"reference": "C1", "type": "capacitor", "value": "100nF", "parsed_value": 100e-9},
                {"reference": "C2", "type": "capacitor", "value": "10uF", "parsed_value": 10e-6},
            ],
            "nets": {
                "VIN": {"pins": [{"component": "U1"}, {"component": "C1"}, {"component": "C2"}]},
                "OUT": {"pins": [{"component": "U1"}]},
                "GND": {"pins": [{"component": "U1"}, {"component": "C1"}, {"component": "C2"}]},
                "FB": {"pins": [{"component": "U1"}]},
                "ON_OFF": {"pins": [{"component": "U1"}]},
            },
            "rail_voltages": {"VIN": 12.0, "OUT": 5.0},
        }
    
    
    def main() -> int:
        if not EXAMPLE_FIXTURE.exists():
            print(f"FAIL: fixture missing: {EXAMPLE_FIXTURE}", file=sys.stderr)
            return 1
    
        with tempfile.TemporaryDirectory() as tmp:
            tmp_path = Path(tmp)
            cache_dir = tmp_path / "datasheets" / "extracted"
            cache_dir.mkdir(parents=True)
            shutil.copy(EXAMPLE_FIXTURE, cache_dir / "LM2596-ADJ.json")
    
            from datasheet_verify import run_datasheet_verification
    
            analysis = build_synthetic_analysis()
            try:
                result = run_datasheet_verification(analysis, project_dir=str(tmp_path))
            except AttributeError as exc:
                print(f"FAIL: AttributeError on v1.4 cache (rc.2 regression): {exc}", file=sys.stderr)
                return 1
            except Exception as exc:
                print(f"FAIL: unexpected exception: {type(exc).__name__}: {exc}", file=sys.stderr)
                return 1
    
            if not isinstance(result, dict):
                print(f"FAIL: result is not a dict: {type(result).__name__}", file=sys.stderr)
                return 1
            if "findings" not in result or "summary" not in result:
                print(f"FAIL: result missing required keys: {sorted(result.keys())}", file=sys.stderr)
                return 1
            if not isinstance(result["findings"], list):
                print(f"FAIL: findings is not a list: {type(result['findings']).__name__}", file=sys.stderr)
                return 1
    
            ics_with_ext = result["summary"].get("ics_with_extractions", 0)
            if ics_with_ext != 1:
                print(
                    f"FAIL: expected ics_with_extractions=1 (LM2596-ADJ should pass v1.4 trust gate), "
                    f"got {ics_with_ext}",
                    file=sys.stderr,
                )
                return 1
    
            # The v2 adapter must emit exactly one extraction_not_verifiable
            # finding (required-external + decoupling unverifiable for v2 format),
            # and must NOT emit any voltage violation (VIN=12V is within limits).
            nv_findings = [f for f in result["findings"]
                           if f.get("type") == "extraction_not_verifiable"]
            if len(nv_findings) != 1:
                print(
                    f"FAIL: expected exactly 1 extraction_not_verifiable finding "
                    f"(v2 format: required-external + decoupling not verifiable), "
                    f"got {len(nv_findings)}: {nv_findings}",
                    file=sys.stderr,
                )
                return 1
            voltage_violations = [f for f in result["findings"]
                                  if f.get("type") in ("pin_voltage_abs_max_exceeded",
                                                       "pin_voltage_operating_exceeded")]
            if voltage_violations:
                print(
                    f"FAIL: unexpected voltage violation findings for 12V input "
                    f"(abs_max=45V, op_max=40V): {voltage_violations}",
                    file=sys.stderr,
                )
                return 1
    
            # The same part on a 50 V rail: pin 1 (VIN) exceeds its 45 V
            # absolute maximum, so the adapter's limits reach a finding.
            over = build_synthetic_analysis()
            over["rail_voltages"]["VIN"] = 50.0
            over_findings = run_datasheet_verification(over, project_dir=str(tmp_path))["findings"]
            abs_max = [(f.get("pin_number"), f.get("abs_max_V"), f.get("severity"))
                       for f in over_findings if f.get("type") == "pin_voltage_abs_max_exceeded"]
            if abs_max != [("1", 45, "error")]:
                print(
                    f"FAIL: expected one abs-max error on pin 1 (50V > 45V), got {abs_max}",
                    file=sys.stderr,
                )
                return 1
    
            print(
                f"PASS: v2 round-trip OK. "
                f"findings={len(result['findings'])} "
                f"(1 extraction_not_verifiable INFO, 0 voltage violations) "
                f"ics_checked={result['summary'].get('ics_checked')} "
                f"ics_with_extractions={ics_with_ext}"
            )
            return 0
    
    
    def test_placement_sniff() -> int:
        """Test that verify_v14_extraction flags absolute_max entries citing
        non-stress-rating sections (the USBLC6 failure mode).
        """
        from datasheet_verify import verify_v14_extraction
    
        # Synthetic v1.4 extraction with the USBLC6-style misplacement:
        # entries citing "Table 2. Electrical characteristics" placed in absolute_max.
        extraction = {
            "schema_version": "1.0",
            "source": {"local_path": "fake.pdf", "sha256": "deadbeef"},
            "extraction": {"quality_score": 77},
            "base": {
                "pinout": [],
                "recommended_operating": {},
                "absolute_max": {
                    # CORRECT: cites stress-rating section
                    "TJ_max": [{
                        "min": None, "max": 150, "typ": None, "unit": "°C",
                        "condition": "Junction temperature",
                        "evidence": {"section": "7.1 Absolute Maximum Ratings", "page": 5},
                    }],
                    # WRONG: cites Table 2 (electrical chars), should be flagged
                    "VBR": [{
                        "min": 6.0, "max": None, "typ": None, "unit": "V",
                        "condition": "Breakdown voltage",
                        "evidence": {"section": "Table 2. Electrical characteristics", "page": 4},
                    }],
                    "VF_max": [{
                        "min": None, "max": 1.1, "typ": None, "unit": "V",
                        "condition": "Forward voltage at IF=10mA",
                        "evidence": {"section": "Table 2. Electrical characteristics", "page": 4},
                    }],
                },
            },
            "categories": [],
        }
    
        issues = verify_v14_extraction(extraction)
        placement_issues = [
            i for i in issues if "absolute_max" in i.get("path", "") and "section" in i.get("path", "")
        ]
    
        if len(placement_issues) != 2:
            print(
                f"FAIL: expected 2 placement-sniff issues (VBR + VF_max), "
                f"got {len(placement_issues)}: {placement_issues}",
                file=sys.stderr,
            )
            return 1
    
        paths = {i["path"] for i in placement_issues}
        expected = {
            "base.absolute_max.VBR[0].evidence.section",
            "base.absolute_max.VF_max[0].evidence.section",
        }
        if paths != expected:
            print(f"FAIL: unexpected paths. got {paths}, expected {expected}", file=sys.stderr)
            return 1
    
        print("PASS: placement-sniff flags Table 2 entries, leaves Absolute Max section alone")
        return 0
    
    
    if __name__ == "__main__":
        for _stream in (sys.stdout, sys.stderr):  # UTF-8 on a cp1252 pipe (Windows)
            if (getattr(_stream, "encoding", "") or "").lower().replace("-", "") != "utf8" \
                    and hasattr(_stream, "reconfigure"):
                _stream.reconfigure(encoding="utf-8", errors="backslashreplace")
        rc1 = main()
        rc2 = test_placement_sniff()
        sys.exit(rc1 | rc2)
    
  • skills/datasheets/scripts/datasheet_extract_cache.py18 KB0f66779436c3upstream ↗
    """
    Cache manager for structured datasheet extractions.
    
    Stores pre-extracted datasheet specs (pin tables, voltage ratings, operating
    conditions, application circuits) as per-MPN JSON files in a project-local
    `datasheets/extracted/` directory. The extraction itself is performed by
    Claude reading selected PDF pages; this module provides the caching framework.
    
    Directory structure:
        <project>/
          design.kicad_sch
          datasheets/
            manifest.json        # PDF manifest (legacy name: index.json)
            TPS61023DRLR.pdf
            extracted/
              manifest.json      # extraction cache index (legacy name: index.json)
              TPS61023DRLR.json  # full structured extraction
              BSS138LT1G.json
    
    The extraction resolution during review:
        1. Check cache (datasheets/extracted/) — instant if fresh
        2. If stale or missing → the agent reads PDF pages and extracts
        3. Score extraction → cache if sufficient (score >= 6.0)
    
    Schema v2 (EXTRACTION_VERSION=2, 2026-04-15):
      pins[].function: canonical pin category, one of:
        'VIN' | 'VOUT' | 'EN' | 'PG' | 'SW' | 'FB' | 'GND' | 'IO' |
        'CLK' | 'RESET' | 'OTHER' | None
      features.has_pg: bool | None — part has a power-good output pin
      features.has_soft_start: bool | None — integrated soft-start
      features.iss_time_us: float | None — soft-start time (microseconds)
      peripherals.usb: {speed, native_phy, series_r_required} | None (MCUs only)
        speed: 'FS' | 'HS' | 'SS' | None
        native_phy: bool | None
        series_r_required: bool | None
      topology: 'boost' | 'buck' | 'ldo' | 'mcu' | 'sensor' | 'adc' |
                'mosfet' | 'bjt' | 'other' | None
    
    Per-pin V_IH/V_IL thresholds already live in pins[].threshold_high_v
    and pins[].threshold_low_v (v1 schema, no change needed).
    
    All v2 fields are nullable. None means 'datasheet didn't specify or
    extraction couldn't verify', distinct from explicit False.
    """
    
    import json
    import os
    import re
    import sys
    from datetime import datetime, timezone
    from pathlib import Path
    
    _TYPES_PARENT = str(Path(__file__).resolve().parent.parent)
    if _TYPES_PARENT not in sys.path:
        sys.path.insert(0, _TYPES_PARENT)
    
    from datasheet_types.cache_reader import (  # noqa: E402
        _compute_pdf_sha256, read_json_object, safe_extract_dir)
    from distributor_core.manifest import manifest_file_path, write_json_atomic  # noqa: E402
    
    _KICAD_SCRIPTS = str(Path(__file__).resolve().parents[2] / "kicad" / "scripts")
    if _KICAD_SCRIPTS not in sys.path:
        sys.path.append(_KICAD_SCRIPTS)
    
    from kicad_core.project import extraction_project_dir  # noqa: E402
    
    # Bump this when the extraction schema changes to trigger re-extraction
    # v2 (2026-04-15): added peripherals.usb, electrical.en_v_ih_max/en_v_il_min,
    # electrical.iss_time_us, features.has_pg, features.has_soft_start,
    # pins[].function (functional category) and top-level topology for
    # structured consumer access via datasheet_features.py.
    EXTRACTION_VERSION = 2
    
    # Extractions older than this are considered stale (days)
    DEFAULT_MAX_AGE_DAYS = 90
    
    # Minimum acceptable score; below this, retry is attempted
    MIN_SCORE = 6.0
    
    # Maximum retry attempts for low-scoring extractions
    MAX_RETRIES = 3
    
    
    # ---------------------------------------------------------------------------
    # Directory resolution
    # ---------------------------------------------------------------------------
    
    def resolve_extract_dir(analysis_json=None, project_dir=None, override_dir=None):
        """Determine the datasheets/extracted/ directory for this project.
    
        Derives the project directory from either an explicit path or the
        analyzer JSON (its "file", else its first ``inputs.source_files`` entry;
        a relative entry only when that file exists from the cwd).
    
        Args:
            analysis_json: Parsed analyzer JSON dict
            project_dir: Explicit project directory path
            override_dir: If set, use this directory directly
    
        Returns:
            Path to datasheets/extracted/ directory (may not exist yet), or None
            when no project is named or the project's datasheets/ or extracted/
            is a symlink. There is no shared fallback directory: a
            temp-dir cache could hold extractions planted by another user.
        """
        if override_dir:
            return Path(override_dir)
    
        if project_dir:
            return safe_extract_dir(Path(project_dir) / "datasheets" / "extracted")
    
        found = extraction_project_dir(analysis_json)
        if found is not None:
            return safe_extract_dir(found / "datasheets" / "extracted")
        return None
    
    
    def resolve_datasheets_dir(extract_dir):
        """Get the parent datasheets/ directory from an extracted/ path.
    
        Args:
            extract_dir: Path to datasheets/extracted/
    
        Returns:
            Path to datasheets/ directory
        """
        p = Path(extract_dir)
        if p.name == "extracted":
            return p.parent
        return p.parent  # best guess
    
    
    # ---------------------------------------------------------------------------
    # PDF hashing for cache invalidation
    # ---------------------------------------------------------------------------
    
    def _pdf_hash(pdf_path):
        """Compute SHA-256 hash of a PDF file.
    
        Args:
            pdf_path: Path to the PDF file
    
        Returns:
            "sha256:<hex>" string, or None if it is not a readable regular file
        """
        return _compute_pdf_sha256(Path(pdf_path))
    
    
    # ---------------------------------------------------------------------------
    # MPN normalization (matches spice_model_generator.sanitize_mpn)
    # ---------------------------------------------------------------------------
    
    def _sanitize_mpn(mpn):
        """Convert an MPN to a safe filename component.
    
        Appends a short hash to avoid collisions when different MPNs
        sanitize to the same string (e.g., STM32F-103 vs STM32F/103).
        """
        import hashlib
        clean = re.sub(r'[^A-Za-z0-9_]', '_', mpn.strip())
        h = hashlib.md5(mpn.strip().encode()).hexdigest()[:6]
        return f"{clean}_{h}"
    
    
    # ---------------------------------------------------------------------------
    # In-memory cache for manifest.json (avoids re-reading on every call)
    # ---------------------------------------------------------------------------
    
    MANIFEST_FILENAME = "manifest.json"
    LEGACY_MANIFEST_FILENAME = "index.json"
    
    _index_cache = {}  # extract_dir_str → (mtime, parsed_index)
    
    
    def _manifest_path_for(extract_dir):
        """Return the manifest path, preferring new manifest.json over legacy index.json."""
        d = Path(extract_dir)
        new = d / MANIFEST_FILENAME
        old = d / LEGACY_MANIFEST_FILENAME
        if new.exists() or not old.exists():
            return new
        return old
    
    
    def _load_index(extract_dir):
        """Load the extraction manifest with mtime-based in-memory caching.
    
        Reads manifest.json preferentially; falls back to legacy index.json on
        projects that haven't been migrated yet.
    
        Args:
            extract_dir: Path to datasheets/extracted/
    
        Returns:
            Parsed manifest dict, or empty structure if not found
        """
        index_path = _manifest_path_for(extract_dir)
        cache_key = str(extract_dir)
    
        try:
            if not index_path.exists():
                return {"version": EXTRACTION_VERSION, "last_updated": "", "extractions": {}}
    
            mtime = index_path.stat().st_mtime
            cached = _index_cache.get(cache_key)
            if cached and cached[0] == mtime:
                return cached[1]
    
            index = read_json_object(index_path)
            if index is None:
                raise ValueError("unreadable manifest")
            _index_cache[cache_key] = (mtime, index)
            return index
        except (ValueError, OSError):
            return {"version": EXTRACTION_VERSION, "last_updated": "", "extractions": {}}
    
    
    def _manifest_unreadable(path):
        """True when a manifest exists but cannot be loaded; it must not be overwritten."""
        path = Path(path)
        return (path.exists() or path.is_symlink()) and read_json_object(path) is None
    
    
    def _warn_manifest_kept(path):
        print(f"Warning: {path} exists but cannot be read; leaving it unchanged", file=sys.stderr)
    
    
    def _save_index(extract_dir, index):
        """Write the extraction manifest atomically. Always writes manifest.json;
        removes any legacy index.json after the new file is in place.
    
        Args:
            extract_dir: Path to datasheets/extracted/
            index: Manifest dict to save
        """
        extract_dir = Path(extract_dir)
        extract_dir.mkdir(parents=True, exist_ok=True)
    
        index["last_updated"] = datetime.now().astimezone().isoformat(timespec='seconds')
        index["version"] = EXTRACTION_VERSION
    
        new_path = extract_dir / MANIFEST_FILENAME
        write_json_atomic(new_path, index)
    
        old_path = extract_dir / LEGACY_MANIFEST_FILENAME
        if old_path.exists() and old_path != new_path:
            try:
                old_path.unlink()
            except OSError:
                pass
    
        # Invalidate in-memory cache
        _index_cache.pop(str(extract_dir), None)
    
    
    # ---------------------------------------------------------------------------
    # Cache read / write / invalidation
    # ---------------------------------------------------------------------------
    
    def get_cached_extraction(extract_dir, mpn):
        """Retrieve a cached extraction by MPN.
    
        Args:
            extract_dir: Path to datasheets/extracted/
            mpn: Manufacturer part number
    
        Returns:
            Parsed extraction dict, or None if not cached (or extract_dir is None)
        """
        if extract_dir is None:
            return None
        extractions = _load_index(extract_dir).get("extractions")
        if not isinstance(extractions, dict):
            return None
        key = _sanitize_mpn(mpn)
    
        entry = extractions.get(key)
        if not entry:
            # Try case-insensitive match
            for k, v in extractions.items():
                if k.upper() == key.upper():
                    entry = v
                    break
        if not entry:
            return None
    
        json_file = manifest_file_path(Path(extract_dir), entry.get("file") if isinstance(entry, dict) else None)
        return read_json_object(json_file) if json_file else None
    
    
    def cache_extraction(extract_dir, mpn, extraction, source_pdf=None):
        """Save an extraction to the cache.
    
        Writes the extraction JSON file and updates the index. If source_pdf
        is provided, computes and stores its hash for invalidation.
    
        Args:
            extract_dir: Path to datasheets/extracted/
            mpn: Manufacturer part number
            extraction: Complete extraction dict (with extraction_metadata)
            source_pdf: Optional path to the source PDF file
        """
        extract_dir = Path(extract_dir)
        extract_dir.mkdir(parents=True, exist_ok=True)
    
        key = _sanitize_mpn(mpn)
        filename = f"{key}.json"
        json_path = extract_dir / filename
    
        # Compute PDF hash if source provided
        if source_pdf:
            pdf_hash = _pdf_hash(source_pdf)
            if pdf_hash:
                extraction.setdefault("extraction_metadata", {})["source_pdf_hash"] = pdf_hash
    
        # Ensure metadata has required fields
        meta = extraction.setdefault("extraction_metadata", {})
        meta.setdefault("extraction_date", datetime.now(timezone.utc).isoformat())
        meta.setdefault("extraction_version", EXTRACTION_VERSION)
        meta.setdefault("retry_count", 0)
    
        # Write extraction file
        write_json_atomic(json_path, extraction)
    
        # Update index
        manifest = _manifest_path_for(extract_dir)
        if _manifest_unreadable(manifest):
            _warn_manifest_kept(manifest)
            return
        index = _load_index(extract_dir)
        index.setdefault("extractions", {})[key] = {
            "file": filename,
            "mpn": mpn,
            "category": extraction.get("category", ""),
            "source_pdf": meta.get("source_pdf", ""),
            "source_pdf_hash": meta.get("source_pdf_hash", ""),
            "extraction_date": meta["extraction_date"],
            "extraction_score": meta.get("extraction_score", 0),
            "extraction_version": meta["extraction_version"],
            "pin_count": len(extraction.get("pins", [])),
        }
        _save_index(extract_dir, index)
    
    
    def is_extraction_stale(extract_dir, mpn, datasheets_dir=None):
        """Check if an extraction needs to be refreshed.
    
        Stale conditions (any one triggers):
        1. Source PDF has changed (sha256 hash mismatch)
        2. Extraction version < current code version (schema upgrade)
        3. Score < MIN_SCORE and retry_count < MAX_RETRIES (worth retrying)
        4. Extraction older than DEFAULT_MAX_AGE_DAYS
    
        Args:
            extract_dir: Path to datasheets/extracted/
            mpn: Manufacturer part number
            datasheets_dir: Path to datasheets/ (for PDF hash comparison)
    
        Returns:
            (is_stale: bool, reason: str) — reason is "" if not stale
        """
        extraction = get_cached_extraction(extract_dir, mpn)
        if extraction is None:
            return True, "not_cached"
    
        meta = extraction.get("extraction_metadata", {})
        if not isinstance(meta, dict) or any(
                k in meta and (not isinstance(meta[k], (int, float)) or isinstance(meta[k], bool))
                for k in ("extraction_version", "extraction_score", "retry_count")):
            return True, "malformed extraction_metadata"
    
        # Check version
        version = meta.get("extraction_version", 0)
        if version < EXTRACTION_VERSION:
            return True, f"schema_upgrade (v{version} < v{EXTRACTION_VERSION})"
    
        # Check PDF hash
        if datasheets_dir:
            source_pdf_name = meta.get("source_pdf", "")
            if source_pdf_name and isinstance(source_pdf_name, str):
                root = Path(datasheets_dir)
                pdf_path = root / source_pdf_name
                try:
                    inside = pdf_path.resolve().is_relative_to(root.resolve())
                except (OSError, RuntimeError, ValueError):
                    inside = False
                if inside:
                    current_hash = _pdf_hash(pdf_path)
                    stored_hash = meta.get("source_pdf_hash", "")
                    if current_hash and stored_hash and current_hash != stored_hash:
                        return True, "pdf_changed"
    
        # Check score + retry budget
        score = meta.get("extraction_score", 0)
        retries = meta.get("retry_count", 0)
        if score < MIN_SCORE and retries < MAX_RETRIES:
            return True, f"low_score ({score:.1f} < {MIN_SCORE}, retry {retries+1}/{MAX_RETRIES})"
    
        # Check age
        date_str = meta.get("extraction_date", "")
        if date_str:
            try:
                # Handle both timezone-aware and naive datetime strings
                ext_date = datetime.fromisoformat(date_str.replace("Z", "+00:00"))
                age_days = (datetime.now(timezone.utc) - ext_date).days
                if age_days > DEFAULT_MAX_AGE_DAYS:
                    return True, f"age ({age_days} days > {DEFAULT_MAX_AGE_DAYS})"
            except (ValueError, TypeError):
                pass
    
        return False, ""
    
    
    def get_extraction_for_review(mpn, extract_dir, datasheets_dir=None):
        """High-level: get extraction for a review, checking freshness.
    
        Resolution:
        1. Check cache — if fresh, return cached extraction
        2. If stale or missing — return None (caller should trigger extraction)
    
        Args:
            mpn: Manufacturer part number
            extract_dir: Path to datasheets/extracted/
            datasheets_dir: Path to datasheets/ (for staleness check)
    
        Returns:
            (extraction_dict, status) where status is one of:
            - "cached" — fresh extraction returned
            - "stale:<reason>" — extraction exists but is stale, needs refresh
            - "missing" — no extraction exists
        """
        extraction = get_cached_extraction(extract_dir, mpn)
    
        if extraction is None:
            return None, "missing"
    
        stale, reason = is_extraction_stale(extract_dir, mpn, datasheets_dir)
        if stale:
            return extraction, f"stale:{reason}"
    
        return extraction, "cached"
    
    
    def list_extractions(extract_dir):
        """List all cached extractions with their metadata.
    
        Args:
            extract_dir: Path to datasheets/extracted/
    
        Returns:
            List of dicts with mpn, category, score, date, version, pin_count
        """
        index = _load_index(extract_dir)
        results = []
        for key, entry in index.get("extractions", {}).items():
            results.append({
                "mpn": entry.get("mpn", key),
                "category": entry.get("category", ""),
                "score": entry.get("extraction_score", 0),
                "date": entry.get("extraction_date", ""),
                "version": entry.get("extraction_version", 0),
                "pin_count": entry.get("pin_count", 0),
                "file": entry.get("file", ""),
            })
        return results
    
    
    def update_datasheets_index(datasheets_dir, mpn, extraction):
        """Add an extraction pointer to the main datasheets/manifest.json.
    
        Adds an "extraction" field to the part's entry in the datasheets
        manifest, so any code reading it can quickly check whether a
        pre-extraction exists and its quality.
    
        Reads manifest.json preferentially, falling back to legacy index.json
        for projects that haven't been migrated. Writes back to whichever file
        was loaded (in-place update only — migration to manifest.json happens
        when the distributor sync script re-runs).
    
        Args:
            datasheets_dir: Path to datasheets/
            mpn: Manufacturer part number
            extraction: The extraction dict (reads metadata from it)
        """
        datasheets_dir = Path(datasheets_dir)
        new_path = datasheets_dir / "manifest.json"
        old_path = datasheets_dir / "index.json"
        if new_path.exists():
            index_path = new_path
        elif old_path.exists():
            index_path = old_path
        else:
            return
    
        index = read_json_object(index_path)
        if index is None:
            _warn_manifest_kept(index_path)
            return
    
        parts = index.get("parts", {})
        entry = parts.get(mpn)
        if not entry:
            # Try case-insensitive match
            for k, v in parts.items():
                if k.upper() == mpn.upper():
                    entry = v
                    mpn = k  # use the original key
                    break
        if not entry:
            return
    
        meta = extraction.get("extraction_metadata", {})
        entry["extraction"] = {
            "file": f"extracted/{_sanitize_mpn(mpn)}.json",
            "score": meta.get("extraction_score", 0),
            "date": meta.get("extraction_date", ""),
        }
    
        write_json_atomic(index_path, index)
    
  • skills/datasheets/scripts/datasheet_features.py17 KB43c7c4228a12upstream ↗
    """Consumer helper API for datasheet extractions.
    
    Thin wrapper over datasheet_extract_cache. Provides field-level accessors for
    IC-aware detectors in kicad, emc, spice, and thermal skills.
    
    Contract:
      - Returns a dict of feature fields on cache hit, with a `quality` flag
        describing score/staleness/version status. Quality gate is applied by each
        caller, not by _load.
      - Returns None only on cache miss or wrong part category (e.g., not a regulator).
      - Individual fields within the dict may be None (datasheet didn't specify).
      - Consumers MUST distinguish None (unknown) from False (explicitly no).
    
    Zero external dependencies — stdlib only.
    """
    from __future__ import annotations
    
    import sys
    from pathlib import Path
    from typing import Optional
    
    from datasheet_extract_cache import (
        resolve_extract_dir,
        get_cached_extraction,
        EXTRACTION_VERSION,
        MIN_SCORE,
    )
    
    # Add datasheet_types to sys.path for direct import of DatasheetFacts etc.
    # This module lives at skills/datasheets/scripts/; the types package is a
    # sibling under skills/datasheets/datasheet_types/.
    _TYPES_PARENT = str(Path(__file__).resolve().parent.parent)
    if _TYPES_PARENT not in sys.path:
        sys.path.insert(0, _TYPES_PARENT)
    
    from datasheet_types.cache_reader import cache_path_for, read_json_object  # noqa: E402
    from datasheet_types.extraction import DatasheetFacts  # noqa: E402
    
    
    _REGULATOR_TOPOLOGIES = ('boost', 'buck', 'ldo')
    _MCU_TOPOLOGIES = ('mcu',)
    
    # v1.3 pin-function name → v1.4 Pin.name patterns. Used by the derivation
    # helper to map a pin to its v1.3 functional category. Includes verbose
    # datasheet variants ("Output", "Ground", etc.) — TI's family datasheets
    # print the long forms verbatim, and the v1.4 pipeline lifts pin names
    # verbatim from the source.
    _PIN_NAME_TO_FUNCTION: dict[str, str] = {
        "VIN": "VIN",
        "VIN+": "VIN",
        "Input": "VIN",
        "OUT": "VOUT",
        "VOUT": "VOUT",
        "VOUT+": "VOUT",
        "Output": "VOUT",
        "GND": "GND",
        "VSS": "GND",
        "AGND": "GND",
        "DGND": "GND",
        "Ground": "GND",
    }
    
    
    def _load(mpn, extract_dir=None, analysis_json=None, project_dir=None):
        """Resolve extract dir and load the cached extraction for mpn.
    
        Returns the extraction dict on any cache hit (quality is described by
        `_quality_v13`, never gated here). Returns None on a
        cache miss or when no project names a cache.
        """
        if extract_dir is None:
            extract_dir = resolve_extract_dir(
                analysis_json=analysis_json, project_dir=project_dir
            )
            if extract_dir is None:
                return None
        ext = get_cached_extraction(extract_dir, mpn)
        if not ext:
            # Direct-file fallback: allows test fixtures (and simple single-file drops)
            # that write {mpn}.json directly without a manifest index.
            ext = read_json_object(cache_path_for(mpn, extract_dir))
        return ext if ext is not None and _legacy_shape_ok(ext) else None
    
    
    def _is_number(v):
        return isinstance(v, (int, float)) and not isinstance(v, bool)
    
    
    def _legacy_shape_ok(ext):
        """True when the nested blocks the helpers read have their expected JSON types."""
        meta = ext.get('extraction_metadata')
        if meta is not None:
            if not isinstance(meta, dict):
                return False
            ver = meta.get('extraction_version')
            if ver is not None and not _is_number(ver):
                return False
        # Falsy wrong-typed blocks were always read as empty; only truthy ones are rejected.
        for key in ('features', 'peripherals'):
            if ext.get(key) and not isinstance(ext[key], dict):
                return False
        periph = ext.get('peripherals')
        if isinstance(periph, dict) and periph.get('usb') \
                and not isinstance(periph['usb'], dict):
            return False
        pins = ext.get('pins')
        if pins:
            if not isinstance(pins, list):
                return False
            for pin in pins:
                if not isinstance(pin, dict):
                    return False
                for k in ('threshold_high_v', 'threshold_low_v'):
                    if pin.get(k) is not None and not _is_number(pin[k]):
                        return False
        return True
    
    
    def _quality_v13(ext):
        """Quality flag for a v1.3-format extraction. Describes — never gates."""
        meta = (ext or {}).get('extraction_metadata') or {}
        score = meta.get('extraction_score')
        version = meta.get('extraction_version') or 0
        reasons = []
        if version < EXTRACTION_VERSION:
            reasons.append(f'stale_version ({version} < {EXTRACTION_VERSION})')
        if not isinstance(score, (int, float)) or score < MIN_SCORE:
            reasons.append(f'low_score ({score} < {MIN_SCORE})')
        return {
            'score': score if isinstance(score, (int, float)) else None,
            'scale': '0-10',
            'trusted': not reasons,
            'reasons': reasons,
        }
    
    
    def _quality_v14(facts):
        """Quality flag for a v1.4 DatasheetFacts. Describes — never gates."""
        score = facts.quality
        reasons = []
        if not isinstance(score, (int, float)) or score < 60:
            reasons.append(f'low_score ({score} < 60)')
        if getattr(facts, 'stale', False):
            reasons.append('stale_pdf')
        return {
            'score': score if isinstance(score, (int, float)) else None,
            'scale': '0-100',
            'trusted': not reasons,
            'reasons': reasons,
        }
    
    
    def _pin_with_function(pins, target_function):
        """Return the first pin whose function matches target_function, or None."""
        for pin in pins or []:
            if pin.get('function') == target_function:
                return pin
        return None
    
    
    # ---------------------------------------------------------------------------
    # v1.4 derivation helpers — translate DatasheetFacts → v1.3 dict shape.
    # Used by the public wrappers below. Pure functions, no filesystem.
    # ---------------------------------------------------------------------------
    
    def _derive_regulator_features_v14(facts: DatasheetFacts) -> Optional[dict]:
        """Translate a v1.4 DatasheetFacts into a v1.3 regulator-features dict.
    
        Returns None when facts has no regulator category (only the regulator
        category is derived).
    
        Fields derived from v1.4:
          topology:  facts.regulator.topology (already v1.3-compatible enum
                     for ldo/buck/boost; other topologies pass through verbatim,
                     so a v1.3 detector checking `topology in ('boost','buck','ldo')`
                     gets the same behavior it always had).
          has_pg:    True iff facts.regulator.power_good_pin is not None.
          en_pin:    facts.regulator.enable_pin.
          pg_pin:    facts.regulator.power_good_pin.
          vin_pin:   pin number of the Pin named VIN (or VIN+) in base.pinout.
          vout_pin:  pin number of the Pin named OUT / VOUT / VOUT+.
    
        Fields with no v1.4 schema v1.0 equivalent (has_soft_start, iss_time_us,
        en_v_ih_max, en_v_il_min) return None. v1.3 contract explicitly allows
        this: None means "datasheet didn't specify."
        """
        if facts.regulator is None:
            return None
    
        topo = facts.regulator.topology
    
        # Find VIN / VOUT pins by name via the Pinout wrapper.
        # Try short forms first, then verbose datasheet variants used by TI etc.
        vin_pin_obj = (facts.base.pinout.find(name="VIN")
                       or facts.base.pinout.find(name="VIN+")
                       or facts.base.pinout.find(name="Input"))
        vout_pin_obj = (facts.base.pinout.find(name="OUT")
                        or facts.base.pinout.find(name="VOUT")
                        or facts.base.pinout.find(name="VOUT+")
                        or facts.base.pinout.find(name="Output"))
    
        def _first_number(pin) -> Optional[str]:
            return pin.numbers[0] if pin is not None and pin.numbers else None
    
        return {
            'topology': topo,
            'has_pg': facts.regulator.power_good_pin is not None,
            'has_soft_start': None,      # No v1.4 equivalent.
            'iss_time_us': None,         # No v1.4 equivalent.
            'en_v_ih_max': None,         # No v1.4 equivalent (per-pin VIH/VIL not in schema v1.0).
            'en_v_il_min': None,
            'vin_pin': _first_number(vin_pin_obj),
            'vout_pin': _first_number(vout_pin_obj),
            'en_pin': facts.regulator.enable_pin,
            'pg_pin': facts.regulator.power_good_pin,
        }
    
    
    def _derive_mcu_features_v14(facts: DatasheetFacts) -> Optional[dict]:
        """Translate a v1.4 DatasheetFacts into a v1.3 mcu-features dict.
    
        DatasheetFacts has no mcu category — this always returns None. The public
        get_mcu_features wrapper falls through to the v1.3 cache read path
        when this returns None.
        """
        # categories is a list[str]; DatasheetFacts has no mcu category.
        return None
    
    
    def _derive_pin_function_v14(facts: DatasheetFacts, pin_id: str) -> Optional[str]:
        """Translate a pin identifier → v1.3-style function string.
    
        Derivation order:
          1. Check regulator pin refs (enable_pin → 'EN', power_good_pin → 'PG',
             feedback_pin → 'FB') — these give definitive function hits for
             regulator parts.
          2. Find the Pin in base.pinout by number (exact) or name (case-insensitive).
             Map Pin.name → v1.3 function via _PIN_NAME_TO_FUNCTION.
          3. Fall back to None if no match.
    
        Returns None when the pin_id does not resolve to any known pin, OR
        when the resolved pin's name is not in _PIN_NAME_TO_FUNCTION (unknown
        function category — caller gets the same None signal v1.3 gave for
        unmapped pins).
        """
        target = str(pin_id).strip()
    
        # 1. Regulator pin refs (strongest signal for regulator parts).
        if facts.regulator is not None:
            if facts.regulator.enable_pin == target:
                return "EN"
            if facts.regulator.power_good_pin == target:
                return "PG"
            if facts.regulator.feedback_pin == target:
                return "FB"
    
        # 2. Pinout name-based map.
        pin = facts.base.pinout.find(pin=target)
        if pin is None:
            pin = facts.base.pinout.find(name=target)
        if pin is None:
            return None
    
        return _PIN_NAME_TO_FUNCTION.get(pin.name)
    
    
    def _try_v14_facts(mpn, *, extract_dir=None, analysis_json=None, project_dir=None):
        """Common v1.4 probe. Returns DatasheetFacts or None.
    
        Resolves extract_dir the same way _load does (via resolve_extract_dir)
        so callers that passed analysis_json / project_dir get the same behavior.
        """
        from datasheet_types.cache_reader import lookup  # Lazy import.
    
        if extract_dir is None:
            extract_dir = resolve_extract_dir(
                analysis_json=analysis_json, project_dir=project_dir
            )
            if extract_dir is None:
                return None
        return lookup(mpn, cache_dir=Path(extract_dir))
    
    
    def get_regulator_features(mpn, *, extract_dir=None,
                                analysis_json=None, project_dir=None) -> Optional[dict]:
        """Return regulator-specific features for mpn, or None if not available.
    
        Dual-cache-read: tries the v1.4 cache first via cache_reader.lookup();
        falls back to the v1.3 (extraction_version=2) cache if no v1.4 cache
        exists. When both exist for the same MPN (mid-migration), v1.4 wins.
    
        Returns None when:
          - No v1.4 cache AND no v1.3 cache for the MPN
          - v1.4 cache parses but has no regulator category
          - v1.3 cache exists but topology not in ('boost', 'buck', 'ldo')
    
        Returned dict fields (any may be None individually):
          topology:          'boost' | 'buck' | 'ldo' (v1.3 enum preserved)
          has_pg:            bool | None
          has_soft_start:    bool | None — always None on v1.4 path (no schema equiv)
          iss_time_us:       float | None — always None on v1.4 path
          en_v_ih_max:       float (V) | None — always None on v1.4 path
          en_v_il_min:       float (V) | None — always None on v1.4 path
          vin_pin:           str | None — pin number of the VIN pin
          vout_pin:          str | None
          en_pin:            str | None
          pg_pin:            str | None
          quality:           {score, scale, trusted, reasons} — always present
        """
        # ---- v1.4 path ----
        facts = _try_v14_facts(mpn, extract_dir=extract_dir,
                               analysis_json=analysis_json, project_dir=project_dir)
        if facts is not None:
            derived = _derive_regulator_features_v14(facts)
            if derived is not None and derived.get('topology') in _REGULATOR_TOPOLOGIES:
                derived['quality'] = _quality_v14(facts)
                return derived
            # v1.4 facts present but not a regulator (or topology outside the
            # v1.3 enum). Fall through to v1.3 below.
    
        # ---- v1.3 fallback ----
        ext = _load(mpn, extract_dir=extract_dir,
                    analysis_json=analysis_json, project_dir=project_dir)
        if not ext:
            return None
        topo = ext.get('topology')
        if topo not in _REGULATOR_TOPOLOGIES:
            return None
        pins = ext.get('pins') or []
        features = ext.get('features') or {}
        en_pin = _pin_with_function(pins, 'EN')
        vin_pin = _pin_with_function(pins, 'VIN')
        vout_pin = _pin_with_function(pins, 'VOUT')
        pg_pin = _pin_with_function(pins, 'PG')
    
        def _pin_number(p):
            if not p:
                return None
            n = p.get('number')
            return str(n) if n is not None else p.get('name')
    
        return {
            'topology': topo,
            'has_pg': features.get('has_pg'),
            'has_soft_start': features.get('has_soft_start'),
            'iss_time_us': features.get('iss_time_us'),
            'en_v_ih_max': (en_pin or {}).get('threshold_high_v'),
            'en_v_il_min': (en_pin or {}).get('threshold_low_v'),
            'vin_pin': _pin_number(vin_pin),
            'vout_pin': _pin_number(vout_pin),
            'en_pin': _pin_number(en_pin),
            'pg_pin': _pin_number(pg_pin),
            'quality': _quality_v13(ext),
        }
    
    
    def get_mcu_features(mpn, *, extract_dir=None,
                         analysis_json=None, project_dir=None) -> Optional[dict]:
        """Return MCU-specific features for mpn, or None if not available.
    
        Dual-cache-read: tries v1.4 first (but DatasheetFacts has no mcu category,
        so this path always returns None), then falls back to the v1.3
        (extraction_version=2) cache.
    
        Returns None when neither cache has data for mpn.
    
        Returned dict fields (any may be None individually):
          usb_speed:              'FS' | 'HS' | 'SS' | None
          has_native_usb_phy:     bool | None
          usb_series_r_required:  bool | None
          quality:                {score, scale, trusted, reasons} — always present
        """
        # ---- v1.4 path ----
        facts = _try_v14_facts(mpn, extract_dir=extract_dir,
                               analysis_json=analysis_json, project_dir=project_dir)
        if facts is not None:
            derived = _derive_mcu_features_v14(facts)
            if derived is not None:
                return derived
            # v1.4 facts present but no mcu category.
            # Fall through to v1.3 for legacy MCU caches.
    
        # ---- v1.3 fallback ----
        ext = _load(mpn, extract_dir=extract_dir,
                    analysis_json=analysis_json, project_dir=project_dir)
        if not ext:
            return None
        if ext.get('topology') not in _MCU_TOPOLOGIES:
            return None
        peripherals = ext.get('peripherals') or {}
        usb = peripherals.get('usb') or {}
        return {
            'usb_speed': usb.get('speed'),
            'has_native_usb_phy': usb.get('native_phy'),
            'usb_series_r_required': usb.get('series_r_required'),
            'quality': _quality_v13(ext),
        }
    
    
    def get_pin_function(mpn, pin_identifier, *, extract_dir=None,
                          analysis_json=None, project_dir=None) -> Optional[str]:
        """Return the functional category of a pin ('EN', 'VIN', etc.), or None.
    
        `pin_identifier` matches against pins[].number (exact) OR pins[].name
        (case-insensitive).
    
        The trusted gate is applied explicitly here: untrusted extractions return
        None (the gate lives in each consumer, not in _load).
    
        Dual-cache-read: v1.4 cache preferred, v1.3 fallback.
        """
        # ---- v1.4 path ----
        facts = _try_v14_facts(mpn, extract_dir=extract_dir,
                               analysis_json=analysis_json, project_dir=project_dir)
        if facts is not None:
            fn = _derive_pin_function_v14(facts, pin_identifier)
            if fn is not None:
                return fn
            # v1.4 facts present but pin not resolved — fall through to v1.3
            # (may have richer pin data with explicit function labels).
    
        # ---- v1.3 fallback ----
        ext = _load(mpn, extract_dir=extract_dir,
                    analysis_json=analysis_json, project_dir=project_dir)
        if not ext or not _quality_v13(ext)['trusted']:
            return None
        target = str(pin_identifier).strip()
        target_lower = target.lower()
        for p in ext.get('pins') or []:
            if str(p.get('number', '')).strip() == target:
                return p.get('function')
            if str(p.get('name', '')).strip().lower() == target_lower:
                return p.get('function')
        return None
    
    
    def is_extraction_available(mpn, *, extract_dir=None,
                                 analysis_json=None, project_dir=None) -> bool:
        """True iff a usable extraction exists for mpn (v1.4 OR v1.3 cache)."""
        # v1.4 path: any non-None DatasheetFacts counts as available.
        if _try_v14_facts(mpn, extract_dir=extract_dir,
                          analysis_json=analysis_json, project_dir=project_dir) is not None:
            return True
        # v1.3 fallback: trusted gate applied explicitly here (the gate lives in
        # each consumer that needs a binary usable/not answer, not in _load).
        ext = _load(mpn, extract_dir=extract_dir,
                    analysis_json=analysis_json, project_dir=project_dir)
        return bool(ext) and _quality_v13(ext)['trusted']
    
  • skills/datasheets/scripts/datasheet_page_selector.py17 KBdc969e61a0feupstream ↗
    """
    Heuristic page selection for datasheet PDF extraction.
    
    Given a downloaded datasheet PDF, identifies which pages contain the
    information Claude needs for structured extraction: pin tables, absolute
    maximum ratings, recommended operating conditions, electrical characteristics,
    and application circuits. This reduces token cost by 60-85% compared to
    feeding the entire PDF.
    
    Uses pdftotext and pdfinfo (poppler-utils) for text extraction. Falls back
    gracefully if these aren't available — returns all pages with low confidence.
    
    Usage:
        from datasheet_page_selector import suggest_pages
    
        selection = suggest_pages("/path/to/datasheet.pdf", "TPS61023DRLR", "switching_regulator")
        print(selection.pages)     # [1, 2, 3, 5, 8, 15]
        print(selection.reasons)   # {1: "front page", 3: "pin description table", ...}
        print(selection.confidence)  # "toc_found" | "keyword_scored" | "fallback"
    """
    
    import re
    import subprocess
    import sys
    from dataclasses import dataclass, field
    from pathlib import Path
    
    _KICAD_SCRIPTS = str(Path(__file__).resolve().parents[2] / "kicad" / "scripts")
    if _KICAD_SCRIPTS not in sys.path:
        sys.path.append(_KICAD_SCRIPTS)
    
    from kicad_core.executables import find_executable  # noqa: E402
    
    
    # ---------------------------------------------------------------------------
    # Result type
    # ---------------------------------------------------------------------------
    
    @dataclass
    class PageSelection:
        """Result of page selection heuristics."""
        pages: list = field(default_factory=list)
        reasons: dict = field(default_factory=dict)  # page_num -> reason string
        confidence: str = "fallback"  # "toc_found", "keyword_scored", "fallback"
        total_pages: int = 0
    
    
    # ---------------------------------------------------------------------------
    # Maximum pages to select (keeps token cost manageable)
    # ---------------------------------------------------------------------------
    
    # Category-specific page limits
    _PAGE_LIMITS = {
        "microcontroller": 15,  # MCUs have complex pin tables
        "fpga": 15,
        "soc": 15,
        "default": 10,
    }
    
    
    # ---------------------------------------------------------------------------
    # TOC section patterns
    # ---------------------------------------------------------------------------
    
    # Each entry: (regex_pattern, reason_label, priority)
    # Priority 1 = always include, 2 = include if space, 3 = nice to have
    _TOC_PATTERNS = [
        (r'pin\s+(?:description|configuration|function|assignment|definition|table|diagram)',
         "pin_table", 1),
        (r'absolute\s+maximum\s+rat',
         "abs_max_ratings", 1),
        (r'(?:recommended|typical)\s+operating\s+cond',
         "operating_conditions", 1),
        (r'electrical\s+characteristics?',
         "electrical_chars", 1),
        (r'(?:typical\s+)?application(?:\s+(?:circuit|schematic|diagram|information))?',
         "application_circuit", 1),
        (r'functional\s+(?:block\s+)?diagram',
         "block_diagram", 2),
        (r'package\s+(?:information|outline|dimensions|marking)',
         "package_info", 2),
        (r'pin(?:out)?\s+(?:diagram|drawing)',
         "pinout_diagram", 1),
        (r'thermal\s+(?:information|characteristics|pad)',
         "thermal_info", 2),
        (r'ordering\s+information',
         "ordering", 3),
        (r'detailed\s+description',
         "detailed_desc", 2),
        (r'design\s+(?:guide|guidelines|considerations|recommendations)',
         "design_guide", 2),
        (r'power\s+(?:supply|management)\s+(?:consideration|recommendation)',
         "power_design", 2),
        (r'layout\s+(?:guide|guidelines|considerations|recommendations)',
         "layout_guide", 2),
    ]
    
    
    # ---------------------------------------------------------------------------
    # Keyword scoring weights for fallback page identification
    # ---------------------------------------------------------------------------
    
    # (regex_pattern, weight)  — higher weight = more relevant page
    _KEYWORD_SCORES = [
        (r'\bpin\b.*\b(?:name|function|description|number)\b', 5.0),
        (r'\babsolute\s+maximum\b', 4.0),
        (r'\brecommended\s+operating\b', 4.0),
        (r'\belectrical\s+characteristics?\b', 3.5),
        (r'\bapplication\s+(?:circuit|schematic)\b', 3.5),
        (r'\btypical\s+application\b', 3.5),
        (r'\bpin\s+(?:configuration|assignment|diagram)\b', 3.0),
        (r'\bblock\s+diagram\b', 2.0),
        (r'\binput\s+voltage\b', 2.0),
        (r'\boutput\s+voltage\b', 2.0),
        (r'\bvoltage\s+range\b', 2.0),
        (r'\bmaximum\s+rat', 2.0),
        (r'\b(?:V_?CC|V_?DD|V_?IN|V_?OUT)\b', 1.5),
        (r'\btemperature\b.*\b(?:range|operating|junction)\b', 1.5),
        (r'\bfeedback\b', 1.5),
        (r'\benable\b.*\bpin\b', 1.0),
        (r'\bschematic\b', 1.0),
        (r'\bthreshold\b', 1.0),
    ]
    
    # Category-specific bonus keywords
    _CATEGORY_KEYWORDS = {
        "operational_amplifier": [
            (r'\bgain[- ]?bandwidth\b', 3.0),
            (r'\bslew\s+rate\b', 3.0),
            (r'\binput\s+offset\b', 2.5),
            (r'\bopen[- ]?loop\s+gain\b', 2.5),
            (r'\brail[- ]?to[- ]?rail\b', 2.0),
        ],
        "linear_regulator": [
            (r'\bdropout\b', 3.0),
            (r'\bquiescent\s+current\b', 3.0),
            (r'\boutput\s+capacitor\b', 2.5),
            (r'\binput\s+capacitor\b', 2.5),
        ],
        "switching_regulator": [
            (r'\bswitching\s+frequency\b', 3.0),
            (r'\binductor\s+selection\b', 3.0),
            (r'\bfeedback\s+divider\b', 3.0),
            (r'\bcompensation\b', 2.5),
            (r'\befficiency\b', 2.0),
            (r'\binput\s+capacitor\b', 2.5),
            (r'\boutput\s+capacitor\b', 2.5),
        ],
        "microcontroller": [
            (r'\bgpio\b', 2.5),
            (r'\balternate\s+function\b', 2.5),
            (r'\bperipheral\b', 2.0),
            (r'\bboot\s+(?:mode|strap|config)\b', 3.0),
            (r'\breset\b.*\bcircuit\b', 2.5),
            (r'\bpower\s+(?:supply|consumption|domain)\b', 2.5),
            (r'\bdecoupling\b', 2.0),
        ],
        "esd_protection": [
            (r'\bclamping\s+voltage\b', 3.0),
            (r'\bleakage\b', 2.5),
            (r'\bcapacitance\b', 2.5),
            (r'\bprotection\b', 2.0),
        ],
    }
    
    
    # ---------------------------------------------------------------------------
    # PDF text extraction helpers
    # ---------------------------------------------------------------------------
    
    def _has_pdftotext():
        """Check if pdftotext is available."""
        return find_executable("pdftotext") is not None
    
    
    def _has_pdfinfo():
        """Check if pdfinfo is available."""
        return find_executable("pdfinfo") is not None
    
    
    def _get_page_count(pdf_path):
        """Get total number of pages in a PDF.
    
        Tries pdfinfo first, falls back to byte scanning.
        """
        pdf_path = str(pdf_path)
    
        pdfinfo = find_executable("pdfinfo")
        if pdfinfo:
            try:
                result = subprocess.run(
                    [pdfinfo, pdf_path],
                    capture_output=True, text=True, timeout=10,
                )
                if result.returncode == 0:
                    m = re.search(r'Pages:\s+(\d+)', result.stdout)
                    if m:
                        return int(m.group(1))
            except (subprocess.TimeoutExpired, OSError):
                pass
    
        # Fallback: count /Type /Page objects in raw PDF bytes
        try:
            with open(pdf_path, "rb") as f:
                content = f.read()
            # Count /Type /Page (not /Type /Pages) — rough estimate
            count = len(re.findall(rb'/Type\s*/Page(?!\s*s)', content))
            return max(count, 1)
        except OSError:
            return 0
    
    
    def _extract_pages_text(pdf_path, first_page=1, last_page=None):
        """Extract text from a range of PDF pages.
    
        Args:
            pdf_path: Path to PDF file
            first_page: First page (1-based)
            last_page: Last page (1-based), or None for all
    
        Returns:
            Dict mapping page_number -> text
        """
        pdftotext = find_executable("pdftotext")
        if not pdftotext:
            return {}
    
        total = _get_page_count(pdf_path)
        if last_page is None:
            last_page = total
    
        # Extract all requested pages at once (faster than per-page)
        try:
            result = subprocess.run(
                [pdftotext, "-f", str(first_page), "-l", str(last_page),
                 "-layout", str(pdf_path), "-"],
                capture_output=True, text=True, timeout=30,
            )
            if result.returncode != 0:
                return {}
        except (subprocess.TimeoutExpired, OSError):
            return {}
    
        # Split on form-feed characters (pdftotext page delimiter)
        pages_text = result.stdout.split("\f")
        page_map = {}
        for i, text in enumerate(pages_text):
            page_num = first_page + i
            if page_num <= last_page:
                page_map[page_num] = text
        return page_map
    
    
    # ---------------------------------------------------------------------------
    # TOC detection
    # ---------------------------------------------------------------------------
    
    def _find_toc_pages(page_texts, max_scan_pages=3):
        """Scan early pages for table of contents references.
    
        Looks for section headings with page numbers in the first few pages.
    
        Args:
            page_texts: Dict of page_number -> text
            max_scan_pages: How many pages to scan for TOC
    
        Returns:
            Dict of page_number -> reason (sections found via TOC)
        """
        toc_pages = {}
    
        for page_num in sorted(page_texts.keys())[:max_scan_pages]:
            text = page_texts.get(page_num, "")
            if not text:
                continue
    
            for pattern, reason, priority in _TOC_PATTERNS:
                # Look for pattern followed by dots/spaces and a page number
                toc_re = re.compile(
                    pattern + r'[\s.…·\-_]*(\d{1,3})',
                    re.IGNORECASE | re.MULTILINE,
                )
                for m in toc_re.finditer(text):
                    try:
                        target_page = int(m.group(1))
                        if target_page > 0:
                            toc_pages[target_page] = reason
                    except (ValueError, IndexError):
                        pass
    
        return toc_pages
    
    
    # ---------------------------------------------------------------------------
    # Keyword-based page scoring (fallback)
    # ---------------------------------------------------------------------------
    
    def _score_page(text, category=""):
        """Score a page's relevance based on keyword density.
    
        Args:
            text: Page text content
            category: Component category for bonus keywords
    
        Returns:
            Float score (higher = more relevant)
        """
        if not text or len(text.strip()) < 50:
            return 0.0
    
        text_lower = text.lower()
        score = 0.0
    
        for pattern, weight in _KEYWORD_SCORES:
            matches = len(re.findall(pattern, text_lower, re.IGNORECASE))
            score += min(matches, 3) * weight  # cap per-pattern contribution
    
        # Category-specific bonus
        category_kws = _CATEGORY_KEYWORDS.get(category, [])
        for pattern, weight in category_kws:
            matches = len(re.findall(pattern, text_lower, re.IGNORECASE))
            score += min(matches, 3) * weight
    
        return score
    
    
    # ---------------------------------------------------------------------------
    # Public API
    # ---------------------------------------------------------------------------
    
    def suggest_pages(pdf_path, mpn="", category="", max_pages=None):
        """Select the most relevant pages from a datasheet PDF.
    
        Uses a multi-strategy approach:
        1. Extract text from early pages, look for table of contents
        2. If TOC found, use referenced page numbers
        3. If no TOC, score all pages by keyword density
        4. Always include page 1 (front page) and last page (package/ordering)
    
        Args:
            pdf_path: Path to the PDF file
            mpn: Manufacturer part number (for context)
            category: Component category (for targeted keyword scoring)
            max_pages: Maximum pages to select (default: category-dependent)
    
        Returns:
            PageSelection with pages, reasons, confidence, total_pages
        """
        pdf_path = Path(pdf_path)
        if not pdf_path.exists():
            return PageSelection(confidence="error")
    
        total_pages = _get_page_count(pdf_path)
        if total_pages == 0:
            return PageSelection(confidence="error")
    
        if max_pages is None:
            max_pages = _PAGE_LIMITS.get(category, _PAGE_LIMITS["default"])
    
        # For short datasheets, just return all pages
        if total_pages <= max_pages:
            pages = list(range(1, total_pages + 1))
            reasons = {p: "all pages (short datasheet)" for p in pages}
            return PageSelection(
                pages=pages, reasons=reasons,
                confidence="all_pages", total_pages=total_pages,
            )
    
        # Check if pdftotext is available
        if not _has_pdftotext():
            # No text extraction available — return front pages + even distribution
            pages = _fallback_page_selection(total_pages, max_pages)
            reasons = {p: "fallback (no pdftotext)" for p in pages}
            return PageSelection(
                pages=pages, reasons=reasons,
                confidence="fallback", total_pages=total_pages,
            )
    
        # Strategy 1: Extract text from all pages
        page_texts = _extract_pages_text(pdf_path, 1, total_pages)
        if not page_texts:
            pages = _fallback_page_selection(total_pages, max_pages)
            reasons = {p: "fallback (extraction failed)" for p in pages}
            return PageSelection(
                pages=pages, reasons=reasons,
                confidence="fallback", total_pages=total_pages,
            )
    
        # Strategy 2: Look for TOC in first 3 pages
        toc_pages = _find_toc_pages(page_texts, max_scan_pages=3)
    
        if toc_pages:
            # TOC found — use referenced pages
            selected = {}
            selected[1] = "front page"
    
            for page_num, reason in toc_pages.items():
                if 1 <= page_num <= total_pages:
                    selected[page_num] = f"TOC: {reason}"
                    # Also include the next page (tables often span 2 pages)
                    if page_num + 1 <= total_pages:
                        selected[page_num + 1] = f"continuation of {reason}"
    
            # Add last page if not already included
            if total_pages not in selected:
                selected[total_pages] = "last page (package/ordering)"
    
            # If still under budget, fill with keyword-scored pages
            if len(selected) < max_pages:
                scored = [(p, _score_page(t, category)) for p, t in page_texts.items()
                          if p not in selected]
                scored.sort(key=lambda x: x[1], reverse=True)
                for p, s in scored:
                    if len(selected) >= max_pages:
                        break
                    if s > 3.0:  # only add pages with meaningful content
                        selected[p] = f"keyword scored ({s:.1f})"
    
            pages = _cap_pages(selected.keys(), max_pages, total_pages)
            reasons = {p: selected[p] for p in pages}
            return PageSelection(
                pages=pages, reasons=reasons,
                confidence="toc_found", total_pages=total_pages,
            )
    
        # Strategy 3: No TOC found — score all pages by keywords
        page_scores = [(p, _score_page(t, category)) for p, t in page_texts.items()]
        page_scores.sort(key=lambda x: x[1], reverse=True)
    
        selected = {}
        selected[1] = "front page"
    
        for page_num, score in page_scores:
            if len(selected) >= max_pages - 1:  # reserve 1 for last page
                break
            if page_num not in selected and score > 1.0:
                selected[page_num] = f"keyword scored ({score:.1f})"
    
        # Always include last page
        if total_pages not in selected and len(selected) < max_pages:
            selected[total_pages] = "last page (package/ordering)"
    
        pages = sorted(selected.keys())[:max_pages]
        reasons = {p: selected[p] for p in pages}
        return PageSelection(
            pages=pages, reasons=reasons,
            confidence="keyword_scored", total_pages=total_pages,
        )
    
    
    def _cap_pages(pages, max_pages, total_pages):
        """Sorted pages limited to max_pages, always keeping page 1 and the last
        page (package and ordering information)."""
        chosen = set(pages)
        keep = [p for p in (1, total_pages) if p in chosen][:max_pages]
        rest = [p for p in sorted(chosen) if p not in keep]
        return sorted(keep + rest[:max(0, max_pages - len(keep))])
    
    
    def _fallback_page_selection(total_pages, max_pages):
        """Generate a fallback page selection without text extraction.
    
        Includes page 1, pages 2-5 (usually contain pin/spec tables),
        and evenly distributed pages through the rest.
    
        Args:
            total_pages: Total number of pages in the PDF
            max_pages: Maximum pages to return
    
        Returns:
            Sorted list of page numbers
        """
        pages = set()
        pages.add(1)
    
        # Front matter (usually has pin tables, abs max, etc.)
        for p in range(2, min(6, total_pages + 1)):
            pages.add(p)
    
        # Last page
        pages.add(total_pages)
    
        # If we need more, distribute evenly through remaining pages
        if len(pages) < max_pages:
            remaining = [p for p in range(6, total_pages) if p not in pages]
            step = max(1, len(remaining) // (max_pages - len(pages)))
            for i in range(0, len(remaining), step):
                if len(pages) >= max_pages:
                    break
                pages.add(remaining[i])
    
        return _cap_pages(pages, max_pages, total_pages)
    
  • skills/datasheets/scripts/datasheet_score.py16 KB8d5e85d0125fupstream ↗
    """
    Quality scoring for structured datasheet extractions.
    
    Evaluates an extraction JSON against a completeness rubric across five
    dimensions: pin coverage, voltage ratings, operating conditions, application
    info, and SPICE-relevant specs. Returns a weighted score 0.0-10.0 with
    per-dimension breakdown and a list of specific issues found.
    
    The scoring is deterministic — same input always produces same score. It
    runs after Claude performs the extraction and tells Claude (and the cache
    manager) whether the extraction is good enough or needs a retry.
    """
    
    
    # ---------------------------------------------------------------------------
    # Category-specific electrical characteristic requirements
    # ---------------------------------------------------------------------------
    
    _ECHAR_REQUIREMENTS = {
        # category: (required_fields, nice_to_have_fields)
        "operational_amplifier": (
            ["gbw_hz", "slew_vus"],
            ["vos_mv", "aol_db", "rin_ohms"],
        ),
        "comparator": (
            ["prop_delay_ns"],
            ["vos_mv", "aol_db"],
        ),
        "linear_regulator": (
            ["vref_v", "quiescent_current_ua"],
            ["dropout_mv", "output_current_max_ma"],
        ),
        "switching_regulator": (
            ["vref_v", "switching_frequency_khz"],
            ["quiescent_current_ua", "output_current_max_ma"],
        ),
        "voltage_reference": (
            ["vref_v", "vref_accuracy_pct"],
            ["temp_coefficient_ppmk"],
        ),
        "microcontroller": (
            [],  # MCUs have diverse e-chars, no universal required set
            ["quiescent_current_ua", "io_voltage_max"],
        ),
        "esd_protection": (
            ["clamping_voltage_v"],
            ["leakage_current_na", "capacitance_pf"],
        ),
    }
    
    # Fallback for categories not listed above
    _ECHAR_DEFAULT = ([], [])
    
    
    # ---------------------------------------------------------------------------
    # SPICE spec requirements per category
    # ---------------------------------------------------------------------------
    
    _SPICE_REQUIREMENTS = {
        "operational_amplifier": ["gbw_hz"],
        "comparator": [],  # comparators rarely need SPICE behavioral models
        "linear_regulator": ["vref", "dropout_mv"],
        "switching_regulator": ["vref"],
        "voltage_reference": ["vref"],
    }
    
    
    # ---------------------------------------------------------------------------
    # Scoring functions
    # ---------------------------------------------------------------------------
    
    def _score_pin_coverage(extraction, expected_pin_count=None):
        """Score pin coverage (0-10).
    
        10.0 = all pins present with name, type, and at least one electrical spec.
        Deductions: -2 per missing pin, -1 per pin with name only (no specs).
        0.0 if fewer than 50% of expected pins documented.
        """
        pins = extraction.get("pins", [])
        if not pins:
            return 0.0, ["No pins documented"]
    
        num_pins = len(pins)
        expected = expected_pin_count or num_pins  # if no expected count, use what we have
    
        # Check if we have enough pins
        if expected > 0 and num_pins < expected * 0.5:
            return 0.0, [f"Only {num_pins}/{expected} pins documented (<50%)"]
    
        issues = []
        pins_with_specs = 0
        pins_with_name_only = 0
    
        for pin in pins:
            has_name = bool(pin.get("name"))
            has_electrical = any([
                pin.get("voltage_abs_max") is not None,
                pin.get("voltage_operating_min") is not None,
                pin.get("voltage_operating_max") is not None,
                pin.get("current_max_ma") is not None,
                pin.get("threshold_high_v") is not None,
                pin.get("threshold_low_v") is not None,
            ])
            has_description = bool(pin.get("description"))
            has_required = bool(pin.get("required_external"))
    
            if has_name and (has_electrical or has_description or has_required):
                pins_with_specs += 1
            elif has_name:
                pins_with_name_only += 1
                issues.append(f"Pin {pin.get('number', '?')} ({pin.get('name', '?')}): name only, no specs")
            else:
                issues.append(f"Pin {pin.get('number', '?')}: missing name")
    
        # Calculate score
        if expected == 0:
            return 10.0, []
    
        missing = max(0, expected - num_pins)
        score = 10.0
        score -= missing * 2.0  # -2 per missing pin
        score -= pins_with_name_only * 1.0  # -1 per name-only pin
    
        if missing > 0:
            issues.insert(0, f"{missing} pin(s) missing from extraction")
    
        return max(0.0, min(10.0, score)), issues[:5]  # cap issues for readability
    
    
    def _score_voltage_ratings(extraction):
        """Score voltage/thermal ratings completeness (0-10).
    
        10.0 = abs max has vin_max + junction_temp, operating has vin range + temp range.
        """
        abs_max = extraction.get("absolute_maximum_ratings", {})
        operating = extraction.get("recommended_operating_conditions", {})
        issues = []
        score = 10.0
    
        # Absolute maximum ratings
        if not abs_max:
            score -= 3.0
            issues.append("No absolute maximum ratings")
        else:
            if abs_max.get("junction_temp_max_c") is None:
                score -= 1.0
                issues.append("Missing junction temperature max")
            # Check for at least one voltage abs max
            voltage_keys = [k for k in abs_max if k.endswith("_max_v") and abs_max[k] is not None]
            if not voltage_keys:
                score -= 2.0
                issues.append("No voltage absolute maximum ratings")
    
        # Recommended operating conditions
        if not operating:
            score -= 3.0
            issues.append("No recommended operating conditions")
        else:
            if operating.get("vin_min_v") is None or operating.get("vin_max_v") is None:
                score -= 1.5
                issues.append("Missing input voltage operating range")
            if operating.get("temp_min_c") is None or operating.get("temp_max_c") is None:
                score -= 1.0
                issues.append("Missing operating temperature range")
    
        return max(0.0, min(10.0, score)), issues
    
    
    def _score_application_info(extraction):
        """Score application circuit information (0-10).
    
        10.0 = has topology + 2+ component recommendations + formula/notes.
        """
        app = extraction.get("application_circuit", {})
        issues = []
        score = 10.0
    
        if not app:
            return 0.0, ["No application circuit information"]
    
        if not app.get("topology"):
            score -= 2.0
            issues.append("Missing circuit topology")
    
        # Count component recommendations
        rec_fields = [
            "inductor_recommended", "input_cap_recommended", "output_cap_recommended",
            "feedback_resistor_top_ohm", "feedback_resistor_bottom_ohm",
            "compensation_cap", "bootstrap_cap", "decoupling_cap",
        ]
        recs = sum(1 for f in rec_fields if app.get(f))
        # Also count any key ending in _recommended
        recs += sum(1 for k, v in app.items()
                    if k.endswith("_recommended") and k not in rec_fields and v)
    
        if recs == 0:
            score -= 3.0
            issues.append("No component recommendations")
        elif recs == 1:
            score -= 1.5
            issues.append("Only 1 component recommendation (expect 2+)")
    
        if not app.get("vout_formula") and not app.get("notes"):
            score -= 2.0
            issues.append("No formula or application notes")
        elif not app.get("notes"):
            score -= 1.0
            issues.append("No layout/application notes")
    
        return max(0.0, min(10.0, score)), issues
    
    
    def _score_electrical_chars(extraction):
        """Score electrical characteristics (0-10), category-dependent.
    
        Uses category-specific required/nice-to-have field lists.
        """
        category = extraction.get("category", "")
        echars = extraction.get("electrical_characteristics", {})
        issues = []
    
        required, nice = _ECHAR_REQUIREMENTS.get(category, _ECHAR_DEFAULT)
    
        if not echars:
            if required:
                return 0.0, [f"No electrical characteristics (need {', '.join(required)})"]
            return 5.0, ["No electrical characteristics (none strictly required for this category)"]
    
        score = 10.0
    
        # Check required fields
        for field in required:
            if echars.get(field) is None:
                score -= 3.0
                issues.append(f"Missing required: {field}")
    
        # Check nice-to-have fields
        for field in nice:
            if echars.get(field) is None:
                score -= 1.0
                issues.append(f"Missing optional: {field}")
    
        # Bonus: if category has no requirements but we got some data, that's good
        if not required and not nice:
            populated = sum(1 for v in echars.values() if v is not None)
            if populated >= 3:
                score = 10.0
            elif populated >= 1:
                score = 7.0
            else:
                score = 5.0
    
        return max(0.0, min(10.0, score)), issues[:5]
    
    
    def _score_spice_specs(extraction):
        """Score SPICE-relevant specs (0-10).
    
        10.0 = enough fields for the component category's behavioral model.
        """
        category = extraction.get("category", "")
        spice = extraction.get("spice_specs", {})
        issues = []
    
        if not spice:
            # Check if we can derive from electrical_characteristics
            echars = extraction.get("electrical_characteristics", {})
            if echars:
                # Some specs can be pulled from e-chars
                return 5.0, ["No spice_specs section (could derive from electrical_characteristics)"]
            return 0.0, ["No SPICE-relevant specs"]
    
        required = _SPICE_REQUIREMENTS.get(category, [])
        score = 10.0
    
        for field in required:
            if spice.get(field) is None:
                score -= 4.0
                issues.append(f"Missing SPICE spec: {field}")
    
        # Count populated fields
        populated = sum(1 for v in spice.values() if v is not None)
        if populated == 0:
            return 0.0, ["spice_specs section is empty"]
        elif populated <= 2 and not required:
            score = min(score, 6.0)
    
        return max(0.0, min(10.0, score)), issues
    
    
    # ---------------------------------------------------------------------------
    # Public API
    # ---------------------------------------------------------------------------
    
    def score_extraction(extraction, expected_pin_count=None):
        """Score a datasheet extraction for completeness.
    
        Evaluates across five weighted dimensions and returns a total score
        (0.0-10.0) along with per-dimension scores and specific issues.
    
        Args:
            extraction: Parsed extraction JSON dict
            expected_pin_count: If known (from schematic), used for pin coverage
    
        Returns:
            {
                "total": 8.2,
                "pin_coverage": 9.0,
                "voltage_ratings": 8.5,
                "application_info": 7.0,
                "electrical_characteristics": 8.0,
                "spice_specs": 8.5,
                "issues": ["Missing abs max for SW pin", ...],
                "sufficient": True  # True if total >= 6.0
            }
        """
        pin_score, pin_issues = _score_pin_coverage(extraction, expected_pin_count)
        volt_score, volt_issues = _score_voltage_ratings(extraction)
        app_score, app_issues = _score_application_info(extraction)
        echar_score, echar_issues = _score_electrical_chars(extraction)
        spice_score, spice_issues = _score_spice_specs(extraction)
    
        # Weighted average (weights sum to 1.0)
        total = (
            pin_score * 0.35 +
            volt_score * 0.25 +
            app_score * 0.20 +
            echar_score * 0.10 +
            spice_score * 0.10
        )
    
        all_issues = pin_issues + volt_issues + app_issues + echar_issues + spice_issues
    
        return {
            "total": round(total, 1),
            "pin_coverage": round(pin_score, 1),
            "voltage_ratings": round(volt_score, 1),
            "application_info": round(app_score, 1),
            "electrical_characteristics": round(echar_score, 1),
            "spice_specs": round(spice_score, 1),
            "issues": all_issues,
            "sufficient": total >= 6.0,
        }
    
    
    # ---------------------------------------------------------------------------
    # v1.4 rubric extension
    # ---------------------------------------------------------------------------
    
    # Per-pin fields scored by population (1 point each, max 4 pts/pin)
    _PIN_FIELDS = ("name", "type", "power_domain", "evidence")
    
    # Base block "fully populated" reference field set
    _BASE_FIELD_REFS = {
        "package":               ["code", "pin_count"],
        "thermal":               ["theta_ja"],
        "absolute_max":          ["VIN_max", "TJ_max"],
        "recommended_operating": ["VIN", "TA"],
        "esd":                   ["HBM"],
    }
    
    # Per-category required+nice-to-have fields
    _CATEGORY_FIELD_REQS = {
        "regulator": {
            "required": ("topology", "vin_range", "vout_range", "iout_max"),
            "nice":     ("reference_voltage", "switching_freq", "cin_min", "cout_min", "enable_pin"),
        },
        # 3b: mcu, opamp, transistor, diode, crystal
    }
    
    
    def _populated(value) -> bool:
        if value is None:
            return False
        if isinstance(value, (list, dict)) and not value:
            return False
        return True
    
    
    def _pinout_completeness(extraction: dict) -> float:
        pinout = ((extraction.get("base") or {}).get("pinout")) or []
        if not isinstance(pinout, list):
            return 0.0  # sentinel ({_extraction_failed: true}) or malformed pinout block
        if not pinout:
            return 0.0
        pts_per_pin = len(_PIN_FIELDS)
        earned = 0
        total = pts_per_pin * len(pinout)
        for p in pinout:
            for f in _PIN_FIELDS:
                if _populated(p.get(f)):
                    earned += 1
        return earned / total if total else 0.0
    
    
    def _base_completeness(extraction: dict) -> float:
        base = extraction.get("base") or {}
        if not base:
            return 0.0
        if isinstance(base, dict) and base.get("_extraction_failed"):
            return 0.0  # base block extraction failed (post-retry sentinel)
        earned = 0
        total = sum(len(v) for v in _BASE_FIELD_REFS.values())
        for block, keys in _BASE_FIELD_REFS.items():
            b = base.get(block) or {}
            if not isinstance(b, dict):
                continue
            for k in keys:
                if _populated(b.get(k)):
                    earned += 1
        return earned / total if total else 0.0
    
    
    def _category_extension_completeness(extraction: dict) -> float:
        cats = extraction.get("categories") or []
        if not cats:
            return 0.0
        fractions = []
        for cat in cats:
            payload = extraction.get(cat)
            if not isinstance(payload, dict) or payload.get("_extraction_failed"):
                fractions.append(0.0)
                continue
            spec = _CATEGORY_FIELD_REQS.get(cat)
            if not spec:
                fractions.append(1.0)  # unknown category — full credit (3b will populate)
                continue
            req = spec["required"]
            nice = spec["nice"]
            req_score = sum(1 for f in req if _populated(payload.get(f))) / len(req) if req else 1.0
            nice_score = sum(1 for f in nice if _populated(payload.get(f))) / len(nice) if nice else 1.0
            fractions.append(0.7 * req_score + 0.3 * nice_score)
        return sum(fractions) / len(fractions) if fractions else 0.0
    
    
    def score_v14_extraction(extraction: dict) -> dict:
        """Quality score (0-100) for a v1.4 extraction JSON.
    
        Weighted dimensions:
          pinout_completeness            20%
          base_completeness              25%
          category_extension_completeness 25%
          reserved                       30%   (always full credit)
        """
        dims = {
            "pinout_completeness": _pinout_completeness(extraction),
            "base_completeness": _base_completeness(extraction),
            "category_extension_completeness": _category_extension_completeness(extraction),
            "reserved_v15": 1.0,
        }
        weights = {
            "pinout_completeness": 0.20,
            "base_completeness": 0.25,
            "category_extension_completeness": 0.25,
            "reserved_v15": 0.30,
        }
        weighted = sum(dims[k] * weights[k] for k in dims)
        score = int(round(weighted * 100))
        return {"score": score, "dimensions": dims, "weights": weights}
    
  • skills/datasheets/scripts/datasheet_verify.py46 KB1532c22d0141upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/scripts/extraction_schema.py1.7 KB50e82c0b2b44upstream ↗
    """JSON Schema validation for datasheet extraction results, shared by merge_results.py and
    validate_extraction_result.py.
    
    jsonschema and referencing are optional packages: they are imported when validation runs, and a
    RuntimeError names them when they are missing.
    """
    
    from __future__ import annotations
    
    import json
    from pathlib import Path
    
    SCHEMA_DIR = Path(__file__).resolve().parents[1] / "schemas"
    
    _MISSING = ("full schema validation needs the optional packages jsonschema and referencing "
                "(pip install jsonschema referencing)")
    
    
    def schema_registry() -> "Registry":
        """Build a referencing Registry so $ref between local schemas resolves."""
        try:
            from referencing import Registry, Resource
        except ImportError as exc:
            raise RuntimeError(_MISSING) from exc
        registry = Registry()
        for schema_path in SCHEMA_DIR.glob("*.schema.json"):
            schema = json.loads(schema_path.read_text(encoding="utf-8"))
            uri = schema.get("$id")
            if uri:
                registry = registry.with_resource(uri, Resource.from_contents(schema))
        return registry
    
    
    def validate_against_schema(data, schema: dict) -> str | None:
        """Return the first error as '<path>: <message>' (errors sorted by path), or None when *data* is valid."""
        try:
            from jsonschema import Draft202012Validator
        except ImportError as exc:
            raise RuntimeError(_MISSING) from exc
        registry = schema_registry()
        validator = Draft202012Validator(schema, registry=registry)
        errors = sorted(validator.iter_errors(data), key=lambda e: list(e.absolute_path))
        if not errors:
            return None
        e0 = errors[0]
        path = "/".join(str(p) for p in e0.absolute_path) or "<root>"
        return f"{path}: {e0.message}"
    
  • skills/datasheets/scripts/merge_results.py13 KBf28ffae7ec0eupstream ↗
    #!/usr/bin/env python3
    """Merge per-task result files into <mpn>.json.
    
    Reads <mpn>.plan.json + per-task wrapped result files, validates each `data`
    against the declared schema (using referencing.Registry for local $ref
    resolution), splices into the canonical extraction shape, and writes <mpn>.json.
    
    Failure semantics:
    - status:"complete" + schema-valid  → splice into top-level field.
    - status:"failed" OR schema-invalid → on first failure, mark task for retry
      (no auto re-dispatch — the dispatcher recipe handles re-run); on second
      failure, partial-merge with {"_extraction_failed": true, "reason": ...}.
    
    Stdlib + jsonschema + referencing (already dev deps). Requires jsonschema and
    referencing — extraction_schema imports them when validation runs and raises
    RuntimeError when they are missing; no fallback. The script's only purpose is
    schema validation; silent-pass on missing deps is dangerous.
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import re
    import sys
    from datetime import datetime, timezone
    from pathlib import Path
    
    from extraction_schema import validate_against_schema
    
    _SKILL_PARENT = str(Path(__file__).resolve().parents[1])
    if _SKILL_PARENT not in sys.path:
        sys.path.insert(0, _SKILL_PARENT)
    
    from datasheet_types.cache_reader import sanitize_mpn  # noqa: E402
    from distributor_core.manifest import write_json_atomic  # noqa: E402
    
    
    SKILL_ROOT = Path(__file__).resolve().parents[1]          # skills/datasheets
    SCHEMA_DIR = SKILL_ROOT / "schemas"
    _LEGACY_PREFIX = "skills/datasheets/"
    
    
    def resolve_skill_resource(rel: str) -> Path:
        """Map a resource reference to a path inside this skill.
    
        Extraction plans persisted before v3 carry repo-relative strings such as
        'skills/datasheets/schemas/base.schema.json'; a copied skill still owns
        that file at SKILL_ROOT / 'schemas/base.schema.json'. Absolute paths pass
        through unchanged (custom schema locations)."""
        p = Path(rel)
        if p.is_absolute():
            return p
        s = p.as_posix()
        if s.startswith(_LEGACY_PREFIX):
            s = s[len(_LEGACY_PREFIX):]
        return SKILL_ROOT / s
    
    
    EXTRACTOR_SCHEMA_VERSION = "1.0"
    
    
    # ---------------------------------------------------------------------------
    # Helpers
    # ---------------------------------------------------------------------------
    
    
    def _now_iso() -> str:
        return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    
    
    _SAFE_TASK_ID = re.compile(r"[A-Za-z0-9_.\-]+")
    
    
    def _read_result(cache: Path, mpn: str, task_id: str) -> "dict | None":
        if not _SAFE_TASK_ID.fullmatch(task_id) or task_id in (".", ".."):
            return {
                "task_id": task_id,
                "status": "failed",
                "error": "task id is not a safe file name component",
                "data": None,
            }
        p = cache / f"{sanitize_mpn(mpn)}.{task_id}.result.json"
        if not p.exists():
            return None
        try:
            return json.loads(p.read_text(encoding="utf-8"))
        except json.JSONDecodeError as exc:
            return {
                "task_id": task_id,
                "status": "failed",
                "error": f"result file is not valid JSON: {exc}",
                "data": None,
            }
    
    
    def _classify(result: "dict | None", schema: dict,
                  pdf_sha256: "str | None" = None) -> "tuple[str, str | None]":
        """Return ('complete'|'failed', error_message). A result must name the
        PDF the plan was built from, so facts from another revision never merge."""
        if result is None:
            return "failed", "no result file found"
        if result.get("status") != "complete":
            return "failed", result.get("error") or "result file status != 'complete'"
        err = validate_against_schema(result.get("data"), schema)
        if err:
            return "failed", f"schema validation: {err}"
        bound = result.get("pdf_sha256")
        if not bound:
            return "failed", ("result has no pdf_sha256, so it cannot be tied to the planned PDF; "
                              "re-dispatch this task and copy pdf_sha256 from the plan into the wrapper")
        if bound != pdf_sha256:
            return "failed", (f"result is bound to PDF {bound}, not the planned PDF "
                              f"{pdf_sha256}; re-dispatch this task")
        return "complete", None
    
    
    def _splice_field(extraction: dict, task_id: str, role: str, data) -> None:
        """Place `data` at the right top-level field per spec.
    
        - base task data  → extraction["base"] (object).
        - pinout task data → extraction["base"]["pinout"] (list[Pin]).
          If base was already merged, the pinout task's data overwrites base.pinout.
        - category tasks  → extraction[role] (e.g. extraction["regulator"]).
        """
        if role == "base":
            # Preserve pinout if it was spliced before base (shouldn't happen, but
            # defend against task ordering variations).
            existing_pinout = (
                extraction.get("base", {}).get("pinout")
                if extraction.get("base")
                else None
            )
            extraction["base"] = data
            if existing_pinout is not None:
                extraction["base"]["pinout"] = existing_pinout
        elif role == "pinout":
            # Pinout task data is list[Pin]; nest under base.pinout.
            extraction.setdefault("base", {})["pinout"] = data
        else:
            # Category extension keyed by role (regulator, mcu, ...).
            extraction[role] = data
            cats = extraction.setdefault("categories", [])
            if role not in cats:
                cats.append(role)
    
    
    def _initial_extraction(plan: dict) -> dict:
        return {
            "schema_version": {"base": "1.0", "categories": {}},
            "source": {
                "manufacturer": "",
                "mpn": plan["mpn"],
                "datasheet_revision": None,
                "datasheet_date": None,
                "source_url": None,
                "local_path": plan["pdf_path"],
                "sha256": plan["pdf_sha256"],
                "page_count": None,
                "family_ref": None,
            },
            "extraction": {
                "extracted_at": _now_iso(),
                "extractor_scout": None,
                "extractor_schema_version": EXTRACTOR_SCHEMA_VERSION,
                "quality_score": None,
                "plan_ref": f"{sanitize_mpn(plan['mpn'])}.plan.json",
            },
            "base": {},
            "categories": [],
        }
    
    
    def _enrich_source_from_scout(extraction: dict, cache: Path, mpn: str) -> None:
        """Overlay scout metadata onto the source block. Silent on missing file."""
        p = cache / f"{sanitize_mpn(mpn)}.scout.json"
        if not p.exists():
            return
        try:
            scout = json.loads(p.read_text(encoding="utf-8"))
        except json.JSONDecodeError:
            return
        md = scout.get("metadata", {}) or {}
        src = extraction["source"]
        src["manufacturer"] = md.get("manufacturer", src["manufacturer"])
        src["datasheet_revision"] = md.get("datasheet_revision", src["datasheet_revision"])
        src["datasheet_date"] = md.get("datasheet_date", src["datasheet_date"])
        src["source_url"] = md.get("source_url", src["source_url"])
        src["page_count"] = md.get("page_count", src["page_count"])
    
    
    def _record_outcome(
        plan: dict,
        task_id: str,
        attempts: int,
        status: str,
        err: "str | None",
    ) -> None:
        outcomes = plan["execution"]["outcomes"]
        for o in outcomes:
            if o["task_id"] == task_id:
                o["attempts"] = attempts
                o["final_status"] = status
                o["last_error"] = err
                return
        outcomes.append(
            {
                "task_id": task_id,
                "attempts": attempts,
                "final_status": status,
                "last_error": err,
            }
        )
    
    
    # ---------------------------------------------------------------------------
    # Core merge logic
    # ---------------------------------------------------------------------------
    
    
    def merge(cache: Path, mpn: str, *, retry_failed: bool = False) -> int:
        plan_path = cache / f"{sanitize_mpn(mpn)}.plan.json"
        if not plan_path.exists():
            print(f"error: plan not found at {plan_path}", file=sys.stderr)
            return 2
        plan = json.loads(plan_path.read_text(encoding="utf-8"))
        if plan["execution"]["started_at"] is None:
            plan["execution"]["started_at"] = _now_iso()
    
        extraction = _initial_extraction(plan)
        _enrich_source_from_scout(extraction, cache, mpn)
    
        failed_tasks: list[tuple[dict, str]] = []
        schema_versions: dict[str, str] = {}
    
        for task in plan["tasks"]:
            tid = task["task_id"]
            role = task["subagent_role"]
            schema_path = resolve_skill_resource(task["schema"])
            schema = json.loads(schema_path.read_text(encoding="utf-8"))
            result = _read_result(cache, mpn, tid)
            status, err = _classify(result, schema, plan.get("pdf_sha256"))
    
            # Track attempt count — start from any prior recorded attempts.
            prior = next(
                (o for o in plan["execution"]["outcomes"] if o["task_id"] == tid), None
            )
            attempts = (prior["attempts"] if prior else 0) + 1
    
            if status == "complete":
                _splice_field(extraction, tid, role, result["data"])
                task["status"] = "complete"
                task["result_ref"] = f"{sanitize_mpn(mpn)}.{tid}.result.json"
                _record_outcome(plan, tid, attempts, "complete", None)
                sv = result.get("schema_version")
                if sv and role not in ("base", "pinout"):
                    schema_versions[role] = sv
            else:
                failed_tasks.append((task, err or "unknown failure"))
                task["status"] = "failed"
                _record_outcome(plan, tid, attempts, "failed", err)
    
        if schema_versions:
            extraction["schema_version"]["categories"].update(schema_versions)
    
        # Failure/retry policy.
        if failed_tasks and not retry_failed:
            # First merge with failures: write plan state, exit nonzero.
            # Caller re-dispatches failed tasks, then re-runs with --retry-failed.
            write_json_atomic(plan_path, plan, sort_keys=True)
            for task, err in failed_tasks:
                print(f"failed: {task['task_id']}: {err}", file=sys.stderr)
            print(
                "re-dispatch failed tasks per dispatcher recipe, "
                "then re-run with --retry-failed",
                file=sys.stderr,
            )
            return 1
    
        if failed_tasks and retry_failed:
            # Second failure after retry: partial-merge with sentinel.
            for task, err in failed_tasks:
                tid = task["task_id"]
                role = task["subagent_role"]
                sentinel = {"_extraction_failed": True, "reason": err}
                if role == "base":
                    extraction["base"] = sentinel
                elif role == "pinout":
                    extraction.setdefault("base", {})["pinout"] = sentinel
                else:
                    extraction[role] = sentinel
                    cats = extraction.setdefault("categories", [])
                    if role not in cats:
                        cats.append(role)
                _record_outcome(plan, tid, 2, "partial", err)
    
        try:
            sys.path.insert(0, str(Path(__file__).resolve().parent))
            from datasheet_score import score_v14_extraction
            extraction["extraction"]["quality_score"] = score_v14_extraction(extraction)["score"]
        except Exception as exc:  # noqa: BLE001
            print(f"warning: quality scoring failed: {exc}", file=sys.stderr)
            extraction["extraction"]["quality_score"] = None
        plan["execution"]["completed_at"] = _now_iso()
    
        out_path = cache / f"{sanitize_mpn(mpn)}.json"
        write_json_atomic(out_path, extraction, sort_keys=True)
        write_json_atomic(plan_path, plan, sort_keys=True)
        print(f"{mpn}: merged → {out_path}")
        return 0
    
    
    # ---------------------------------------------------------------------------
    # CLI
    # ---------------------------------------------------------------------------
    
    
    def main(argv: "list[str] | None" = None) -> int:
        ap = argparse.ArgumentParser(
            description="Merge per-task result files into <mpn>.json."
        )
        ap.add_argument("mpn")
        ap.add_argument(
            "--cache-dir",
            type=Path,
            default=Path("datasheets/extracted"),
        )
        ap.add_argument(
            "--retry-failed",
            action="store_true",
            help=(
                "Treat this as the post-retry merge — "
                "partial-merge any still-failing tasks."
            ),
        )
        args = ap.parse_args(argv)
        return merge(args.cache_dir, args.mpn, retry_failed=args.retry_failed)
    
    
    if __name__ == "__main__":
        for _stream in (sys.stdout, sys.stderr):  # UTF-8 on a cp1252 pipe (Windows)
            if (getattr(_stream, "encoding", "") or "").lower().replace("-", "") != "utf8" \
                    and hasattr(_stream, "reconfigure"):
                _stream.reconfigure(encoding="utf-8", errors="backslashreplace")
        sys.exit(main())
    
  • skills/datasheets/scripts/plan_extraction.py9.5 KB87ef94d302bbupstream ↗
    #!/usr/bin/env python3
    """Plan an extraction run.
    
    Computes PDF SHA, dispatches scout (or reads cached scout), generates
    <mpn>.plan.json. Stdlib-only. The dispatcher (separate concern) reads the
    plan and executes per-task subagents per dispatcher-contract.md.
    """
    
    from __future__ import annotations
    
    import argparse
    import hashlib
    import json
    import sys
    from datetime import datetime, timezone
    from pathlib import Path
    
    _SKILL_ROOT = str(Path(__file__).resolve().parent.parent)
    if _SKILL_ROOT not in sys.path:
        sys.path.insert(0, _SKILL_ROOT)
    
    from datasheet_types.cache_reader import read_json_object, sanitize_mpn  # noqa: E402
    from distributor_core.manifest import write_json_atomic  # noqa: E402
    from merge_results import _SAFE_TASK_ID  # noqa: E402
    
    PLAN_VERSION = "1.0"
    
    # task_id → (subagent_role, tier, schema_path, prompt_path) relative to repo root
    ALWAYS_TASKS = {
        "base":   ("base",   "B", "skills/datasheets/schemas/base.schema.json",
                   "skills/datasheets/prompts/base.md"),
        "pinout": ("pinout", "A", "skills/datasheets/schemas/pinout.schema.json",
                   "skills/datasheets/prompts/pinout.md"),
    }
    CATEGORY_TASKS = {
        "regulator":  ("regulator",  "B", "skills/datasheets/schemas/regulator.schema.json",
                       "skills/datasheets/prompts/regulator.md"),
        "diode":      ("diode",      "B", "skills/datasheets/schemas/diode.schema.json",
                       "skills/datasheets/prompts/diode.md"),
        "transistor": ("transistor", "B", "skills/datasheets/schemas/transistor.schema.json",
                       "skills/datasheets/prompts/transistor.md"),
        "opamp":      ("opamp",      "B", "skills/datasheets/schemas/opamp.schema.json",
                       "skills/datasheets/prompts/opamp.md"),
        "mcu":        ("mcu",        "B", "skills/datasheets/schemas/mcu.schema.json",
                       "skills/datasheets/prompts/mcu.md"),
        "crystal":    ("crystal",    "B", "skills/datasheets/schemas/crystal.schema.json",
                       "skills/datasheets/prompts/crystal.md"),
    }
    
    
    def _sha256(path: Path) -> str:
        h = hashlib.sha256()
        with path.open("rb") as f:
            for chunk in iter(lambda: f.read(65536), b""):
                h.update(chunk)
        return "sha256:" + h.hexdigest()
    
    
    def _now_iso() -> str:
        return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    
    
    def _load_existing_extraction(cache_dir: Path, mpn: str) -> dict | None:
        p = cache_dir / f"{sanitize_mpn(mpn)}.json"
        if not p.exists():
            return None
        try:
            return json.loads(p.read_text(encoding="utf-8"))
        except json.JSONDecodeError:
            return None
    
    
    def _is_cache_current(existing: dict, pdf_sha: str) -> bool:
        """Cache is current iff the recorded SHA matches the PDF's SHA."""
        src_sha = (existing.get("source") or {}).get("sha256")
        return src_sha == pdf_sha
    
    
    def _drop_stale_results(cache_dir: Path, mpn: str, pdf_sha: str) -> None:
        """Remove the previous plan's task results when the PDF changed, so a
        re-dispatch cannot reuse facts read from another revision."""
        old = read_json_object(cache_dir / f"{sanitize_mpn(mpn)}.plan.json")
        if not old or old.get("pdf_sha256") == pdf_sha:
            return
        removed = 0
        for task in old.get("tasks") or []:
            tid = task.get("task_id") if isinstance(task, dict) else None
            if not isinstance(tid, str) or tid in (".", "..") or not _SAFE_TASK_ID.fullmatch(tid):
                continue
            p = cache_dir / f"{sanitize_mpn(mpn)}.{tid}.result.json"
            if p.parent == cache_dir and (p.is_file() or p.is_symlink()):
                p.unlink()
                removed += 1
        print(f"{mpn}: PDF changed since the previous plan; removed {removed} stale task result(s)",
              file=sys.stderr)
    
    
    def _load_scout(cache_dir: Path, mpn: str) -> dict:
        p = cache_dir / f"{sanitize_mpn(mpn)}.scout.json"
        if not p.exists():
            raise FileNotFoundError(
                f"Cached scout not found at {p}. Run scout subagent and write its data here, "
                f"or invoke without --use-cached-scout (live scout requires a dispatcher)."
            )
        return json.loads(p.read_text(encoding="utf-8"))
    
    
    def _build_plan(mpn: str, pdf_path: Path, pdf_sha: str, cache_dir: Path, scout: dict) -> dict:
        tasks: list[dict] = []
        for tid, (role, tier, schema, prompt) in ALWAYS_TASKS.items():
            tasks.append({
                "task_id": tid,
                "subagent_role": role,
                "tier": tier,
                "schema": schema,
                "prompt_template": prompt,
                "pages": list(scout["extraction_pages"].get(tid, [])),
                "depends_on": [],
                "status": "pending",
                "result_ref": None,
            })
        for cat in scout.get("categories", []):
            if cat not in CATEGORY_TASKS:
                print(
                    f"warning: unsupported category {cat!r} in scout output "
                    f"(not yet registered in CATEGORY_TASKS — supported: "
                    f"{sorted(CATEGORY_TASKS)}); skipping",
                    file=sys.stderr,
                )
                continue
            role, tier, schema, prompt = CATEGORY_TASKS[cat]
            tasks.append({
                "task_id": cat,
                "subagent_role": role,
                "tier": tier,
                "schema": schema,
                "prompt_template": prompt,
                "pages": list(scout["extraction_pages"].get(cat, [])),
                "depends_on": [],
                "status": "pending",
                "result_ref": None,
            })
        return {
            "plan_version": PLAN_VERSION,
            "mpn": mpn,
            "pdf_path": str(pdf_path),
            "pdf_sha256": pdf_sha,
            "cache_dir": str(cache_dir),
            "created_at": _now_iso(),
            "scout_ref": f"{sanitize_mpn(mpn)}.scout.json",
            "tasks": tasks,
            "execution": {"started_at": None, "completed_at": None, "outcomes": []},
        }
    
    
    def _build_skip_plan(mpn: str, pdf_path: Path, pdf_sha: str, cache_dir: Path) -> dict:
        return {
            "plan_version": PLAN_VERSION,
            "mpn": mpn,
            "pdf_path": str(pdf_path),
            "pdf_sha256": pdf_sha,
            "cache_dir": str(cache_dir),
            "created_at": _now_iso(),
            "scout_ref": f"{sanitize_mpn(mpn)}.scout.json",
            "tasks": [],
            "execution": {"started_at": None, "completed_at": None, "outcomes": []},
        }
    
    
    def main(argv: list[str] | None = None) -> int:
        ap = argparse.ArgumentParser(description="Plan an extraction run.")
        ap.add_argument("mpn")
        ap.add_argument("pdf_path", type=Path)
        ap.add_argument("--cache-dir", type=Path, default=Path("datasheets/extracted"))
        ap.add_argument("--force", action="store_true",
                        help="Regenerate plan even if cache is current.")
        ap.add_argument("--use-cached-scout", action="store_true",
                        help="Read <mpn>.scout.json from cache_dir instead of dispatching scout. "
                             "The stdlib path requires this; live scout dispatch is the dispatcher's job.")
        args = ap.parse_args(argv)
    
        if not args.pdf_path.exists():
            print(f"error: PDF not found: {args.pdf_path}", file=sys.stderr)
            return 2
        args.cache_dir.mkdir(parents=True, exist_ok=True)
    
        pdf_sha = _sha256(args.pdf_path)
        existing = _load_existing_extraction(args.cache_dir, args.mpn)
        if existing and _is_cache_current(existing, pdf_sha) and not args.force:
            print(f"{args.mpn}: already up-to-date (PDF SHA matches cache)")
            return 0
    
        if not args.use_cached_scout:
            print(
                "error: live scout dispatch is not implemented in plan_extraction.py. "
                "Run the scout subagent via dispatch-claude-code.md (or any dispatcher), "
                "write its output to <cache>/<mpn>.scout.json, then re-invoke with --use-cached-scout.",
                file=sys.stderr,
            )
            return 2
    
        scout = _load_scout(args.cache_dir, args.mpn)
        scout_sha = scout.get("pdf_sha256")
        if scout_sha is None:
            print(f"warning: cached scout for {args.mpn} has no PDF SHA; re-scout if the PDF changed",
                  file=sys.stderr)
        elif scout_sha != pdf_sha:
            print(
                f"error: the cached scout for {args.mpn} was not made from this PDF "
                f"(scout pdf_sha256 {scout_sha!r}, PDF {pdf_sha}); "
                "re-run the scout on this PDF and write its pdf_sha256 into the scout file.",
                file=sys.stderr,
            )
            return 2
        _drop_stale_results(args.cache_dir, args.mpn, pdf_sha)
    
        verdict = (scout.get("quality_verdict") or {}).get("verdict")
        plan_path = args.cache_dir / f"{sanitize_mpn(args.mpn)}.plan.json"
        if verdict == "skip":
            plan = _build_skip_plan(args.mpn, args.pdf_path, pdf_sha, args.cache_dir)
            write_json_atomic(plan_path, plan)
            reason = (scout.get("quality_verdict") or {}).get("reason")
            print(
                f"{args.mpn}: scout verdict 'skip' ({reason}); empty plan written, no extraction dispatched",
                file=sys.stderr,
            )
            return 1
    
        plan = _build_plan(args.mpn, args.pdf_path, pdf_sha, args.cache_dir, scout)
        write_json_atomic(plan_path, plan)
        print(f"{args.mpn}: plan written to {plan_path} ({len(plan['tasks'])} tasks)")
        print(f"  next: run dispatcher per skills/datasheets/references/dispatch-claude-code.md")
        return 0
    
    
    if __name__ == "__main__":
        for _stream in (sys.stdout, sys.stderr):  # UTF-8 on a cp1252 pipe (Windows)
            if (getattr(_stream, "encoding", "") or "").lower().replace("-", "") != "utf8" \
                    and hasattr(_stream, "reconfigure"):
                _stream.reconfigure(encoding="utf-8", errors="backslashreplace")
        sys.exit(main())
    
  • skills/datasheets/scripts/validate_extraction_result.py3.9 KB824228f7bb27upstream ↗
    #!/usr/bin/env python3
    """Validate a single per-task extraction result against its schema.
    
    Wraps the inline per-task validator that `merge_results.py` runs at extract
    time, exposed as a standalone CLI for the harness 4-check acceptance gate
    (Check 1).
    
    Input: a wrapped result file
        {task_id, schema_version, status, extracted_at, model_tier, model_id, data}
    
    The `data` field is validated against `skills/datasheets/schemas/<task-type>.schema.json`.
    
    Exit codes:
        0  data validates and status == "complete"
        1  schema validation failed, status != "complete", or unreadable result
        2  CLI / I/O error (missing schema, missing file, bad JSON in result wrapper)
    
    Designed for `--task-type` to be any task name backed by a sibling
    `<name>.schema.json` — the mcu/opamp/transistor/diode/crystal tasks work
    without code changes.
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import sys
    from pathlib import Path
    
    from extraction_schema import validate_against_schema
    
    
    SCHEMA_DIR = Path(__file__).resolve().parents[1] / "schemas"
    
    
    def _resolve_schema(task_type: str) -> Path:
        return SCHEMA_DIR / f"{task_type}.schema.json"
    
    
    def validate_result(result_file: Path, task_type: str) -> "tuple[int, str]":
        """Return (exit_code, message). exit_code semantics match module docstring."""
        schema_path = _resolve_schema(task_type)
        if not schema_path.exists():
            return 2, f"schema not found for task-type {task_type!r}: {schema_path}"
    
        if not result_file.exists():
            return 2, f"result file not found: {result_file}"
    
        try:
            wrapper = json.loads(result_file.read_text(encoding="utf-8"))
        except json.JSONDecodeError as exc:
            return 2, f"result file is not valid JSON: {exc}"
    
        status = wrapper.get("status")
        if status != "complete":
            err = wrapper.get("error") or "no error message provided"
            return 1, f"result status is {status!r} (expected 'complete'): {err}"
    
        if "data" not in wrapper:
            return 1, "result wrapper has no 'data' field"
    
        schema = json.loads(schema_path.read_text(encoding="utf-8"))
        err = validate_against_schema(wrapper["data"], schema)
        if err:
            return 1, f"schema validation: {err}"
    
        return 0, f"valid: {result_file.name} matches {schema_path.name}"
    
    
    def main(argv: "list[str] | None" = None) -> int:
        ap = argparse.ArgumentParser(
            description=(
                "Validate a single per-task extraction result against its declared schema. "
                "Used by the harness 4-check gate (Check 1)."
            ),
        )
        ap.add_argument(
            "--result-file",
            type=Path,
            required=True,
            help="path to <mpn>.<task_id>.result.json",
        )
        ap.add_argument(
            "--task-type",
            required=True,
            help="task name; resolves to skills/datasheets/schemas/<task-type>.schema.json",
        )
        ap.add_argument(
            "--json",
            dest="emit_json",
            action="store_true",
            help="emit a JSON summary to stdout instead of human text",
        )
        args = ap.parse_args(argv)
    
        code, message = validate_result(args.result_file, args.task_type)
    
        if args.emit_json:
            json.dump(
                {
                    "valid": code == 0,
                    "exit_code": code,
                    "message": message,
                    "result_file": str(args.result_file),
                    "task_type": args.task_type,
                },
                sys.stdout,
                indent=2,
            )
            sys.stdout.write("\n")
        else:
            stream = sys.stdout if code == 0 else sys.stderr
            print(message, file=stream)
    
        return code
    
    
    if __name__ == "__main__":
        for _stream in (sys.stdout, sys.stderr):  # UTF-8 on a cp1252 pipe (Windows)
            if (getattr(_stream, "encoding", "") or "").lower().replace("-", "") != "utf8" \
                    and hasattr(_stream, "reconfigure"):
                _stream.reconfigure(encoding="utf-8", errors="backslashreplace")
        sys.exit(main())
    
  • skills/datasheets/scripts/validate_sanity_vector.py6.2 KBad13484249bcupstream ↗
    #!/usr/bin/env python3
    """Diff a harness sanity vector against an extraction JSON.
    
    Used by the harness 4-check acceptance gate (check #4). Pure function: no
    LLM, no state, no network. Loads YAML vector, walks per-field expected
    values, resolves dotted paths against extraction JSON, compares within
    declared tolerance.
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import sys
    from pathlib import Path
    
    
    DEFAULT_TOLERANCE_PCT = 5.0
    
    
    def _resolve_path(extraction: dict, dotted: str):
        """Resolve a dotted path. Each segment is a dict key.
    
        Lists at the terminal position are unwrapped via `_spec_value_pick`;
        intermediate list indices are NOT supported (e.g., 'regulator.vin_range[0].max').
        Only single-condition SpecValue lists are supported; multi-condition
        indexing is not.
        """
        cur = extraction
        for part in dotted.split("."):
            if not isinstance(cur, dict) or part not in cur:
                return None
            cur = cur[part]
        return cur
    
    
    def _spec_value_pick(value, key: str):
        """SpecValue lists carry [{min, typ, max, unit, ...}]. Scalars are used as-is.
    
        For list values, we pick the first element's `key` (single-condition lists only).
        For scalar values (int/float/bool/str), only the `typ` key has meaning — the
        scalar IS the typical value, with no min/max/unit. This lets sanity vectors
        write `expected: {typ: 5}` for plain scalar fields like `package.pin_count`.
        """
        if isinstance(value, list) and value:
            return value[0].get(key) if isinstance(value[0], dict) else None
        if isinstance(value, dict):
            return value.get(key)
        if isinstance(value, (int, float, bool, str)):
            return value if key == "typ" else None
        return None
    
    
    def _within_tolerance(actual: float, expected: float, tol_pct: float) -> bool:
        if expected == 0:
            return abs(actual) <= tol_pct / 100.0
        return abs((actual - expected) / expected) * 100.0 <= tol_pct
    
    
    def _check_numeric_spec(actual_value, expected: dict, tol_pct: float) -> tuple[bool, str | None, dict]:
        """`expected` like {min: 4.5, typ: null, max: 40, unit: 'V'}.
        Each populated key (min/typ/max) must match within tolerance.
        Unit must match exactly (string compare).
        Returns (pass, reason, debug_dict).
        """
        debug = {"checked": [], "actual": {}}
        if actual_value is None:
            return False, "missing in extraction", debug
    
        if "unit" in expected:
            actual_unit = _spec_value_pick(actual_value, "unit")
            debug["actual"]["unit"] = actual_unit
            if actual_unit != expected["unit"]:
                return False, f"unit mismatch: expected {expected['unit']!r}, got {actual_unit!r}", debug
    
        for key in ("min", "typ", "max"):
            if key not in expected or expected[key] is None:
                continue
            actual_n = _spec_value_pick(actual_value, key)
            debug["actual"][key] = actual_n
            if actual_n is None:
                return False, f"{key} missing", debug
            try:
                actual_f = float(actual_n)
                exp_f = float(expected[key])
            except (TypeError, ValueError):
                return False, f"{key} not numeric (actual={actual_n!r}, expected={expected[key]!r})", debug
            if not _within_tolerance(actual_f, exp_f, tol_pct):
                return False, f"{key}={actual_f} outside ±{tol_pct}% of {exp_f}", debug
            debug["checked"].append(key)
        return True, None, debug
    
    
    def _check_enum(actual_value, expected_enum) -> tuple[bool, str | None]:
        """`expected_enum` is either a scalar string or a list of strings."""
        allowed = expected_enum if isinstance(expected_enum, list) else [expected_enum]
        if actual_value in allowed:
            return True, None
        return False, f"expected one of {allowed}, got {actual_value!r}"
    
    
    def diff(vector: dict, extraction: dict) -> dict:
        fields_out = []
        for f in vector.get("fields", []):
            path = f["path"]
            actual = _resolve_path(extraction, path)
            entry = {"path": path, "expected": None, "actual": None, "pass": False, "reason": None}
            if "expected_enum" in f:
                entry["expected"] = f["expected_enum"]
                entry["actual"] = actual if not isinstance(actual, dict) else _spec_value_pick(actual, "value")
                if actual is None:
                    entry["pass"] = False
                    entry["reason"] = "missing in extraction"
                else:
                    ok, why = _check_enum(actual, f["expected_enum"])
                    entry["pass"] = ok
                    entry["reason"] = why
            elif "expected" in f:
                tol = float(f.get("tolerance_pct", DEFAULT_TOLERANCE_PCT))
                entry["expected"] = f["expected"]
                ok, why, dbg = _check_numeric_spec(actual, f["expected"], tol)
                entry["pass"] = ok
                entry["reason"] = why
                entry["actual"] = dbg["actual"]
                entry["tolerance_pct"] = tol
            else:
                entry["reason"] = "vector field has neither expected nor expected_enum"
            if "page" in f:
                entry["expected_page"] = f["page"]
            fields_out.append(entry)
        passed = sum(1 for x in fields_out if x["pass"])
        failed = len(fields_out) - passed
        return {
            "schema_version": "1.0",
            "mpn": vector.get("mpn"),
            "fields": fields_out,
            "summary": {"total": len(fields_out), "passed": passed, "failed": failed},
        }
    
    
    def main(argv: list[str] | None = None) -> int:
        ap = argparse.ArgumentParser(description="Diff a sanity vector against an extraction JSON.")
        ap.add_argument("vector_path", type=Path)
        ap.add_argument("extraction_path", type=Path)
        args = ap.parse_args(argv)
    
        import yaml
        vector = yaml.safe_load(args.vector_path.read_text(encoding="utf-8"))
        extraction = json.loads(args.extraction_path.read_text(encoding="utf-8"))
        report = diff(vector, extraction)
        print(json.dumps(report, indent=2, default=str))
        return 0 if report["summary"]["failed"] == 0 else 1
    
    
    if __name__ == "__main__":
        for _stream in (sys.stdout, sys.stderr):  # UTF-8 on a cp1252 pipe (Windows)
            if (getattr(_stream, "encoding", "") or "").lower().replace("-", "") != "utf8" \
                    and hasattr(_stream, "reconfigure"):
                _stream.reconfigure(encoding="utf-8", errors="backslashreplace")
        sys.exit(main())
    
  • skills/datasheets/references/cache-layout.md8.2 KB9c96869f18f5upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/references/consumer-api.md11 KB01cf7057546dupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/references/dispatch-claude-code.md9.4 KBecffaa2bf8c5upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/references/dispatcher-contract.md8.2 KBca1f130f7985upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/references/extraction-pipeline.md1.7 KB7a79422a6971upstream ↗
    # Extraction Pipeline — Full Procedure
    
    Full extraction-pipeline procedure. The SKILL.md main flow points here; scripts do the orchestration — this is depth-on-demand, not a required read.
    
    ---
    
    ## Extraction workflow
    
    **v1.4 pipeline (current, used for all new extractions):**
    1. A scout subagent reads the PDF (TOC, headings) for the requested MPN; its schema-valid, unwrapped output, with the PDF's `pdf_sha256` added, is saved as `<cache_dir>/<sanitized_mpn>.scout.json` (the MPN with every character outside `[A-Za-z0-9_.-]` replaced by `_`: `LM317/NOPB` → `LM317_NOPB`).
    2. `plan_extraction.py <mpn> <pdf_path> --use-cached-scout --cache-dir <cache_dir>` builds the orchestration plan JSON. A scout `skip` verdict writes an empty plan and exits 1; stop extraction for that part.
    3. Category extractor prompts (base, pinout, regulator, …) run per `dispatch-claude-code.md` and `dispatcher-contract.md`.
    4. `merge_results.py` validates per-task result files against schemas and merges into `<project>/datasheets/extracted/<sanitized_mpn>.json`.
    5. Three-dimension quality score lives at `facts.extraction.quality_score`.
    6. Consumers query via `lookup(mpn, cache_dir=…)` or via the v1.3 compat helpers in `datasheet_features.py`.
    
    **v1.3 legacy pipeline (read-only in v1.4):**
    1. User runs an analyzer or requests extraction.
    2. Skill checks the cache (`<project>/datasheets/extracted/<sanitized_mpn>.json`).
    3. On cache miss / stale / low score: Claude reads selected PDF pages and extracts structured data.
    4. Extraction is scored; if score ≥ 6.0, cached.
    5. Consumers query via `datasheet_features.py`.
    
    For dispatcher dispatch recipes and subagent recipes, see `references/dispatch-claude-code.md` and `references/dispatcher-contract.md`.
    
  • skills/datasheets/references/extraction-schema.md18 KB7c65fad30372upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/references/field-extraction-guide.md15 KB786119af4423upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/references/quality-scoring.md8.4 KB6e275b32649eupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/datasheet_types/__init__.py1.9 KB23284a4fbdb6upstream ↗
    """Typed Python access layer for datasheet v2 extraction.
    
    Mirrors the JSON schemas under skills/datasheets/schemas/.
    Dataclasses are ergonomic access; the JSON Schema remains source of truth.
    
    Public API:
        DatasheetFacts — top-level per-MPN fact envelope
        SpecValue, Evidence — atomic primitives
        Pin, AltFunction, Pinout — pinout types
        BaseBlock, Package, ComplianceMark, PinRelationship — base block types
        Regulator, StabilityConditions, Sequencing — regulator category
        lookup — resolve MPN to DatasheetFacts (cache_reader)
        best, trusted, has_data — trust-gating helpers
    
    Consumers import:
        from datasheet_types.extraction import DatasheetFacts
        from datasheet_types.codec import from_dict, to_dict
        facts = from_dict(DatasheetFacts, json.load(open('cache/LM2596-ADJ.json')))
    
        # Trust-gating pattern:
        from datasheet_types import best, trusted, has_data, lookup
        ds = lookup("LM2596-ADJ", cache_dir=...)
        if has_data(ds.regulator.vin_range):
            v = best(ds.regulator.vin_range, min_confidence="medium")
    """
    from .spec_value import SpecValue, Evidence
    from .pinout import Pin, AltFunction, Pinout
    from .base_block import (
        BaseBlock,
        Package,
        BodyMm,
        MoistureSensitivity,
        ComplianceMark,
        PinRelationship,
    )
    from .regulator import Regulator, StabilityConditions, Sequencing
    from .extraction import DatasheetFacts, Source, ExtractionMeta, SchemaVersion
    from .trust_gating import best, trusted, has_data
    from .cache_reader import lookup
    
    __all__ = [
        "DatasheetFacts",
        "SchemaVersion",
        "Source",
        "ExtractionMeta",
        "BaseBlock",
        "Package",
        "BodyMm",
        "MoistureSensitivity",
        "ComplianceMark",
        "PinRelationship",
        "Pinout",
        "Pin",
        "AltFunction",
        "SpecValue",
        "Evidence",
        "Regulator",
        "StabilityConditions",
        "Sequencing",
        "lookup",
        "best",
        "trusted",
        "has_data",
    ]
    
    
  • skills/datasheets/datasheet_types/base_block.py6.5 KB23feba4ec159upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/datasheet_types/cache_reader.py12 KB93c9f3301581upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/datasheet_types/codec.py6.3 KB8aab158088acupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/datasheet_types/extraction.py8.0 KB6e548a074e1eupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/datasheet_types/pinout.py5.2 KB3d1c9b55e268upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/datasheet_types/regulator.py5.3 KB501c2be0883dupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/datasheet_types/spec_value.py2.4 KB80448bf62901upstream ↗
    """SpecValue + Evidence dataclasses — the atomic primitives.
    
    Mirrors spec_value.schema.json. Every numeric datasheet fact in the
    v2 cache is a list of SpecValue; every SpecValue carries an Evidence
    block (page + confidence + method).
    """
    from __future__ import annotations
    
    from dataclasses import dataclass, field
    from typing import Optional
    
    
    @dataclass
    class Evidence:
        """Provenance for a single fact.
    
        Mirrors the inline evidence block shape used across every schema.
        """
        page: int = field(metadata={
            "description": "PDF page number (1-based) where the fact was sourced."})
        confidence: str = field(metadata={
            "description": "'high' | 'medium' | 'low' — extractor confidence. Pivots trust gating."})
        method: str = field(metadata={
            "description": "'table' | 'prose' | 'curve' | 'calculated' | 'derived'."})
        section: Optional[str] = field(default=None, metadata={
            "description": "Datasheet section label when available."})
    
    
    @dataclass
    class SpecValue:
        """One electrical/physical specification value with provenance.
    
        Numeric fields (min/typ/max) are all Optional[float] — the datasheet
        may publish any subset. unit is always populated (canonical SI).
        evidence is always populated (no fact without provenance).
        """
        unit: str = field(metadata={
            "description": "Canonical SI unit (V, A, s, Hz, Ω, F, H, K, W, °C, %, ppm, K/W, °C/W). "
                           "Extraction converts prefixed units (mV, kHz, µF, ...) before caching."})
        evidence: Evidence = field(metadata={
            "description": "Provenance for this value. Never null — every fact carries evidence."})
        min: Optional[float] = field(default=None, metadata={
            "description": "Minimum published value. None when datasheet doesn't specify."})
        typ: Optional[float] = field(default=None, metadata={
            "description": "Typical published value. None when datasheet doesn't specify."})
        max: Optional[float] = field(default=None, metadata={
            "description": "Maximum published value. None when datasheet doesn't specify."})
        condition: Optional[str] = field(default=None, metadata={
            "description": "Test conditions (e.g. 'VIN=5V, IL=100mA, TJ=25°C')."})
        notes: Optional[str] = field(default=None, metadata={
            "description": "Additional notes or original symbolic form if resolved at extraction."})
    
  • skills/datasheets/datasheet_types/trust_gating.py4.1 KB6b6a6c6694b3upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/distributor_core/__init__.py0 Be69de29bb2d1upstream ↗
  • skills/datasheets/distributor_core/cli.py2.9 KBd2d72219b1ffupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/distributor_core/download.py25 KB7e6b8b6f4fd7upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/distributor_core/inputs.py8.9 KBa186bdb7c849upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/distributor_core/manifest.py5.7 KB036de67d47d5upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/distributor_core/sync.py20 KB00779c4310ccupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/distributor_core/verify.py12 KBa19911e7ee5eupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/examples/abm8g-106-12.000mhz-t.json9.4 KBebc5803b64afupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/examples/irlml6344.json16 KB4bc517b75d1cupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/examples/lm2596-adj.json14 KB0eb22ee96258upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/examples/lm358.json16 KB757b03fa9962upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/examples/mbrs540t3g.json12 KB30ef563d9124upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/examples/stm32f103c8t6.json51 KB1f05f1114eb3upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/base.md4.5 KB1e8ec7c322dbupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/crystal.md8.8 KB940af69b64b0upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/diode.md8.3 KBf147d81be51bupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/mcu.md9.6 KBc7be25460f43upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/opamp.md11 KB436df12319fcupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/pinout.md4.7 KB6ee035653e34upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/regulator.md4.4 KB324ffb098c85upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/scout.md3.5 KB652e2a0ae5ccupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/prompts/transistor.md13 KB151887b7cb77upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/base.schema.json9.3 KB5e1d739f272bupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/CHANGELOG.md16 KB6a4244121faaupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/crystal.schema.json7.2 KB32b42883d4d9upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/diode.schema.json7.9 KBea2f85d4f3deupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/extraction.schema.json3.7 KBa0bd0a366cbfupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/manifest.schema.json3.0 KBb65f6cede56eupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/mcu.schema.json12 KB3b98f2ca02c7upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/opamp.schema.json9.2 KB9d481195e572upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/pinout.schema.json5.9 KB1388982f6b4cupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/plan.schema.json4.6 KB8268c5d9a947upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/regulator.schema.json8.4 KBea56da123adeupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/scout.schema.json2.9 KBf46d370d1f31upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/spec_value.schema.json3.0 KB22c02370c6d9upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/datasheets/schemas/transistor.schema.json12 KBefe1444baf56upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/aklofas-kicad-happy-skills-datasheets.md

// REPO STATS

1.7k stars
6 open issues
Last commit: 9d ago