# pid-to-kg

Adopt this skill: https://proskills.md/skills/other/scottduncanai-pid-to-knowledge-graph-skills-pid-to-kg

Page: https://proskills.md/skills/other/scottduncanai-pid-to-knowledge-graph-skills-pid-to-kg
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: skills/pid-to-kg/SKILL.md
size: 9519
sha: 352f8b8fd5f2acd407f654f054312d7c75f2b888
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/SKILL.md

````text
---
name: pid-to-kg
description: Converts P&IDs (piping and instrumentation diagrams) and PFDs in PDF form, either vector or scanned, into a process knowledge graph covering equipment connectivity (upstream/downstream), instrument locations and measured variables, and control loops. Output is JSON plus a DuckDB file plus an LLM-ready Markdown summary. Use when the user wants to digitize, extract, analyze or build a knowledge graph or context layer from P&IDs, or asks what is upstream/downstream of a piece of equipment, which sensors measure what, or how a plant's control loops are structured.
---

# P&ID to knowledge graph

Work is split between deterministic scripts and you:

- **Scripts** classify pages (vector vs scanned), render them into readable tiles, collect OCR/text hints, then build, check, store and query the graph.
- **You** read the drawings and write one structured extraction file per sheet. Symbol recognition and line tracing are visual reasoning, and that is the part scripts cannot do well.

All scripts are in `scripts/` next to this file. Call them by their full path (the commands below abbreviate it as `scripts/`), and keep the work directory in the user's current folder, e.g. `work/<plant>`.

**Setup.** Many users are engineers, not programmers, so handle setup yourself. The scripts need Python 3.10+. Before the first run, install the dependencies with `pip install -r <skill dir>/scripts/requirements.txt` (use a virtual environment if the user's Python is shared with other work). If Python is missing, tell the user to install it from python.org and stop there.

## Ground rules

- **Only the P&ID pages.** Drawings often sit inside a long permit or design report. If the user names the pages, pass them with `--pages`. If not, and the PDF has more than a few pages, probe it, look at the page overviews or ask the user, then re-run the probe with `--pages` so later steps skip reports, legends and non-P&ID drawings.
- **One plant, one work directory, one graph.** Never put sheets from different plants in the same work directory: tags would merge across unrelated plants. If the user supplies several plants, run the whole workflow separately for each.
- **One configuration per graph.** If the PDF contains two revisions of the same drawing (same title or area, overlapping tags, e.g. an as-built and an older draft), do not merge them. Ask the user which revision to use, extract only that one, and record the excluded pages and the reason in the report. QA flags likely duplicates as `possible_duplicate_revision`. Record each sheet's `issue_status` from the title block or stamps.
- **Record only what is drawn.** Never invent tags, connections or loops to make the graph look complete. Anything unclear goes in with a lower `confidence`, or into `unreadable`.
- **The QA loop is not optional.** Rebuild until there are no errors, and re-check every warning on the drawing.

## Workflow

Copy this checklist into your notes and tick it off:

```
- [ ] 1. Probe:    python scripts/probe_pdf.py <pdf> --work <work> --plant-name "<name>" [--pages 47-60]
- [ ] 2. Render:   python scripts/render_pages.py <work>
- [ ] 3. Orient:   open each pages/page_NN/overview_grid.png; fix rotation if text is not upright
- [ ] 4. Hints:    python scripts/make_hints.py <work>
- [ ] 5. Extract:  for each sheet, write <work>/extract/page_NN.json (references/extraction_procedure.md)
- [ ] 6. Build:    python scripts/build_kg.py <work>
- [ ] 7. QA loop:  fix every error, re-check every warning, rebuild (repeat)
- [ ] 8. Report:   summarize the results and open issues for the user
```

### 1. Probe
`probe_pdf.py` assigns each page a class and an extraction route, and writes `<work>/manifest.json`. With `--pages`, the later steps render and read only those pages.

| Class | What it means | Route |
|---|---|---|
| `vector_text` | Vector drawing with a text layer | Render from vectors; exact text from the text layer as hints |
| `vector_outlined` | Vector drawing, text drawn as outlines (no text layer) | High-resolution vector render; OCR hints |
| `raster` | Scanned image | Native-resolution render, upscaled and contrast-enhanced; OCR hints |
| `raster_with_text` | Scan with an OCR text layer | As raster; text layer as (untrusted) hints |
| `hybrid` | Vectors plus large images | Both routes |

Report the probe result to the user in one line, e.g. "14 vector sheets with outlined text."

Low scan resolution is flagged in the notes. Expect to record more items in `unreadable` on those sheets.

### 2–3. Render and check orientation
`render_pages.py` writes the following for each page:

- `overview.png`
- `overview_grid.png`, which shows the tile names
- overlapping tiles in `tiles/`
- `tiles.json`, which records which tiles are blank

Orientation is detected with OCR when it is installed. Always confirm it on `overview_grid.png`. If the main drawing text is not upright, re-render that page with an explicit rotation, given clockwise: `render_pages.py <work> --pages N --rotation 90`.

Title blocks are sometimes rotated differently from the drawing. Orient by the drawing labels, not by the title block.

### 4. Hints
`make_hints.py` writes `hints/page_NN.json`. Each file holds candidate tags with page positions, taken from the text layer or from OCR. They are only hints: OCR misreads, and it misses things. You don't need to read the hint files during extraction. QA uses them to flag tags you may have missed. If OCR is not installed the step is skipped; note this in the final report.

### 5. Extract each sheet
This is the core of the job. Follow **[references/extraction_procedure.md](references/extraction_procedure.md)** exactly. It covers:

- the reading order: overview, then every non-blank tile, then zoomed crops
- how to model pipe runs, tees, taps, signals and off-page connectors
- confidence calibration
- a self-check list to run before saving

Reference material, to read when needed:

- [references/isa_5_1.md](references/isa_5_1.md): instrument letter codes, bubble symbols, signal line types
- [references/symbol_guide.md](references/symbol_guide.md): how equipment and valve symbols map to `kind` values
- [references/ontology.md](references/ontology.md): the graph's node and edge types and how they are derived
- `references/schema/extraction.schema.json`: the exact JSON schema of a page file

Write `extract/page_NN.json` and save it before starting the next sheet, so a long job never holds more than one sheet in context. With many sheets, and if your environment supports subagents, you can extract sheets in parallel. Give each subagent the procedure file and its own page directory.

### 6–7. Build and the QA loop
`build_kg.py` validates every page file and stops with a precise message on schema errors. It then writes these files to `<work>/kg/`:

- `graph.json`
- `graph.duckdb`
- `summary.md`
- `qa_report.md`

Work through `qa_report.md`:

- **Errors**, such as `unresolved_reference`, are mistakes in the page files. Fix them and rebuild.
- **Warnings** name the tiles to re-check. Look at those tiles again, or use `crop.py` for a zoomed view, and correct the extraction if the drawing shows something different. Common warnings:
  - `instrument_not_located`: a sensor with no `measures`
  - `equipment_not_connected`
  - `unmatched_off_page`
  - `possible_missed_tags`: an OCR tag that is not in your extraction
  - `tiles_not_reviewed`
- A warning that turns out correct on the drawing is fine to leave. Examples: an off-page connector whose other drawing was not supplied, or an OCR misread. List these in your report.

Stop when there are no errors and every warning has been checked.

```bash
python scripts/crop.py <work> --page 5 --bbox 0.40 0.20 0.55 0.35   # zoomed view of a region
```

### 8. Report
Tell the user:

- where the outputs are
- the headline counts (equipment, instruments, loops)
- two or three example flow paths and loops taken from `summary.md`
- what remains uncertain

Point to `qa_report.md` for the details. State plainly that the graph is machine-extracted and needs engineering review before it is used for design, operations or safety.

## Using the graph

- `summary.md` is the compact, token-efficient context for an LLM: equipment-level flow, an instrument table with locations, and a loop table.
- `graph.json` is the canonical graph, which any tool can read.
- `graph.duckdb` is for SQL. Its views are `equipment`, `equipment_flow`, `instruments`, `loops` and `signals`, and its table macros are `upstream(id)` and `downstream(id)`.

Use `kg_query.py` to answer questions; add `--json` when another tool will consume the output:

```bash
python scripts/kg_query.py <work>/kg downstream 660-TK-001 --equipment
python scripts/kg_query.py <work>/kg upstream FV-0502
python scripts/kg_query.py <work>/kg path 660-TK-001 660-TK-004
python scripts/kg_query.py <work>/kg instruments --on 660-PP-011
python scripts/kg_query.py <work>/kg loop FIC-0502
python scripts/kg_query.py <work>/kg sql "SELECT * FROM equipment_flow"
```

To measure extraction quality on a sheet, hand-check it into a small gold file (format in `scripts/eval_kg.py`) and run `python scripts/eval_kg.py <work>/kg gold.json`: it reports tag, flow and loop precision/recall and instrument placement.

To move a graph into other tools: `python scripts/export.py <work>/kg graphml` produces a file for yEd, Gephi or NetworkX, and `python scripts/export.py <work>/kg neo4j` produces CSVs plus a Cypher import script.
````

## File: skills/pid-to-kg/scripts/build_kg.py
size: 4642
sha: 9de11c3fe63fec3ded90b5fcaee1d304b59f0e66
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/build_kg.py

```text
#!/usr/bin/env python3
"""Build the knowledge graph for one plant from its page extractions.

Reads <work>/extract/page_*.json (validated against the extraction schema),
merges items across sheets by tag, expands pipe runs into segments, links
off-page connectors, infers control loops and equipment-level flow, runs QA,
and writes to <work>/kg/:

    graph.json     canonical knowledge graph (node-link JSON)
    graph.duckdb   the same graph as DuckDB tables, views and traversal macros
    summary.md     compact LLM-ready summary of flow, instruments and loops
    qa_report.md   issues to fix or re-check, with the tiles to look at

Usage:
    python build_kg.py work/my-plant
    python build_kg.py --write-schemas      # regenerate references/schema/*.json
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))

from pidkg.build import BuildError, build_graph, load_extractions  # noqa: E402
from pidkg.models import KnowledgeGraph, PageExtraction  # noqa: E402
from pidkg.qa import run_qa, write_report  # noqa: E402
from pidkg.store import write_duckdb  # noqa: E402
from pidkg.summary import write_summary  # noqa: E402

SCHEMA_DIR = Path(__file__).resolve().parent.parent / "references" / "schema"

def write_schemas() -> None:
    SCHEMA_DIR.mkdir(parents=True, exist_ok=True)
    for name, model in (("extraction.schema.json", PageExtraction), ("graph.schema.json", KnowledgeGraph)):
        schema = model.model_json_schema(by_alias=True)
        schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
        (SCHEMA_DIR / name).write_text(json.dumps(schema, indent=2) + "\n", encoding="utf-8")
        print(f"wrote {SCHEMA_DIR / name}")

def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("work", type=Path, nargs="?")
    ap.add_argument("--out", type=Path, help="output directory (default: <work>/kg)")
    ap.add_argument("--plant-name")
    ap.add_argument("--plant-id")
    ap.add_argument("--pages", help="only build from these pages, e.g. '1-2' (default: all extracted)")
    ap.add_argument("--write-schemas", action="store_true")
    args = ap.parse_args()
    if hasattr(sys.stdout, "reconfigure"):
        sys.stdout.reconfigure(encoding="utf-8")

    if args.write_schemas:
        write_schemas()
        return 0
    if not args.work:
        ap.error("work directory is required")

    manifest = {}
    mpath = args.work / "manifest.json"
    if mpath.exists():
        manifest = json.loads(mpath.read_text(encoding="utf-8"))
    plant_name = args.plant_name or manifest.get("plant_name") or args.work.name
    plant_id = args.plant_id or manifest.get("plant_id") or args.work.name
    source = Path(manifest["source_pdf"]).name if manifest.get("source_pdf") else None

    try:
        pages = load_extractions(args.work / "extract")
    except BuildError as e:
        print(f"error: {e}", file=sys.stderr)
        return 1
    if args.pages:
        from render_pages import parse_pages

        keep = set(parse_pages(args.pages, max(p.page for p in pages)))
        pages = [p for p in pages if p.page in keep]
        if not pages:
            print(f"error: no extracted pages in {args.pages}", file=sys.stderr)
            return 1

    kg = build_graph(pages, plant_id, plant_name, source)
    kg.issues.extend(run_qa(kg, args.work))
    sev: dict[str, int] = {}
    for i in kg.issues:
        sev[i["severity"]] = sev.get(i["severity"], 0) + 1
    kg.stats["issues_by_severity"] = sev

    out = args.out or (args.work / "kg")
    out.mkdir(parents=True, exist_ok=True)
    (out / "graph.json").write_text(kg.model_dump_json(indent=1, exclude_none=True), encoding="utf-8")
    write_duckdb(kg, out / "graph.duckdb")
    (out / "summary.md").write_text(write_summary(kg), encoding="utf-8")
    write_report(kg, kg.issues, out / "qa_report.md", args.work)

    s = kg.stats
    print(f"{plant_name}: {s['nodes']} nodes, {s['edges']} edges -> {out}")
    print("  nodes: " + ", ".join(f"{k} {v}" for k, v in s["nodes_by_type"].items()))
    print("  edges: " + ", ".join(f"{k} {v}" for k, v in s["edges_by_type"].items()))
    if s.get("loops_by_type"):
        print("  loops: " + ", ".join(f"{k} {v}" for k, v in s["loops_by_type"].items()))
    print(f"  issues: {sev.get('error', 0)} errors, {sev.get('warning', 0)} warnings, "
          f"{sev.get('info', 0)} info (see qa_report.md)")
    return 1 if sev.get("error") else 0

if __name__ == "__main__":
    sys.exit(main())
```

## File: skills/pid-to-kg/scripts/crop.py
size: 1796
sha: cb6e1a7f8cbeb511b265118926b9b7fd28a2ef8f
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/crop.py

```text
#!/usr/bin/env python3
"""Cut a zoomed view of a page region for a closer look.

The bbox is in normalized page coordinates (0..1, origin top-left), the same
system used for tiles and for ``bbox`` in extractions. The crop is upscaled
so its long edge is --size pixels.

Usage:
    python crop.py work/my-plant --page 5 --bbox 0.40 0.20 0.55 0.35
"""

from __future__ import annotations

import argparse
import sys
from pathlib import Path

def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("work", type=Path)
    ap.add_argument("--page", type=int, required=True)
    ap.add_argument("--bbox", type=float, nargs=4, required=True, metavar=("X0", "Y0", "X1", "Y1"))
    ap.add_argument("--size", type=int, default=1400, help="long edge of the output in px")
    ap.add_argument("--out", type=Path)
    a = ap.parse_args()

    from PIL import Image

    page_dir = a.work / "pages" / f"page_{a.page:02d}"
    src = page_dir / "page.png"
    if not src.exists():
        print(f"error: {src} not found; run render_pages.py first", file=sys.stderr)
        return 2
    img = Image.open(src)
    x0, y0, x1, y1 = a.bbox
    box = (round(min(x0, x1) * img.width), round(min(y0, y1) * img.height),
           round(max(x0, x1) * img.width), round(max(y0, y1) * img.height))
    crop = img.crop(box)
    scale = a.size / max(crop.size)
    crop = crop.resize((max(1, round(crop.width * scale)), max(1, round(crop.height * scale))),
                       Image.LANCZOS)
    out = a.out or page_dir / "crops" / f"crop_{x0:.2f}_{y0:.2f}_{x1:.2f}_{y1:.2f}.png"
    out.parent.mkdir(parents=True, exist_ok=True)
    crop.save(out)
    print(out)
    return 0

if __name__ == "__main__":
    sys.exit(main())
```

## File: skills/pid-to-kg/scripts/eval_kg.py
size: 7632
sha: 282d8a37aeaee84b55f4d483f43a63ec2083f6ba
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/eval_kg.py

```text
#!/usr/bin/env python3
"""Score a built graph against a hand-checked gold annotation.

Gold file (JSON), usually covering one or a few sheets:

    {
      "sheets": ["660-PI-203"],                      # drawing numbers the gold covers
      "tags": ["660-TK-004", "FIT-0502", ...],       # every tagged item on those sheets
      "feeds": [["660-EQ-002", "660-PP-011"], ...],  # equipment-level flow (upstream, downstream)
      "instrument_at": {"FIT-0502": "660-PP-011"},   # nearest equipment to where it senses
      "loops": {"F-0502": ["FE-0502", "FIT-0502"]},  # loop key -> member tags
      "feeds_scope": "within_sheets"                 # optional: "all" (default), "within_sheets"
                                                     # (both ends on gold sheets), "equipment" (both ends equipment)
    }

Reports precision/recall/F1 for tags and flow, instrument placement accuracy,
and loop recall with member overlap. Only graph items on the gold sheets are
scored, so a partial gold set gives a fair spot check.

Usage:
    python eval_kg.py work/my-plant/kg gold.json [--json]
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))

from pidkg.isa import normalize_tag  # noqa: E402
from pidkg.store import Graph  # noqa: E402

def prf(pred: set, gold: set) -> dict:
    tp = len(pred & gold)
    p = tp / len(pred) if pred else 0.0
    r = tp / len(gold) if gold else 0.0
    f = 2 * p * r / (p + r) if p + r else 0.0
    return {"precision": round(p, 3), "recall": round(r, 3), "f1": round(f, 3), "tp": tp,
            "false_positives": sorted(map(str, pred - gold)), "missed": sorted(map(str, gold - pred))}

def near_equipment(g: Graph, nid: str, hops: int = 60) -> set[str]:
    """Equipment directly measured, or nearest up/downstream of a measured pipe or item."""
    n = g.nodes[nid]
    if n.type == "Equipment":
        return {n.tag or n.label}
    found: set[str] = set()
    for direction in ("up", "down"):
        best = None
        for other, d in g.walk(nid, direction, max_depth=hops):
            if g.nodes[other].type == "Equipment":
                best = (d, g.nodes[other].tag or g.nodes[other].label)
                break
        if best:
            found.add(best[1])
    return found

def evaluate(g: Graph, gold: dict) -> dict:
    sheets = {f"SHEET:{normalize_tag(s)}" for s in gold.get("sheets", [])}
    on_sheet = {n.id for n in g.kg.nodes if not sheets or set(n.sheets) & sheets}

    def find(tag: str):
        """Prefer the node with this tag that is drawn on the gold sheets."""
        t = normalize_tag(tag)
        hits = [n.id for n in g.kg.nodes if n.tag == t and n.id in on_sheet]
        return hits[0] if hits else g.find(tag)

    pred_tags = {n.tag for n in g.kg.nodes if n.tag and n.id in on_sheet
                 and n.type in ("Equipment", "Instrument", "Component")}
    gold_tags = {normalize_tag(t) for t in gold.get("tags", [])}
    res: dict = {"tags": prf(pred_tags, gold_tags)}

    scope = gold.get("feeds_scope", "all")
    feeds = [e for es in g.out.values() for e in es if e.type == "FEEDS"]  # honours the PFD/P&ID view
    if scope == "within_sheets":
        feeds = [e for e in feeds if e.source in on_sheet and e.target in on_sheet]
    else:
        feeds = [e for e in feeds if e.source in on_sheet or e.target in on_sheet]
    if scope in ("equipment", "within_sheets"):
        feeds = [e for e in feeds if g.nodes[e.source].type == "Equipment" and g.nodes[e.target].type == "Equipment"]
    pred_feeds = {(g.nodes[e.source].tag or g.nodes[e.source].label, g.nodes[e.target].tag or g.nodes[e.target].label)
                  for e in feeds}
    gold_feeds = {(normalize_tag(a) or a, normalize_tag(b) or b) for a, b in gold.get("feeds", [])}
    norm_pred = {(normalize_tag(a) or a, normalize_tag(b) or b) for a, b in pred_feeds}
    res["flow"] = prf(norm_pred, gold_feeds)

    placed, wrong = 0, []
    for tag, eq in gold.get("instrument_at", {}).items():
        nid = find(tag)
        if nid is None:
            wrong.append({"tag": tag, "expected": eq, "got": "missing"})
            continue
        targets = [e.target for e in g.out[nid] if e.type == "MEASURES"]
        if g.nodes[nid].attributes.get("inline"):
            targets.append(nid)
        got = set().union(*(near_equipment(g, t) for t in targets)) if targets else set()
        if normalize_tag(eq) in {normalize_tag(x.split("@")[0]) for x in got}:
            placed += 1
        else:
            wrong.append({"tag": tag, "expected": eq, "got": sorted(got) or "unlocated"})
    n_inst = len(gold.get("instrument_at", {}))
    res["instrument_placement"] = {"accuracy": round(placed / n_inst, 3) if n_inst else None,
                                   "correct": placed, "total": n_inst, "errors": wrong}

    loops_found, overlaps, loop_detail = 0, [], []
    for key, members in gold.get("loops", {}).items():
        lid = next((n.id for n in g.kg.nodes if n.type == "ControlLoop" and n.id in on_sheet | {n.id}
                    and n.label.split("@")[0] == key and set(n.sheets) & (sheets or set(n.sheets))), None)
        gm = {normalize_tag(m) for m in members}
        if lid is None:
            loop_detail.append({"loop": key, "found": False})
            continue
        loops_found += 1
        pm = {g.nodes[e.source].tag or g.nodes[e.source].label for e in g.inn[lid] if e.type == "MEMBER_OF"}
        jac = len(pm & gm) / len(pm | gm) if pm | gm else 1.0
        overlaps.append(jac)
        loop_detail.append({"loop": key, "found": True, "member_jaccard": round(jac, 3),
                            "missing_members": sorted(gm - pm), "extra_members": sorted(pm - gm)})
    n_loops = len(gold.get("loops", {}))
    res["loops"] = {"recall": round(loops_found / n_loops, 3) if n_loops else None,
                    "mean_member_jaccard": round(sum(overlaps) / len(overlaps), 3) if overlaps else None,
                    "detail": loop_detail}
    return res

def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("kg", type=Path)
    ap.add_argument("gold", type=Path)
    ap.add_argument("--json", action="store_true")
    a = ap.parse_args()
    if hasattr(sys.stdout, "reconfigure"):
        sys.stdout.reconfigure(encoding="utf-8")
    g = Graph.load(a.kg / "graph.json" if a.kg.is_dir() else a.kg)
    res = evaluate(g, json.loads(a.gold.read_text(encoding="utf-8")))
    if a.json:
        print(json.dumps(res, indent=1))
        return 0
    t, f, ip, lp = res["tags"], res["flow"], res["instrument_placement"], res["loops"]
    print(f"Tags:   P {t['precision']:.2f}  R {t['recall']:.2f}  F1 {t['f1']:.2f}"
          f"   missed: {', '.join(t['missed']) or '-'}   extra: {', '.join(t['false_positives']) or '-'}")
    print(f"Flow:   P {f['precision']:.2f}  R {f['recall']:.2f}  F1 {f['f1']:.2f}"
          f"   missed: {f['missed'] or '-'}   extra: {f['false_positives'] or '-'}")
    if ip["total"]:
        print(f"Instrument placement: {ip['correct']}/{ip['total']}"
              + "".join(f"\n   {w['tag']}: expected {w['expected']}, got {w['got']}" for w in ip["errors"]))
    if lp["recall"] is not None:
        print(f"Loops:  recall {lp['recall']:.2f}, mean member overlap {lp['mean_member_jaccard']}")
        for d in lp["detail"]:
            if not d["found"] or d.get("missing_members") or d.get("extra_members"):
                print(f"   {d}")
    return 0

if __name__ == "__main__":
    sys.exit(main())
```

## File: skills/pid-to-kg/scripts/export.py
size: 4832
sha: 24d5caa75ea6444ba776f72be23c9922386dc8fc
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/export.py

```text
#!/usr/bin/env python3
"""Export a knowledge graph to GraphML or a Neo4j import bundle.

    python export.py work/my-plant/kg graphml          # -> graph.graphml (yEd, Gephi, NetworkX)
    python export.py work/my-plant/kg neo4j            # -> neo4j/nodes.csv, edges.csv, import.cypher

Node types become Neo4j labels and edge types become relationship types.
Nested attributes are flattened to scalar properties where possible and kept
whole as a JSON string in ``attributes_json``.
"""

from __future__ import annotations

import argparse
import csv
import json
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))

from pidkg.store import Graph  # noqa: E402

def _scalar(v):
    if v is None or isinstance(v, (str, int, float, bool)):
        return v
    return json.dumps(v)

def _node_props(n) -> dict:
    props = {"id": n.id, "type": n.type, "label": n.label, "tag": n.tag, "kind": n.kind,
             "name": n.name, "status": n.status, "dexpi_class": n.dexpi_class,
             "sheets": ";".join(n.sheets), "confidence": n.confidence,
             "attributes_json": json.dumps(n.attributes)}
    for k, v in n.attributes.items():
        props.setdefault(f"a_{k}", _scalar(v))
    return {k: v for k, v in props.items() if v is not None}

def export_graphml(g: Graph, path: Path) -> None:
    import networkx as nx

    G = nx.MultiDiGraph(plant=g.kg.plant.name)
    for n in g.kg.nodes:
        G.add_node(n.id, **_node_props(n))
    for e in g.kg.edges:
        attrs = {"type": e.type, "confidence": e.confidence, "derived": e.derived,
                 "attributes_json": json.dumps(e.attributes)}
        if e.sheet:
            attrs["sheet"] = e.sheet
        G.add_edge(e.source, e.target, key=e.id, **attrs)
    nx.write_graphml(G, path)

def export_neo4j(g: Graph, out: Path) -> None:
    out.mkdir(parents=True, exist_ok=True)
    keys = ["id", "type", "label", "tag", "kind", "name", "status", "dexpi_class", "sheets",
            "confidence", "attributes_json"]
    with open(out / "nodes.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=keys, extrasaction="ignore")
        w.writeheader()
        for n in g.kg.nodes:
            w.writerow(_node_props(n))
    with open(out / "edges.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["id", "source", "target", "type", "confidence", "derived", "sheet", "attributes_json"])
        for e in g.kg.edges:
            w.writerow([e.id, e.source, e.target, e.type, e.confidence, e.derived, e.sheet or "",
                        json.dumps(e.attributes)])
    node_types = sorted({n.type for n in g.kg.nodes})
    edge_types = sorted({e.type for e in g.kg.edges})
    plant = g.kg.plant.id
    lines = [
        f"// Neo4j import for plant '{g.kg.plant.name}'. Copy nodes.csv and edges.csv into the",
        "// Neo4j import directory, then run this script (e.g. with cypher-shell).",
        "CREATE CONSTRAINT pid_node_id IF NOT EXISTS FOR (n:PidNode) REQUIRE (n.plant, n.id) IS UNIQUE;",
        "",
        "LOAD CSV WITH HEADERS FROM 'file:///nodes.csv' AS row",
        f"MERGE (n:PidNode {{plant: '{plant}', id: row.id}})",
        "SET n += row, n.confidence = toFloat(row.confidence);",
        "",
    ]
    for t in node_types:
        lines.append(f"MATCH (n:PidNode {{plant: '{plant}', type: '{t}'}}) SET n:{t};")
    lines.append("")
    for t in edge_types:
        lines += [
            "LOAD CSV WITH HEADERS FROM 'file:///edges.csv' AS row",
            f"WITH row WHERE row.type = '{t}'",
            f"MATCH (a:PidNode {{plant: '{plant}', id: row.source}}), (b:PidNode {{plant: '{plant}', id: row.target}})",
            f"MERGE (a)-[r:{t} {{id: row.id}}]->(b)",
            "SET r.confidence = toFloat(row.confidence), r.derived = row.derived = 'True',",
            "    r.sheet = row.sheet, r.attributes_json = row.attributes_json;",
            "",
        ]
    (out / "import.cypher").write_text("\n".join(lines), encoding="utf-8")

def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("kg", type=Path, help="kg/ directory or graph.json")
    ap.add_argument("format", choices=["graphml", "neo4j"])
    a = ap.parse_args()
    if hasattr(sys.stdout, "reconfigure"):
        sys.stdout.reconfigure(encoding="utf-8")
    json_path = a.kg / "graph.json" if a.kg.is_dir() else a.kg
    g = Graph.load(json_path)
    if a.format == "graphml":
        target = json_path.with_name("graph.graphml")
        export_graphml(g, target)
    else:
        target = json_path.with_name("neo4j")
        export_neo4j(g, target)
    print(f"wrote {target}")
    return 0

if __name__ == "__main__":
    sys.exit(main())
```

## File: skills/pid-to-kg/scripts/kg_query.py
size: 9046
sha: d27d262fad34eab2a84418131a775300589bd512
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/kg_query.py

```text
#!/usr/bin/env python3
"""Query a P&ID knowledge graph (for people, and as a tool for LLM agents).

Usage:
    python kg_query.py KG stats
    python kg_query.py KG describe 660-TK-004
    python kg_query.py KG upstream FV-0502 [--equipment] [--depth 3]
    python kg_query.py KG downstream 660-TK-001 [--equipment]
    python kg_query.py KG path 660-TK-001 660-TK-004
    python kg_query.py KG loops [--type control]
    python kg_query.py KG loop F-0502
    python kg_query.py KG instruments [--on 660-PP-011] [--variable flow]
    python kg_query.py KG sql "SELECT * FROM equipment_flow"

KG is a kg/ directory, graph.json or graph.duckdb. Add --json for machine-readable output.
Item references accept ids, tags or labels; tags are matched loosely (FIT0502 = FIT-0502).
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))

from pidkg.loops import point_label, sensed_points  # noqa: E402
from pidkg.store import Graph  # noqa: E402

TERMINAL = ("Equipment", "Boundary")

def _resolve_paths(kg: Path) -> tuple[Path, Path]:
    if kg.is_dir():
        return kg / "graph.json", kg / "graph.duckdb"
    if kg.suffix == ".duckdb":
        return kg.with_name("graph.json"), kg
    return kg, kg.with_name("graph.duckdb")

def _node_brief(g: Graph, nid: str) -> dict:
    n = g.nodes[nid]
    d = {"id": n.id, "type": n.type, "label": n.label}
    if n.kind:
        d["kind"] = n.kind
    if n.name:
        d["name"] = n.name
    return d

def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("kg", type=Path)
    ap.add_argument("command", choices=["stats", "describe", "upstream", "downstream", "path",
                                        "loops", "loop", "instruments", "sql"])
    ap.add_argument("args", nargs="*")
    ap.add_argument("--equipment", action="store_true", help="only equipment and boundaries")
    ap.add_argument("--depth", type=int)
    ap.add_argument("--type", help="loop type filter")
    ap.add_argument("--on", help="instruments measuring this item (or anything on it)")
    ap.add_argument("--variable", help="instrument variable filter, e.g. flow")
    ap.add_argument("--include-pfd", action="store_true",
                    help="also traverse PFD-sheet flow when the plant has P&IDs (default: P&ID detail only)")
    ap.add_argument("--json", action="store_true")
    a = ap.parse_args()
    if hasattr(sys.stdout, "reconfigure"):
        sys.stdout.reconfigure(encoding="utf-8")

    json_path, db_path = _resolve_paths(a.kg)

    if a.command == "sql":
        import duckdb

        con = duckdb.connect(str(db_path), read_only=True)
        rel = con.sql(" ".join(a.args))
        if a.json:
            cols = rel.columns
            print(json.dumps([dict(zip(cols, r)) for r in rel.fetchall()], indent=1, default=str))
        else:
            rel.show(max_width=200, max_rows=200)
        return 0

    g = Graph.load(json_path, include_pfd=a.include_pfd)

    def need(n: int) -> list[str]:
        if len(a.args) < n:
            ap.error(f"'{a.command}' needs {n} item reference(s)")
        ids = []
        for ref in a.args[:n]:
            nid = g.find(ref)
            if nid is None:
                print(f"error: '{ref}' not found in the graph", file=sys.stderr)
                sys.exit(2)
            ids.append(nid)
        return ids

    result: object
    if a.command == "stats":
        result = {"plant": g.kg.plant.model_dump(), **g.kg.stats}
    elif a.command == "describe":
        (nid,) = need(1)
        n = g.nodes[nid]
        result = {**n.model_dump(exclude_none=True, exclude={"provenance"}),
                  "relations": g.neighbors(nid)}
        if n.type == "PipeSegment":
            result["location"] = point_label(nid, g.nodes, g.kg.edges)
    elif a.command in ("upstream", "downstream"):
        (nid,) = need(1)
        hits = g.walk(nid, "down" if a.command == "downstream" else "up", max_depth=a.depth)
        rows = [dict(_node_brief(g, h), hops=d) for h, d in hits
                if not a.equipment or g.nodes[h].type in TERMINAL]
        result = {"from": _node_brief(g, nid), "direction": a.command, "items": rows}
    elif a.command == "path":
        x, y = need(2)
        p = g.path(x, y) or g.path(y, x)
        result = {"path": [_node_brief(g, i) for i in p]} if p else {"path": None,
                  "note": "no directed flow path between these items"}
    elif a.command == "loops":
        loops = [n for n in g.kg.nodes if n.type == "ControlLoop" and (not a.type or n.kind == a.type)]
        result = [{"id": n.id, "loop": n.label, "type": n.kind,
                   "description": n.attributes.get("description")} for n in loops]
    elif a.command == "loop":
        ref = a.args[0] if a.args else ap.error("'loop' needs a loop id")
        nid = g.find(ref) or g.find(f"LOOP:{ref}")
        if nid is None or g.nodes[nid].type != "ControlLoop":
            # Accept a member tag, e.g. 'loop FIC-0502'.
            member = g.find(ref)
            nid = next((e.target for e in g.out.get(member, []) if e.type == "MEMBER_OF"), None) \
                if member else None
        if nid is None:
            print(f"error: loop '{ref}' not found", file=sys.stderr)
            return 2
        n = g.nodes[nid]
        members = [dict(_node_brief(g, e.source), role=e.attributes.get("role"))
                   for e in g.inn[nid] if e.type == "MEMBER_OF"]
        result = {**n.model_dump(exclude_none=True, exclude={"provenance"}), "members": members}
    else:  # instruments
        target_ids: set[str] | None = None
        if a.on:
            t = g.find(a.on)
            if t is None:
                print(f"error: '{a.on}' not found", file=sys.stderr)
                return 2
            # The item plus its own piping: everything reachable up- and downstream
            # before the next piece of equipment.
            target_ids = {t} | _local_piping(g, t)
        rows = []
        for n in g.kg.nodes:
            if n.type != "Instrument":
                continue
            if a.variable and a.variable.lower() not in (n.attributes.get("variable") or "").lower():
                continue
            meas = {e.target for e in g.out[n.id] if e.type == "MEASURES"}
            if target_ids is not None and not ((meas | {n.id}) & target_ids):
                continue
            rows.append({"tag": n.label, "variable": n.attributes.get("variable"),
                         "role": n.attributes.get("role"), "location": n.attributes.get("location"),
                         "measures": sensed_points(n.id, g.nodes, g.kg.edges),
                         "loop": [g.nodes[e.target].label for e in g.out[n.id] if e.type == "MEMBER_OF"]})
        result = rows

    if a.json:
        print(json.dumps(result, indent=1, default=str))
    else:
        _print_text(result)
    return 0

def _local_piping(g: Graph, start: str, limit: int = 8) -> set[str]:
    """Segments and fittings within ``limit`` hops of an item, stopping at equipment."""
    found: set[str] = set()
    for direction in ("down", "up"):
        frontier = [start]
        while frontier and len(found) < 500:
            nxt = []
            for cur in frontier:
                edges = g.out[cur] if direction == "down" else g.inn[cur]
                for e in edges:
                    if e.type != "FLOWS_TO":
                        continue
                    other = e.target if direction == "down" else e.source
                    if other in found or other == start or g.nodes[other].type in TERMINAL:
                        continue
                    found.add(other)
                    nxt.append(other)
            frontier = nxt
            limit -= 1
            if limit <= 0:
                break
    return found

def _print_text(obj, indent: int = 0) -> None:
    pad = "  " * indent
    if isinstance(obj, list):
        if not obj:
            print(pad + "(none)")
        for item in obj:
            if isinstance(item, dict):
                print(pad + "- " + ", ".join(f"{k}: {_fmt(v)}" for k, v in item.items()))
            else:
                print(pad + f"- {item}")
    elif isinstance(obj, dict):
        for k, v in obj.items():
            if isinstance(v, (dict, list)) and v and any(isinstance(x, (dict, list)) for x in
                                                        (v.values() if isinstance(v, dict) else v)):
                print(f"{pad}{k}:")
                _print_text(v, indent + 1)
            else:
                print(f"{pad}{k}: {_fmt(v)}")
    else:
        print(pad + str(obj))

def _fmt(v) -> str:
    if isinstance(v, list):
        return "[" + ", ".join(_fmt(x) for x in v) + "]"
    if isinstance(v, dict):
        return "{" + ", ".join(f"{k}: {_fmt(x)}" for k, x in v.items()) + "}"
    return str(v)

if __name__ == "__main__":
    sys.exit(main())
```

## File: skills/pid-to-kg/scripts/make_hints.py
size: 4049
sha: c45e830a7b63f4c65377c90a7f35695245c84d12
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/make_hints.py

```text
#!/usr/bin/env python3
"""Collect deterministic text hints for each rendered page.

Hints are candidate tags with page positions, gathered without the LLM:

* text layer (vector_text, raster_with_text, hybrid pages): exact text and
  positions straight from the PDF;
* OCR (vector_outlined, raster, hybrid pages; needs rapidocr-onnxruntime):
  each non-blank tile is OCR'd and mapped back to page coordinates.

ISA bubbles split over two lines ('FIT' over '0502') are joined. QA compares
these candidates with the extraction to flag tags that may have been missed.
Writes <work>/hints/page_NN.json.

Usage:
    python make_hints.py work/my-plant [--pages 1-3] [--no-ocr]
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))

from pidkg import ocr  # noqa: E402
from pidkg.pdf import text_runs  # noqa: E402
from render_pages import parse_pages  # noqa: E402

TEXT_LAYER_CLASSES = {"vector_text", "raster_with_text", "hybrid"}
OCR_CLASSES = {"vector_outlined", "raster", "hybrid"}

def tiles_for(bbox: list[float], tiles: list[dict]) -> list[str]:
    cx, cy = (bbox[0] + bbox[2]) / 2, (bbox[1] + bbox[3]) / 2
    return [t["name"] for t in tiles if t["norm"][0] <= cx <= t["norm"][2]
            and t["norm"][1] <= cy <= t["norm"][3]]

def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("work", type=Path)
    ap.add_argument("--pages")
    ap.add_argument("--no-ocr", action="store_true")
    args = ap.parse_args()
    if hasattr(sys.stdout, "reconfigure"):
        sys.stdout.reconfigure(encoding="utf-8", line_buffering=True)

    from PIL import Image

    manifest = json.loads((args.work / "manifest.json").read_text(encoding="utf-8"))
    use_ocr = not args.no_ocr and ocr.available()
    if not args.no_ocr and not use_ocr:
        print(f"note: {ocr.maybe_import_error()}")
    (args.work / "hints").mkdir(exist_ok=True)

    for page_no in parse_pages(args.pages, len(manifest["pages"]), manifest.get("selected_pages")):
        info = manifest["pages"][page_no - 1]
        cls = info["page_class"]
        page_dir = args.work / "pages" / f"page_{page_no:02d}"
        tiles_file = page_dir / "tiles.json"
        if not tiles_file.exists():
            print(f"page {page_no}: not rendered yet (run render_pages.py), skipped")
            continue
        meta = json.loads(tiles_file.read_text(encoding="utf-8"))
        runs: list[dict] = []
        sources: list[str] = []
        if cls in TEXT_LAYER_CLASSES:
            runs += [dict(r, source="text_layer")
                     for r in text_runs(manifest["source_pdf"], page_no, meta["rotation"])]
            sources.append("text_layer")
        if cls in OCR_CLASSES and use_ocr:
            for t in meta["tiles"]:
                if t.get("blank"):
                    continue
                img = Image.open(page_dir / "tiles" / f"{t['name']}.png")
                for r in ocr.ocr_image(img):
                    if r["conf"] < 0.5 or not r["text"]:
                        continue
                    runs.append({"text": r["text"], "conf": round(r["conf"], 3), "source": "ocr",
                                 "bbox": ocr.to_norm(r["bbox"], t["px"], meta["width"], meta["height"])})
            sources.append("ocr")
        runs = ocr.dedupe_runs(runs)
        runs = ocr.join_bubbles(runs)
        cands = ocr.tag_candidates(runs)
        for c in cands:
            c["tiles"] = tiles_for(c["bbox"], meta["tiles"])
        out = {"page": page_no, "source": "+".join(sources) or "none", "page_class": cls,
               "tag_candidates": cands, "runs": runs}
        (args.work / "hints" / f"page_{page_no:02d}.json").write_text(
            json.dumps(out, indent=1), encoding="utf-8")
        print(f"page {page_no}: {len(runs)} text runs, {len(cands)} tag candidates ({out['source']})")
    return 0

if __name__ == "__main__":
    sys.exit(main())
```

## File: skills/pid-to-kg/scripts/pidkg/__init__.py
size: 153
sha: cdc430366f41d181fdfbf6a0391e539ec0388b01
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/__init__.py

```text
"""pidkg: build process knowledge graphs from P&ID extractions."""

__version__ = "0.1.0"
EXTRACTION_SCHEMA_VERSION = "1.0"
GRAPH_SCHEMA_VERSION = "1.0"
```

## File: skills/pid-to-kg/scripts/pidkg/build.py
size: 45253
sha: 7ecaf83f96f37d2a241d231ebebc91f426ab771f
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/build.py

Content is not inlined in the catalog. Read the upstream file: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/build.py

## File: skills/pid-to-kg/scripts/pidkg/isa.py
size: 5016
sha: 3ea93fc81e20caed15a46ff02a217ecfd122d3c5
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/isa.py

```text
"""Tag normalization and ISA-5.1 instrument tag parsing."""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import Optional

from .ontology import ISA_FUNCTIONS, ISA_MEASURED, ISA_VARIABLE_MODIFIERS

_DASHES = dict.fromkeys(map(ord, "‐‑‒–—−"), "-")
_COMPACT = re.compile(r"^([A-Z]{1,6})(\d{1,6}[A-Z]{0,2})$")

def normalize_tag(raw: Optional[str]) -> Optional[str]:
    """Normalize a printed tag so the same item read twice compares equal.

    ``"fit 0502"``, ``"FIT-0502"``, ``"FIT_0502"`` and ``"FIT0502"`` all become
    ``"FIT-0502"``. Area prefixes and suffixes are preserved
    (``"660 TK 004"`` -> ``"660-TK-004"``).
    """
    if raw is None:
        return None
    s = str(raw).translate(_DASHES).strip().strip("\"'").upper()
    if not s:
        return None
    s = re.sub(r"[\s_/]*-[\s_/]*", "-", s)
    s = re.sub(r"[\s_]+", "-", s)
    s = re.sub(r"-{2,}", "-", s).strip("-")
    m = _COMPACT.match(s)
    if m:
        s = f"{m.group(1)}-{m.group(2)}"
    return s

def normalize_label(raw: Optional[str]) -> str:
    """Normalize free text (boundary names, off-page labels) for matching."""
    s = (raw or "").translate(_DASHES).upper()
    s = re.sub(r"[^A-Z0-9#\-\. ]+", " ", s)
    return re.sub(r"\s+", " ", s).strip()

_ISA_TAG = re.compile(
    r"^(?:(?P<prefix>[0-9][0-9A-Z]*)-)?"
    r"(?P<letters>[A-Z]{1,6})-"
    r"(?P<number>[A-Z]?\d{1,5})(?P<suffix>[A-Z]{0,2})"
    r"(?:-(?P<ext>[A-Z0-9]{1,4}))?$"
)

_RELIEF = {"PSV", "PRV", "PSE", "TSV", "TSE", "VRV", "PVRV", "PVSV"}

ROLE_ORDER = (
    "relief_device",
    "hand_switch",
    "final_element",
    "relay",
    "controller",
    "transmitter",
    "switch",
    "element",
    "gauge",
    "indicator",
    "recorder",
    "totalizer",
    "alarm",
    "other",
)

@dataclass
class IsaTag:
    tag: str
    prefix: Optional[str]
    letters: str
    number: str
    suffix: str
    variable_code: str
    variable: str
    modifiers: list[str] = field(default_factory=list)
    functions: list[str] = field(default_factory=list)
    role: str = "other"

    @property
    def loop_key(self) -> str:
        base = f"{self.variable_code[0]}-{self.number}"
        return f"{self.prefix}-{base}" if self.prefix else base

def parse_isa_tag(tag: Optional[str]) -> Optional[IsaTag]:
    """Parse an ISA-5.1 style instrument tag. Returns None if it does not fit."""
    t = normalize_tag(tag)
    if not t:
        return None
    m = _ISA_TAG.match(t)
    if not m:
        return None
    letters = m.group("letters")
    first, rest = letters[0], letters[1:]
    if first not in ISA_MEASURED:
        return None

    variable_code = first
    modifiers: list[str] = []
    state = None
    # Open/closed state suffix on position and event tags: ZSO, ZSC, ZIO, ZIC, XSO, XSC.
    if first in "XYZ" and len(rest) >= 2 and rest[-1] in "OC" and rest[-2] in "IS":
        state = "open" if rest[-1] == "O" else "closed"
        rest = rest[:-1]
    if letters in _RELIEF:
        functions = ["safety", "valve"]
        role = "relief_device"
        variable_code = first
    else:
        if rest and rest[0] in ISA_VARIABLE_MODIFIERS and len(rest) > 1:
            variable_code = first + rest[0]
            modifiers.append(ISA_VARIABLE_MODIFIERS[rest[0]])
            rest = rest[1:]
        functions = [ISA_FUNCTIONS.get(ch, ch) for ch in rest]
        if state:
            functions.append(state)
        role = _role(rest)
        if first == "H" and role in ("switch", "controller", "indicator", "other"):
            role = "hand_switch"  # operator input (HS, HIC), not a process sensor

    variable = ISA_MEASURED[first]
    if modifiers:
        variable = f"{modifiers[0]} {variable}"
    return IsaTag(
        tag=t,
        prefix=m.group("prefix"),
        letters=letters,
        number=m.group("number"),
        suffix=m.group("suffix") or "",
        variable_code=variable_code,
        variable=variable,
        modifiers=modifiers,
        functions=functions,
        role=role,
    )

def _role(fn: str) -> str:
    if fn.endswith("V") or fn.endswith("Z"):
        return "final_element"
    if "Y" in fn:
        return "relay"
    if "C" in fn or fn.endswith("K"):
        return "controller"
    if "T" in fn:
        return "transmitter"
    if "S" in fn:
        return "switch"
    if "E" in fn:
        return "element"
    if "G" in fn:
        return "gauge"
    if "I" in fn:
        return "indicator"
    if "R" in fn:
        return "recorder"
    if "Q" in fn:
        return "totalizer"
    if "A" in fn:
        return "alarm"
    return "other"

# Roles that sense the process directly and therefore should have a MEASURES edge.
SENSING_ROLES = {"transmitter", "switch", "element", "gauge"}

# Candidate tags in free text (OCR or vector text). Deliberately loose; used only
# to suggest items the extraction may have missed.
TAG_CANDIDATE = re.compile(
    r"\b(?:\d{2,4}[- ]?)?[A-Z]{1,5}[- ]?\d{2,5}[A-Z]?\b"
)
```

## File: skills/pid-to-kg/scripts/pidkg/loops.py
size: 14977
sha: f95338ee95ac6ef870b96f4e321913c8a608103e
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/loops.py

```text
"""Control-loop inference.

Loops are inferred in two steps:

1. **Tag grouping (ISA-5.1).** Instruments and actuated components sharing a
   measured-variable letter and loop number (FT-101, FIC-101, FV-101) form a
   loop ``F-101``. An explicit ``loop`` on an item overrides this.
2. **Signal tracing.** Signal lines refine the groups: a controller's output
   is followed (through relays such as I/P converters) to the final element it
   moves, even when the numbers differ; controller-to-controller signals
   become cascades; and sensors wired into a controller from another loop
   number are added as inputs.
"""

from __future__ import annotations

from collections import defaultdict
from typing import TYPE_CHECKING, Optional

from .models import Node
from .ontology import ACTUATED_COMPONENT_KINDS

if TYPE_CHECKING:  # pragma: no cover
    from .build import GraphBuilder

SENSOR_ROLES = {"transmitter", "element", "switch", "gauge", "indicator", "recorder", "totalizer", "alarm"}
READOUT_ROLES = {"indicator", "recorder", "totalizer", "alarm"}
PANEL_LOCATIONS = {"main_panel", "behind_panel", "dcs", "plc", "scada", "sis", "computer"}

def _member_role(n: Node) -> str:
    """Role within a loop: panel indicators and alarms are read-outs, not sensors."""
    r = _role(n)
    if r in READOUT_ROLES and n.attributes.get("location") in PANEL_LOCATIONS:
        return "readout"
    return r

def loop_variable(key: str) -> Optional[str]:
    """Measured variable named by a loop key such as 'F-101', '660-F-0502' or 'S-1@M702'."""
    from .ontology import ISA_MEASURED, ISA_VARIABLE_MODIFIERS

    core = key.split("@")[0].split("#")[0]
    parts = core.split("-")
    code = parts[-2] if len(parts) >= 2 else ""
    if not code or code[0] not in ISA_MEASURED:
        return None
    var = ISA_MEASURED[code[0]]
    if len(code) > 1 and code[1] in ISA_VARIABLE_MODIFIERS:
        var = f"{ISA_VARIABLE_MODIFIERS[code[1]]} {var}"
    return var

def _role(n: Node) -> str:
    if n.type == "Component":
        if n.kind == "relief_valve" or n.attributes.get("role") == "relief_device":
            return "relief_device"
        return "final_element"
    if n.type == "Equipment":
        return "final_element"
    return n.attributes.get("role", "other")

def _is_final(n: Node) -> bool:
    if n.type == "Component":
        return n.kind in ACTUATED_COMPONENT_KINDS or n.attributes.get("role") == "final_element" \
            or n.attributes.get("actuator") not in (None, "manual", "none", "self")
    if n.type == "Equipment":
        return True  # e.g. a pump motor / VFD driven by a speed controller
    return n.attributes.get("role") == "final_element"

def derive_loops(b: "GraphBuilder") -> None:
    nodes = b.nodes
    sig_out: dict[str, list[str]] = defaultdict(list)
    sig_in: dict[str, list[str]] = defaultdict(list)
    sig_purpose: dict[tuple[str, str], str] = {}
    for e in b.edges:
        if e.type == "SIGNAL_TO":
            sig_out[e.source].append(e.target)
            sig_in[e.target].append(e.source)
            sig_purpose[(e.source, e.target)] = e.attributes.get("purpose", "unknown")

    # 1. Group by loop key.
    groups: dict[str, dict[str, str]] = defaultdict(dict)  # loop key -> {node id: role}
    for n in nodes.values():
        key = n.attributes.get("loop_key")
        if key and n.type in ("Instrument", "Component"):
            groups[key][n.id] = _member_role(n)

    # 2. Refine with signals.
    cascades: set[tuple[str, str]] = set()
    loop_of = {nid: key for key, members in groups.items() for nid in members}
    for key, members in list(groups.items()):
        for cid in [m for m, r in members.items() if r == "controller"]:
            for target, via in _trace_outputs(cid, sig_out, nodes):
                tn = nodes[target]
                if tn.type == "Instrument" and tn.attributes.get("role") == "controller":
                    other = loop_of.get(target)
                    if other and other != key:
                        cascades.add((key, other))
                    continue
                if _is_final(tn):
                    members.setdefault(target, "final_element")
                    for r in via:
                        members.setdefault(r, "relay")
                    b._edge(cid, target, "ACTUATES", derived=True,
                            via=[nodes[r].label for r in via] or None)
            for src in sig_in.get(cid, []):
                sn = nodes[src]
                if sn.type == "Instrument" and sn.attributes.get("role") in SENSOR_ROLES:
                    members.setdefault(src, "sensor_input")

        # Sensors or hand switches wired straight to a final element (interlocks,
        # on/off logic in a PLC that is not drawn as a controller).
        for sid in [m for m, r in members.items() if r in ("switch", "hand_switch", "transmitter", "alarm", "other")]:
            for target, via in _trace_outputs(sid, sig_out, nodes):
                if _is_final(nodes[target]) and target not in members:
                    members[target] = "final_element"
                    b._edge(sid, target, "ACTUATES", derived=True,
                            via=[nodes[r].label for r in via] or None)

    # 3. Emit loop nodes for meaningful groups.
    loop_ids: dict[str, str] = {}
    for key, members in sorted(groups.items()):
        roles = set(members.values())
        if len(members) < 2 and "controller" not in roles:
            continue
        lid = f"LOOP:{key}"
        loop_ids[key] = lid
        loop_type = _loop_type(members, nodes, sig_purpose)
        variable = loop_variable(key) or next((nodes[m].attributes.get("variable") for m in members
                                               if nodes[m].attributes.get("variable")), None)
        sheets: list[str] = []
        for m in members:
            for s in nodes[m].sheets:
                if s not in sheets:
                    sheets.append(s)
        sensors = [m for m, r in members.items() if r in SENSOR_ROLES or r == "sensor_input"]
        controllers = [m for m, r in members.items() if r == "controller"]
        finals = [m for m, r in members.items() if r == "final_element"]
        measured_at = _measured_points(sensors, b)
        manipulated_at = _manipulated_points(finals, b)
        node = Node(
            id=lid, type="ControlLoop", label=key, kind=loop_type, sheets=sheets,
            confidence=min(nodes[m].confidence for m in members),
            dexpi_class="InstrumentationLoopFunction",
            attributes={k: v for k, v in dict(
                variable=variable, loop_type=loop_type, sensors=sensors, controllers=controllers,
                final_elements=finals, relays=[m for m, r in members.items() if r == "relay"],
                readouts=[m for m, r in members.items() if r == "readout"],
                measured_at=measured_at, manipulated_at=manipulated_at).items() if v})
        node.attributes["description"] = _describe(node, b)
        b._upsert(node)
        for m, role in members.items():
            b._edge(m, lid, "MEMBER_OF", derived=True, role=role)
        if loop_type == "interlock":
            for s in sensors:
                for f in finals:
                    b._edge(s, f, "TRIPS", derived=True, loop=lid)

    for master, slave in sorted(cascades):
        if master in loop_ids and slave in loop_ids:
            b._edge(loop_ids[master], loop_ids[slave], "CASCADES_TO", derived=True)
            b.nodes[loop_ids[master]].attributes.setdefault("cascade_slaves", []).append(loop_ids[slave])
            b.nodes[loop_ids[slave]].attributes["cascade_master"] = loop_ids[master]
    for lid in loop_ids.values():
        ln = b.nodes[lid]
        if ln.attributes.get("cascade_slaves") or ln.attributes.get("cascade_master"):
            ln.attributes["description"] = _describe(ln, b)

    # 4. QA on loop completeness.
    for key, lid in loop_ids.items():
        ln = b.nodes[lid]
        a = ln.attributes
        if ln.kind == "control":
            if not a.get("final_elements") and not a.get("cascade_slaves"):
                b.issue("warning", "loop_no_final_element",
                        f"control loop {key} has a controller but no final element", node=lid)
            if not a.get("sensors"):
                # Common for speed/position controllers that only take an operator setpoint.
                b.issue("info", "loop_no_sensor",
                        f"control loop {key} has a controller but no sensor", node=lid)
        for c in a.get("controllers", []):
            if not sig_in.get(c) and not sig_out.get(c):
                b.issue("info", "loop_no_signals",
                        f"controller {c} has no signal lines; loop {key} built from tag numbers only",
                        node=c)

def _trace_outputs(start: str, sig_out: dict[str, list[str]], nodes: dict[str, Node],
                   max_depth: int = 4) -> list[tuple[str, list[str]]]:
    """Follow signals from ``start`` through relays; return (endpoint, relays) pairs."""
    results: list[tuple[str, list[str]]] = []
    stack = [(t, []) for t in sig_out.get(start, [])]
    seen = {start}
    while stack:
        nid, via = stack.pop()
        if nid in seen or len(via) > max_depth:
            continue
        seen.add(nid)
        n = nodes[nid]
        if n.type == "Instrument" and n.attributes.get("role") in ("relay", "other") and sig_out.get(nid):
            for t in sig_out[nid]:
                stack.append((t, via + [nid]))
            continue
        results.append((nid, via))
    return results

def _loop_type(members: dict[str, str], nodes: dict[str, Node],
               sig_purpose: dict[tuple[str, str], str]) -> str:
    roles = set(members.values())
    if any(p == "trip" for (a, _), p in sig_purpose.items() if a in members):
        return "interlock"
    if "controller" in roles:
        return "control"
    variables = {nodes[m].attributes.get("variable") for m in members}
    if "final_element" in roles:
        if "hand" in variables or "hand_switch" in roles:
            return "manual"
        process_switch = any(r in ("switch", "alarm") and nodes[m].attributes.get("variable") not in
                             ("event/state", "unclassified", "position", "hand")
                             for m, r in members.items())
        if process_switch:
            return "interlock"
        if roles & {"transmitter", "element", "sensor_input"}:
            # Sensor and final element share a loop but no controller is drawn:
            # the control function lives in a PLC/DCS not shown on the sheet.
            return "control"
        return "on_off"
    return "monitoring"

def sensed_points(nid: str, nodes: dict[str, Node], edges: list) -> list[str]:
    """Where an instrument senses the process, as human-readable labels."""
    if nodes[nid].attributes.get("inline"):
        return [point_label(nid, nodes, edges)]
    return [point_label(e.target, nodes, edges) for e in edges
            if e.type == "MEASURES" and e.source == nid]

def _measured_points(sensors: list[str], b: "GraphBuilder") -> list[str]:
    out: list[str] = []
    for s in sensors:
        for lbl in sensed_points(s, b.nodes, b.edges):
            if lbl not in out:
                out.append(lbl)
    return out

def _manipulated_points(finals: list[str], b: "GraphBuilder") -> list[str]:
    out: list[str] = []
    for f in finals:
        n = b.nodes[f]
        if n.type == "Equipment":
            continue
        lines = {b.nodes[e.source].attributes.get("line_number") for e in b.edges
                 if e.type == "FLOWS_TO" and e.target == f and b.nodes[e.source].type == "PipeSegment"}
        lines.discard(None)
        if lines:
            out.append(f"{n.label} on {', '.join(sorted(lines))}")
    return out

def point_label(nid: str, nodes: dict[str, Node], edges: list) -> str:
    """Human-readable process location.

    An item gives its label; a pipe segment or an in-line instrument gives
    ``line (nearest upstream item -> nearest downstream item)``.
    """
    n = nodes[nid]
    inline = n.type == "Instrument" and n.attributes.get("inline")
    if n.type != "PipeSegment" and not inline:
        return n.label
    up = _nearest(nid, nodes, edges, upstream=True)
    down = _nearest(nid, nodes, edges, upstream=False)
    line = n.attributes.get("line_number")
    if not line:
        line = next((nodes[e.source].attributes.get("line_number") for e in edges
                     if e.type == "FLOWS_TO" and e.target == nid
                     and nodes[e.source].type == "PipeSegment"), None)
    where = f"{line or 'line'} ({up or '?'} -> {down or '?'})"
    return f"in-line on {where}" if inline else where

def _nearest(start: str, nodes: dict[str, Node], edges: list, upstream: bool,
             limit: int = 50) -> Optional[str]:
    """Nearest equipment/component/boundary label upstream or downstream of a segment."""
    adj: dict[str, list[str]] = defaultdict(list)
    for e in edges:
        if e.type == "FLOWS_TO":
            if upstream:
                adj[e.target].append(e.source)
            else:
                adj[e.source].append(e.target)
    frontier, seen = [start], {start}
    for _ in range(limit):
        nxt = []
        for nid in frontier:
            for other in adj[nid]:
                if other in seen:
                    continue
                seen.add(other)
                if nodes[other].type in ("Equipment", "Boundary", "Component", "Instrument",
                                         "OffPageConnector"):
                    return nodes[other].label
                nxt.append(other)
        if not nxt:
            return None
        frontier = nxt
    return None

def _describe(loop: Node, b: "GraphBuilder") -> str:
    a = loop.attributes
    lbl = lambda ids: ", ".join(b.nodes[i].label for i in ids)  # noqa: E731
    var = a.get("variable") or "process"
    parts = [f"{loop.kind.replace('_', ' ').capitalize()} loop {loop.label} ({var})."]
    if a.get("sensors"):
        where = f" ({'; '.join(a['measured_at'])})" if a.get("measured_at") else ""
        parts.append(f"Sensed by {lbl(a['sensors'])}{where}.")
    if a.get("controllers"):
        locs = {b.nodes[c].attributes.get('location', 'unknown') for c in a["controllers"]}
        parts.append(f"Controlled by {lbl(a['controllers'])} ({', '.join(sorted(locs))}).")
    if a.get("final_elements"):
        placed = {m.split(" on ")[0]: m for m in a.get("manipulated_at", [])}
        acts = [placed.get(b.nodes[f].label, b.nodes[f].label) for f in a["final_elements"]]
        parts.append(f"Acts on {'; '.join(acts)}.")
    if a.get("cascade_slaves"):
        parts.append(f"Cascade master: sets the setpoint of {lbl(a['cascade_slaves'])}.")
    if a.get("cascade_master"):
        parts.append(f"Cascade slave of {b.nodes[a['cascade_master']].label}.")
    return " ".join(parts)
```

## File: skills/pid-to-kg/scripts/pidkg/models.py
size: 9778
sha: 99bc109f63b000ece225b99187bcd83592141053
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/models.py

```text
"""Pydantic models for page extractions and the knowledge graph.

Two schemas live here:

* ``PageExtraction`` is what Claude writes for every P&ID sheet after reading
  it visually (``extract/page_NN.json``). It mirrors how an engineer reads a
  drawing: items on the sheet, the pipe runs between them, and the signal
  lines between instruments.
* ``KnowledgeGraph`` is the canonical output (``graph.json``) produced by
  ``build_kg.py``. It is a typed property graph in node-link form.

JSON Schemas for both are generated into ``references/schema/`` by
``python scripts/build_kg.py --write-schemas``.
"""

from __future__ import annotations

from typing import Any, Literal, Optional, Union

from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator

from . import EXTRACTION_SCHEMA_VERSION, GRAPH_SCHEMA_VERSION

BBox = list[float]  # normalized [x0, y0, x1, y1] on the upright page, 0..1

class _Strict(BaseModel):
    model_config = ConfigDict(extra="forbid", populate_by_name=True)

def _check_bbox(v: Optional[BBox]) -> Optional[BBox]:
    if v is None:
        return v
    if len(v) != 4:
        raise ValueError("bbox must be [x0, y0, x1, y1]")
    if any(c < -0.01 or c > 1.01 for c in v):
        raise ValueError("bbox must be normalized to 0..1 of the page")
    x0, y0, x1, y1 = v
    return [min(x0, x1), min(y0, y1), max(x0, x1), max(y0, y1)]

# ==========================================================================
# Page extraction (written by Claude, one file per sheet)
# ==========================================================================

class ProcessRef(_Strict):
    """A point in the process: an item, or a position on a pipe run.

    Use ``item`` for equipment, components and in-line instruments. Use
    ``line`` (plus optional ``after``) for a tap or tee on a pipe run, where
    ``after`` is the path item immediately upstream of the tap.
    """

    item: Optional[str] = None
    line: Optional[str] = None
    after: Optional[str] = None

    @model_validator(mode="after")
    def _one_target(self) -> "ProcessRef":
        if bool(self.item) == bool(self.line):
            raise ValueError("ProcessRef needs exactly one of 'item' or 'line'")
        if self.after and not self.line:
            raise ValueError("'after' is only valid together with 'line'")
        return self

Endpoint = Union[str, ProcessRef]

class _Item(_Strict):
    id: str = Field(description="Unique within the page. Use the tag when the item has one.")
    tag: Optional[str] = Field(None, description="Tag exactly as printed, e.g. 'FIT-0502'.")
    name: Optional[str] = Field(None, description="Descriptive text printed with the item.")
    status: Optional[Literal["new", "existing", "future", "demolish", "unknown"]] = None
    confidence: float = Field(0.9, ge=0.0, le=1.0)
    bbox: Optional[BBox] = None
    notes: Optional[str] = None
    attributes: dict[str, Any] = Field(default_factory=dict)

    @field_validator("bbox")
    @classmethod
    def _bbox_ok(cls, v: Optional[BBox]) -> Optional[BBox]:
        return _check_bbox(v)

class Equipment(_Item):
    kind: str = Field(description="Vocabulary in references/ontology.md, e.g. 'tank', 'centrifugal_pump'.")
    part_of: Optional[str] = Field(None, description="Id of the equipment this is mounted on (agitator on a tank, drive on a pump).")

class Component(_Item):
    kind: str = Field(description="Vocabulary in references/ontology.md, e.g. 'gate_valve', 'control_valve'.")
    actuator: Optional[
        Literal["manual", "pneumatic", "electric", "hydraulic", "solenoid", "self", "none"]
    ] = None
    fail_position: Optional[Literal["FO", "FC", "FL", "FI"]] = None
    normal_position: Optional[Literal["open", "closed"]] = None
    size: Optional[str] = None
    loop: Optional[str] = Field(None, description="Override the ISA loop this item belongs to.")

class Instrument(_Item):
    tag: Optional[str] = Field(None, description="Printed tag. Leave empty for untagged gauges and give 'function'.")
    function: Optional[str] = Field(None, description="ISA letters for an untagged instrument, e.g. 'PI' for a bare pressure gauge.")
    location: str = Field("unknown", description="field | local_panel | main_panel | behind_panel | dcs | plc | scada | sis | computer | unknown")
    measures: Optional[ProcessRef] = Field(None, description="Where the instrument senses the process.")
    loop: Optional[str] = Field(None, description="Override the ISA loop this instrument belongs to.")

    @model_validator(mode="after")
    def _tag_or_function(self) -> "Instrument":
        if not self.tag and not self.function:
            raise ValueError("an instrument needs a 'tag', or a 'function' (e.g. 'PI') if untagged")
        return self

class Line(_Strict):
    id: str = Field(description="Unique within the page, e.g. 'L1'.")
    line_number: Optional[str] = None
    size: Optional[str] = None
    service: Optional[str] = Field(None, description="Fluid/service code or description.")
    from_: Endpoint = Field(alias="from")
    to: Endpoint
    path: list[str] = Field(default_factory=list, description="Ids of in-line items in flow order.")
    flow_direction: Literal["shown", "inferred"] = "shown"
    status: Optional[Literal["new", "existing", "future", "demolish", "unknown"]] = None
    confidence: float = Field(0.85, ge=0.0, le=1.0)
    notes: Optional[str] = None

class Signal(_Strict):
    from_: str = Field(alias="from")
    to: str
    type: str = Field("unknown", description="electrical | pneumatic | software | hydraulic | capillary | wireless | mechanical | unknown")
    purpose: Literal["control", "measurement", "trip", "alarm", "status", "unknown"] = "unknown"
    confidence: float = Field(0.85, ge=0.0, le=1.0)
    notes: Optional[str] = None

class OffPageConnector(_Item):
    direction: Literal["in", "out"]
    other_drawing: Optional[str] = Field(None, description="Drawing number the flow continues on/comes from.")
    other_item: Optional[str] = Field(None, description="Item named at the far end, e.g. '660-TK-004'.")
    line_number: Optional[str] = None

class Boundary(_Item):
    direction: Literal["source", "sink", "area"] = Field(
        description="source/sink for streams entering or leaving; 'area' for a room or zone that "
                    "area instruments (gas detectors, room temperature) measure")

class SheetInfo(_Strict):
    drawing_number: Optional[str] = None
    title: Optional[str] = None
    revision: Optional[str] = None
    drawing_type: Literal["pid", "pfd", "other"] = "pid"
    issue_status: Optional[str] = Field(None, description="As printed: 'as-built', 'issued for construction', 'draft', 'superseded'...")
    rotation: Optional[int] = None
    legend_notes: list[str] = Field(default_factory=list)

class PageExtraction(_Strict):
    schema_version: str = EXTRACTION_SCHEMA_VERSION
    page: int = Field(ge=1, description="1-based page number in the input PDF.")
    sheet: SheetInfo = Field(default_factory=SheetInfo)
    equipment: list[Equipment] = Field(default_factory=list)
    components: list[Component] = Field(default_factory=list)
    instruments: list[Instrument] = Field(default_factory=list)
    lines: list[Line] = Field(default_factory=list)
    signals: list[Signal] = Field(default_factory=list)
    off_page_connectors: list[OffPageConnector] = Field(default_factory=list)
    boundaries: list[Boundary] = Field(default_factory=list)
    tiles_reviewed: list[str] = Field(default_factory=list)
    unreadable: list[str] = Field(default_factory=list, description="Things seen but not legible.")

    @model_validator(mode="after")
    def _unique_ids(self) -> "PageExtraction":
        seen: dict[str, str] = {}
        groups = {
            "equipment": self.equipment,
            "components": self.components,
            "instruments": self.instruments,
            "off_page_connectors": self.off_page_connectors,
            "boundaries": self.boundaries,
            "lines": self.lines,
        }
        for group, items in groups.items():
            for it in items:
                if it.id in seen:
                    raise ValueError(f"duplicate id '{it.id}' in {group} and {seen[it.id]}")
                seen[it.id] = group
        return self

# ==========================================================================
# Knowledge graph (written by build_kg.py)
# ==========================================================================

class Provenance(_Strict):
    sheet: str
    page: int
    bbox: Optional[BBox] = None
    source: str = "vision"
    local_id: Optional[str] = None

class Node(_Strict):
    id: str
    type: str
    label: str
    tag: Optional[str] = None
    kind: Optional[str] = None
    name: Optional[str] = None
    status: Optional[str] = None
    dexpi_class: Optional[str] = None
    sheets: list[str] = Field(default_factory=list)
    confidence: float = 1.0
    attributes: dict[str, Any] = Field(default_factory=dict)
    provenance: list[Provenance] = Field(default_factory=list)

class Edge(_Strict):
    id: str
    source: str
    target: str
    type: str
    confidence: float = 1.0
    derived: bool = False
    attributes: dict[str, Any] = Field(default_factory=dict)
    sheet: Optional[str] = None

class PlantInfo(_Strict):
    id: str
    name: str
    source_file: Optional[str] = None
    sheets: list[str] = Field(default_factory=list)

class KnowledgeGraph(_Strict):
    schema_version: str = GRAPH_SCHEMA_VERSION
    generator: str
    generated_at: str
    plant: PlantInfo
    nodes: list[Node]
    edges: list[Edge]
    stats: dict[str, Any] = Field(default_factory=dict)
    issues: list[dict[str, Any]] = Field(default_factory=list)
```

## File: skills/pid-to-kg/scripts/pidkg/ocr.py
size: 6694
sha: 9ba3ecc29a6ec58101933c1bfe1515ef99819fe3
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/ocr.py

```text
"""Optional OCR (RapidOCR, ONNX runtime; pip-installable, no system binaries).

OCR is used for two things only:

* detecting page orientation, and
* producing *hints*: candidate tags with positions that QA compares against
  Claude's visual extraction to flag possible misses.

It never writes to the graph directly.
"""

from __future__ import annotations

import re
from typing import Optional

from .isa import normalize_tag, parse_isa_tag

_engine = None

EQUIPMENT_LIKE = re.compile(r"^(\d{2,4}-)?[A-Z]{2,4}-\d{2,5}[A-Z]?$")
_LETTERS = re.compile(r"^[A-Z]{1,5}$")
_DIGITS = re.compile(r"^\d{2,5}[A-Z]?$")

def available() -> bool:
    try:
        import rapidocr_onnxruntime  # noqa: F401
        return True
    except ImportError:
        return False

def _get_engine():
    global _engine
    if _engine is None:
        from rapidocr_onnxruntime import RapidOCR

        _engine = RapidOCR()
    return _engine

def ocr_image(img) -> list[dict]:
    """OCR a PIL image. Returns [{'text', 'bbox' (pixels x0,y0,x1,y1), 'conf'}]."""
    import numpy as np

    arr = np.array(img.convert("RGB"))
    result, _ = _get_engine()(arr, use_cls=False)
    out = []
    for box, text, conf in result or []:
        xs = [p[0] for p in box]
        ys = [p[1] for p in box]
        out.append({"text": str(text).strip(), "bbox": [min(xs), min(ys), max(xs), max(ys)],
                    "conf": float(conf)})
    return out

def _upright_boxes(results: list[dict]) -> list[dict]:
    """Confident, horizontal, alphanumeric OCR boxes.

    Only horizontal boxes count: the recognizer silently turns tall crops, so
    sideways text would otherwise read in every orientation.
    """
    keep = []
    for r in results:
        t = re.sub(r"[^A-Za-z0-9]", "", r["text"])
        x0, y0, x1, y1 = r["bbox"]
        if (x1 - x0) > 1.2 * (y1 - y0) and len(t) >= 2 and r["conf"] >= 0.6:
            keep.append(r)
    return keep

def detect_rotation(img, rotations=(0, 90, 180, 270)) -> tuple[int, dict[int, float], Optional[float]]:
    """Find the clockwise rotation that makes the text upright, and measure text size.

    Returns (rotation, score per rotation, 25th-percentile text box height in
    pixels of ``img`` at that rotation, or None if no text was found).
    """
    scores: dict[int, float] = {}
    heights: dict[int, list[float]] = {}
    for rot in rotations:
        # PIL rotates counter-clockwise; negate to match the clockwise convention.
        test = img.rotate(-rot, expand=True) if rot else img
        boxes = _upright_boxes(ocr_image(test))
        scores[rot] = round(sum(r["conf"] * len(re.sub(r"[^A-Za-z0-9]", "", r["text"]))
                                for r in boxes), 1)
        heights[rot] = sorted(r["bbox"][3] - r["bbox"][1] for r in boxes)
    best = max(scores, key=scores.get)
    hs = heights[best]
    p25 = hs[len(hs) // 4] if len(hs) >= 8 else None
    return best, scores, p25

def join_bubbles(runs: list[dict]) -> list[dict]:
    """Join ISA bubble text split over two lines ('FIT' above '0502') into one run.

    ``runs`` use normalized page bboxes. Joined runs are added; originals kept.
    """
    letters = [r for r in runs if _LETTERS.match(r["text"].replace(" ", ""))]
    digits = [r for r in runs if _DIGITS.match(r["text"].replace(" ", ""))]
    joined = []
    for a in letters:
        ax0, ay0, ax1, ay1 = a["bbox"]
        h = ay1 - ay0
        acx = (ax0 + ax1) / 2
        best, best_gap = None, None
        for d in digits:
            dx0, dy0, dx1, dy1 = d["bbox"]
            dcx = (dx0 + dx1) / 2
            gap = dy0 - ay1
            width = max(ax1 - ax0, dx1 - dx0)
            if -0.5 * h <= gap <= 1.5 * h and abs(acx - dcx) <= 0.6 * width:
                if best_gap is None or gap < best_gap:
                    best, best_gap = d, gap
        if best is not None:
            text = f"{a['text'].replace(' ', '')}-{best['text'].replace(' ', '')}"
            bb = [min(ax0, best["bbox"][0]), ay0, max(ax1, best["bbox"][2]), best["bbox"][3]]
            joined.append({"text": text, "bbox": [round(v, 4) for v in bb],
                           "conf": round(min(a.get("conf", 1), best.get("conf", 1)), 3),
                           "joined": True})
    return runs + joined

def tag_candidates(runs: list[dict]) -> list[dict]:
    """Pick tag-like strings from text runs, de-duplicated by normalized tag."""
    seen: dict[str, dict] = {}
    for r in runs:
        tokens = [t for t in re.split(r"[\s,;()]+", r["text"]) if t]
        # Tags are often printed with a space ('FIT 0502'): also try adjacent pairs.
        pairs = [f"{a}-{b}" for a, b in zip(tokens, tokens[1:])
                 if _LETTERS.match(a) and _DIGITS.match(b)]
        for token in tokens + pairs:
            t = normalize_tag(token)
            if not t or len(t) < 4:
                continue
            isa = parse_isa_tag(t)
            eq = bool(EQUIPMENT_LIKE.match(t))
            if not isa and not eq:
                continue
            if t not in seen or r.get("conf", 0) > seen[t]["conf"]:
                seen[t] = {"text": t, "bbox": r["bbox"], "conf": r.get("conf", 1.0),
                           "isa": bool(isa), "equipment_like": eq}
    return sorted(seen.values(), key=lambda c: (c["bbox"][1], c["bbox"][0]))

def dedupe_runs(runs: list[dict], tol: float = 0.004) -> list[dict]:
    """Drop runs seen twice in overlapping tiles (same text, nearly same position)."""
    kept: list[dict] = []
    for r in sorted(runs, key=lambda r: -r.get("conf", 0)):
        cx = (r["bbox"][0] + r["bbox"][2]) / 2
        cy = (r["bbox"][1] + r["bbox"][3]) / 2
        dup = False
        for k in kept:
            if k["text"] != r["text"]:
                continue
            kx = (k["bbox"][0] + k["bbox"][2]) / 2
            ky = (k["bbox"][1] + k["bbox"][3]) / 2
            if abs(kx - cx) < tol and abs(ky - cy) < tol:
                dup = True
                break
        if not dup:
            kept.append(r)
    return kept

def to_norm(bbox_px: list[float], tile_px: list[int], page_w: int, page_h: int,
            tile_scale: float = 1.0) -> list[float]:
    """Convert a bbox in tile pixels to normalized page coordinates."""
    x0, y0, x1, y1 = (v / tile_scale for v in bbox_px)
    return [round((tile_px[0] + x0) / page_w, 4), round((tile_px[1] + y0) / page_h, 4),
            round((tile_px[0] + x1) / page_w, 4), round((tile_px[1] + y1) / page_h, 4)]

def maybe_import_error() -> Optional[str]:
    if available():
        return None
    return ("RapidOCR is not installed; OCR hints and automatic rotation are skipped. "
            "Install with: pip install rapidocr-onnxruntime")
```

## File: skills/pid-to-kg/scripts/pidkg/ontology.py
size: 7961
sha: ef58d2c495857001008e7f653642b11e02b44f0f
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/ontology.py

```text
"""Controlled vocabularies for the P&ID knowledge graph.

Vocabularies are deliberately permissive: an unknown ``kind`` is accepted and
reported by QA rather than rejected, because real drawings always contain
something the vocabulary did not anticipate.

``DEXPI_CLASS`` maps each kind to the closest DEXPI (ISO 15926-based P&ID
exchange standard) class name. The mapping is informative: it helps users
align the graph with DEXPI tooling but is not a certified DEXPI export.
"""

from __future__ import annotations

# --------------------------------------------------------------------------
# Node and edge types
# --------------------------------------------------------------------------

NODE_TYPES = (
    "Plant",
    "Sheet",
    "Equipment",
    "PipeSegment",
    "Component",
    "Instrument",
    "ControlLoop",
    "OffPageConnector",
    "Boundary",
)

EDGE_TYPES = (
    # Physical process connectivity (directed in the direction of flow).
    "FLOWS_TO",
    # Derived equipment-to-equipment flow, collapsing pipes and fittings.
    "FEEDS",
    # Instrument -> the equipment, pipe segment or component it senses.
    "MEASURES",
    # Instrument/control signal (directed from sender to receiver).
    "SIGNAL_TO",
    # Controller or logic -> the final element it moves (valve, VFD/motor).
    "ACTUATES",
    # Instrument/component -> ControlLoop, with a ``role`` attribute.
    "MEMBER_OF",
    # ControlLoop -> ControlLoop (master -> slave in a cascade).
    "CASCADES_TO",
    # Safety/interlock action: initiator -> the item it trips.
    "TRIPS",
    # Sheet -> Plant; mounted equipment -> its parent (agitator -> tank).
    "PART_OF",
)

# Edge types followed when tracing upstream/downstream process flow.
FLOW_EDGE_TYPES = ("FLOWS_TO",)

# --------------------------------------------------------------------------
# Equipment
# --------------------------------------------------------------------------

EQUIPMENT_DEXPI = {
    "tank": "Tank",
    "vessel": "Vessel",
    "pressure_vessel": "PressureVessel",
    "pump": "Pump",
    "centrifugal_pump": "CentrifugalPump",
    "positive_displacement_pump": "ReciprocatingPump",
    "metering_pump": "ReciprocatingPump",
    "slurry_pump": "CentrifugalPump",
    "sump_pump": "CentrifugalPump",
    "compressor": "Compressor",
    "blower": "Blower",
    "fan": "Fan",
    "heat_exchanger": "HeatExchanger",
    "cooler": "HeatExchanger",
    "heater": "HeatExchanger",
    "boiler": "Boiler",
    "furnace": "Furnace",
    "column": "Column",
    "reactor": "Vessel",
    "agitated_tank": "Tank",
    "agitator": "Agitator",
    "mixer": "Mixer",
    "static_mixer": "StaticMixer",
    "filter": "Filter",
    "filter_press": "Filter",
    "membrane_filter": "Filter",
    "strainer_unit": "Filter",
    "screen": "Sieve",
    "cyclone": "CycloneSeparator",
    "hydrocyclone": "CycloneSeparator",
    "separator": "Separator",
    "clarifier": "Separator",
    "thickener": "Separator",
    "flotation_cell": "Separator",
    "centrifuge": "Centrifuge",
    "dryer": "Dryer",
    "mill": "Mill",
    "crusher": "Crusher",
    "conveyor": "Conveyor",
    "feeder": "Feeder",
    "hopper": "Silo",
    "bin": "Silo",
    "silo": "Silo",
    "sump": "Tank",
    "pond": "Tank",
    "box": "Tank",
    "electrowinning_cell": "Vessel",
    "kiln": "Furnace",
    "scrubber": "Column",
    "ion_exchange": "Column",
    "uv_unit": "Equipment",
    "sampler": "Equipment",
    "package_unit": "Equipment",
    "motor": "ElectricMotor",
    "other": "Equipment",
}
EQUIPMENT_KINDS = tuple(EQUIPMENT_DEXPI)

# --------------------------------------------------------------------------
# In-line piping components
# --------------------------------------------------------------------------

COMPONENT_DEXPI = {
    "manual_valve": "OperatedValve",
    "gate_valve": "GateValve",
    "globe_valve": "GlobeValve",
    "ball_valve": "BallValve",
    "butterfly_valve": "ButterflyValve",
    "plug_valve": "PlugValve",
    "knife_gate_valve": "GateValve",
    "diaphragm_valve": "DiaphragmValve",
    "pinch_valve": "OperatedValve",
    "needle_valve": "NeedleValve",
    "three_way_valve": "OperatedValve",
    "check_valve": "CheckValve",
    "backflow_preventer": "CheckValve",
    "control_valve": "OperatedValve",
    "on_off_valve": "OperatedValve",
    "solenoid_valve": "OperatedValve",
    "motor_operated_valve": "OperatedValve",
    "pressure_regulator": "OperatedValve",
    "relief_valve": "SafetyValveOrFitting",
    "rupture_disc": "RuptureDisc",
    "vacuum_breaker": "SafetyValveOrFitting",
    "strainer": "Strainer",
    "filter": "Filter",
    "reducer": "PipeReducer",
    "blind": "Blind",
    "spectacle_blind": "SpectacleBlind",
    "flexible_hose": "Hose",
    "expansion_joint": "Compensator",
    "sight_glass": "SightGlass",
    "steam_trap": "SteamTrap",
    "flame_arrestor": "FlameArrestor",
    "sample_point": "PipingComponent",
    "drain": "PipingComponent",
    "vent": "PipingComponent",
    "quick_connect": "PipingComponent",
    "tee": "PipeTee",
    "other": "PipingComponent",
}
COMPONENT_KINDS = tuple(COMPONENT_DEXPI)

# Components that can be the final element of a control or trip loop.
ACTUATED_COMPONENT_KINDS = {
    "control_valve",
    "on_off_valve",
    "solenoid_valve",
    "motor_operated_valve",
    "three_way_valve",
}

# --------------------------------------------------------------------------
# Instruments
# --------------------------------------------------------------------------

INSTRUMENT_LOCATIONS = (
    "field",          # plain circle
    "local_panel",    # circle with line, dashed or in a local/field panel
    "main_panel",     # circle with solid line: main control room panel
    "behind_panel",   # circle with dashed line
    "dcs",            # circle in square: shared display / DCS
    "plc",            # diamond in square
    "scada",
    "sis",            # safety instrumented system
    "computer",       # hexagon
    "unknown",
)

SIGNAL_TYPES = (
    "electrical",   # dashed line
    "pneumatic",    # line with double cross-hatches
    "software",     # line with circles / data link
    "hydraulic",
    "capillary",    # line with x's
    "wireless",     # zig-zag / wave
    "mechanical",
    "unknown",
)

STATUSES = ("new", "existing", "future", "demolish", "unknown")

# ISA-5.1 first letters (measured or initiating variable).
ISA_MEASURED = {
    "A": "analysis",
    "B": "burner/flame",
    "C": "conductivity",      # user's choice; conductivity is the most common
    "D": "density",           # user's choice
    "E": "voltage",
    "F": "flow",
    "G": "gauging/position",  # user's choice
    "H": "hand",
    "I": "current",
    "J": "power",
    "K": "time",
    "L": "level",
    "M": "moisture",          # user's choice
    "N": "user-defined",
    "O": "user-defined",
    "P": "pressure",
    "Q": "quantity",
    "R": "radiation",
    "S": "speed/frequency",
    "T": "temperature",
    "U": "multivariable",
    "V": "vibration",
    "W": "weight/force",
    "X": "unclassified",
    "Y": "event/state",
    "Z": "position",
}

# Variable modifiers that may follow the first letter.
ISA_VARIABLE_MODIFIERS = {
    "D": "differential",
    "F": "ratio",
    "J": "scan",
    "K": "rate of change",
    "Q": "integrate/totalize",
}

# Succeeding letters (readout/passive or output functions).
ISA_FUNCTIONS = {
    "A": "alarm",
    "B": "user's choice",
    "C": "control",
    "D": "differential",
    "E": "primary element",
    "G": "glass/gauge",
    "H": "high",
    "I": "indicate",
    "K": "control station",
    "L": "light/low",
    "M": "middle",
    "N": "user's choice",
    "O": "orifice/restriction",
    "P": "test point",
    "Q": "totalize",
    "R": "record",
    "S": "switch",
    "T": "transmit",
    "U": "multifunction",
    "V": "valve/damper",
    "W": "well",
    "X": "unclassified",
    "Y": "relay/compute/convert",
    "Z": "driver/actuator",
}
```

## File: skills/pid-to-kg/scripts/pidkg/pdf.py
size: 8529
sha: 305445be2e91b819ade57cd2279faf2ca02baa13
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/pdf.py

```text
"""PDF inspection and rendering (pypdfium2: Apache-2.0/BSD, no system dependencies)."""

from __future__ import annotations

import math
from dataclasses import asdict, dataclass, field
from typing import Optional

import pypdfium2 as pdfium
import pypdfium2.raw as pdfium_c

# Page classes and how each is extracted.
ROUTES = {
    "vector_text": "Vector drawing with a text layer: render from vectors, take exact text and "
                   "positions from the text layer as hints, then read visually.",
    "vector_outlined": "Vector drawing whose text was converted to outlines (no text layer): "
                       "render from vectors at high resolution, OCR the render for hints, then "
                       "read visually.",
    "raster": "Scanned image: render at the scan's native resolution, upscale and enhance "
              "contrast, OCR for hints, then read visually.",
    "raster_with_text": "Scanned image with an OCR text layer: as raster, and use the existing "
                        "text layer as (untrusted) hints.",
    "hybrid": "Mix of vector content and large images: render from vectors at high "
              "resolution and use both text layer and OCR hints.",
    "blank": "No meaningful content.",
}

MIN_TEXT_CHARS = 50
MIN_VECTOR_PATHS = 200
BIG_IMAGE_COVERAGE = 0.5

@dataclass
class PageProbe:
    page: int
    width_pt: float
    height_pt: float
    pdf_rotation: int
    text_chars: int
    path_objects: int
    text_objects: int
    image_objects: int
    max_image_coverage: float
    image_px: Optional[list[int]]
    effective_image_dpi: Optional[float]
    page_class: str
    route: str = ""
    rotation: Optional[int] = None
    notes: list[str] = field(default_factory=list)

    def to_dict(self) -> dict:
        return asdict(self)

def probe_pdf(path: str) -> list[PageProbe]:
    pdf = pdfium.PdfDocument(path)
    try:
        return [probe_page(pdf, i) for i in range(len(pdf))]
    finally:
        pdf.close()

def probe_page(pdf: "pdfium.PdfDocument", index: int) -> PageProbe:
    page = pdf[index]
    w, h = page.get_size()
    area = max(w * h, 1.0)
    tp = page.get_textpage()
    text = tp.get_text_range() or ""
    chars = sum(1 for ch in text if not ch.isspace())
    counts = {pdfium_c.FPDF_PAGEOBJ_PATH: 0, pdfium_c.FPDF_PAGEOBJ_TEXT: 0, pdfium_c.FPDF_PAGEOBJ_IMAGE: 0}
    best_cov, best_px, best_dpi = 0.0, None, None
    for obj in page.get_objects(max_depth=8):
        if obj.type in counts:
            counts[obj.type] += 1
        if obj.type == pdfium_c.FPDF_PAGEOBJ_IMAGE:
            l, b, r, t = obj.get_bounds()
            cov = max(0.0, (r - l) * (t - b)) / area
            if cov > best_cov:
                best_cov = cov
                try:
                    px = list(obj.get_px_size())
                    best_px = px
                    best_dpi = round(72.0 * max(px) / max(r - l, t - b, 1e-6), 1)
                except Exception:  # pragma: no cover - exotic image types
                    best_px, best_dpi = None, None
    paths = counts[pdfium_c.FPDF_PAGEOBJ_PATH]
    has_text = chars >= MIN_TEXT_CHARS
    has_vec = paths >= MIN_VECTOR_PATHS
    big_img = best_cov >= BIG_IMAGE_COVERAGE
    if big_img and not has_vec:
        cls = "raster_with_text" if has_text else "raster"
    elif big_img and has_vec:
        cls = "hybrid"
    elif has_text:
        cls = "vector_text"
    elif has_vec:
        cls = "vector_outlined"
    elif paths or counts[pdfium_c.FPDF_PAGEOBJ_IMAGE]:
        cls = "vector_outlined" if paths else "raster"
    else:
        cls = "blank"
    probe = PageProbe(
        page=index + 1, width_pt=round(w, 1), height_pt=round(h, 1),
        pdf_rotation=page.get_rotation(), text_chars=chars, path_objects=paths,
        text_objects=counts[pdfium_c.FPDF_PAGEOBJ_TEXT],
        image_objects=counts[pdfium_c.FPDF_PAGEOBJ_IMAGE],
        max_image_coverage=round(best_cov, 3), image_px=best_px, effective_image_dpi=best_dpi,
        page_class=cls, route=ROUTES[cls])
    if cls.startswith("raster") and best_dpi and best_dpi < 200:
        probe.notes.append(f"low scan resolution (~{best_dpi:.0f} dpi): small text may be "
                           "illegible; record unreadable items in 'unreadable'")
    if max(w, h) < 1000:
        probe.notes.append("page is smaller than a typical drawing sheet; the drawing was "
                           "probably reduced, so render at high resolution")
    return probe

def render_page(pdf_path: str, page_no: int, page_class: str, rotation: int = 0,
                vector_long_edge: int = 5000, raster_long_edge: int = 4000,
                max_upscale: float = 2.5):
    """Render a page to a PIL image following the route for its class."""
    from PIL import Image, ImageOps

    pdf = pdfium.PdfDocument(pdf_path)
    try:
        page = pdf[page_no - 1]
        w, h = page.get_size()
        if page_class.startswith("raster"):
            probe = probe_page(pdf, page_no - 1)
            native = (probe.effective_image_dpi or 200) / 72.0
            img = page.render(scale=native, rotation=rotation, grayscale=True).to_pil()
            long_edge = max(img.size)
            factor = min(max_upscale, max(1.0, raster_long_edge / long_edge))
            if factor > 1.0:
                img = img.resize((round(img.width * factor), round(img.height * factor)),
                                 Image.LANCZOS)
            img = ImageOps.autocontrast(img.convert("L"), cutoff=0.5)
            return img
        scale = vector_long_edge / max(w, h)
        return page.render(scale=scale, rotation=rotation).to_pil().convert("RGB")
    finally:
        pdf.close()

def tile_grid(width: int, height: int, tile: int = 1400, overlap: int = 200) -> list[dict]:
    """Evenly spaced overlapping tiles covering the image."""
    def spans(total: int) -> list[tuple[int, int]]:
        if total <= tile:
            return [(0, total)]
        n = math.ceil((total - overlap) / (tile - overlap))
        step = (total - tile) / (n - 1)
        return [(round(i * step), round(i * step) + tile) for i in range(n)]

    tiles = []
    for r, (y0, y1) in enumerate(spans(height), 1):
        for c, (x0, x1) in enumerate(spans(width), 1):
            tiles.append({"name": f"r{r}c{c}", "row": r, "col": c, "px": [x0, y0, x1, y1],
                          "norm": [round(x0 / width, 4), round(y0 / height, 4),
                                   round(x1 / width, 4), round(y1 / height, 4)]})
    return tiles

def ink_fraction(img, threshold: int = 200) -> float:
    """Fraction of pixels darker than ``threshold``, ignoring frame lines.

    Rows and columns that are dark across most of the tile (the drawing
    border, title-block rules) are not content, so they are excluded; a tile
    holding only a border line then counts as blank.
    """
    g = img.convert("L")
    try:
        import numpy as np
    except ImportError:  # pragma: no cover - numpy ships with the OCR extra
        hist = g.histogram()
        return sum(hist[:threshold]) / max(1, g.width * g.height)
    dark = np.asarray(g) < threshold
    rows = dark.mean(axis=1) > 0.6
    cols = dark.mean(axis=0) > 0.6
    dark[rows, :] = False
    dark[:, cols] = False
    return float(dark.mean())

def rotate_norm_bbox(b: list[float], rotation: int) -> list[float]:
    """Map a normalized bbox on the unrotated page onto the page rotated clockwise."""
    x0, y0, x1, y1 = b
    pts = [(x0, y0), (x1, y1)]
    if rotation == 90:
        pts = [(1 - y, x) for x, y in pts]
    elif rotation == 180:
        pts = [(1 - x, 1 - y) for x, y in pts]
    elif rotation == 270:
        pts = [(y, 1 - x) for x, y in pts]
    (a, b2), (c, d) = pts
    return [round(min(a, c), 4), round(min(b2, d), 4), round(max(a, c), 4), round(max(b2, d), 4)]

def text_runs(pdf_path: str, page_no: int, rotation: int = 0) -> list[dict]:
    """Text-layer runs with normalized bboxes on the (rotated) page."""
    pdf = pdfium.PdfDocument(pdf_path)
    try:
        page = pdf[page_no - 1]
        w, h = page.get_size()
        tp = page.get_textpage()
        runs = []
        for i in range(tp.count_rects()):
            l, b, r, t = tp.get_rect(i)
            txt = (tp.get_text_bounded(l, b, r, t) or "").strip()
            if not txt:
                continue
            norm = [l / w, 1 - t / h, r / w, 1 - b / h]
            runs.append({"text": txt, "bbox": rotate_norm_bbox(norm, rotation), "conf": 1.0})
        return runs
    finally:
        pdf.close()
```

## File: skills/pid-to-kg/scripts/pidkg/qa.py
size: 12039
sha: 24bf8fa4eaa2f485aa6cb87f2630c97bb6c07cea
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/qa.py

```text
"""Quality checks on a built graph, and the QA report Claude uses for its second pass."""

from __future__ import annotations

import json
import re
from collections import defaultdict
from pathlib import Path
from typing import Any, Optional

from .isa import SENSING_ROLES, normalize_tag, parse_isa_tag
from .models import KnowledgeGraph

LOW_CONFIDENCE = 0.6
_MONTHS = re.compile(r"^(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)[A-Z]*-\d")

def run_qa(kg: KnowledgeGraph, work_dir: Optional[Path] = None) -> list[dict[str, Any]]:
    """Return graph-level issues (build issues are already in ``kg.issues``)."""
    issues: list[dict[str, Any]] = []
    nodes = {n.id: n for n in kg.nodes}
    flow_deg: dict[str, int] = defaultdict(int)
    any_deg: dict[str, int] = defaultdict(int)
    measured: set[str] = set()
    for e in kg.edges:
        any_deg[e.source] += 1
        any_deg[e.target] += 1
        if e.type == "FLOWS_TO":
            flow_deg[e.source] += 1
            flow_deg[e.target] += 1
        elif e.type == "MEASURES":
            measured.add(e.source)

    def add(sev: str, code: str, msg: str, node=None) -> None:
        n = nodes.get(node) if node else None
        bbox = next((p.bbox for p in n.provenance if p.bbox), None) if n else None
        sheet = n.sheets[0] if n and n.sheets else None
        page = n.provenance[0].page if n and n.provenance else None
        issues.append({k: v for k, v in dict(severity=sev, code=code, message=msg, node=node,
                                             sheet=sheet, page=page, bbox=bbox).items() if v is not None})

    for n in kg.nodes:
        if n.type == "Equipment" and flow_deg[n.id] == 0 and not n.attributes.get("part_of"):
            add("warning", "equipment_not_connected",
                f"equipment {n.label} has no process connections", n.id)
        elif n.type == "Component" and flow_deg[n.id] == 0:
            add("warning", "component_not_connected",
                f"component {n.label} is not on any line (add it to a line 'path')", n.id)
        elif n.type == "Instrument":
            role = n.attributes.get("role")
            loc = n.attributes.get("location", "unknown")
            if role in SENSING_ROLES and loc in ("field", "unknown") and n.id not in measured \
                    and not n.attributes.get("inline"):
                add("warning", "instrument_not_located",
                    f"{n.label} ({role}) senses the process but has no 'measures' location", n.id)
            if any_deg[n.id] == 0:
                add("info", "isolated_instrument", f"{n.label} has no connections at all", n.id)
        if n.type not in ("Plant", "Sheet") and n.confidence < LOW_CONFIDENCE:
            add("info", "low_confidence", f"{n.label} was read with confidence {n.confidence:.2f}", n.id)

    # Tags that differ only in zero padding ('660-TK-06' vs '660-TK-006').
    padded: dict[str, list[str]] = defaultdict(list)
    for n in kg.nodes:
        if n.tag and n.type == "Equipment":  # instrument loop numbers legitimately vary
            padded[re.sub(r"(?<![0-9])0+(?=[0-9])", "", n.tag.split("@")[0].split("#")[0])].append(n.id)
    for ids in padded.values():
        tags = sorted({nodes[i].tag for i in ids})
        if len(tags) > 1:
            add("warning", "near_duplicate_tags",
                f"tags {', '.join(tags)} differ only in zero padding; probably one item "
                "drawn with inconsistent tags (check the sheets)", ids[0])

    issues.extend(_revision_overlap(kg))

    inferred = sum(1 for e in kg.edges if e.type == "FLOWS_TO"
                   and e.attributes.get("flow_direction") == "inferred")
    if inferred:
        issues.append({"severity": "info", "code": "inferred_flow_direction",
                       "message": f"{inferred} flow edges have inferred (not drawn) direction"})

    if work_dir:
        issues.extend(_coverage_issues(kg, Path(work_dir)))
        issues.extend(_hint_issues(kg, Path(work_dir)))
    return issues

def _revision_overlap(kg: KnowledgeGraph) -> list[dict[str, Any]]:
    """Two sheets sharing most of their tags are probably two revisions of one drawing."""
    tags: dict[str, set[str]] = defaultdict(set)
    for n in kg.nodes:
        if n.tag and n.type in ("Equipment", "Instrument", "Component"):
            for s in {p.sheet for p in n.provenance}:
                tags[s].add(n.tag)
    out = []
    sheets = sorted(t for t in tags if len(tags[t]) >= 5)
    for i, a in enumerate(sheets):
        for b in sheets[i + 1:]:
            shared = tags[a] & tags[b]
            frac = len(shared) / min(len(tags[a]), len(tags[b]))
            if frac >= 0.5:
                out.append({"severity": "warning", "code": "possible_duplicate_revision", "sheet": b,
                            "message": f"{a} and {b} share {len(shared)} tags ({frac:.0%} of the smaller sheet); "
                                       "if they are two revisions of the same drawing, keep only the current one "
                                       "in this graph and ask the user what to do with the other"})
    return out

def _coverage_issues(kg: KnowledgeGraph, work: Path) -> list[dict[str, Any]]:
    out = []
    for n in kg.nodes:
        if n.type != "Sheet":
            continue
        page = n.attributes.get("page")
        tiles_file = work / "pages" / f"page_{page:02d}" / "tiles.json"
        if not tiles_file.exists():
            continue
        tiles = json.loads(tiles_file.read_text(encoding="utf-8"))
        expected = {t["name"] for t in tiles["tiles"] if not t.get("blank")}
        reviewed = set(n.attributes.get("tiles_reviewed", []))
        missing = sorted(expected - reviewed)
        if missing:
            out.append({"severity": "warning", "code": "tiles_not_reviewed", "sheet": n.id,
                        "page": page, "message": f"{len(missing)} non-blank tiles not listed in "
                        f"tiles_reviewed: {', '.join(missing)}"})
    return out

def _hint_issues(kg: KnowledgeGraph, work: Path) -> list[dict[str, Any]]:
    """Compare tags found by OCR / the text layer with tags in the graph."""
    out = []
    known = {n.tag for n in kg.nodes if n.tag} | {n.id for n in kg.nodes}
    known_core = {_core(t) for t in known if t}
    # Drawing numbers (title blocks, off-page flags) look like tags but are not items.
    drawings = {_core(normalize_tag(v)) for n in kg.nodes for v in
                (n.attributes.get("drawing_number"), n.attributes.get("other_drawing")) if v}
    # Text already captured in names/attributes (off-page 'other_item', tie-in labels...).
    blob = _core(normalize_tag(" ".join(
        f"{n.name or ''} {n.label} {json.dumps(n.attributes)}" for n in kg.nodes)) or "")
    by_number: dict[str, set[str]] = defaultdict(set)
    for t in known:
        p = parse_isa_tag(t) if t else None
        if p:
            by_number[p.number].add(p.letters)
    for n in kg.nodes:
        if n.type != "Sheet":
            continue
        page = n.attributes.get("page")
        hint_file = work / "hints" / f"page_{page:02d}.json"
        if not hint_file.exists():
            continue
        hints = json.loads(hint_file.read_text(encoding="utf-8"))
        missed = []
        for cand in hints.get("tag_candidates", []):
            t = normalize_tag(cand["text"])
            if not t or t in known:
                continue
            core = _core(t)
            if core in known_core or core in drawings or _MONTHS.match(t):
                continue
            # OCR fragments cut at a tile edge ('0-BX-002', '660-TK-00').
            if any(core in k or k in core for k in known_core if len(k) >= 4) or core in blob:
                continue
            isa = parse_isa_tag(t)
            if not isa and not cand.get("equipment_like"):
                continue
            # Likely OCR misread of a known tag: same number, letters one edit apart.
            if isa and any(_close(isa.letters, l) for l in by_number.get(isa.number, ())):
                continue
            missed.append(cand)
        if missed:
            listed = ", ".join(f"{c['text']}" for c in missed[:40])
            out.append({"severity": "warning", "code": "possible_missed_tags", "sheet": n.id,
                        "page": page, "count": len(missed),
                        "message": f"{len(missed)} tags found by {hints.get('source', 'hints')} are not "
                                   f"in the graph: {listed}",
                        "candidates": missed})
    return out

def _edit1(a: str, b: str) -> bool:
    """True if a and b differ by at most one substitution, insertion or deletion."""
    if abs(len(a) - len(b)) > 1:
        return False
    if len(a) == len(b):
        return sum(x != y for x, y in zip(a, b)) <= 1
    if len(a) > len(b):
        a, b = b, a
    return any(a == b[:i] + b[i + 1:] for i in range(len(b)))

def _core(tag: str) -> str:
    """Tag without separators, so 'FIT-0502' and 'FIT0502' match."""
    return tag.replace("-", "")

def tiles_for_bbox(work: Path, page: int, bbox: Optional[list[float]]) -> list[str]:
    if not bbox:
        return []
    f = work / "pages" / f"page_{page:02d}" / "tiles.json"
    if not f.exists():
        return []
    tiles = json.loads(f.read_text(encoding="utf-8"))["tiles"]
    x0, y0, x1, y1 = bbox
    cx, cy = (x0 + x1) / 2, (y0 + y1) / 2
    hits = [t["name"] for t in tiles
            if t["norm"][0] <= cx <= t["norm"][2] and t["norm"][1] <= cy <= t["norm"][3]]
    return hits[:2]

def write_report(kg: KnowledgeGraph, issues: list[dict[str, Any]], path: Path,
                 work_dir: Optional[Path] = None) -> None:
    sev_order = {"error": 0, "warning": 1, "info": 2}
    nodes = {n.id: n for n in kg.nodes}
    for i in issues:  # fill page/bbox from the node's provenance when missing
        n = nodes.get(i.get("node", ""))
        if n and n.provenance:
            i.setdefault("page", n.provenance[0].page)
            if not i.get("bbox"):
                bb = next((p.bbox for p in n.provenance if p.bbox), None)
                if bb:
                    i["bbox"] = bb
    issues = sorted(issues, key=lambda i: (sev_order.get(i["severity"], 3), i.get("page") or 0,
                                           i["code"]))
    counts: dict[str, int] = defaultdict(int)
    by_code: dict[tuple[str, str], int] = defaultdict(int)
    for i in issues:
        counts[i["severity"]] += 1
        by_code[(i["severity"], i["code"])] += 1

    lines = [f"# QA report: {kg.plant.name}", "",
             f"Generated {kg.generated_at} by {kg.generator}.", "",
             f"**{counts.get('error', 0)} errors, {counts.get('warning', 0)} warnings, "
             f"{counts.get('info', 0)} info.**", "",
             "Errors must be fixed in the page extraction files before the graph is trusted. "
             "Warnings are things to re-check on the drawing (the *Re-check* column names the tiles). "
             "Info items are for the record.", "",
             "| Severity | Code | Count |", "|---|---|---|"]
    for (sev, code), c in sorted(by_code.items(), key=lambda kv: (sev_order.get(kv[0][0], 3), kv[0][1])):
        lines.append(f"| {sev} | `{code}` | {c} |")
    lines += ["", "## Details", "", "| Severity | Page | Code | Item | Message | Re-check |",
              "|---|---|---|---|---|---|"]
    for i in issues:
        tiles = tiles_for_bbox(work_dir, i["page"], i.get("bbox")) if work_dir and i.get("page") else []
        msg = i["message"].replace("|", "\\|")
        lines.append(f"| {i['severity']} | {i.get('page', '')} | `{i['code']}` | {i.get('node', '')} | "
                     f"{msg} | {', '.join(tiles)} |")
    path.write_text("\n".join(lines) + "\n", encoding="utf-8")

def _close(a: str, b: str) -> bool:
    """Letter codes close enough to be an OCR misread (same first letter, <=2 slips)."""
    if _edit1(a, b):
        return True
    return len(a) == len(b) and a[:1] == b[:1] and sum(x != y for x, y in zip(a, b)) <= 2
```

## File: skills/pid-to-kg/scripts/pidkg/store.py
size: 9596
sha: 233c319230d618fd24ce087c9484cc2595875893
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/store.py

```text
"""DuckDB storage and graph traversal.

The DuckDB file is a query layer over ``graph.json``: two tables (``nodes``,
``edges``), convenience views, and table macros for flow traversal, e.g.::

    SELECT * FROM downstream('660-TK-001');
    SELECT * FROM upstream('FV-0502');
    SELECT * FROM instruments WHERE variable = 'flow';
"""

from __future__ import annotations

import json
from collections import defaultdict, deque
from pathlib import Path
from typing import Any, Optional

from .models import KnowledgeGraph

SCHEMA_SQL = """
CREATE TABLE plant (
    id VARCHAR PRIMARY KEY, name VARCHAR, source_file VARCHAR,
    generated_at VARCHAR, generator VARCHAR, schema_version VARCHAR, stats JSON
);
CREATE TABLE nodes (
    id VARCHAR PRIMARY KEY, type VARCHAR NOT NULL, label VARCHAR, tag VARCHAR,
    kind VARCHAR, name VARCHAR, status VARCHAR, dexpi_class VARCHAR,
    sheets VARCHAR[], confidence DOUBLE, attributes JSON, provenance JSON
);
CREATE TABLE edges (
    id VARCHAR PRIMARY KEY, source VARCHAR NOT NULL, target VARCHAR NOT NULL,
    type VARCHAR NOT NULL, confidence DOUBLE, derived BOOLEAN, sheet VARCHAR,
    attributes JSON
);
CREATE TABLE issues (
    severity VARCHAR, code VARCHAR, message VARCHAR, node VARCHAR, sheet VARCHAR, page INTEGER
);
"""

FLOW_VIEW_SQL = """
CREATE VIEW flow_edges AS
SELECT source, target, sheet, attributes FROM edges
WHERE type = 'FLOWS_TO' {pfd_filter};
"""

VIEWS_SQL = """
CREATE VIEW equipment AS
SELECT id, tag, kind, name, status, sheets, confidence FROM nodes WHERE type = 'Equipment';

CREATE VIEW equipment_flow AS
SELECT e.source AS from_id, s.label AS from_label, e.target AS to_id, t.label AS to_label,
       e.attributes->>'via_lines' AS via_lines, e.attributes->>'via_items' AS via_items
FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target
WHERE e.type = 'FEEDS' {feeds_filter};

CREATE VIEW instruments AS
SELECT n.id, n.tag, n.attributes->>'variable' AS variable, n.attributes->>'role' AS role,
       n.attributes->>'location' AS location, n.name,
       m.target AS measures_id, mt.label AS measures_label, mt.type AS measures_type,
       l.target AS loop_id, n.sheets, n.confidence
FROM nodes n
LEFT JOIN edges m ON m.source = n.id AND m.type = 'MEASURES'
LEFT JOIN nodes mt ON mt.id = m.target
LEFT JOIN edges l ON l.source = n.id AND l.type = 'MEMBER_OF'
WHERE n.type = 'Instrument';

CREATE VIEW loops AS
SELECT id, label AS loop, kind AS loop_type, attributes->>'variable' AS variable,
       attributes->'sensors' AS sensors, attributes->'controllers' AS controllers,
       attributes->'final_elements' AS final_elements,
       attributes->>'description' AS description, sheets
FROM nodes WHERE type = 'ControlLoop';

CREATE VIEW signals AS
SELECT e.source, e.target, e.attributes->>'signal_type' AS signal_type,
       e.attributes->>'purpose' AS purpose, e.sheet
FROM edges e WHERE e.type = 'SIGNAL_TO';

CREATE MACRO downstream(start_id) AS TABLE
WITH RECURSIVE walk(id) AS (
    SELECT target FROM flow_edges WHERE source = start_id
    UNION
    SELECT e.target FROM flow_edges e JOIN walk w ON e.source = w.id
)
SELECT n.id, n.type, n.label, n.kind, n.name FROM walk JOIN nodes n ON n.id = walk.id;

CREATE MACRO upstream(start_id) AS TABLE
WITH RECURSIVE walk(id) AS (
    SELECT source FROM flow_edges WHERE target = start_id
    UNION
    SELECT e.source FROM flow_edges e JOIN walk w ON e.target = w.id
)
SELECT n.id, n.type, n.label, n.kind, n.name FROM walk JOIN nodes n ON n.id = walk.id;
"""

def write_duckdb(kg: KnowledgeGraph, path: Path) -> None:
    import duckdb

    path = Path(path)
    if path.exists():
        path.unlink()
    con = duckdb.connect(str(path))
    try:
        con.execute(SCHEMA_SQL)
        con.execute("INSERT INTO plant VALUES (?, ?, ?, ?, ?, ?, ?)",
                    [kg.plant.id, kg.plant.name, kg.plant.source_file, kg.generated_at,
                     kg.generator, kg.schema_version, json.dumps(kg.stats)])
        con.executemany(
            "INSERT INTO nodes VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
            [[n.id, n.type, n.label, n.tag, n.kind, n.name, n.status, n.dexpi_class, n.sheets,
              n.confidence, json.dumps(n.attributes),
              json.dumps([p.model_dump(exclude_none=True) for p in n.provenance])]
             for n in kg.nodes])
        con.executemany(
            "INSERT INTO edges VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
            [[e.id, e.source, e.target, e.type, e.confidence, e.derived, e.sheet,
              json.dumps(e.attributes)] for e in kg.edges])
        if kg.issues:
            con.executemany(
                "INSERT INTO issues VALUES (?, ?, ?, ?, ?, ?)",
                [[i.get("severity"), i.get("code"), i.get("message"), i.get("node"),
                  i.get("sheet"), i.get("page")] for i in kg.issues])
        pfd_filter = ("AND coalesce(attributes->>'drawing_type', '') <> 'pfd'"
                      if has_pid_sheets(kg) else "")
        con.execute(FLOW_VIEW_SQL.format(pfd_filter=pfd_filter))
        con.execute(VIEWS_SQL.replace("{feeds_filter}", "AND coalesce(e.attributes->>'view', '') <> 'pfd'"
                                      if pfd_filter else ""))
    finally:
        con.close()

def has_pid_sheets(kg: KnowledgeGraph) -> bool:
    sheets = [n for n in kg.nodes if n.type == "Sheet"]
    return any(n.attributes.get("drawing_type", "pid") != "pfd" for n in sheets) and         any(n.attributes.get("drawing_type") == "pfd" for n in sheets)

class Graph:
    """In-memory view of a knowledge graph for traversal queries."""

    def __init__(self, kg: KnowledgeGraph, include_pfd: bool = False):
        """``include_pfd``: when a plant has P&IDs, PFD-sheet flow is skipped in
        traversals by default because the P&IDs show the same flow in detail."""
        self.kg = kg
        self.nodes = {n.id: n for n in kg.nodes}
        self.out: dict[str, list] = defaultdict(list)
        self.inn: dict[str, list] = defaultdict(list)
        skip_pfd = not include_pfd and has_pid_sheets(kg)
        for e in kg.edges:
            if skip_pfd and e.attributes.get("drawing_type") == "pfd":
                continue
            if skip_pfd and e.type == "FEEDS" and e.attributes.get("view") == "pfd":
                continue
            self.out[e.source].append(e)
            self.inn[e.target].append(e)
        self._by_tag = {}
        for n in kg.nodes:
            for key in {n.id, n.tag, n.label}:
                if key:
                    self._by_tag.setdefault(key.upper(), n.id)

    @classmethod
    def load(cls, path: Path, include_pfd: bool = False) -> "Graph":
        path = Path(path)
        if path.suffix == ".duckdb":
            path = path.with_name("graph.json")
        return cls(KnowledgeGraph.model_validate_json(path.read_text(encoding="utf-8")), include_pfd)

    def find(self, ref: str) -> Optional[str]:
        from .isa import normalize_tag

        if ref in self.nodes:
            return ref
        for key in (ref.upper(), normalize_tag(ref)):
            if key and key in self._by_tag:
                return self._by_tag[key]
        # Untagged items: a unique local id ('P12.OXIDANT_TOTE') or a unique name.
        r = ref.upper()
        hits = [n.id for n in self.kg.nodes if n.id.upper().endswith("." + r)]
        if len(hits) != 1:
            hits = [n.id for n in self.kg.nodes if n.name and n.name.upper() == r]
        return hits[0] if len(hits) == 1 else None

    def walk(self, start: str, direction: str = "down", edge_types=("FLOWS_TO",),
             max_depth: Optional[int] = None) -> list[tuple[str, int]]:
        """Breadth-first traversal returning (node id, hops) in visiting order."""
        adj = self.out if direction == "down" else self.inn
        seen = {start: 0}
        order: list[tuple[str, int]] = []
        q = deque([start])
        while q:
            cur = q.popleft()
            d = seen[cur]
            if max_depth is not None and d >= max_depth:
                continue
            for e in adj[cur]:
                if e.type not in edge_types:
                    continue
                nxt = e.target if direction == "down" else e.source
                if nxt not in seen:
                    seen[nxt] = d + 1
                    order.append((nxt, d + 1))
                    q.append(nxt)
        return order

    def path(self, a: str, b: str, edge_types=("FLOWS_TO",)) -> Optional[list[str]]:
        prev: dict[str, Optional[str]] = {a: None}
        q = deque([a])
        while q:
            cur = q.popleft()
            if cur == b:
                out = []
                while cur is not None:
                    out.append(cur)
                    cur = prev[cur]
                return out[::-1]
            for e in self.out[cur]:
                if e.type in edge_types and e.target not in prev:
                    prev[e.target] = cur
                    q.append(e.target)
        return None

    def neighbors(self, nid: str) -> dict[str, list[dict[str, Any]]]:
        rel: dict[str, list[dict[str, Any]]] = defaultdict(list)
        for e in self.out[nid]:
            rel[f"{e.type} ->"].append({"id": e.target, "label": self.nodes[e.target].label,
                                        **({"attributes": e.attributes} if e.attributes else {})})
        for e in self.inn[nid]:
            rel[f"<- {e.type}"].append({"id": e.source, "label": self.nodes[e.source].label,
                                        **({"attributes": e.attributes} if e.attributes else {})})
        return dict(rel)
```

## File: skills/pid-to-kg/scripts/pidkg/summary.py
size: 7814
sha: 24a99daf74210c3956a08615c94b760669a4f9c7
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/pidkg/summary.py

```text
"""Compact Markdown summary of a knowledge graph, written for LLM context windows."""

from __future__ import annotations

from collections import defaultdict
from typing import Optional

from .loops import sensed_points
from .models import KnowledgeGraph
from .store import Graph

def _esc(s: Optional[str]) -> str:
    return (s or "").replace("|", "\\|").replace("\n", " ")

def _flow_order(g: Graph) -> list[str]:
    """Equipment/boundaries in reading order: sources first, then breadth-first along FEEDS."""
    feeds_in: dict[str, int] = defaultdict(int)
    terminals = [n.id for n in g.kg.nodes if n.type in ("Equipment", "Boundary")]
    for e in g.kg.edges:
        if e.type == "FEEDS":
            feeds_in[e.target] += 1
    starts = [t for t in terminals if feeds_in[t] == 0]
    starts.sort(key=lambda t: (g.nodes[t].type != "Boundary", g.nodes[t].sheets[:1], t))
    order: list[str] = []
    seen: set[str] = set()
    for s in starts + terminals:
        if s in seen:
            continue
        queue = [s]
        while queue:
            cur = queue.pop(0)
            if cur in seen:
                continue
            seen.add(cur)
            order.append(cur)
            queue.extend(e.target for e in g.out[cur] if e.type == "FEEDS")
    return order

def write_summary(kg: KnowledgeGraph) -> str:
    g = Graph(kg)
    nodes = g.nodes
    sheets = [n for n in kg.nodes if n.type == "Sheet"]
    sheets.sort(key=lambda n: n.attributes.get("page", 0))
    st = kg.stats
    out: list[str] = []
    w = out.append

    w(f"# {kg.plant.name}: process knowledge graph summary")
    w("")
    w(f"Built from `{kg.plant.source_file or 'P&IDs'}` ({len(sheets)} sheets) by {kg.generator} "
      f"on {kg.generated_at[:10]}. The graph has {st.get('nodes', 0)} nodes and "
      f"{st.get('edges', 0)} edges; this summary covers equipment-level flow, instruments and "
      "control loops. Query `graph.duckdb` or `graph.json` for pipe-segment and valve detail.")
    w("")
    nbt = st.get("nodes_by_type", {})
    w("| Equipment | Instruments | Valves & fittings | Pipe segments | Control loops | Off-page links |")
    w("|---|---|---|---|---|---|")
    w(f"| {nbt.get('Equipment', 0)} | {nbt.get('Instrument', 0)} | {nbt.get('Component', 0)} | "
      f"{nbt.get('PipeSegment', 0)} | {nbt.get('ControlLoop', 0)} | {nbt.get('OffPageConnector', 0)} |")
    w("")

    w("## Sheets")
    w("")
    w("| Page | Drawing | Title | Type |")
    w("|---|---|---|---|")
    for s in sheets:
        a = s.attributes
        w(f"| {a.get('page', '')} | {_esc(a.get('drawing_number'))} | {_esc(a.get('title'))} | "
          f"{a.get('drawing_type', '')} |")
    w("")

    w("## Process flow (equipment to equipment)")
    w("")
    w("Each line reads *upstream → downstream*, with the pipe lines and in-line items between them. "
      "Pipes, valves and instruments are collapsed; use `kg_query.py path` for the full route.")
    w("")
    drains: dict[str, set[str]] = defaultdict(set)
    for nid in _flow_order(g):
        feeds = [e for e in g.out[nid] if e.type == "FEEDS"]
        for e in [e for e in feeds if _is_drain(nodes[e.target])]:
            drains[nodes[e.target].label].add(nodes[nid].label)
        feeds = [e for e in feeds if not _is_drain(nodes[e.target])]
        if not feeds:
            continue
        n = nodes[nid]
        head = f"**{n.label}**" + (f" ({_esc(n.name)})" if n.name and n.name != n.label else "")
        for e in feeds:
            t = nodes[e.target]
            via = []
            if e.attributes.get("via_lines"):
                via.append("lines " + ", ".join(e.attributes["via_lines"]))
            if e.attributes.get("via_items"):
                via.append("through " + ", ".join(e.attributes["via_items"]))
            if e.attributes.get("off_page"):
                via.append("crosses sheets")
            tname = f" ({_esc(t.name)})" if t.name and t.name != t.label else ""
            w(f"- {head} → **{t.label}**{tname}" + (f": {'; '.join(via)}" if via else ""))
    if drains:
        w("")
        w("Drains, vents and overflows (not process flow):")
        for d, srcs in sorted(drains.items()):
            w(f"- {d} ← {', '.join(sorted(srcs))}")
    w("")

    w("## Equipment")
    w("")
    w("| Tag | Kind | Description | Fed by | Feeds | Status | Sheets |")
    w("|---|---|---|---|---|---|---|")
    for n in sorted((n for n in kg.nodes if n.type == "Equipment"), key=lambda n: n.label):
        ups = sorted({nodes[e.source].label for e in g.inn[n.id] if e.type == "FEEDS"})
        downs = sorted({nodes[e.target].label for e in g.out[n.id] if e.type == "FEEDS"})
        w(f"| {n.label} | {n.kind or ''} | {_esc(n.name)} | {', '.join(ups)} | {', '.join(downs)} | "
          f"{n.status or ''} | {', '.join(_sheet_labels(n.sheets, nodes))} |")
    w("")

    w("## Instruments")
    w("")
    w("Location is where the instrument senses the process. Pipe locations read "
      "`line (upstream item -> downstream item)`.")
    w("")
    w("| Tag | Variable | Function | Mounted | Senses | Loop |")
    w("|---|---|---|---|---|---|")
    for n in sorted((n for n in kg.nodes if n.type == "Instrument"), key=lambda n: n.label):
        a = n.attributes
        senses = sensed_points(n.id, nodes, kg.edges)
        loops = [nodes[e.target].label for e in g.out[n.id] if e.type == "MEMBER_OF"]
        w(f"| {n.label} | {a.get('variable', '')} | {a.get('role', '')} | {a.get('location', '')} | "
          f"{_esc('; '.join(senses))} | {', '.join(loops)} |")
    w("")

    loops = [n for n in kg.nodes if n.type == "ControlLoop"]
    w("## Control loops")
    w("")
    if not loops:
        w("No control loops were identified.")
    else:
        w("| Loop | Type | Sensor(s) | Controller(s) | Final element(s) | Description |")
        w("|---|---|---|---|---|---|")
        lab = lambda ids: ", ".join(nodes[i].label for i in ids if i in nodes)  # noqa: E731
        for n in sorted(loops, key=lambda n: n.label):
            a = n.attributes
            w(f"| {n.label} | {n.kind} | {lab(a.get('sensors', []))} | {lab(a.get('controllers', []))} | "
              f"{lab(a.get('final_elements', []))} | {_esc(a.get('description'))} |")
    w("")

    opcs = [n for n in kg.nodes if n.type == "OffPageConnector"]
    if opcs:
        w("## Off-page connections")
        w("")
        w("| Sheet | Connector | Direction | Other drawing | Linked to |")
        w("|---|---|---|---|---|")
        for n in sorted(opcs, key=lambda n: n.id):
            a = n.attributes
            link = a.get("linked_to") or a.get("linked_from") or a.get("linked_item") or "unmatched"
            if link in nodes:
                link = f"{nodes[link].label} ({', '.join(_sheet_labels(nodes[link].sheets, nodes))})"
            w(f"| {', '.join(_sheet_labels(n.sheets, nodes))} | {_esc(n.label)} | {a.get('direction', '')} | "
              f"{_esc(a.get('other_drawing'))} | {_esc(link)} |")
        w("")

    sev = st.get("issues_by_severity", {})
    w("## Data quality")
    w("")
    w(f"{sev.get('error', 0)} errors, {sev.get('warning', 0)} warnings and {sev.get('info', 0)} "
      "informational items; see `qa_report.md`. Every node carries `confidence` and the sheet "
      "region it was read from (`provenance`). This graph was machine-extracted from drawings "
      "and must be verified by a qualified engineer before use in design, operations or safety work.")
    return "\n".join(out) + "\n"

_DRAIN_WORDS = ("DRAIN", "VENT", "OVERFLOW", "SUMP")

def _is_drain(n) -> bool:
    return n.type == "Boundary" and any(w in (n.label or "").upper() for w in _DRAIN_WORDS[:3])

def _sheet_labels(sheet_ids: list[str], nodes) -> list[str]:
    return [nodes[s].label if s in nodes else s for s in sheet_ids]
```

## File: skills/pid-to-kg/scripts/probe_pdf.py
size: 3534
sha: c3343eda29a702a6ad26f43c76d160bbddff7729
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/probe_pdf.py

```text
#!/usr/bin/env python3
"""Classify every page of a P&ID PDF and start a work directory.

Each page is classified as vector_text, vector_outlined, raster,
raster_with_text, hybrid or blank, which decides how it is rendered and
which hints are available. Writes <work>/manifest.json.

Usage:
    python probe_pdf.py drawings.pdf --work work/my-plant --plant-name "My Plant"
    python probe_pdf.py permit.pdf --work work/my-plant --pages 47-60   # only the P&ID pages
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))

from pidkg.pdf import probe_pdf  # noqa: E402
from render_pages import parse_pages  # noqa: E402

def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("pdf", type=Path)
    ap.add_argument("--work", type=Path, required=True, help="work directory for this plant")
    ap.add_argument("--plant-name", help="human-readable plant name (default: PDF file name)")
    ap.add_argument("--plant-id", help="short id (default: slug of the plant name)")
    ap.add_argument("--pages", help="pages that hold the P&IDs, e.g. '47-60' or '50-51,62-63' "
                                    "(default: all). Later steps use only these pages.")
    args = ap.parse_args()

    if not args.pdf.exists():
        print(f"error: {args.pdf} not found", file=sys.stderr)
        return 2
    name = args.plant_name or args.pdf.stem
    plant_id = args.plant_id or re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-")
    probes = probe_pdf(str(args.pdf))

    args.work.mkdir(parents=True, exist_ok=True)
    manifest_path = args.work / "manifest.json"
    previous = {}
    if manifest_path.exists():
        old = json.loads(manifest_path.read_text(encoding="utf-8"))
        previous = {p["page"]: p for p in old.get("pages", [])}
    pages = []
    for p in probes:
        d = p.to_dict()
        # Keep a rotation that was already confirmed for this page.
        if previous.get(p.page, {}).get("rotation") is not None:
            d["rotation"] = previous[p.page]["rotation"]
            d["rotation_source"] = previous[p.page].get("rotation_source")
        pages.append(d)
    selected = parse_pages(args.pages, len(probes)) if args.pages else None
    if args.pages and not selected:
        print(f"error: --pages {args.pages} selects no page of a {len(probes)}-page PDF", file=sys.stderr)
        return 2
    manifest = {"source_pdf": str(args.pdf.resolve()), "plant_id": plant_id, "plant_name": name,
                "pages": pages}
    if selected:
        manifest["selected_pages"] = selected
    manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")

    shown = [p for p in probes if not selected or p.page in selected]
    print(f"{args.pdf.name}: {len(probes)} pages -> {manifest_path}")
    if selected:
        print(f"using {len(selected)} selected pages: {args.pages}")
    print(f"{'page':>4}  {'class':<17} {'chars':>6} {'paths':>6} {'img cov':>7}  notes")
    for p in shown:
        print(f"{p.page:>4}  {p.page_class:<17} {p.text_chars:>6} {p.path_objects:>6} "
              f"{p.max_image_coverage:>7.2f}  {'; '.join(p.notes)}")
    classes = sorted({p.page_class for p in probes})
    print("\nRoutes:")
    for c in classes:
        print(f"  {c}: {next(p.route for p in probes if p.page_class == c)}")
    return 0

if __name__ == "__main__":
    sys.exit(main())
```

## File: skills/pid-to-kg/scripts/render_pages.py
size: 8352
sha: f1722ca0f502bff4ffe95f5174c71c64cac3a77f
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/render_pages.py

Content is not inlined in the catalog. Read the upstream file: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/render_pages.py

## File: skills/pid-to-kg/scripts/requirements.txt
size: 283
sha: e71cc9e5c43404e58fce304e78df14be090c0fa4
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/scripts/requirements.txt

```text
pypdfium2>=4.30
pillow>=10.0
pydantic>=2.6
duckdb>=1.1
networkx>=3.2
# Optional but recommended: OCR hints and automatic page orientation.
# The OCR package supports Python 3.10-3.12 only; on newer Python the skill runs without it.
rapidocr-onnxruntime>=1.3; python_version < "3.13"
```

## File: skills/pid-to-kg/references/extraction_procedure.md
size: 15461
sha: 2c609c62fd86da4d75f4ad41c34ed6cf80f3c10d
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/extraction_procedure.md

Content is not inlined in the catalog. Read the upstream file: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/extraction_procedure.md

## File: skills/pid-to-kg/references/isa_5_1.md
size: 5944
sha: 1addb63fff8c72039f71b77a32bfbc5c04b5ad9b
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/isa_5_1.md

Content is not inlined in the catalog. Read the upstream file: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/isa_5_1.md

## File: skills/pid-to-kg/references/ontology.md
size: 7738
sha: 26823553594de51f8401de36789ec50697a1297b
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/ontology.md

Content is not inlined in the catalog. Read the upstream file: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/ontology.md

## File: skills/pid-to-kg/references/schema/extraction.schema.json
size: 25130
sha: d62ea8a88fbf3506a3907b5799c1d3f9391c4568
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/schema/extraction.schema.json

Content is not inlined in the catalog. Read the upstream file: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/schema/extraction.schema.json

## File: skills/pid-to-kg/references/schema/graph.schema.json
size: 6308
sha: b7b1ab2d094d48018f978fa6a19878564d972a89
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/schema/graph.schema.json

Content is not inlined in the catalog. Read the upstream file: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/schema/graph.schema.json

## File: skills/pid-to-kg/references/symbol_guide.md
size: 4371
sha: 95a2c84d7828a23df120f57f72ad70a91eeac38c
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/symbol_guide.md

Content is not inlined in the catalog. Read the upstream file: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/skills/pid-to-kg/references/symbol_guide.md

## File: README.md
upstream: https://github.com/scottduncanai/pid-to-knowledge-graph/blob/main/README.md

```text
# P&ID to Knowledge Graph

By Scott Duncan. Reach me on [LinkedIn](https://www.linkedin.com/in/s-r-duncan/) or at [scottduncanai.com](https://scottduncanai.com/).

**Turn P&ID drawings into a process knowledge graph that an LLM can reason over.**

`pid-to-kg` is a [Claude skill](https://code.claude.com/docs/en/skills) that reads piping and instrumentation diagrams (P&IDs) and PFDs from PDF and builds a knowledge graph of the process. The PDFs can be digital (exported from CAD) or scanned. The graph captures:

- **Process flow:** what is upstream and downstream of every tank, pump, valve and pipe segment.
- **Measurement:** which instrument measures which variable, and exactly where it sits in the flow.
- **Control:** control loops from sensor through controller to final element, including I/P relays, cascades, interlocks and VFD drives.

The output is meant to be one layer of a chemical-manufacturing **context layer**. An LLM application can query it to answer questions such as "what feeds this clarifier?", "which transmitters sit downstream of P-101?" or "what does FIC-0502 manipulate?" from the drawings themselves.

## Quick start

No programming is needed. You need [Claude Code](https://code.claude.com/docs/en/overview) (the desktop app or the terminal version) and [Python](https://www.python.org/downloads/) 3.10 or newer. Python 3.12 is the best choice: on 3.13 and later the skill still works, but without the text-recognition (OCR) cross-check and automatic page orientation.

**1. Download the skill.** On this page, click the green **Code** button, then **Download ZIP**. Unzip the file.

**2. Install it.** In the unzipped folder, open `skills` and copy the `pid-to-kg` folder into your personal Claude skills folder:

| System | Skills folder |
|---|---|
| Windows | `C:\Users\<your user name>\.claude\skills\` |
| macOS | `/Users/<your user name>/.claude/skills/` |

If the `skills` folder doesn't exist yet, create it. The `.claude` folder is hidden by default: in Windows File Explorer use **View → Show → Hidden items**, and in macOS Finder press **Cmd + Shift + .** to show it.

**3. Add your P&IDs.** Make a new folder for the project and put your P&ID PDF in it. Open that folder in Claude Code: in the desktop app, start a new Code session and choose the folder.

No P&IDs at hand? My [public chemical manufacturing datasets](https://github.com/ScottDuncanAI/public-chemical-manufacturing-datasets#pids) repo lists public P&ID sets from real permitted facilities, with the page numbers where the drawings are. Both digital and scanned sets are included.

**4. Ask Claude to convert them.** For example:

> Convert `haile.pdf` into a knowledge graph. The P&IDs are on pages 47–60.

The skill loads by itself; there is no need to name it. The first time, Claude installs the Python packages it needs, and it asks your permission before running commands. It then reads every sheet in detail, so a large drawing set takes a while. Claude may ask a question along the way, such as which revision to use when a PDF holds two versions of the same drawing. When it is done, it reports what it found and what remains uncertain.

**5. Check the graph by asking questions.** The best test is to ask about things you can verify on the drawings:

> Name all the equipment upstream of the clarifier.
>
> What sensor measures the flow into TK-101, and where exactly is it?
>
> What does FIC-0502 control, and what valve does it move?

The results are saved in `work/<plant name>/kg/` inside your project folder. `summary.md` is a readable overview of the whole plant, and `qa_report.md` lists anything Claude could not confirm.

## How it works

![How the skill works: your P&ID PDF, Claude reads it, build and check, ask questions](docs/how-it-works.svg)

**Claude reads the drawings; Python scripts do everything else.** Recognising symbols and tracing lines needs visual judgement, so that part is Claude's. Preparing pages, building the graph and checking it are routine, so tested scripts do them the same way every time.

### 1. Preparing the pages

A script first looks at every page and works out what kind of drawing it is. Digital and scanned P&IDs need different handling:

| Page type | How to recognise it | What the skill does |
|---|---|---|
| **Digital, searchable text** | Exported from CAD; you can select the text in a PDF viewer | Renders the page straight from the vector data, so lines stay sharp at any zoom. The exact text in the PDF is used to cross-check the tags Claude reads. |
| **Digital, outlined text** | Looks digital, but the text can't be selected: every letter was drawn as shapes. Common in CAD exports. | Renders from the vector data at high resolution. Since there is no text to pull out, OCR (text recognition) reads the tags for the cross-check. |
| **Scanned** | Each page is a picture of a paper drawing | Takes the scanned image at its original resolution, enlarges it up to 2.5×, and boosts the contrast so faded lines and small text stand out. OCR reads the tags for the cross-check. Low-resolution scans are flagged, because small text may be lost. |
| **Scanned with OCR text** | A scan with a searchable text layer added | Handled like a scan. The added text layer is only used as a hint, since it often contains errors. |

Then, for every page:
- **Orientation:** the page is turned so the text is upright, since drawings are often stored sideways in a PDF.
- **Resolution:** chosen from the size of the drawing's text, so the smallest tags stay legible.
- **Tiles:** the sheet is cut into overlapping tiles, and blank tiles (empty margins, border lines) are skipped.

### 2. Claude reads each sheet

- Claude reads the whole sheet first, then every tile up close.
- It records equipment, pipe runs, valves, instruments, signal lines and off-page connectors.
- Each item gets the place on the sheet where it was found and a confidence level. Anything unreadable is marked as such, never guessed.
- Each sheet is saved to its own file before the next one starts, so large drawing sets don't overload Claude.

### 3. Building and checking the knowledge graph

The scripts turn Claude's readings into one connected knowledge graph:
- **Process flow:** each pipe is split into segments between its valves and fittings, and off-page connectors link the sheets, so flow can be traced upstream or downstream across the whole plant.
- **Measurement:** every instrument is attached to the exact pipe segment or vessel it measures.
- **Control:** loops are assembled from the instrument tags (per ISA-5.1) and the signal lines, from sensor to controller to valve or drive.

A QA report then lists anything doubtful: a sensor with no location, an off-page connector with no partner, a tag OCR saw that Claude didn't record. Claude re-checks each one on the drawing and rebuilds until the report is clean.

### 4. Asking questions

- `summary.md` is a readable overview to give an LLM as context.
- `graph.json` holds the full graph for any tool to read.
- `graph.duckdb` is a small database file for precise queries. Nothing needs a server.
- Every item records the sheet it came from, so any answer can be traced back to the drawing.

## Outputs

| File | What it is |
|---|---|
| `summary.md` | A readable overview of the plant: equipment-level flow, equipment, instruments with their locations, control loops and off-page links. This is the file to give an LLM as context. |
| `qa_report.md` | Errors, warnings and notes, each pointing to the part of the drawing to re-check |
| `graph.json` | The complete graph (schema: [`graph.schema.json`](skills/pid-to-kg/references/schema/graph.schema.json)) |
| `graph.duckdb` | The same graph as a [DuckDB](https://duckdb.org) database file, for SQL queries, including recursive upstream and downstream searches |

For larger projects, the graph can also be exported to Neo4j or to GraphML (for yEd, Gephi or NetworkX). Ask Claude to do it.

The node and edge types are described in [`refere
```
