⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

v0.1.0

pid-to-kg

scottduncanai@scottduncanai⭐ 260 stars· last commit 7d ago· 1 open issues

Converts P&IDs (piping and instrumentation diagrams) and PFDs in PDF form, either vector or scanned, into a process knowledge graph covering equipment connectivity (upstream/downstream), instrument locations and measured variables, and control loops. Output is JSON plus a DuckDB file plus an LLM-ready Markdown summary. Use when the user wants to digitize, extract, analyze or build a knowledge graph or context layer from P&IDs, or asks what is upstream/downstream of a piece of equipment, which sensors measure what, or how a plant's control loops are structured.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐⭐⭐ 260 on GitHubGitHub ↗

Popular

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

P&ID to Knowledge Graph

By Scott Duncan. Reach me on LinkedIn or at scottduncanai.com.

Turn P&ID drawings into a process knowledge graph that an LLM can reason over.

pid-to-kg is a Claude skill that reads piping and instrumentation diagrams (P&IDs) and PFDs from PDF and builds a knowledge graph of the process. The PDFs can be digital (exported from CAD) or scanned. The graph captures:

  • Process flow: what is upstream and downstream of every tank, pump, valve and pipe segment.
  • Measurement: which instrument measures which variable, and exactly where it sits in the flow.
  • Control: control loops from sensor through controller to final element, including I/P relays, cascades, interlocks and VFD drives.

The output is meant to be one layer of a chemical-manufacturing context layer. An LLM application can query it to answer questions such as "what feeds this clarifier?", "which transmitters sit downstream of P-101?" or "what does FIC-0502 manipulate?" from the drawings themselves.

Quick start

No programming is needed. You need Claude Code (the desktop app or the terminal version) and Python 3.10 or newer. Python 3.12 is the best choice: on 3.13 and later the skill still works, but without the text-recognition (OCR) cross-check and automatic page orientation.

1. Download the skill. On this page, click the green Code button, then Download ZIP. Unzip the file.

2. Install it. In the unzipped folder, open skills and copy the pid-to-kg folder into your personal Claude skills folder:

SystemSkills folder
WindowsC:\Users\<your user name>\.claude\skills\
macOS/Users/<your user name>/.claude/skills/

If the skills folder doesn't exist yet, create it. The .claude folder is hidden by default: in Windows File Explorer use View → Show → Hidden items, and in macOS Finder press Cmd + Shift + . to show it.

3. Add your P&IDs. Make a new folder for the project and put your P&ID PDF in it. Open that folder in Claude Code: in the desktop app, start a new Code session and choose the folder.

No P&IDs at hand? My public chemical manufacturing datasets repo lists public P&ID sets from real permitted facilities, with the page numbers where the drawings are. Both digital and scanned sets are included.

4. Ask Claude to convert them. For example:

Convert haile.pdf into a knowledge graph. The P&IDs are on pages 47–60.

The skill loads by itself; there is no need to name it. The first time, Claude installs the Python packages it needs, and it asks your permission before running commands. It then reads every sheet in detail, so a large drawing set takes a while. Claude may ask a question along the way, such as which revision to use when a PDF holds two versions of the same drawing. When it is done, it reports what it found and what remains uncertain.

5. Check the graph by asking questions. The best test is to ask about things you can verify on the drawings:

Name all the equipment upstream of the clarifier.

What sensor measures the flow into TK-101, and where exactly is it?

What does FIC-0502 control, and what valve does it move?

The results are saved in work/<plant name>/kg/ inside your project folder. summary.md is a readable overview of the whole plant, and qa_report.md lists anything Claude could not confirm.

How it works

How the skill works: your P&ID PDF, Claude reads it, build and check, ask questions

Claude reads the drawings; Python scripts do everything else. Recognising symbols and tracing lines needs visual judgement, so that part is Claude's. Preparing pages, building the graph and checking it are routine, so tested scripts do them the same way every time.

1. Preparing the pages

A script first looks at every page and works out what kind of drawing it is. Digital and scanned P&IDs need different handling:

Page typeHow to recognise itWhat the skill does
Digital, searchable textExported from CAD; you can select the text in a PDF viewerRenders the page straight from the vector data, so lines stay sharp at any zoom. The exact text in the PDF is used to cross-check the tags Claude reads.
Digital, outlined textLooks digital, but the text can't be selected: every letter was drawn as shapes. Common in CAD exports.Renders from the vector data at high resolution. Since there is no text to pull out, OCR (text recognition) reads the tags for the cross-check.
ScannedEach page is a picture of a paper drawingTakes the scanned image at its original resolution, enlarges it up to 2.5×, and boosts the contrast so faded lines and small text stand out. OCR reads the tags for the cross-check. Low-resolution scans are flagged, because small text may be lost.
Scanned with OCR textA scan with a searchable text layer addedHandled like a scan. The added text layer is only used as a hint, since it often contains errors.

Then, for every page:

  • Orientation: the page is turned so the text is upright, since drawings are often stored sideways in a PDF.
  • Resolution: chosen from the size of the drawing's text, so the smallest tags stay legible.
  • Tiles: the sheet is cut into overlapping tiles, and blank tiles (empty margins, border lines) are skipped.

2. Claude reads each sheet

  • Claude reads the whole sheet first, then every tile up close.
  • It records equipment, pipe runs, valves, instruments, signal lines and off-page connectors.
  • Each item gets the place on the sheet where it was found and a confidence level. Anything unreadable is marked as such, never guessed.
  • Each sheet is saved to its own file before the next one starts, so large drawing sets don't overload Claude.

3. Building and checking the knowledge graph

The scripts turn Claude's readings into one connected knowledge graph:

  • Process flow: each pipe is split into segments between its valves and fittings, and off-page connectors link the sheets, so flow can be traced upstream or downstream across the whole plant.
  • Measurement: every instrument is attached to the exact pipe segment or vessel it measures.
  • Control: loops are assembled from the instrument tags (per ISA-5.1) and the signal lines, from sensor to controller to valve or drive.

A QA report then lists anything doubtful: a sensor with no location, an off-page connector with no partner, a tag OCR saw that Claude didn't record. Claude re-checks each one on the drawing and rebuilds until the report is clean.

4. Asking questions

  • summary.md is a readable overview to give an LLM as context.
  • graph.json holds the full graph for any tool to read.
  • graph.duckdb is a small database file for precise queries. Nothing needs a server.
  • Every item records the sheet it came from, so any answer can be traced back to the drawing.

Outputs

FileWhat it is
summary.mdA readable overview of the plant: equipment-level flow, equipment, instruments with their locations, control loops and off-page links. This is the file to give an LLM as context.
qa_report.mdErrors, warnings and notes, each pointing to the part of the drawing to re-check
graph.jsonThe complete graph (schema: graph.schema.json)
graph.duckdbThe same graph as a DuckDB database file, for SQL queries, including recursive upstream and downstream searches

For larger projects, the graph can also be exported to Neo4j or to GraphML (for yEd, Gephi or NetworkX). Ask Claude to do it.

The node and edge types are described in [`refere

// HOW IT'S BUILT

// FILES

28 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • skills/pid-to-kg/SKILL.md9.3 KB352f8b8fd5f2upstream ↗
    ---
    name: pid-to-kg
    description: Converts P&IDs (piping and instrumentation diagrams) and PFDs in PDF form, either vector or scanned, into a process knowledge graph covering equipment connectivity (upstream/downstream), instrument locations and measured variables, and control loops. Output is JSON plus a DuckDB file plus an LLM-ready Markdown summary. Use when the user wants to digitize, extract, analyze or build a knowledge graph or context layer from P&IDs, or asks what is upstream/downstream of a piece of equipment, which sensors measure what, or how a plant's control loops are structured.
    ---
    
    # P&ID to knowledge graph
    
    Work is split between deterministic scripts and you:
    
    - **Scripts** classify pages (vector vs scanned), render them into readable tiles, collect OCR/text hints, then build, check, store and query the graph.
    - **You** read the drawings and write one structured extraction file per sheet. Symbol recognition and line tracing are visual reasoning, and that is the part scripts cannot do well.
    
    All scripts are in `scripts/` next to this file. Call them by their full path (the commands below abbreviate it as `scripts/`), and keep the work directory in the user's current folder, e.g. `work/<plant>`.
    
    **Setup.** Many users are engineers, not programmers, so handle setup yourself. The scripts need Python 3.10+. Before the first run, install the dependencies with `pip install -r <skill dir>/scripts/requirements.txt` (use a virtual environment if the user's Python is shared with other work). If Python is missing, tell the user to install it from python.org and stop there.
    
    ## Ground rules
    
    - **Only the P&ID pages.** Drawings often sit inside a long permit or design report. If the user names the pages, pass them with `--pages`. If not, and the PDF has more than a few pages, probe it, look at the page overviews or ask the user, then re-run the probe with `--pages` so later steps skip reports, legends and non-P&ID drawings.
    - **One plant, one work directory, one graph.** Never put sheets from different plants in the same work directory: tags would merge across unrelated plants. If the user supplies several plants, run the whole workflow separately for each.
    - **One configuration per graph.** If the PDF contains two revisions of the same drawing (same title or area, overlapping tags, e.g. an as-built and an older draft), do not merge them. Ask the user which revision to use, extract only that one, and record the excluded pages and the reason in the report. QA flags likely duplicates as `possible_duplicate_revision`. Record each sheet's `issue_status` from the title block or stamps.
    - **Record only what is drawn.** Never invent tags, connections or loops to make the graph look complete. Anything unclear goes in with a lower `confidence`, or into `unreadable`.
    - **The QA loop is not optional.** Rebuild until there are no errors, and re-check every warning on the drawing.
    
    ## Workflow
    
    Copy this checklist into your notes and tick it off:
    
    ```
    - [ ] 1. Probe:    python scripts/probe_pdf.py <pdf> --work <work> --plant-name "<name>" [--pages 47-60]
    - [ ] 2. Render:   python scripts/render_pages.py <work>
    - [ ] 3. Orient:   open each pages/page_NN/overview_grid.png; fix rotation if text is not upright
    - [ ] 4. Hints:    python scripts/make_hints.py <work>
    - [ ] 5. Extract:  for each sheet, write <work>/extract/page_NN.json (references/extraction_procedure.md)
    - [ ] 6. Build:    python scripts/build_kg.py <work>
    - [ ] 7. QA loop:  fix every error, re-check every warning, rebuild (repeat)
    - [ ] 8. Report:   summarize the results and open issues for the user
    ```
    
    ### 1. Probe
    `probe_pdf.py` assigns each page a class and an extraction route, and writes `<work>/manifest.json`. With `--pages`, the later steps render and read only those pages.
    
    | Class | What it means | Route |
    |---|---|---|
    | `vector_text` | Vector drawing with a text layer | Render from vectors; exact text from the text layer as hints |
    | `vector_outlined` | Vector drawing, text drawn as outlines (no text layer) | High-resolution vector render; OCR hints |
    | `raster` | Scanned image | Native-resolution render, upscaled and contrast-enhanced; OCR hints |
    | `raster_with_text` | Scan with an OCR text layer | As raster; text layer as (untrusted) hints |
    | `hybrid` | Vectors plus large images | Both routes |
    
    Report the probe result to the user in one line, e.g. "14 vector sheets with outlined text."
    
    Low scan resolution is flagged in the notes. Expect to record more items in `unreadable` on those sheets.
    
    ### 2–3. Render and check orientation
    `render_pages.py` writes the following for each page:
    
    - `overview.png`
    - `overview_grid.png`, which shows the tile names
    - overlapping tiles in `tiles/`
    - `tiles.json`, which records which tiles are blank
    
    Orientation is detected with OCR when it is installed. Always confirm it on `overview_grid.png`. If the main drawing text is not upright, re-render that page with an explicit rotation, given clockwise: `render_pages.py <work> --pages N --rotation 90`.
    
    Title blocks are sometimes rotated differently from the drawing. Orient by the drawing labels, not by the title block.
    
    ### 4. Hints
    `make_hints.py` writes `hints/page_NN.json`. Each file holds candidate tags with page positions, taken from the text layer or from OCR. They are only hints: OCR misreads, and it misses things. You don't need to read the hint files during extraction. QA uses them to flag tags you may have missed. If OCR is not installed the step is skipped; note this in the final report.
    
    ### 5. Extract each sheet
    This is the core of the job. Follow **[references/extraction_procedure.md](references/extraction_procedure.md)** exactly. It covers:
    
    - the reading order: overview, then every non-blank tile, then zoomed crops
    - how to model pipe runs, tees, taps, signals and off-page connectors
    - confidence calibration
    - a self-check list to run before saving
    
    Reference material, to read when needed:
    
    - [references/isa_5_1.md](references/isa_5_1.md): instrument letter codes, bubble symbols, signal line types
    - [references/symbol_guide.md](references/symbol_guide.md): how equipment and valve symbols map to `kind` values
    - [references/ontology.md](references/ontology.md): the graph's node and edge types and how they are derived
    - `references/schema/extraction.schema.json`: the exact JSON schema of a page file
    
    Write `extract/page_NN.json` and save it before starting the next sheet, so a long job never holds more than one sheet in context. With many sheets, and if your environment supports subagents, you can extract sheets in parallel. Give each subagent the procedure file and its own page directory.
    
    ### 6–7. Build and the QA loop
    `build_kg.py` validates every page file and stops with a precise message on schema errors. It then writes these files to `<work>/kg/`:
    
    - `graph.json`
    - `graph.duckdb`
    - `summary.md`
    - `qa_report.md`
    
    Work through `qa_report.md`:
    
    - **Errors**, such as `unresolved_reference`, are mistakes in the page files. Fix them and rebuild.
    - **Warnings** name the tiles to re-check. Look at those tiles again, or use `crop.py` for a zoomed view, and correct the extraction if the drawing shows something different. Common warnings:
      - `instrument_not_located`: a sensor with no `measures`
      - `equipment_not_connected`
      - `unmatched_off_page`
      - `possible_missed_tags`: an OCR tag that is not in your extraction
      - `tiles_not_reviewed`
    - A warning that turns out correct on the drawing is fine to leave. Examples: an off-page connector whose other drawing was not supplied, or an OCR misread. List these in your report.
    
    Stop when there are no errors and every warning has been checked.
    
    ```bash
    python scripts/crop.py <work> --page 5 --bbox 0.40 0.20 0.55 0.35   # zoomed view of a region
    ```
    
    ### 8. Report
    Tell the user:
    
    - where the outputs are
    - the headline counts (equipment, instruments, loops)
    - two or three example flow paths and loops taken from `summary.md`
    - what remains uncertain
    
    Point to `qa_report.md` for the details. State plainly that the graph is machine-extracted and needs engineering review before it is used for design, operations or safety.
    
    ## Using the graph
    
    - `summary.md` is the compact, token-efficient context for an LLM: equipment-level flow, an instrument table with locations, and a loop table.
    - `graph.json` is the canonical graph, which any tool can read.
    - `graph.duckdb` is for SQL. Its views are `equipment`, `equipment_flow`, `instruments`, `loops` and `signals`, and its table macros are `upstream(id)` and `downstream(id)`.
    
    Use `kg_query.py` to answer questions; add `--json` when another tool will consume the output:
    
    ```bash
    python scripts/kg_query.py <work>/kg downstream 660-TK-001 --equipment
    python scripts/kg_query.py <work>/kg upstream FV-0502
    python scripts/kg_query.py <work>/kg path 660-TK-001 660-TK-004
    python scripts/kg_query.py <work>/kg instruments --on 660-PP-011
    python scripts/kg_query.py <work>/kg loop FIC-0502
    python scripts/kg_query.py <work>/kg sql "SELECT * FROM equipment_flow"
    ```
    
    To measure extraction quality on a sheet, hand-check it into a small gold file (format in `scripts/eval_kg.py`) and run `python scripts/eval_kg.py <work>/kg gold.json`: it reports tag, flow and loop precision/recall and instrument placement.
    
    To move a graph into other tools: `python scripts/export.py <work>/kg graphml` produces a file for yEd, Gephi or NetworkX, and `python scripts/export.py <work>/kg neo4j` produces CSVs plus a Cypher import script.
    
  • skills/pid-to-kg/scripts/build_kg.py4.5 KB9de11c3fe63fupstream ↗
    #!/usr/bin/env python3
    """Build the knowledge graph for one plant from its page extractions.
    
    Reads <work>/extract/page_*.json (validated against the extraction schema),
    merges items across sheets by tag, expands pipe runs into segments, links
    off-page connectors, infers control loops and equipment-level flow, runs QA,
    and writes to <work>/kg/:
    
        graph.json     canonical knowledge graph (node-link JSON)
        graph.duckdb   the same graph as DuckDB tables, views and traversal macros
        summary.md     compact LLM-ready summary of flow, instruments and loops
        qa_report.md   issues to fix or re-check, with the tiles to look at
    
    Usage:
        python build_kg.py work/my-plant
        python build_kg.py --write-schemas      # regenerate references/schema/*.json
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import sys
    from pathlib import Path
    
    sys.path.insert(0, str(Path(__file__).resolve().parent))
    
    from pidkg.build import BuildError, build_graph, load_extractions  # noqa: E402
    from pidkg.models import KnowledgeGraph, PageExtraction  # noqa: E402
    from pidkg.qa import run_qa, write_report  # noqa: E402
    from pidkg.store import write_duckdb  # noqa: E402
    from pidkg.summary import write_summary  # noqa: E402
    
    SCHEMA_DIR = Path(__file__).resolve().parent.parent / "references" / "schema"
    
    
    def write_schemas() -> None:
        SCHEMA_DIR.mkdir(parents=True, exist_ok=True)
        for name, model in (("extraction.schema.json", PageExtraction), ("graph.schema.json", KnowledgeGraph)):
            schema = model.model_json_schema(by_alias=True)
            schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
            (SCHEMA_DIR / name).write_text(json.dumps(schema, indent=2) + "\n", encoding="utf-8")
            print(f"wrote {SCHEMA_DIR / name}")
    
    
    def main() -> int:
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument("work", type=Path, nargs="?")
        ap.add_argument("--out", type=Path, help="output directory (default: <work>/kg)")
        ap.add_argument("--plant-name")
        ap.add_argument("--plant-id")
        ap.add_argument("--pages", help="only build from these pages, e.g. '1-2' (default: all extracted)")
        ap.add_argument("--write-schemas", action="store_true")
        args = ap.parse_args()
        if hasattr(sys.stdout, "reconfigure"):
            sys.stdout.reconfigure(encoding="utf-8")
    
        if args.write_schemas:
            write_schemas()
            return 0
        if not args.work:
            ap.error("work directory is required")
    
        manifest = {}
        mpath = args.work / "manifest.json"
        if mpath.exists():
            manifest = json.loads(mpath.read_text(encoding="utf-8"))
        plant_name = args.plant_name or manifest.get("plant_name") or args.work.name
        plant_id = args.plant_id or manifest.get("plant_id") or args.work.name
        source = Path(manifest["source_pdf"]).name if manifest.get("source_pdf") else None
    
        try:
            pages = load_extractions(args.work / "extract")
        except BuildError as e:
            print(f"error: {e}", file=sys.stderr)
            return 1
        if args.pages:
            from render_pages import parse_pages
    
            keep = set(parse_pages(args.pages, max(p.page for p in pages)))
            pages = [p for p in pages if p.page in keep]
            if not pages:
                print(f"error: no extracted pages in {args.pages}", file=sys.stderr)
                return 1
    
        kg = build_graph(pages, plant_id, plant_name, source)
        kg.issues.extend(run_qa(kg, args.work))
        sev: dict[str, int] = {}
        for i in kg.issues:
            sev[i["severity"]] = sev.get(i["severity"], 0) + 1
        kg.stats["issues_by_severity"] = sev
    
        out = args.out or (args.work / "kg")
        out.mkdir(parents=True, exist_ok=True)
        (out / "graph.json").write_text(kg.model_dump_json(indent=1, exclude_none=True), encoding="utf-8")
        write_duckdb(kg, out / "graph.duckdb")
        (out / "summary.md").write_text(write_summary(kg), encoding="utf-8")
        write_report(kg, kg.issues, out / "qa_report.md", args.work)
    
        s = kg.stats
        print(f"{plant_name}: {s['nodes']} nodes, {s['edges']} edges -> {out}")
        print("  nodes: " + ", ".join(f"{k} {v}" for k, v in s["nodes_by_type"].items()))
        print("  edges: " + ", ".join(f"{k} {v}" for k, v in s["edges_by_type"].items()))
        if s.get("loops_by_type"):
            print("  loops: " + ", ".join(f"{k} {v}" for k, v in s["loops_by_type"].items()))
        print(f"  issues: {sev.get('error', 0)} errors, {sev.get('warning', 0)} warnings, "
              f"{sev.get('info', 0)} info (see qa_report.md)")
        return 1 if sev.get("error") else 0
    
    
    if __name__ == "__main__":
        sys.exit(main())
    
  • skills/pid-to-kg/scripts/crop.py1.8 KBcb6e1a7f8cbeupstream ↗
    #!/usr/bin/env python3
    """Cut a zoomed view of a page region for a closer look.
    
    The bbox is in normalized page coordinates (0..1, origin top-left), the same
    system used for tiles and for ``bbox`` in extractions. The crop is upscaled
    so its long edge is --size pixels.
    
    Usage:
        python crop.py work/my-plant --page 5 --bbox 0.40 0.20 0.55 0.35
    """
    
    from __future__ import annotations
    
    import argparse
    import sys
    from pathlib import Path
    
    
    def main() -> int:
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument("work", type=Path)
        ap.add_argument("--page", type=int, required=True)
        ap.add_argument("--bbox", type=float, nargs=4, required=True, metavar=("X0", "Y0", "X1", "Y1"))
        ap.add_argument("--size", type=int, default=1400, help="long edge of the output in px")
        ap.add_argument("--out", type=Path)
        a = ap.parse_args()
    
        from PIL import Image
    
        page_dir = a.work / "pages" / f"page_{a.page:02d}"
        src = page_dir / "page.png"
        if not src.exists():
            print(f"error: {src} not found; run render_pages.py first", file=sys.stderr)
            return 2
        img = Image.open(src)
        x0, y0, x1, y1 = a.bbox
        box = (round(min(x0, x1) * img.width), round(min(y0, y1) * img.height),
               round(max(x0, x1) * img.width), round(max(y0, y1) * img.height))
        crop = img.crop(box)
        scale = a.size / max(crop.size)
        crop = crop.resize((max(1, round(crop.width * scale)), max(1, round(crop.height * scale))),
                           Image.LANCZOS)
        out = a.out or page_dir / "crops" / f"crop_{x0:.2f}_{y0:.2f}_{x1:.2f}_{y1:.2f}.png"
        out.parent.mkdir(parents=True, exist_ok=True)
        crop.save(out)
        print(out)
        return 0
    
    
    if __name__ == "__main__":
        sys.exit(main())
    
  • skills/pid-to-kg/scripts/eval_kg.py7.5 KB282d8a37aeaeupstream ↗
    #!/usr/bin/env python3
    """Score a built graph against a hand-checked gold annotation.
    
    Gold file (JSON), usually covering one or a few sheets:
    
        {
          "sheets": ["660-PI-203"],                      # drawing numbers the gold covers
          "tags": ["660-TK-004", "FIT-0502", ...],       # every tagged item on those sheets
          "feeds": [["660-EQ-002", "660-PP-011"], ...],  # equipment-level flow (upstream, downstream)
          "instrument_at": {"FIT-0502": "660-PP-011"},   # nearest equipment to where it senses
          "loops": {"F-0502": ["FE-0502", "FIT-0502"]},  # loop key -> member tags
          "feeds_scope": "within_sheets"                 # optional: "all" (default), "within_sheets"
                                                         # (both ends on gold sheets), "equipment" (both ends equipment)
        }
    
    Reports precision/recall/F1 for tags and flow, instrument placement accuracy,
    and loop recall with member overlap. Only graph items on the gold sheets are
    scored, so a partial gold set gives a fair spot check.
    
    Usage:
        python eval_kg.py work/my-plant/kg gold.json [--json]
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import sys
    from pathlib import Path
    
    sys.path.insert(0, str(Path(__file__).resolve().parent))
    
    from pidkg.isa import normalize_tag  # noqa: E402
    from pidkg.store import Graph  # noqa: E402
    
    
    def prf(pred: set, gold: set) -> dict:
        tp = len(pred & gold)
        p = tp / len(pred) if pred else 0.0
        r = tp / len(gold) if gold else 0.0
        f = 2 * p * r / (p + r) if p + r else 0.0
        return {"precision": round(p, 3), "recall": round(r, 3), "f1": round(f, 3), "tp": tp,
                "false_positives": sorted(map(str, pred - gold)), "missed": sorted(map(str, gold - pred))}
    
    
    def near_equipment(g: Graph, nid: str, hops: int = 60) -> set[str]:
        """Equipment directly measured, or nearest up/downstream of a measured pipe or item."""
        n = g.nodes[nid]
        if n.type == "Equipment":
            return {n.tag or n.label}
        found: set[str] = set()
        for direction in ("up", "down"):
            best = None
            for other, d in g.walk(nid, direction, max_depth=hops):
                if g.nodes[other].type == "Equipment":
                    best = (d, g.nodes[other].tag or g.nodes[other].label)
                    break
            if best:
                found.add(best[1])
        return found
    
    
    def evaluate(g: Graph, gold: dict) -> dict:
        sheets = {f"SHEET:{normalize_tag(s)}" for s in gold.get("sheets", [])}
        on_sheet = {n.id for n in g.kg.nodes if not sheets or set(n.sheets) & sheets}
    
        def find(tag: str):
            """Prefer the node with this tag that is drawn on the gold sheets."""
            t = normalize_tag(tag)
            hits = [n.id for n in g.kg.nodes if n.tag == t and n.id in on_sheet]
            return hits[0] if hits else g.find(tag)
    
        pred_tags = {n.tag for n in g.kg.nodes if n.tag and n.id in on_sheet
                     and n.type in ("Equipment", "Instrument", "Component")}
        gold_tags = {normalize_tag(t) for t in gold.get("tags", [])}
        res: dict = {"tags": prf(pred_tags, gold_tags)}
    
        scope = gold.get("feeds_scope", "all")
        feeds = [e for es in g.out.values() for e in es if e.type == "FEEDS"]  # honours the PFD/P&ID view
        if scope == "within_sheets":
            feeds = [e for e in feeds if e.source in on_sheet and e.target in on_sheet]
        else:
            feeds = [e for e in feeds if e.source in on_sheet or e.target in on_sheet]
        if scope in ("equipment", "within_sheets"):
            feeds = [e for e in feeds if g.nodes[e.source].type == "Equipment" and g.nodes[e.target].type == "Equipment"]
        pred_feeds = {(g.nodes[e.source].tag or g.nodes[e.source].label, g.nodes[e.target].tag or g.nodes[e.target].label)
                      for e in feeds}
        gold_feeds = {(normalize_tag(a) or a, normalize_tag(b) or b) for a, b in gold.get("feeds", [])}
        norm_pred = {(normalize_tag(a) or a, normalize_tag(b) or b) for a, b in pred_feeds}
        res["flow"] = prf(norm_pred, gold_feeds)
    
        placed, wrong = 0, []
        for tag, eq in gold.get("instrument_at", {}).items():
            nid = find(tag)
            if nid is None:
                wrong.append({"tag": tag, "expected": eq, "got": "missing"})
                continue
            targets = [e.target for e in g.out[nid] if e.type == "MEASURES"]
            if g.nodes[nid].attributes.get("inline"):
                targets.append(nid)
            got = set().union(*(near_equipment(g, t) for t in targets)) if targets else set()
            if normalize_tag(eq) in {normalize_tag(x.split("@")[0]) for x in got}:
                placed += 1
            else:
                wrong.append({"tag": tag, "expected": eq, "got": sorted(got) or "unlocated"})
        n_inst = len(gold.get("instrument_at", {}))
        res["instrument_placement"] = {"accuracy": round(placed / n_inst, 3) if n_inst else None,
                                       "correct": placed, "total": n_inst, "errors": wrong}
    
        loops_found, overlaps, loop_detail = 0, [], []
        for key, members in gold.get("loops", {}).items():
            lid = next((n.id for n in g.kg.nodes if n.type == "ControlLoop" and n.id in on_sheet | {n.id}
                        and n.label.split("@")[0] == key and set(n.sheets) & (sheets or set(n.sheets))), None)
            gm = {normalize_tag(m) for m in members}
            if lid is None:
                loop_detail.append({"loop": key, "found": False})
                continue
            loops_found += 1
            pm = {g.nodes[e.source].tag or g.nodes[e.source].label for e in g.inn[lid] if e.type == "MEMBER_OF"}
            jac = len(pm & gm) / len(pm | gm) if pm | gm else 1.0
            overlaps.append(jac)
            loop_detail.append({"loop": key, "found": True, "member_jaccard": round(jac, 3),
                                "missing_members": sorted(gm - pm), "extra_members": sorted(pm - gm)})
        n_loops = len(gold.get("loops", {}))
        res["loops"] = {"recall": round(loops_found / n_loops, 3) if n_loops else None,
                        "mean_member_jaccard": round(sum(overlaps) / len(overlaps), 3) if overlaps else None,
                        "detail": loop_detail}
        return res
    
    
    def main() -> int:
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument("kg", type=Path)
        ap.add_argument("gold", type=Path)
        ap.add_argument("--json", action="store_true")
        a = ap.parse_args()
        if hasattr(sys.stdout, "reconfigure"):
            sys.stdout.reconfigure(encoding="utf-8")
        g = Graph.load(a.kg / "graph.json" if a.kg.is_dir() else a.kg)
        res = evaluate(g, json.loads(a.gold.read_text(encoding="utf-8")))
        if a.json:
            print(json.dumps(res, indent=1))
            return 0
        t, f, ip, lp = res["tags"], res["flow"], res["instrument_placement"], res["loops"]
        print(f"Tags:   P {t['precision']:.2f}  R {t['recall']:.2f}  F1 {t['f1']:.2f}"
              f"   missed: {', '.join(t['missed']) or '-'}   extra: {', '.join(t['false_positives']) or '-'}")
        print(f"Flow:   P {f['precision']:.2f}  R {f['recall']:.2f}  F1 {f['f1']:.2f}"
              f"   missed: {f['missed'] or '-'}   extra: {f['false_positives'] or '-'}")
        if ip["total"]:
            print(f"Instrument placement: {ip['correct']}/{ip['total']}"
                  + "".join(f"\n   {w['tag']}: expected {w['expected']}, got {w['got']}" for w in ip["errors"]))
        if lp["recall"] is not None:
            print(f"Loops:  recall {lp['recall']:.2f}, mean member overlap {lp['mean_member_jaccard']}")
            for d in lp["detail"]:
                if not d["found"] or d.get("missing_members") or d.get("extra_members"):
                    print(f"   {d}")
        return 0
    
    
    if __name__ == "__main__":
        sys.exit(main())
    
  • skills/pid-to-kg/scripts/export.py4.7 KB24d5caa75ea6upstream ↗
    #!/usr/bin/env python3
    """Export a knowledge graph to GraphML or a Neo4j import bundle.
    
        python export.py work/my-plant/kg graphml          # -> graph.graphml (yEd, Gephi, NetworkX)
        python export.py work/my-plant/kg neo4j            # -> neo4j/nodes.csv, edges.csv, import.cypher
    
    Node types become Neo4j labels and edge types become relationship types.
    Nested attributes are flattened to scalar properties where possible and kept
    whole as a JSON string in ``attributes_json``.
    """
    
    from __future__ import annotations
    
    import argparse
    import csv
    import json
    import sys
    from pathlib import Path
    
    sys.path.insert(0, str(Path(__file__).resolve().parent))
    
    from pidkg.store import Graph  # noqa: E402
    
    
    def _scalar(v):
        if v is None or isinstance(v, (str, int, float, bool)):
            return v
        return json.dumps(v)
    
    
    def _node_props(n) -> dict:
        props = {"id": n.id, "type": n.type, "label": n.label, "tag": n.tag, "kind": n.kind,
                 "name": n.name, "status": n.status, "dexpi_class": n.dexpi_class,
                 "sheets": ";".join(n.sheets), "confidence": n.confidence,
                 "attributes_json": json.dumps(n.attributes)}
        for k, v in n.attributes.items():
            props.setdefault(f"a_{k}", _scalar(v))
        return {k: v for k, v in props.items() if v is not None}
    
    
    def export_graphml(g: Graph, path: Path) -> None:
        import networkx as nx
    
        G = nx.MultiDiGraph(plant=g.kg.plant.name)
        for n in g.kg.nodes:
            G.add_node(n.id, **_node_props(n))
        for e in g.kg.edges:
            attrs = {"type": e.type, "confidence": e.confidence, "derived": e.derived,
                     "attributes_json": json.dumps(e.attributes)}
            if e.sheet:
                attrs["sheet"] = e.sheet
            G.add_edge(e.source, e.target, key=e.id, **attrs)
        nx.write_graphml(G, path)
    
    
    def export_neo4j(g: Graph, out: Path) -> None:
        out.mkdir(parents=True, exist_ok=True)
        keys = ["id", "type", "label", "tag", "kind", "name", "status", "dexpi_class", "sheets",
                "confidence", "attributes_json"]
        with open(out / "nodes.csv", "w", newline="", encoding="utf-8") as f:
            w = csv.DictWriter(f, fieldnames=keys, extrasaction="ignore")
            w.writeheader()
            for n in g.kg.nodes:
                w.writerow(_node_props(n))
        with open(out / "edges.csv", "w", newline="", encoding="utf-8") as f:
            w = csv.writer(f)
            w.writerow(["id", "source", "target", "type", "confidence", "derived", "sheet", "attributes_json"])
            for e in g.kg.edges:
                w.writerow([e.id, e.source, e.target, e.type, e.confidence, e.derived, e.sheet or "",
                            json.dumps(e.attributes)])
        node_types = sorted({n.type for n in g.kg.nodes})
        edge_types = sorted({e.type for e in g.kg.edges})
        plant = g.kg.plant.id
        lines = [
            f"// Neo4j import for plant '{g.kg.plant.name}'. Copy nodes.csv and edges.csv into the",
            "// Neo4j import directory, then run this script (e.g. with cypher-shell).",
            "CREATE CONSTRAINT pid_node_id IF NOT EXISTS FOR (n:PidNode) REQUIRE (n.plant, n.id) IS UNIQUE;",
            "",
            "LOAD CSV WITH HEADERS FROM 'file:///nodes.csv' AS row",
            f"MERGE (n:PidNode {{plant: '{plant}', id: row.id}})",
            "SET n += row, n.confidence = toFloat(row.confidence);",
            "",
        ]
        for t in node_types:
            lines.append(f"MATCH (n:PidNode {{plant: '{plant}', type: '{t}'}}) SET n:{t};")
        lines.append("")
        for t in edge_types:
            lines += [
                "LOAD CSV WITH HEADERS FROM 'file:///edges.csv' AS row",
                f"WITH row WHERE row.type = '{t}'",
                f"MATCH (a:PidNode {{plant: '{plant}', id: row.source}}), (b:PidNode {{plant: '{plant}', id: row.target}})",
                f"MERGE (a)-[r:{t} {{id: row.id}}]->(b)",
                "SET r.confidence = toFloat(row.confidence), r.derived = row.derived = 'True',",
                "    r.sheet = row.sheet, r.attributes_json = row.attributes_json;",
                "",
            ]
        (out / "import.cypher").write_text("\n".join(lines), encoding="utf-8")
    
    
    def main() -> int:
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument("kg", type=Path, help="kg/ directory or graph.json")
        ap.add_argument("format", choices=["graphml", "neo4j"])
        a = ap.parse_args()
        if hasattr(sys.stdout, "reconfigure"):
            sys.stdout.reconfigure(encoding="utf-8")
        json_path = a.kg / "graph.json" if a.kg.is_dir() else a.kg
        g = Graph.load(json_path)
        if a.format == "graphml":
            target = json_path.with_name("graph.graphml")
            export_graphml(g, target)
        else:
            target = json_path.with_name("neo4j")
            export_neo4j(g, target)
        print(f"wrote {target}")
        return 0
    
    
    if __name__ == "__main__":
        sys.exit(main())
    
  • skills/pid-to-kg/scripts/kg_query.py8.8 KBd27d262fad34upstream ↗
    #!/usr/bin/env python3
    """Query a P&ID knowledge graph (for people, and as a tool for LLM agents).
    
    Usage:
        python kg_query.py KG stats
        python kg_query.py KG describe 660-TK-004
        python kg_query.py KG upstream FV-0502 [--equipment] [--depth 3]
        python kg_query.py KG downstream 660-TK-001 [--equipment]
        python kg_query.py KG path 660-TK-001 660-TK-004
        python kg_query.py KG loops [--type control]
        python kg_query.py KG loop F-0502
        python kg_query.py KG instruments [--on 660-PP-011] [--variable flow]
        python kg_query.py KG sql "SELECT * FROM equipment_flow"
    
    KG is a kg/ directory, graph.json or graph.duckdb. Add --json for machine-readable output.
    Item references accept ids, tags or labels; tags are matched loosely (FIT0502 = FIT-0502).
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import sys
    from pathlib import Path
    
    sys.path.insert(0, str(Path(__file__).resolve().parent))
    
    from pidkg.loops import point_label, sensed_points  # noqa: E402
    from pidkg.store import Graph  # noqa: E402
    
    TERMINAL = ("Equipment", "Boundary")
    
    
    def _resolve_paths(kg: Path) -> tuple[Path, Path]:
        if kg.is_dir():
            return kg / "graph.json", kg / "graph.duckdb"
        if kg.suffix == ".duckdb":
            return kg.with_name("graph.json"), kg
        return kg, kg.with_name("graph.duckdb")
    
    
    def _node_brief(g: Graph, nid: str) -> dict:
        n = g.nodes[nid]
        d = {"id": n.id, "type": n.type, "label": n.label}
        if n.kind:
            d["kind"] = n.kind
        if n.name:
            d["name"] = n.name
        return d
    
    
    def main() -> int:
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument("kg", type=Path)
        ap.add_argument("command", choices=["stats", "describe", "upstream", "downstream", "path",
                                            "loops", "loop", "instruments", "sql"])
        ap.add_argument("args", nargs="*")
        ap.add_argument("--equipment", action="store_true", help="only equipment and boundaries")
        ap.add_argument("--depth", type=int)
        ap.add_argument("--type", help="loop type filter")
        ap.add_argument("--on", help="instruments measuring this item (or anything on it)")
        ap.add_argument("--variable", help="instrument variable filter, e.g. flow")
        ap.add_argument("--include-pfd", action="store_true",
                        help="also traverse PFD-sheet flow when the plant has P&IDs (default: P&ID detail only)")
        ap.add_argument("--json", action="store_true")
        a = ap.parse_args()
        if hasattr(sys.stdout, "reconfigure"):
            sys.stdout.reconfigure(encoding="utf-8")
    
        json_path, db_path = _resolve_paths(a.kg)
    
        if a.command == "sql":
            import duckdb
    
            con = duckdb.connect(str(db_path), read_only=True)
            rel = con.sql(" ".join(a.args))
            if a.json:
                cols = rel.columns
                print(json.dumps([dict(zip(cols, r)) for r in rel.fetchall()], indent=1, default=str))
            else:
                rel.show(max_width=200, max_rows=200)
            return 0
    
        g = Graph.load(json_path, include_pfd=a.include_pfd)
    
        def need(n: int) -> list[str]:
            if len(a.args) < n:
                ap.error(f"'{a.command}' needs {n} item reference(s)")
            ids = []
            for ref in a.args[:n]:
                nid = g.find(ref)
                if nid is None:
                    print(f"error: '{ref}' not found in the graph", file=sys.stderr)
                    sys.exit(2)
                ids.append(nid)
            return ids
    
        result: object
        if a.command == "stats":
            result = {"plant": g.kg.plant.model_dump(), **g.kg.stats}
        elif a.command == "describe":
            (nid,) = need(1)
            n = g.nodes[nid]
            result = {**n.model_dump(exclude_none=True, exclude={"provenance"}),
                      "relations": g.neighbors(nid)}
            if n.type == "PipeSegment":
                result["location"] = point_label(nid, g.nodes, g.kg.edges)
        elif a.command in ("upstream", "downstream"):
            (nid,) = need(1)
            hits = g.walk(nid, "down" if a.command == "downstream" else "up", max_depth=a.depth)
            rows = [dict(_node_brief(g, h), hops=d) for h, d in hits
                    if not a.equipment or g.nodes[h].type in TERMINAL]
            result = {"from": _node_brief(g, nid), "direction": a.command, "items": rows}
        elif a.command == "path":
            x, y = need(2)
            p = g.path(x, y) or g.path(y, x)
            result = {"path": [_node_brief(g, i) for i in p]} if p else {"path": None,
                      "note": "no directed flow path between these items"}
        elif a.command == "loops":
            loops = [n for n in g.kg.nodes if n.type == "ControlLoop" and (not a.type or n.kind == a.type)]
            result = [{"id": n.id, "loop": n.label, "type": n.kind,
                       "description": n.attributes.get("description")} for n in loops]
        elif a.command == "loop":
            ref = a.args[0] if a.args else ap.error("'loop' needs a loop id")
            nid = g.find(ref) or g.find(f"LOOP:{ref}")
            if nid is None or g.nodes[nid].type != "ControlLoop":
                # Accept a member tag, e.g. 'loop FIC-0502'.
                member = g.find(ref)
                nid = next((e.target for e in g.out.get(member, []) if e.type == "MEMBER_OF"), None) \
                    if member else None
            if nid is None:
                print(f"error: loop '{ref}' not found", file=sys.stderr)
                return 2
            n = g.nodes[nid]
            members = [dict(_node_brief(g, e.source), role=e.attributes.get("role"))
                       for e in g.inn[nid] if e.type == "MEMBER_OF"]
            result = {**n.model_dump(exclude_none=True, exclude={"provenance"}), "members": members}
        else:  # instruments
            target_ids: set[str] | None = None
            if a.on:
                t = g.find(a.on)
                if t is None:
                    print(f"error: '{a.on}' not found", file=sys.stderr)
                    return 2
                # The item plus its own piping: everything reachable up- and downstream
                # before the next piece of equipment.
                target_ids = {t} | _local_piping(g, t)
            rows = []
            for n in g.kg.nodes:
                if n.type != "Instrument":
                    continue
                if a.variable and a.variable.lower() not in (n.attributes.get("variable") or "").lower():
                    continue
                meas = {e.target for e in g.out[n.id] if e.type == "MEASURES"}
                if target_ids is not None and not ((meas | {n.id}) & target_ids):
                    continue
                rows.append({"tag": n.label, "variable": n.attributes.get("variable"),
                             "role": n.attributes.get("role"), "location": n.attributes.get("location"),
                             "measures": sensed_points(n.id, g.nodes, g.kg.edges),
                             "loop": [g.nodes[e.target].label for e in g.out[n.id] if e.type == "MEMBER_OF"]})
            result = rows
    
        if a.json:
            print(json.dumps(result, indent=1, default=str))
        else:
            _print_text(result)
        return 0
    
    
    def _local_piping(g: Graph, start: str, limit: int = 8) -> set[str]:
        """Segments and fittings within ``limit`` hops of an item, stopping at equipment."""
        found: set[str] = set()
        for direction in ("down", "up"):
            frontier = [start]
            while frontier and len(found) < 500:
                nxt = []
                for cur in frontier:
                    edges = g.out[cur] if direction == "down" else g.inn[cur]
                    for e in edges:
                        if e.type != "FLOWS_TO":
                            continue
                        other = e.target if direction == "down" else e.source
                        if other in found or other == start or g.nodes[other].type in TERMINAL:
                            continue
                        found.add(other)
                        nxt.append(other)
                frontier = nxt
                limit -= 1
                if limit <= 0:
                    break
        return found
    
    
    def _print_text(obj, indent: int = 0) -> None:
        pad = "  " * indent
        if isinstance(obj, list):
            if not obj:
                print(pad + "(none)")
            for item in obj:
                if isinstance(item, dict):
                    print(pad + "- " + ", ".join(f"{k}: {_fmt(v)}" for k, v in item.items()))
                else:
                    print(pad + f"- {item}")
        elif isinstance(obj, dict):
            for k, v in obj.items():
                if isinstance(v, (dict, list)) and v and any(isinstance(x, (dict, list)) for x in
                                                            (v.values() if isinstance(v, dict) else v)):
                    print(f"{pad}{k}:")
                    _print_text(v, indent + 1)
                else:
                    print(f"{pad}{k}: {_fmt(v)}")
        else:
            print(pad + str(obj))
    
    
    def _fmt(v) -> str:
        if isinstance(v, list):
            return "[" + ", ".join(_fmt(x) for x in v) + "]"
        if isinstance(v, dict):
            return "{" + ", ".join(f"{k}: {_fmt(x)}" for k, x in v.items()) + "}"
        return str(v)
    
    
    if __name__ == "__main__":
        sys.exit(main())
    
  • skills/pid-to-kg/scripts/make_hints.py4.0 KBc45e830a7b63upstream ↗
    #!/usr/bin/env python3
    """Collect deterministic text hints for each rendered page.
    
    Hints are candidate tags with page positions, gathered without the LLM:
    
    * text layer (vector_text, raster_with_text, hybrid pages): exact text and
      positions straight from the PDF;
    * OCR (vector_outlined, raster, hybrid pages; needs rapidocr-onnxruntime):
      each non-blank tile is OCR'd and mapped back to page coordinates.
    
    ISA bubbles split over two lines ('FIT' over '0502') are joined. QA compares
    these candidates with the extraction to flag tags that may have been missed.
    Writes <work>/hints/page_NN.json.
    
    Usage:
        python make_hints.py work/my-plant [--pages 1-3] [--no-ocr]
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import sys
    from pathlib import Path
    
    sys.path.insert(0, str(Path(__file__).resolve().parent))
    
    from pidkg import ocr  # noqa: E402
    from pidkg.pdf import text_runs  # noqa: E402
    from render_pages import parse_pages  # noqa: E402
    
    TEXT_LAYER_CLASSES = {"vector_text", "raster_with_text", "hybrid"}
    OCR_CLASSES = {"vector_outlined", "raster", "hybrid"}
    
    
    def tiles_for(bbox: list[float], tiles: list[dict]) -> list[str]:
        cx, cy = (bbox[0] + bbox[2]) / 2, (bbox[1] + bbox[3]) / 2
        return [t["name"] for t in tiles if t["norm"][0] <= cx <= t["norm"][2]
                and t["norm"][1] <= cy <= t["norm"][3]]
    
    
    def main() -> int:
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument("work", type=Path)
        ap.add_argument("--pages")
        ap.add_argument("--no-ocr", action="store_true")
        args = ap.parse_args()
        if hasattr(sys.stdout, "reconfigure"):
            sys.stdout.reconfigure(encoding="utf-8", line_buffering=True)
    
        from PIL import Image
    
        manifest = json.loads((args.work / "manifest.json").read_text(encoding="utf-8"))
        use_ocr = not args.no_ocr and ocr.available()
        if not args.no_ocr and not use_ocr:
            print(f"note: {ocr.maybe_import_error()}")
        (args.work / "hints").mkdir(exist_ok=True)
    
        for page_no in parse_pages(args.pages, len(manifest["pages"]), manifest.get("selected_pages")):
            info = manifest["pages"][page_no - 1]
            cls = info["page_class"]
            page_dir = args.work / "pages" / f"page_{page_no:02d}"
            tiles_file = page_dir / "tiles.json"
            if not tiles_file.exists():
                print(f"page {page_no}: not rendered yet (run render_pages.py), skipped")
                continue
            meta = json.loads(tiles_file.read_text(encoding="utf-8"))
            runs: list[dict] = []
            sources: list[str] = []
            if cls in TEXT_LAYER_CLASSES:
                runs += [dict(r, source="text_layer")
                         for r in text_runs(manifest["source_pdf"], page_no, meta["rotation"])]
                sources.append("text_layer")
            if cls in OCR_CLASSES and use_ocr:
                for t in meta["tiles"]:
                    if t.get("blank"):
                        continue
                    img = Image.open(page_dir / "tiles" / f"{t['name']}.png")
                    for r in ocr.ocr_image(img):
                        if r["conf"] < 0.5 or not r["text"]:
                            continue
                        runs.append({"text": r["text"], "conf": round(r["conf"], 3), "source": "ocr",
                                     "bbox": ocr.to_norm(r["bbox"], t["px"], meta["width"], meta["height"])})
                sources.append("ocr")
            runs = ocr.dedupe_runs(runs)
            runs = ocr.join_bubbles(runs)
            cands = ocr.tag_candidates(runs)
            for c in cands:
                c["tiles"] = tiles_for(c["bbox"], meta["tiles"])
            out = {"page": page_no, "source": "+".join(sources) or "none", "page_class": cls,
                   "tag_candidates": cands, "runs": runs}
            (args.work / "hints" / f"page_{page_no:02d}.json").write_text(
                json.dumps(out, indent=1), encoding="utf-8")
            print(f"page {page_no}: {len(runs)} text runs, {len(cands)} tag candidates ({out['source']})")
        return 0
    
    
    if __name__ == "__main__":
        sys.exit(main())
    
  • skills/pid-to-kg/scripts/pidkg/__init__.py153 Bcdc430366f41upstream ↗
    """pidkg: build process knowledge graphs from P&ID extractions."""
    
    __version__ = "0.1.0"
    EXTRACTION_SCHEMA_VERSION = "1.0"
    GRAPH_SCHEMA_VERSION = "1.0"
    
  • skills/pid-to-kg/scripts/pidkg/build.py44 KB7ecaf83f96f3upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/pid-to-kg/scripts/pidkg/isa.py4.9 KB3ea93fc81e20upstream ↗
    """Tag normalization and ISA-5.1 instrument tag parsing."""
    
    from __future__ import annotations
    
    import re
    from dataclasses import dataclass, field
    from typing import Optional
    
    from .ontology import ISA_FUNCTIONS, ISA_MEASURED, ISA_VARIABLE_MODIFIERS
    
    _DASHES = dict.fromkeys(map(ord, "‐‑‒–—−"), "-")
    _COMPACT = re.compile(r"^([A-Z]{1,6})(\d{1,6}[A-Z]{0,2})$")
    
    
    def normalize_tag(raw: Optional[str]) -> Optional[str]:
        """Normalize a printed tag so the same item read twice compares equal.
    
        ``"fit 0502"``, ``"FIT-0502"``, ``"FIT_0502"`` and ``"FIT0502"`` all become
        ``"FIT-0502"``. Area prefixes and suffixes are preserved
        (``"660 TK 004"`` -> ``"660-TK-004"``).
        """
        if raw is None:
            return None
        s = str(raw).translate(_DASHES).strip().strip("\"'").upper()
        if not s:
            return None
        s = re.sub(r"[\s_/]*-[\s_/]*", "-", s)
        s = re.sub(r"[\s_]+", "-", s)
        s = re.sub(r"-{2,}", "-", s).strip("-")
        m = _COMPACT.match(s)
        if m:
            s = f"{m.group(1)}-{m.group(2)}"
        return s
    
    
    def normalize_label(raw: Optional[str]) -> str:
        """Normalize free text (boundary names, off-page labels) for matching."""
        s = (raw or "").translate(_DASHES).upper()
        s = re.sub(r"[^A-Z0-9#\-\. ]+", " ", s)
        return re.sub(r"\s+", " ", s).strip()
    
    
    _ISA_TAG = re.compile(
        r"^(?:(?P<prefix>[0-9][0-9A-Z]*)-)?"
        r"(?P<letters>[A-Z]{1,6})-"
        r"(?P<number>[A-Z]?\d{1,5})(?P<suffix>[A-Z]{0,2})"
        r"(?:-(?P<ext>[A-Z0-9]{1,4}))?$"
    )
    
    _RELIEF = {"PSV", "PRV", "PSE", "TSV", "TSE", "VRV", "PVRV", "PVSV"}
    
    ROLE_ORDER = (
        "relief_device",
        "hand_switch",
        "final_element",
        "relay",
        "controller",
        "transmitter",
        "switch",
        "element",
        "gauge",
        "indicator",
        "recorder",
        "totalizer",
        "alarm",
        "other",
    )
    
    
    @dataclass
    class IsaTag:
        tag: str
        prefix: Optional[str]
        letters: str
        number: str
        suffix: str
        variable_code: str
        variable: str
        modifiers: list[str] = field(default_factory=list)
        functions: list[str] = field(default_factory=list)
        role: str = "other"
    
        @property
        def loop_key(self) -> str:
            base = f"{self.variable_code[0]}-{self.number}"
            return f"{self.prefix}-{base}" if self.prefix else base
    
    
    def parse_isa_tag(tag: Optional[str]) -> Optional[IsaTag]:
        """Parse an ISA-5.1 style instrument tag. Returns None if it does not fit."""
        t = normalize_tag(tag)
        if not t:
            return None
        m = _ISA_TAG.match(t)
        if not m:
            return None
        letters = m.group("letters")
        first, rest = letters[0], letters[1:]
        if first not in ISA_MEASURED:
            return None
    
        variable_code = first
        modifiers: list[str] = []
        state = None
        # Open/closed state suffix on position and event tags: ZSO, ZSC, ZIO, ZIC, XSO, XSC.
        if first in "XYZ" and len(rest) >= 2 and rest[-1] in "OC" and rest[-2] in "IS":
            state = "open" if rest[-1] == "O" else "closed"
            rest = rest[:-1]
        if letters in _RELIEF:
            functions = ["safety", "valve"]
            role = "relief_device"
            variable_code = first
        else:
            if rest and rest[0] in ISA_VARIABLE_MODIFIERS and len(rest) > 1:
                variable_code = first + rest[0]
                modifiers.append(ISA_VARIABLE_MODIFIERS[rest[0]])
                rest = rest[1:]
            functions = [ISA_FUNCTIONS.get(ch, ch) for ch in rest]
            if state:
                functions.append(state)
            role = _role(rest)
            if first == "H" and role in ("switch", "controller", "indicator", "other"):
                role = "hand_switch"  # operator input (HS, HIC), not a process sensor
    
        variable = ISA_MEASURED[first]
        if modifiers:
            variable = f"{modifiers[0]} {variable}"
        return IsaTag(
            tag=t,
            prefix=m.group("prefix"),
            letters=letters,
            number=m.group("number"),
            suffix=m.group("suffix") or "",
            variable_code=variable_code,
            variable=variable,
            modifiers=modifiers,
            functions=functions,
            role=role,
        )
    
    
    def _role(fn: str) -> str:
        if fn.endswith("V") or fn.endswith("Z"):
            return "final_element"
        if "Y" in fn:
            return "relay"
        if "C" in fn or fn.endswith("K"):
            return "controller"
        if "T" in fn:
            return "transmitter"
        if "S" in fn:
            return "switch"
        if "E" in fn:
            return "element"
        if "G" in fn:
            return "gauge"
        if "I" in fn:
            return "indicator"
        if "R" in fn:
            return "recorder"
        if "Q" in fn:
            return "totalizer"
        if "A" in fn:
            return "alarm"
        return "other"
    
    
    # Roles that sense the process directly and therefore should have a MEASURES edge.
    SENSING_ROLES = {"transmitter", "switch", "element", "gauge"}
    
    # Candidate tags in free text (OCR or vector text). Deliberately loose; used only
    # to suggest items the extraction may have missed.
    TAG_CANDIDATE = re.compile(
        r"\b(?:\d{2,4}[- ]?)?[A-Z]{1,5}[- ]?\d{2,5}[A-Z]?\b"
    )
    
  • skills/pid-to-kg/scripts/pidkg/loops.py15 KBf95338ee95acupstream ↗
    """Control-loop inference.
    
    Loops are inferred in two steps:
    
    1. **Tag grouping (ISA-5.1).** Instruments and actuated components sharing a
       measured-variable letter and loop number (FT-101, FIC-101, FV-101) form a
       loop ``F-101``. An explicit ``loop`` on an item overrides this.
    2. **Signal tracing.** Signal lines refine the groups: a controller's output
       is followed (through relays such as I/P converters) to the final element it
       moves, even when the numbers differ; controller-to-controller signals
       become cascades; and sensors wired into a controller from another loop
       number are added as inputs.
    """
    
    from __future__ import annotations
    
    from collections import defaultdict
    from typing import TYPE_CHECKING, Optional
    
    from .models import Node
    from .ontology import ACTUATED_COMPONENT_KINDS
    
    if TYPE_CHECKING:  # pragma: no cover
        from .build import GraphBuilder
    
    SENSOR_ROLES = {"transmitter", "element", "switch", "gauge", "indicator", "recorder", "totalizer", "alarm"}
    READOUT_ROLES = {"indicator", "recorder", "totalizer", "alarm"}
    PANEL_LOCATIONS = {"main_panel", "behind_panel", "dcs", "plc", "scada", "sis", "computer"}
    
    
    def _member_role(n: Node) -> str:
        """Role within a loop: panel indicators and alarms are read-outs, not sensors."""
        r = _role(n)
        if r in READOUT_ROLES and n.attributes.get("location") in PANEL_LOCATIONS:
            return "readout"
        return r
    
    
    def loop_variable(key: str) -> Optional[str]:
        """Measured variable named by a loop key such as 'F-101', '660-F-0502' or 'S-1@M702'."""
        from .ontology import ISA_MEASURED, ISA_VARIABLE_MODIFIERS
    
        core = key.split("@")[0].split("#")[0]
        parts = core.split("-")
        code = parts[-2] if len(parts) >= 2 else ""
        if not code or code[0] not in ISA_MEASURED:
            return None
        var = ISA_MEASURED[code[0]]
        if len(code) > 1 and code[1] in ISA_VARIABLE_MODIFIERS:
            var = f"{ISA_VARIABLE_MODIFIERS[code[1]]} {var}"
        return var
    
    
    def _role(n: Node) -> str:
        if n.type == "Component":
            if n.kind == "relief_valve" or n.attributes.get("role") == "relief_device":
                return "relief_device"
            return "final_element"
        if n.type == "Equipment":
            return "final_element"
        return n.attributes.get("role", "other")
    
    
    def _is_final(n: Node) -> bool:
        if n.type == "Component":
            return n.kind in ACTUATED_COMPONENT_KINDS or n.attributes.get("role") == "final_element" \
                or n.attributes.get("actuator") not in (None, "manual", "none", "self")
        if n.type == "Equipment":
            return True  # e.g. a pump motor / VFD driven by a speed controller
        return n.attributes.get("role") == "final_element"
    
    
    def derive_loops(b: "GraphBuilder") -> None:
        nodes = b.nodes
        sig_out: dict[str, list[str]] = defaultdict(list)
        sig_in: dict[str, list[str]] = defaultdict(list)
        sig_purpose: dict[tuple[str, str], str] = {}
        for e in b.edges:
            if e.type == "SIGNAL_TO":
                sig_out[e.source].append(e.target)
                sig_in[e.target].append(e.source)
                sig_purpose[(e.source, e.target)] = e.attributes.get("purpose", "unknown")
    
        # 1. Group by loop key.
        groups: dict[str, dict[str, str]] = defaultdict(dict)  # loop key -> {node id: role}
        for n in nodes.values():
            key = n.attributes.get("loop_key")
            if key and n.type in ("Instrument", "Component"):
                groups[key][n.id] = _member_role(n)
    
        # 2. Refine with signals.
        cascades: set[tuple[str, str]] = set()
        loop_of = {nid: key for key, members in groups.items() for nid in members}
        for key, members in list(groups.items()):
            for cid in [m for m, r in members.items() if r == "controller"]:
                for target, via in _trace_outputs(cid, sig_out, nodes):
                    tn = nodes[target]
                    if tn.type == "Instrument" and tn.attributes.get("role") == "controller":
                        other = loop_of.get(target)
                        if other and other != key:
                            cascades.add((key, other))
                        continue
                    if _is_final(tn):
                        members.setdefault(target, "final_element")
                        for r in via:
                            members.setdefault(r, "relay")
                        b._edge(cid, target, "ACTUATES", derived=True,
                                via=[nodes[r].label for r in via] or None)
                for src in sig_in.get(cid, []):
                    sn = nodes[src]
                    if sn.type == "Instrument" and sn.attributes.get("role") in SENSOR_ROLES:
                        members.setdefault(src, "sensor_input")
    
            # Sensors or hand switches wired straight to a final element (interlocks,
            # on/off logic in a PLC that is not drawn as a controller).
            for sid in [m for m, r in members.items() if r in ("switch", "hand_switch", "transmitter", "alarm", "other")]:
                for target, via in _trace_outputs(sid, sig_out, nodes):
                    if _is_final(nodes[target]) and target not in members:
                        members[target] = "final_element"
                        b._edge(sid, target, "ACTUATES", derived=True,
                                via=[nodes[r].label for r in via] or None)
    
        # 3. Emit loop nodes for meaningful groups.
        loop_ids: dict[str, str] = {}
        for key, members in sorted(groups.items()):
            roles = set(members.values())
            if len(members) < 2 and "controller" not in roles:
                continue
            lid = f"LOOP:{key}"
            loop_ids[key] = lid
            loop_type = _loop_type(members, nodes, sig_purpose)
            variable = loop_variable(key) or next((nodes[m].attributes.get("variable") for m in members
                                                   if nodes[m].attributes.get("variable")), None)
            sheets: list[str] = []
            for m in members:
                for s in nodes[m].sheets:
                    if s not in sheets:
                        sheets.append(s)
            sensors = [m for m, r in members.items() if r in SENSOR_ROLES or r == "sensor_input"]
            controllers = [m for m, r in members.items() if r == "controller"]
            finals = [m for m, r in members.items() if r == "final_element"]
            measured_at = _measured_points(sensors, b)
            manipulated_at = _manipulated_points(finals, b)
            node = Node(
                id=lid, type="ControlLoop", label=key, kind=loop_type, sheets=sheets,
                confidence=min(nodes[m].confidence for m in members),
                dexpi_class="InstrumentationLoopFunction",
                attributes={k: v for k, v in dict(
                    variable=variable, loop_type=loop_type, sensors=sensors, controllers=controllers,
                    final_elements=finals, relays=[m for m, r in members.items() if r == "relay"],
                    readouts=[m for m, r in members.items() if r == "readout"],
                    measured_at=measured_at, manipulated_at=manipulated_at).items() if v})
            node.attributes["description"] = _describe(node, b)
            b._upsert(node)
            for m, role in members.items():
                b._edge(m, lid, "MEMBER_OF", derived=True, role=role)
            if loop_type == "interlock":
                for s in sensors:
                    for f in finals:
                        b._edge(s, f, "TRIPS", derived=True, loop=lid)
    
        for master, slave in sorted(cascades):
            if master in loop_ids and slave in loop_ids:
                b._edge(loop_ids[master], loop_ids[slave], "CASCADES_TO", derived=True)
                b.nodes[loop_ids[master]].attributes.setdefault("cascade_slaves", []).append(loop_ids[slave])
                b.nodes[loop_ids[slave]].attributes["cascade_master"] = loop_ids[master]
        for lid in loop_ids.values():
            ln = b.nodes[lid]
            if ln.attributes.get("cascade_slaves") or ln.attributes.get("cascade_master"):
                ln.attributes["description"] = _describe(ln, b)
    
        # 4. QA on loop completeness.
        for key, lid in loop_ids.items():
            ln = b.nodes[lid]
            a = ln.attributes
            if ln.kind == "control":
                if not a.get("final_elements") and not a.get("cascade_slaves"):
                    b.issue("warning", "loop_no_final_element",
                            f"control loop {key} has a controller but no final element", node=lid)
                if not a.get("sensors"):
                    # Common for speed/position controllers that only take an operator setpoint.
                    b.issue("info", "loop_no_sensor",
                            f"control loop {key} has a controller but no sensor", node=lid)
            for c in a.get("controllers", []):
                if not sig_in.get(c) and not sig_out.get(c):
                    b.issue("info", "loop_no_signals",
                            f"controller {c} has no signal lines; loop {key} built from tag numbers only",
                            node=c)
    
    
    def _trace_outputs(start: str, sig_out: dict[str, list[str]], nodes: dict[str, Node],
                       max_depth: int = 4) -> list[tuple[str, list[str]]]:
        """Follow signals from ``start`` through relays; return (endpoint, relays) pairs."""
        results: list[tuple[str, list[str]]] = []
        stack = [(t, []) for t in sig_out.get(start, [])]
        seen = {start}
        while stack:
            nid, via = stack.pop()
            if nid in seen or len(via) > max_depth:
                continue
            seen.add(nid)
            n = nodes[nid]
            if n.type == "Instrument" and n.attributes.get("role") in ("relay", "other") and sig_out.get(nid):
                for t in sig_out[nid]:
                    stack.append((t, via + [nid]))
                continue
            results.append((nid, via))
        return results
    
    
    def _loop_type(members: dict[str, str], nodes: dict[str, Node],
                   sig_purpose: dict[tuple[str, str], str]) -> str:
        roles = set(members.values())
        if any(p == "trip" for (a, _), p in sig_purpose.items() if a in members):
            return "interlock"
        if "controller" in roles:
            return "control"
        variables = {nodes[m].attributes.get("variable") for m in members}
        if "final_element" in roles:
            if "hand" in variables or "hand_switch" in roles:
                return "manual"
            process_switch = any(r in ("switch", "alarm") and nodes[m].attributes.get("variable") not in
                                 ("event/state", "unclassified", "position", "hand")
                                 for m, r in members.items())
            if process_switch:
                return "interlock"
            if roles & {"transmitter", "element", "sensor_input"}:
                # Sensor and final element share a loop but no controller is drawn:
                # the control function lives in a PLC/DCS not shown on the sheet.
                return "control"
            return "on_off"
        return "monitoring"
    
    
    def sensed_points(nid: str, nodes: dict[str, Node], edges: list) -> list[str]:
        """Where an instrument senses the process, as human-readable labels."""
        if nodes[nid].attributes.get("inline"):
            return [point_label(nid, nodes, edges)]
        return [point_label(e.target, nodes, edges) for e in edges
                if e.type == "MEASURES" and e.source == nid]
    
    
    def _measured_points(sensors: list[str], b: "GraphBuilder") -> list[str]:
        out: list[str] = []
        for s in sensors:
            for lbl in sensed_points(s, b.nodes, b.edges):
                if lbl not in out:
                    out.append(lbl)
        return out
    
    
    def _manipulated_points(finals: list[str], b: "GraphBuilder") -> list[str]:
        out: list[str] = []
        for f in finals:
            n = b.nodes[f]
            if n.type == "Equipment":
                continue
            lines = {b.nodes[e.source].attributes.get("line_number") for e in b.edges
                     if e.type == "FLOWS_TO" and e.target == f and b.nodes[e.source].type == "PipeSegment"}
            lines.discard(None)
            if lines:
                out.append(f"{n.label} on {', '.join(sorted(lines))}")
        return out
    
    
    def point_label(nid: str, nodes: dict[str, Node], edges: list) -> str:
        """Human-readable process location.
    
        An item gives its label; a pipe segment or an in-line instrument gives
        ``line (nearest upstream item -> nearest downstream item)``.
        """
        n = nodes[nid]
        inline = n.type == "Instrument" and n.attributes.get("inline")
        if n.type != "PipeSegment" and not inline:
            return n.label
        up = _nearest(nid, nodes, edges, upstream=True)
        down = _nearest(nid, nodes, edges, upstream=False)
        line = n.attributes.get("line_number")
        if not line:
            line = next((nodes[e.source].attributes.get("line_number") for e in edges
                         if e.type == "FLOWS_TO" and e.target == nid
                         and nodes[e.source].type == "PipeSegment"), None)
        where = f"{line or 'line'} ({up or '?'} -> {down or '?'})"
        return f"in-line on {where}" if inline else where
    
    
    def _nearest(start: str, nodes: dict[str, Node], edges: list, upstream: bool,
                 limit: int = 50) -> Optional[str]:
        """Nearest equipment/component/boundary label upstream or downstream of a segment."""
        adj: dict[str, list[str]] = defaultdict(list)
        for e in edges:
            if e.type == "FLOWS_TO":
                if upstream:
                    adj[e.target].append(e.source)
                else:
                    adj[e.source].append(e.target)
        frontier, seen = [start], {start}
        for _ in range(limit):
            nxt = []
            for nid in frontier:
                for other in adj[nid]:
                    if other in seen:
                        continue
                    seen.add(other)
                    if nodes[other].type in ("Equipment", "Boundary", "Component", "Instrument",
                                             "OffPageConnector"):
                        return nodes[other].label
                    nxt.append(other)
            if not nxt:
                return None
            frontier = nxt
        return None
    
    
    def _describe(loop: Node, b: "GraphBuilder") -> str:
        a = loop.attributes
        lbl = lambda ids: ", ".join(b.nodes[i].label for i in ids)  # noqa: E731
        var = a.get("variable") or "process"
        parts = [f"{loop.kind.replace('_', ' ').capitalize()} loop {loop.label} ({var})."]
        if a.get("sensors"):
            where = f" ({'; '.join(a['measured_at'])})" if a.get("measured_at") else ""
            parts.append(f"Sensed by {lbl(a['sensors'])}{where}.")
        if a.get("controllers"):
            locs = {b.nodes[c].attributes.get('location', 'unknown') for c in a["controllers"]}
            parts.append(f"Controlled by {lbl(a['controllers'])} ({', '.join(sorted(locs))}).")
        if a.get("final_elements"):
            placed = {m.split(" on ")[0]: m for m in a.get("manipulated_at", [])}
            acts = [placed.get(b.nodes[f].label, b.nodes[f].label) for f in a["final_elements"]]
            parts.append(f"Acts on {'; '.join(acts)}.")
        if a.get("cascade_slaves"):
            parts.append(f"Cascade master: sets the setpoint of {lbl(a['cascade_slaves'])}.")
        if a.get("cascade_master"):
            parts.append(f"Cascade slave of {b.nodes[a['cascade_master']].label}.")
        return " ".join(parts)
    
  • skills/pid-to-kg/scripts/pidkg/models.py9.5 KB99bc109f63b0upstream ↗
    """Pydantic models for page extractions and the knowledge graph.
    
    Two schemas live here:
    
    * ``PageExtraction`` is what Claude writes for every P&ID sheet after reading
      it visually (``extract/page_NN.json``). It mirrors how an engineer reads a
      drawing: items on the sheet, the pipe runs between them, and the signal
      lines between instruments.
    * ``KnowledgeGraph`` is the canonical output (``graph.json``) produced by
      ``build_kg.py``. It is a typed property graph in node-link form.
    
    JSON Schemas for both are generated into ``references/schema/`` by
    ``python scripts/build_kg.py --write-schemas``.
    """
    
    from __future__ import annotations
    
    from typing import Any, Literal, Optional, Union
    
    from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
    
    from . import EXTRACTION_SCHEMA_VERSION, GRAPH_SCHEMA_VERSION
    
    BBox = list[float]  # normalized [x0, y0, x1, y1] on the upright page, 0..1
    
    
    class _Strict(BaseModel):
        model_config = ConfigDict(extra="forbid", populate_by_name=True)
    
    
    def _check_bbox(v: Optional[BBox]) -> Optional[BBox]:
        if v is None:
            return v
        if len(v) != 4:
            raise ValueError("bbox must be [x0, y0, x1, y1]")
        if any(c < -0.01 or c > 1.01 for c in v):
            raise ValueError("bbox must be normalized to 0..1 of the page")
        x0, y0, x1, y1 = v
        return [min(x0, x1), min(y0, y1), max(x0, x1), max(y0, y1)]
    
    
    # ==========================================================================
    # Page extraction (written by Claude, one file per sheet)
    # ==========================================================================
    
    
    class ProcessRef(_Strict):
        """A point in the process: an item, or a position on a pipe run.
    
        Use ``item`` for equipment, components and in-line instruments. Use
        ``line`` (plus optional ``after``) for a tap or tee on a pipe run, where
        ``after`` is the path item immediately upstream of the tap.
        """
    
        item: Optional[str] = None
        line: Optional[str] = None
        after: Optional[str] = None
    
        @model_validator(mode="after")
        def _one_target(self) -> "ProcessRef":
            if bool(self.item) == bool(self.line):
                raise ValueError("ProcessRef needs exactly one of 'item' or 'line'")
            if self.after and not self.line:
                raise ValueError("'after' is only valid together with 'line'")
            return self
    
    
    Endpoint = Union[str, ProcessRef]
    
    
    class _Item(_Strict):
        id: str = Field(description="Unique within the page. Use the tag when the item has one.")
        tag: Optional[str] = Field(None, description="Tag exactly as printed, e.g. 'FIT-0502'.")
        name: Optional[str] = Field(None, description="Descriptive text printed with the item.")
        status: Optional[Literal["new", "existing", "future", "demolish", "unknown"]] = None
        confidence: float = Field(0.9, ge=0.0, le=1.0)
        bbox: Optional[BBox] = None
        notes: Optional[str] = None
        attributes: dict[str, Any] = Field(default_factory=dict)
    
        @field_validator("bbox")
        @classmethod
        def _bbox_ok(cls, v: Optional[BBox]) -> Optional[BBox]:
            return _check_bbox(v)
    
    
    class Equipment(_Item):
        kind: str = Field(description="Vocabulary in references/ontology.md, e.g. 'tank', 'centrifugal_pump'.")
        part_of: Optional[str] = Field(None, description="Id of the equipment this is mounted on (agitator on a tank, drive on a pump).")
    
    
    class Component(_Item):
        kind: str = Field(description="Vocabulary in references/ontology.md, e.g. 'gate_valve', 'control_valve'.")
        actuator: Optional[
            Literal["manual", "pneumatic", "electric", "hydraulic", "solenoid", "self", "none"]
        ] = None
        fail_position: Optional[Literal["FO", "FC", "FL", "FI"]] = None
        normal_position: Optional[Literal["open", "closed"]] = None
        size: Optional[str] = None
        loop: Optional[str] = Field(None, description="Override the ISA loop this item belongs to.")
    
    
    class Instrument(_Item):
        tag: Optional[str] = Field(None, description="Printed tag. Leave empty for untagged gauges and give 'function'.")
        function: Optional[str] = Field(None, description="ISA letters for an untagged instrument, e.g. 'PI' for a bare pressure gauge.")
        location: str = Field("unknown", description="field | local_panel | main_panel | behind_panel | dcs | plc | scada | sis | computer | unknown")
        measures: Optional[ProcessRef] = Field(None, description="Where the instrument senses the process.")
        loop: Optional[str] = Field(None, description="Override the ISA loop this instrument belongs to.")
    
        @model_validator(mode="after")
        def _tag_or_function(self) -> "Instrument":
            if not self.tag and not self.function:
                raise ValueError("an instrument needs a 'tag', or a 'function' (e.g. 'PI') if untagged")
            return self
    
    
    class Line(_Strict):
        id: str = Field(description="Unique within the page, e.g. 'L1'.")
        line_number: Optional[str] = None
        size: Optional[str] = None
        service: Optional[str] = Field(None, description="Fluid/service code or description.")
        from_: Endpoint = Field(alias="from")
        to: Endpoint
        path: list[str] = Field(default_factory=list, description="Ids of in-line items in flow order.")
        flow_direction: Literal["shown", "inferred"] = "shown"
        status: Optional[Literal["new", "existing", "future", "demolish", "unknown"]] = None
        confidence: float = Field(0.85, ge=0.0, le=1.0)
        notes: Optional[str] = None
    
    
    class Signal(_Strict):
        from_: str = Field(alias="from")
        to: str
        type: str = Field("unknown", description="electrical | pneumatic | software | hydraulic | capillary | wireless | mechanical | unknown")
        purpose: Literal["control", "measurement", "trip", "alarm", "status", "unknown"] = "unknown"
        confidence: float = Field(0.85, ge=0.0, le=1.0)
        notes: Optional[str] = None
    
    
    class OffPageConnector(_Item):
        direction: Literal["in", "out"]
        other_drawing: Optional[str] = Field(None, description="Drawing number the flow continues on/comes from.")
        other_item: Optional[str] = Field(None, description="Item named at the far end, e.g. '660-TK-004'.")
        line_number: Optional[str] = None
    
    
    class Boundary(_Item):
        direction: Literal["source", "sink", "area"] = Field(
            description="source/sink for streams entering or leaving; 'area' for a room or zone that "
                        "area instruments (gas detectors, room temperature) measure")
    
    
    class SheetInfo(_Strict):
        drawing_number: Optional[str] = None
        title: Optional[str] = None
        revision: Optional[str] = None
        drawing_type: Literal["pid", "pfd", "other"] = "pid"
        issue_status: Optional[str] = Field(None, description="As printed: 'as-built', 'issued for construction', 'draft', 'superseded'...")
        rotation: Optional[int] = None
        legend_notes: list[str] = Field(default_factory=list)
    
    
    class PageExtraction(_Strict):
        schema_version: str = EXTRACTION_SCHEMA_VERSION
        page: int = Field(ge=1, description="1-based page number in the input PDF.")
        sheet: SheetInfo = Field(default_factory=SheetInfo)
        equipment: list[Equipment] = Field(default_factory=list)
        components: list[Component] = Field(default_factory=list)
        instruments: list[Instrument] = Field(default_factory=list)
        lines: list[Line] = Field(default_factory=list)
        signals: list[Signal] = Field(default_factory=list)
        off_page_connectors: list[OffPageConnector] = Field(default_factory=list)
        boundaries: list[Boundary] = Field(default_factory=list)
        tiles_reviewed: list[str] = Field(default_factory=list)
        unreadable: list[str] = Field(default_factory=list, description="Things seen but not legible.")
    
        @model_validator(mode="after")
        def _unique_ids(self) -> "PageExtraction":
            seen: dict[str, str] = {}
            groups = {
                "equipment": self.equipment,
                "components": self.components,
                "instruments": self.instruments,
                "off_page_connectors": self.off_page_connectors,
                "boundaries": self.boundaries,
                "lines": self.lines,
            }
            for group, items in groups.items():
                for it in items:
                    if it.id in seen:
                        raise ValueError(f"duplicate id '{it.id}' in {group} and {seen[it.id]}")
                    seen[it.id] = group
            return self
    
    
    # ==========================================================================
    # Knowledge graph (written by build_kg.py)
    # ==========================================================================
    
    
    class Provenance(_Strict):
        sheet: str
        page: int
        bbox: Optional[BBox] = None
        source: str = "vision"
        local_id: Optional[str] = None
    
    
    class Node(_Strict):
        id: str
        type: str
        label: str
        tag: Optional[str] = None
        kind: Optional[str] = None
        name: Optional[str] = None
        status: Optional[str] = None
        dexpi_class: Optional[str] = None
        sheets: list[str] = Field(default_factory=list)
        confidence: float = 1.0
        attributes: dict[str, Any] = Field(default_factory=dict)
        provenance: list[Provenance] = Field(default_factory=list)
    
    
    class Edge(_Strict):
        id: str
        source: str
        target: str
        type: str
        confidence: float = 1.0
        derived: bool = False
        attributes: dict[str, Any] = Field(default_factory=dict)
        sheet: Optional[str] = None
    
    
    class PlantInfo(_Strict):
        id: str
        name: str
        source_file: Optional[str] = None
        sheets: list[str] = Field(default_factory=list)
    
    
    class KnowledgeGraph(_Strict):
        schema_version: str = GRAPH_SCHEMA_VERSION
        generator: str
        generated_at: str
        plant: PlantInfo
        nodes: list[Node]
        edges: list[Edge]
        stats: dict[str, Any] = Field(default_factory=dict)
        issues: list[dict[str, Any]] = Field(default_factory=list)
    
  • skills/pid-to-kg/scripts/pidkg/ocr.py6.5 KB9ba3ecc29a6eupstream ↗
    """Optional OCR (RapidOCR, ONNX runtime; pip-installable, no system binaries).
    
    OCR is used for two things only:
    
    * detecting page orientation, and
    * producing *hints*: candidate tags with positions that QA compares against
      Claude's visual extraction to flag possible misses.
    
    It never writes to the graph directly.
    """
    
    from __future__ import annotations
    
    import re
    from typing import Optional
    
    from .isa import normalize_tag, parse_isa_tag
    
    _engine = None
    
    EQUIPMENT_LIKE = re.compile(r"^(\d{2,4}-)?[A-Z]{2,4}-\d{2,5}[A-Z]?$")
    _LETTERS = re.compile(r"^[A-Z]{1,5}$")
    _DIGITS = re.compile(r"^\d{2,5}[A-Z]?$")
    
    
    def available() -> bool:
        try:
            import rapidocr_onnxruntime  # noqa: F401
            return True
        except ImportError:
            return False
    
    
    def _get_engine():
        global _engine
        if _engine is None:
            from rapidocr_onnxruntime import RapidOCR
    
            _engine = RapidOCR()
        return _engine
    
    
    def ocr_image(img) -> list[dict]:
        """OCR a PIL image. Returns [{'text', 'bbox' (pixels x0,y0,x1,y1), 'conf'}]."""
        import numpy as np
    
        arr = np.array(img.convert("RGB"))
        result, _ = _get_engine()(arr, use_cls=False)
        out = []
        for box, text, conf in result or []:
            xs = [p[0] for p in box]
            ys = [p[1] for p in box]
            out.append({"text": str(text).strip(), "bbox": [min(xs), min(ys), max(xs), max(ys)],
                        "conf": float(conf)})
        return out
    
    
    def _upright_boxes(results: list[dict]) -> list[dict]:
        """Confident, horizontal, alphanumeric OCR boxes.
    
        Only horizontal boxes count: the recognizer silently turns tall crops, so
        sideways text would otherwise read in every orientation.
        """
        keep = []
        for r in results:
            t = re.sub(r"[^A-Za-z0-9]", "", r["text"])
            x0, y0, x1, y1 = r["bbox"]
            if (x1 - x0) > 1.2 * (y1 - y0) and len(t) >= 2 and r["conf"] >= 0.6:
                keep.append(r)
        return keep
    
    
    def detect_rotation(img, rotations=(0, 90, 180, 270)) -> tuple[int, dict[int, float], Optional[float]]:
        """Find the clockwise rotation that makes the text upright, and measure text size.
    
        Returns (rotation, score per rotation, 25th-percentile text box height in
        pixels of ``img`` at that rotation, or None if no text was found).
        """
        scores: dict[int, float] = {}
        heights: dict[int, list[float]] = {}
        for rot in rotations:
            # PIL rotates counter-clockwise; negate to match the clockwise convention.
            test = img.rotate(-rot, expand=True) if rot else img
            boxes = _upright_boxes(ocr_image(test))
            scores[rot] = round(sum(r["conf"] * len(re.sub(r"[^A-Za-z0-9]", "", r["text"]))
                                    for r in boxes), 1)
            heights[rot] = sorted(r["bbox"][3] - r["bbox"][1] for r in boxes)
        best = max(scores, key=scores.get)
        hs = heights[best]
        p25 = hs[len(hs) // 4] if len(hs) >= 8 else None
        return best, scores, p25
    
    
    def join_bubbles(runs: list[dict]) -> list[dict]:
        """Join ISA bubble text split over two lines ('FIT' above '0502') into one run.
    
        ``runs`` use normalized page bboxes. Joined runs are added; originals kept.
        """
        letters = [r for r in runs if _LETTERS.match(r["text"].replace(" ", ""))]
        digits = [r for r in runs if _DIGITS.match(r["text"].replace(" ", ""))]
        joined = []
        for a in letters:
            ax0, ay0, ax1, ay1 = a["bbox"]
            h = ay1 - ay0
            acx = (ax0 + ax1) / 2
            best, best_gap = None, None
            for d in digits:
                dx0, dy0, dx1, dy1 = d["bbox"]
                dcx = (dx0 + dx1) / 2
                gap = dy0 - ay1
                width = max(ax1 - ax0, dx1 - dx0)
                if -0.5 * h <= gap <= 1.5 * h and abs(acx - dcx) <= 0.6 * width:
                    if best_gap is None or gap < best_gap:
                        best, best_gap = d, gap
            if best is not None:
                text = f"{a['text'].replace(' ', '')}-{best['text'].replace(' ', '')}"
                bb = [min(ax0, best["bbox"][0]), ay0, max(ax1, best["bbox"][2]), best["bbox"][3]]
                joined.append({"text": text, "bbox": [round(v, 4) for v in bb],
                               "conf": round(min(a.get("conf", 1), best.get("conf", 1)), 3),
                               "joined": True})
        return runs + joined
    
    
    def tag_candidates(runs: list[dict]) -> list[dict]:
        """Pick tag-like strings from text runs, de-duplicated by normalized tag."""
        seen: dict[str, dict] = {}
        for r in runs:
            tokens = [t for t in re.split(r"[\s,;()]+", r["text"]) if t]
            # Tags are often printed with a space ('FIT 0502'): also try adjacent pairs.
            pairs = [f"{a}-{b}" for a, b in zip(tokens, tokens[1:])
                     if _LETTERS.match(a) and _DIGITS.match(b)]
            for token in tokens + pairs:
                t = normalize_tag(token)
                if not t or len(t) < 4:
                    continue
                isa = parse_isa_tag(t)
                eq = bool(EQUIPMENT_LIKE.match(t))
                if not isa and not eq:
                    continue
                if t not in seen or r.get("conf", 0) > seen[t]["conf"]:
                    seen[t] = {"text": t, "bbox": r["bbox"], "conf": r.get("conf", 1.0),
                               "isa": bool(isa), "equipment_like": eq}
        return sorted(seen.values(), key=lambda c: (c["bbox"][1], c["bbox"][0]))
    
    
    def dedupe_runs(runs: list[dict], tol: float = 0.004) -> list[dict]:
        """Drop runs seen twice in overlapping tiles (same text, nearly same position)."""
        kept: list[dict] = []
        for r in sorted(runs, key=lambda r: -r.get("conf", 0)):
            cx = (r["bbox"][0] + r["bbox"][2]) / 2
            cy = (r["bbox"][1] + r["bbox"][3]) / 2
            dup = False
            for k in kept:
                if k["text"] != r["text"]:
                    continue
                kx = (k["bbox"][0] + k["bbox"][2]) / 2
                ky = (k["bbox"][1] + k["bbox"][3]) / 2
                if abs(kx - cx) < tol and abs(ky - cy) < tol:
                    dup = True
                    break
            if not dup:
                kept.append(r)
        return kept
    
    
    def to_norm(bbox_px: list[float], tile_px: list[int], page_w: int, page_h: int,
                tile_scale: float = 1.0) -> list[float]:
        """Convert a bbox in tile pixels to normalized page coordinates."""
        x0, y0, x1, y1 = (v / tile_scale for v in bbox_px)
        return [round((tile_px[0] + x0) / page_w, 4), round((tile_px[1] + y0) / page_h, 4),
                round((tile_px[0] + x1) / page_w, 4), round((tile_px[1] + y1) / page_h, 4)]
    
    
    def maybe_import_error() -> Optional[str]:
        if available():
            return None
        return ("RapidOCR is not installed; OCR hints and automatic rotation are skipped. "
                "Install with: pip install rapidocr-onnxruntime")
    
  • skills/pid-to-kg/scripts/pidkg/ontology.py7.8 KBef58d2c49585upstream ↗
    """Controlled vocabularies for the P&ID knowledge graph.
    
    Vocabularies are deliberately permissive: an unknown ``kind`` is accepted and
    reported by QA rather than rejected, because real drawings always contain
    something the vocabulary did not anticipate.
    
    ``DEXPI_CLASS`` maps each kind to the closest DEXPI (ISO 15926-based P&ID
    exchange standard) class name. The mapping is informative: it helps users
    align the graph with DEXPI tooling but is not a certified DEXPI export.
    """
    
    from __future__ import annotations
    
    # --------------------------------------------------------------------------
    # Node and edge types
    # --------------------------------------------------------------------------
    
    NODE_TYPES = (
        "Plant",
        "Sheet",
        "Equipment",
        "PipeSegment",
        "Component",
        "Instrument",
        "ControlLoop",
        "OffPageConnector",
        "Boundary",
    )
    
    EDGE_TYPES = (
        # Physical process connectivity (directed in the direction of flow).
        "FLOWS_TO",
        # Derived equipment-to-equipment flow, collapsing pipes and fittings.
        "FEEDS",
        # Instrument -> the equipment, pipe segment or component it senses.
        "MEASURES",
        # Instrument/control signal (directed from sender to receiver).
        "SIGNAL_TO",
        # Controller or logic -> the final element it moves (valve, VFD/motor).
        "ACTUATES",
        # Instrument/component -> ControlLoop, with a ``role`` attribute.
        "MEMBER_OF",
        # ControlLoop -> ControlLoop (master -> slave in a cascade).
        "CASCADES_TO",
        # Safety/interlock action: initiator -> the item it trips.
        "TRIPS",
        # Sheet -> Plant; mounted equipment -> its parent (agitator -> tank).
        "PART_OF",
    )
    
    # Edge types followed when tracing upstream/downstream process flow.
    FLOW_EDGE_TYPES = ("FLOWS_TO",)
    
    # --------------------------------------------------------------------------
    # Equipment
    # --------------------------------------------------------------------------
    
    EQUIPMENT_DEXPI = {
        "tank": "Tank",
        "vessel": "Vessel",
        "pressure_vessel": "PressureVessel",
        "pump": "Pump",
        "centrifugal_pump": "CentrifugalPump",
        "positive_displacement_pump": "ReciprocatingPump",
        "metering_pump": "ReciprocatingPump",
        "slurry_pump": "CentrifugalPump",
        "sump_pump": "CentrifugalPump",
        "compressor": "Compressor",
        "blower": "Blower",
        "fan": "Fan",
        "heat_exchanger": "HeatExchanger",
        "cooler": "HeatExchanger",
        "heater": "HeatExchanger",
        "boiler": "Boiler",
        "furnace": "Furnace",
        "column": "Column",
        "reactor": "Vessel",
        "agitated_tank": "Tank",
        "agitator": "Agitator",
        "mixer": "Mixer",
        "static_mixer": "StaticMixer",
        "filter": "Filter",
        "filter_press": "Filter",
        "membrane_filter": "Filter",
        "strainer_unit": "Filter",
        "screen": "Sieve",
        "cyclone": "CycloneSeparator",
        "hydrocyclone": "CycloneSeparator",
        "separator": "Separator",
        "clarifier": "Separator",
        "thickener": "Separator",
        "flotation_cell": "Separator",
        "centrifuge": "Centrifuge",
        "dryer": "Dryer",
        "mill": "Mill",
        "crusher": "Crusher",
        "conveyor": "Conveyor",
        "feeder": "Feeder",
        "hopper": "Silo",
        "bin": "Silo",
        "silo": "Silo",
        "sump": "Tank",
        "pond": "Tank",
        "box": "Tank",
        "electrowinning_cell": "Vessel",
        "kiln": "Furnace",
        "scrubber": "Column",
        "ion_exchange": "Column",
        "uv_unit": "Equipment",
        "sampler": "Equipment",
        "package_unit": "Equipment",
        "motor": "ElectricMotor",
        "other": "Equipment",
    }
    EQUIPMENT_KINDS = tuple(EQUIPMENT_DEXPI)
    
    # --------------------------------------------------------------------------
    # In-line piping components
    # --------------------------------------------------------------------------
    
    COMPONENT_DEXPI = {
        "manual_valve": "OperatedValve",
        "gate_valve": "GateValve",
        "globe_valve": "GlobeValve",
        "ball_valve": "BallValve",
        "butterfly_valve": "ButterflyValve",
        "plug_valve": "PlugValve",
        "knife_gate_valve": "GateValve",
        "diaphragm_valve": "DiaphragmValve",
        "pinch_valve": "OperatedValve",
        "needle_valve": "NeedleValve",
        "three_way_valve": "OperatedValve",
        "check_valve": "CheckValve",
        "backflow_preventer": "CheckValve",
        "control_valve": "OperatedValve",
        "on_off_valve": "OperatedValve",
        "solenoid_valve": "OperatedValve",
        "motor_operated_valve": "OperatedValve",
        "pressure_regulator": "OperatedValve",
        "relief_valve": "SafetyValveOrFitting",
        "rupture_disc": "RuptureDisc",
        "vacuum_breaker": "SafetyValveOrFitting",
        "strainer": "Strainer",
        "filter": "Filter",
        "reducer": "PipeReducer",
        "blind": "Blind",
        "spectacle_blind": "SpectacleBlind",
        "flexible_hose": "Hose",
        "expansion_joint": "Compensator",
        "sight_glass": "SightGlass",
        "steam_trap": "SteamTrap",
        "flame_arrestor": "FlameArrestor",
        "sample_point": "PipingComponent",
        "drain": "PipingComponent",
        "vent": "PipingComponent",
        "quick_connect": "PipingComponent",
        "tee": "PipeTee",
        "other": "PipingComponent",
    }
    COMPONENT_KINDS = tuple(COMPONENT_DEXPI)
    
    # Components that can be the final element of a control or trip loop.
    ACTUATED_COMPONENT_KINDS = {
        "control_valve",
        "on_off_valve",
        "solenoid_valve",
        "motor_operated_valve",
        "three_way_valve",
    }
    
    # --------------------------------------------------------------------------
    # Instruments
    # --------------------------------------------------------------------------
    
    INSTRUMENT_LOCATIONS = (
        "field",          # plain circle
        "local_panel",    # circle with line, dashed or in a local/field panel
        "main_panel",     # circle with solid line: main control room panel
        "behind_panel",   # circle with dashed line
        "dcs",            # circle in square: shared display / DCS
        "plc",            # diamond in square
        "scada",
        "sis",            # safety instrumented system
        "computer",       # hexagon
        "unknown",
    )
    
    SIGNAL_TYPES = (
        "electrical",   # dashed line
        "pneumatic",    # line with double cross-hatches
        "software",     # line with circles / data link
        "hydraulic",
        "capillary",    # line with x's
        "wireless",     # zig-zag / wave
        "mechanical",
        "unknown",
    )
    
    STATUSES = ("new", "existing", "future", "demolish", "unknown")
    
    # ISA-5.1 first letters (measured or initiating variable).
    ISA_MEASURED = {
        "A": "analysis",
        "B": "burner/flame",
        "C": "conductivity",      # user's choice; conductivity is the most common
        "D": "density",           # user's choice
        "E": "voltage",
        "F": "flow",
        "G": "gauging/position",  # user's choice
        "H": "hand",
        "I": "current",
        "J": "power",
        "K": "time",
        "L": "level",
        "M": "moisture",          # user's choice
        "N": "user-defined",
        "O": "user-defined",
        "P": "pressure",
        "Q": "quantity",
        "R": "radiation",
        "S": "speed/frequency",
        "T": "temperature",
        "U": "multivariable",
        "V": "vibration",
        "W": "weight/force",
        "X": "unclassified",
        "Y": "event/state",
        "Z": "position",
    }
    
    # Variable modifiers that may follow the first letter.
    ISA_VARIABLE_MODIFIERS = {
        "D": "differential",
        "F": "ratio",
        "J": "scan",
        "K": "rate of change",
        "Q": "integrate/totalize",
    }
    
    # Succeeding letters (readout/passive or output functions).
    ISA_FUNCTIONS = {
        "A": "alarm",
        "B": "user's choice",
        "C": "control",
        "D": "differential",
        "E": "primary element",
        "G": "glass/gauge",
        "H": "high",
        "I": "indicate",
        "K": "control station",
        "L": "light/low",
        "M": "middle",
        "N": "user's choice",
        "O": "orifice/restriction",
        "P": "test point",
        "Q": "totalize",
        "R": "record",
        "S": "switch",
        "T": "transmit",
        "U": "multifunction",
        "V": "valve/damper",
        "W": "well",
        "X": "unclassified",
        "Y": "relay/compute/convert",
        "Z": "driver/actuator",
    }
    
  • skills/pid-to-kg/scripts/pidkg/pdf.py8.3 KB305445be2e91upstream ↗
    """PDF inspection and rendering (pypdfium2: Apache-2.0/BSD, no system dependencies)."""
    
    from __future__ import annotations
    
    import math
    from dataclasses import asdict, dataclass, field
    from typing import Optional
    
    import pypdfium2 as pdfium
    import pypdfium2.raw as pdfium_c
    
    # Page classes and how each is extracted.
    ROUTES = {
        "vector_text": "Vector drawing with a text layer: render from vectors, take exact text and "
                       "positions from the text layer as hints, then read visually.",
        "vector_outlined": "Vector drawing whose text was converted to outlines (no text layer): "
                           "render from vectors at high resolution, OCR the render for hints, then "
                           "read visually.",
        "raster": "Scanned image: render at the scan's native resolution, upscale and enhance "
                  "contrast, OCR for hints, then read visually.",
        "raster_with_text": "Scanned image with an OCR text layer: as raster, and use the existing "
                            "text layer as (untrusted) hints.",
        "hybrid": "Mix of vector content and large images: render from vectors at high "
                  "resolution and use both text layer and OCR hints.",
        "blank": "No meaningful content.",
    }
    
    MIN_TEXT_CHARS = 50
    MIN_VECTOR_PATHS = 200
    BIG_IMAGE_COVERAGE = 0.5
    
    
    @dataclass
    class PageProbe:
        page: int
        width_pt: float
        height_pt: float
        pdf_rotation: int
        text_chars: int
        path_objects: int
        text_objects: int
        image_objects: int
        max_image_coverage: float
        image_px: Optional[list[int]]
        effective_image_dpi: Optional[float]
        page_class: str
        route: str = ""
        rotation: Optional[int] = None
        notes: list[str] = field(default_factory=list)
    
        def to_dict(self) -> dict:
            return asdict(self)
    
    
    def probe_pdf(path: str) -> list[PageProbe]:
        pdf = pdfium.PdfDocument(path)
        try:
            return [probe_page(pdf, i) for i in range(len(pdf))]
        finally:
            pdf.close()
    
    
    def probe_page(pdf: "pdfium.PdfDocument", index: int) -> PageProbe:
        page = pdf[index]
        w, h = page.get_size()
        area = max(w * h, 1.0)
        tp = page.get_textpage()
        text = tp.get_text_range() or ""
        chars = sum(1 for ch in text if not ch.isspace())
        counts = {pdfium_c.FPDF_PAGEOBJ_PATH: 0, pdfium_c.FPDF_PAGEOBJ_TEXT: 0, pdfium_c.FPDF_PAGEOBJ_IMAGE: 0}
        best_cov, best_px, best_dpi = 0.0, None, None
        for obj in page.get_objects(max_depth=8):
            if obj.type in counts:
                counts[obj.type] += 1
            if obj.type == pdfium_c.FPDF_PAGEOBJ_IMAGE:
                l, b, r, t = obj.get_bounds()
                cov = max(0.0, (r - l) * (t - b)) / area
                if cov > best_cov:
                    best_cov = cov
                    try:
                        px = list(obj.get_px_size())
                        best_px = px
                        best_dpi = round(72.0 * max(px) / max(r - l, t - b, 1e-6), 1)
                    except Exception:  # pragma: no cover - exotic image types
                        best_px, best_dpi = None, None
        paths = counts[pdfium_c.FPDF_PAGEOBJ_PATH]
        has_text = chars >= MIN_TEXT_CHARS
        has_vec = paths >= MIN_VECTOR_PATHS
        big_img = best_cov >= BIG_IMAGE_COVERAGE
        if big_img and not has_vec:
            cls = "raster_with_text" if has_text else "raster"
        elif big_img and has_vec:
            cls = "hybrid"
        elif has_text:
            cls = "vector_text"
        elif has_vec:
            cls = "vector_outlined"
        elif paths or counts[pdfium_c.FPDF_PAGEOBJ_IMAGE]:
            cls = "vector_outlined" if paths else "raster"
        else:
            cls = "blank"
        probe = PageProbe(
            page=index + 1, width_pt=round(w, 1), height_pt=round(h, 1),
            pdf_rotation=page.get_rotation(), text_chars=chars, path_objects=paths,
            text_objects=counts[pdfium_c.FPDF_PAGEOBJ_TEXT],
            image_objects=counts[pdfium_c.FPDF_PAGEOBJ_IMAGE],
            max_image_coverage=round(best_cov, 3), image_px=best_px, effective_image_dpi=best_dpi,
            page_class=cls, route=ROUTES[cls])
        if cls.startswith("raster") and best_dpi and best_dpi < 200:
            probe.notes.append(f"low scan resolution (~{best_dpi:.0f} dpi): small text may be "
                               "illegible; record unreadable items in 'unreadable'")
        if max(w, h) < 1000:
            probe.notes.append("page is smaller than a typical drawing sheet; the drawing was "
                               "probably reduced, so render at high resolution")
        return probe
    
    
    def render_page(pdf_path: str, page_no: int, page_class: str, rotation: int = 0,
                    vector_long_edge: int = 5000, raster_long_edge: int = 4000,
                    max_upscale: float = 2.5):
        """Render a page to a PIL image following the route for its class."""
        from PIL import Image, ImageOps
    
        pdf = pdfium.PdfDocument(pdf_path)
        try:
            page = pdf[page_no - 1]
            w, h = page.get_size()
            if page_class.startswith("raster"):
                probe = probe_page(pdf, page_no - 1)
                native = (probe.effective_image_dpi or 200) / 72.0
                img = page.render(scale=native, rotation=rotation, grayscale=True).to_pil()
                long_edge = max(img.size)
                factor = min(max_upscale, max(1.0, raster_long_edge / long_edge))
                if factor > 1.0:
                    img = img.resize((round(img.width * factor), round(img.height * factor)),
                                     Image.LANCZOS)
                img = ImageOps.autocontrast(img.convert("L"), cutoff=0.5)
                return img
            scale = vector_long_edge / max(w, h)
            return page.render(scale=scale, rotation=rotation).to_pil().convert("RGB")
        finally:
            pdf.close()
    
    
    def tile_grid(width: int, height: int, tile: int = 1400, overlap: int = 200) -> list[dict]:
        """Evenly spaced overlapping tiles covering the image."""
        def spans(total: int) -> list[tuple[int, int]]:
            if total <= tile:
                return [(0, total)]
            n = math.ceil((total - overlap) / (tile - overlap))
            step = (total - tile) / (n - 1)
            return [(round(i * step), round(i * step) + tile) for i in range(n)]
    
        tiles = []
        for r, (y0, y1) in enumerate(spans(height), 1):
            for c, (x0, x1) in enumerate(spans(width), 1):
                tiles.append({"name": f"r{r}c{c}", "row": r, "col": c, "px": [x0, y0, x1, y1],
                              "norm": [round(x0 / width, 4), round(y0 / height, 4),
                                       round(x1 / width, 4), round(y1 / height, 4)]})
        return tiles
    
    
    def ink_fraction(img, threshold: int = 200) -> float:
        """Fraction of pixels darker than ``threshold``, ignoring frame lines.
    
        Rows and columns that are dark across most of the tile (the drawing
        border, title-block rules) are not content, so they are excluded; a tile
        holding only a border line then counts as blank.
        """
        g = img.convert("L")
        try:
            import numpy as np
        except ImportError:  # pragma: no cover - numpy ships with the OCR extra
            hist = g.histogram()
            return sum(hist[:threshold]) / max(1, g.width * g.height)
        dark = np.asarray(g) < threshold
        rows = dark.mean(axis=1) > 0.6
        cols = dark.mean(axis=0) > 0.6
        dark[rows, :] = False
        dark[:, cols] = False
        return float(dark.mean())
    
    
    def rotate_norm_bbox(b: list[float], rotation: int) -> list[float]:
        """Map a normalized bbox on the unrotated page onto the page rotated clockwise."""
        x0, y0, x1, y1 = b
        pts = [(x0, y0), (x1, y1)]
        if rotation == 90:
            pts = [(1 - y, x) for x, y in pts]
        elif rotation == 180:
            pts = [(1 - x, 1 - y) for x, y in pts]
        elif rotation == 270:
            pts = [(y, 1 - x) for x, y in pts]
        (a, b2), (c, d) = pts
        return [round(min(a, c), 4), round(min(b2, d), 4), round(max(a, c), 4), round(max(b2, d), 4)]
    
    
    def text_runs(pdf_path: str, page_no: int, rotation: int = 0) -> list[dict]:
        """Text-layer runs with normalized bboxes on the (rotated) page."""
        pdf = pdfium.PdfDocument(pdf_path)
        try:
            page = pdf[page_no - 1]
            w, h = page.get_size()
            tp = page.get_textpage()
            runs = []
            for i in range(tp.count_rects()):
                l, b, r, t = tp.get_rect(i)
                txt = (tp.get_text_bounded(l, b, r, t) or "").strip()
                if not txt:
                    continue
                norm = [l / w, 1 - t / h, r / w, 1 - b / h]
                runs.append({"text": txt, "bbox": rotate_norm_bbox(norm, rotation), "conf": 1.0})
            return runs
        finally:
            pdf.close()
    
  • skills/pid-to-kg/scripts/pidkg/qa.py12 KB24bf8fa4eaa2upstream ↗
    """Quality checks on a built graph, and the QA report Claude uses for its second pass."""
    
    from __future__ import annotations
    
    import json
    import re
    from collections import defaultdict
    from pathlib import Path
    from typing import Any, Optional
    
    from .isa import SENSING_ROLES, normalize_tag, parse_isa_tag
    from .models import KnowledgeGraph
    
    LOW_CONFIDENCE = 0.6
    _MONTHS = re.compile(r"^(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)[A-Z]*-\d")
    
    
    def run_qa(kg: KnowledgeGraph, work_dir: Optional[Path] = None) -> list[dict[str, Any]]:
        """Return graph-level issues (build issues are already in ``kg.issues``)."""
        issues: list[dict[str, Any]] = []
        nodes = {n.id: n for n in kg.nodes}
        flow_deg: dict[str, int] = defaultdict(int)
        any_deg: dict[str, int] = defaultdict(int)
        measured: set[str] = set()
        for e in kg.edges:
            any_deg[e.source] += 1
            any_deg[e.target] += 1
            if e.type == "FLOWS_TO":
                flow_deg[e.source] += 1
                flow_deg[e.target] += 1
            elif e.type == "MEASURES":
                measured.add(e.source)
    
        def add(sev: str, code: str, msg: str, node=None) -> None:
            n = nodes.get(node) if node else None
            bbox = next((p.bbox for p in n.provenance if p.bbox), None) if n else None
            sheet = n.sheets[0] if n and n.sheets else None
            page = n.provenance[0].page if n and n.provenance else None
            issues.append({k: v for k, v in dict(severity=sev, code=code, message=msg, node=node,
                                                 sheet=sheet, page=page, bbox=bbox).items() if v is not None})
    
        for n in kg.nodes:
            if n.type == "Equipment" and flow_deg[n.id] == 0 and not n.attributes.get("part_of"):
                add("warning", "equipment_not_connected",
                    f"equipment {n.label} has no process connections", n.id)
            elif n.type == "Component" and flow_deg[n.id] == 0:
                add("warning", "component_not_connected",
                    f"component {n.label} is not on any line (add it to a line 'path')", n.id)
            elif n.type == "Instrument":
                role = n.attributes.get("role")
                loc = n.attributes.get("location", "unknown")
                if role in SENSING_ROLES and loc in ("field", "unknown") and n.id not in measured \
                        and not n.attributes.get("inline"):
                    add("warning", "instrument_not_located",
                        f"{n.label} ({role}) senses the process but has no 'measures' location", n.id)
                if any_deg[n.id] == 0:
                    add("info", "isolated_instrument", f"{n.label} has no connections at all", n.id)
            if n.type not in ("Plant", "Sheet") and n.confidence < LOW_CONFIDENCE:
                add("info", "low_confidence", f"{n.label} was read with confidence {n.confidence:.2f}", n.id)
    
        # Tags that differ only in zero padding ('660-TK-06' vs '660-TK-006').
        padded: dict[str, list[str]] = defaultdict(list)
        for n in kg.nodes:
            if n.tag and n.type == "Equipment":  # instrument loop numbers legitimately vary
                padded[re.sub(r"(?<![0-9])0+(?=[0-9])", "", n.tag.split("@")[0].split("#")[0])].append(n.id)
        for ids in padded.values():
            tags = sorted({nodes[i].tag for i in ids})
            if len(tags) > 1:
                add("warning", "near_duplicate_tags",
                    f"tags {', '.join(tags)} differ only in zero padding; probably one item "
                    "drawn with inconsistent tags (check the sheets)", ids[0])
    
        issues.extend(_revision_overlap(kg))
    
    
        inferred = sum(1 for e in kg.edges if e.type == "FLOWS_TO"
                       and e.attributes.get("flow_direction") == "inferred")
        if inferred:
            issues.append({"severity": "info", "code": "inferred_flow_direction",
                           "message": f"{inferred} flow edges have inferred (not drawn) direction"})
    
        if work_dir:
            issues.extend(_coverage_issues(kg, Path(work_dir)))
            issues.extend(_hint_issues(kg, Path(work_dir)))
        return issues
    
    
    def _revision_overlap(kg: KnowledgeGraph) -> list[dict[str, Any]]:
        """Two sheets sharing most of their tags are probably two revisions of one drawing."""
        tags: dict[str, set[str]] = defaultdict(set)
        for n in kg.nodes:
            if n.tag and n.type in ("Equipment", "Instrument", "Component"):
                for s in {p.sheet for p in n.provenance}:
                    tags[s].add(n.tag)
        out = []
        sheets = sorted(t for t in tags if len(tags[t]) >= 5)
        for i, a in enumerate(sheets):
            for b in sheets[i + 1:]:
                shared = tags[a] & tags[b]
                frac = len(shared) / min(len(tags[a]), len(tags[b]))
                if frac >= 0.5:
                    out.append({"severity": "warning", "code": "possible_duplicate_revision", "sheet": b,
                                "message": f"{a} and {b} share {len(shared)} tags ({frac:.0%} of the smaller sheet); "
                                           "if they are two revisions of the same drawing, keep only the current one "
                                           "in this graph and ask the user what to do with the other"})
        return out
    
    
    def _coverage_issues(kg: KnowledgeGraph, work: Path) -> list[dict[str, Any]]:
        out = []
        for n in kg.nodes:
            if n.type != "Sheet":
                continue
            page = n.attributes.get("page")
            tiles_file = work / "pages" / f"page_{page:02d}" / "tiles.json"
            if not tiles_file.exists():
                continue
            tiles = json.loads(tiles_file.read_text(encoding="utf-8"))
            expected = {t["name"] for t in tiles["tiles"] if not t.get("blank")}
            reviewed = set(n.attributes.get("tiles_reviewed", []))
            missing = sorted(expected - reviewed)
            if missing:
                out.append({"severity": "warning", "code": "tiles_not_reviewed", "sheet": n.id,
                            "page": page, "message": f"{len(missing)} non-blank tiles not listed in "
                            f"tiles_reviewed: {', '.join(missing)}"})
        return out
    
    
    def _hint_issues(kg: KnowledgeGraph, work: Path) -> list[dict[str, Any]]:
        """Compare tags found by OCR / the text layer with tags in the graph."""
        out = []
        known = {n.tag for n in kg.nodes if n.tag} | {n.id for n in kg.nodes}
        known_core = {_core(t) for t in known if t}
        # Drawing numbers (title blocks, off-page flags) look like tags but are not items.
        drawings = {_core(normalize_tag(v)) for n in kg.nodes for v in
                    (n.attributes.get("drawing_number"), n.attributes.get("other_drawing")) if v}
        # Text already captured in names/attributes (off-page 'other_item', tie-in labels...).
        blob = _core(normalize_tag(" ".join(
            f"{n.name or ''} {n.label} {json.dumps(n.attributes)}" for n in kg.nodes)) or "")
        by_number: dict[str, set[str]] = defaultdict(set)
        for t in known:
            p = parse_isa_tag(t) if t else None
            if p:
                by_number[p.number].add(p.letters)
        for n in kg.nodes:
            if n.type != "Sheet":
                continue
            page = n.attributes.get("page")
            hint_file = work / "hints" / f"page_{page:02d}.json"
            if not hint_file.exists():
                continue
            hints = json.loads(hint_file.read_text(encoding="utf-8"))
            missed = []
            for cand in hints.get("tag_candidates", []):
                t = normalize_tag(cand["text"])
                if not t or t in known:
                    continue
                core = _core(t)
                if core in known_core or core in drawings or _MONTHS.match(t):
                    continue
                # OCR fragments cut at a tile edge ('0-BX-002', '660-TK-00').
                if any(core in k or k in core for k in known_core if len(k) >= 4) or core in blob:
                    continue
                isa = parse_isa_tag(t)
                if not isa and not cand.get("equipment_like"):
                    continue
                # Likely OCR misread of a known tag: same number, letters one edit apart.
                if isa and any(_close(isa.letters, l) for l in by_number.get(isa.number, ())):
                    continue
                missed.append(cand)
            if missed:
                listed = ", ".join(f"{c['text']}" for c in missed[:40])
                out.append({"severity": "warning", "code": "possible_missed_tags", "sheet": n.id,
                            "page": page, "count": len(missed),
                            "message": f"{len(missed)} tags found by {hints.get('source', 'hints')} are not "
                                       f"in the graph: {listed}",
                            "candidates": missed})
        return out
    
    
    def _edit1(a: str, b: str) -> bool:
        """True if a and b differ by at most one substitution, insertion or deletion."""
        if abs(len(a) - len(b)) > 1:
            return False
        if len(a) == len(b):
            return sum(x != y for x, y in zip(a, b)) <= 1
        if len(a) > len(b):
            a, b = b, a
        return any(a == b[:i] + b[i + 1:] for i in range(len(b)))
    
    
    def _core(tag: str) -> str:
        """Tag without separators, so 'FIT-0502' and 'FIT0502' match."""
        return tag.replace("-", "")
    
    
    def tiles_for_bbox(work: Path, page: int, bbox: Optional[list[float]]) -> list[str]:
        if not bbox:
            return []
        f = work / "pages" / f"page_{page:02d}" / "tiles.json"
        if not f.exists():
            return []
        tiles = json.loads(f.read_text(encoding="utf-8"))["tiles"]
        x0, y0, x1, y1 = bbox
        cx, cy = (x0 + x1) / 2, (y0 + y1) / 2
        hits = [t["name"] for t in tiles
                if t["norm"][0] <= cx <= t["norm"][2] and t["norm"][1] <= cy <= t["norm"][3]]
        return hits[:2]
    
    
    def write_report(kg: KnowledgeGraph, issues: list[dict[str, Any]], path: Path,
                     work_dir: Optional[Path] = None) -> None:
        sev_order = {"error": 0, "warning": 1, "info": 2}
        nodes = {n.id: n for n in kg.nodes}
        for i in issues:  # fill page/bbox from the node's provenance when missing
            n = nodes.get(i.get("node", ""))
            if n and n.provenance:
                i.setdefault("page", n.provenance[0].page)
                if not i.get("bbox"):
                    bb = next((p.bbox for p in n.provenance if p.bbox), None)
                    if bb:
                        i["bbox"] = bb
        issues = sorted(issues, key=lambda i: (sev_order.get(i["severity"], 3), i.get("page") or 0,
                                               i["code"]))
        counts: dict[str, int] = defaultdict(int)
        by_code: dict[tuple[str, str], int] = defaultdict(int)
        for i in issues:
            counts[i["severity"]] += 1
            by_code[(i["severity"], i["code"])] += 1
    
        lines = [f"# QA report: {kg.plant.name}", "",
                 f"Generated {kg.generated_at} by {kg.generator}.", "",
                 f"**{counts.get('error', 0)} errors, {counts.get('warning', 0)} warnings, "
                 f"{counts.get('info', 0)} info.**", "",
                 "Errors must be fixed in the page extraction files before the graph is trusted. "
                 "Warnings are things to re-check on the drawing (the *Re-check* column names the tiles). "
                 "Info items are for the record.", "",
                 "| Severity | Code | Count |", "|---|---|---|"]
        for (sev, code), c in sorted(by_code.items(), key=lambda kv: (sev_order.get(kv[0][0], 3), kv[0][1])):
            lines.append(f"| {sev} | `{code}` | {c} |")
        lines += ["", "## Details", "", "| Severity | Page | Code | Item | Message | Re-check |",
                  "|---|---|---|---|---|---|"]
        for i in issues:
            tiles = tiles_for_bbox(work_dir, i["page"], i.get("bbox")) if work_dir and i.get("page") else []
            msg = i["message"].replace("|", "\\|")
            lines.append(f"| {i['severity']} | {i.get('page', '')} | `{i['code']}` | {i.get('node', '')} | "
                         f"{msg} | {', '.join(tiles)} |")
        path.write_text("\n".join(lines) + "\n", encoding="utf-8")
    
    
    def _close(a: str, b: str) -> bool:
        """Letter codes close enough to be an OCR misread (same first letter, <=2 slips)."""
        if _edit1(a, b):
            return True
        return len(a) == len(b) and a[:1] == b[:1] and sum(x != y for x, y in zip(a, b)) <= 2
    
  • skills/pid-to-kg/scripts/pidkg/store.py9.4 KB233c319230d6upstream ↗
    """DuckDB storage and graph traversal.
    
    The DuckDB file is a query layer over ``graph.json``: two tables (``nodes``,
    ``edges``), convenience views, and table macros for flow traversal, e.g.::
    
        SELECT * FROM downstream('660-TK-001');
        SELECT * FROM upstream('FV-0502');
        SELECT * FROM instruments WHERE variable = 'flow';
    """
    
    from __future__ import annotations
    
    import json
    from collections import defaultdict, deque
    from pathlib import Path
    from typing import Any, Optional
    
    from .models import KnowledgeGraph
    
    SCHEMA_SQL = """
    CREATE TABLE plant (
        id VARCHAR PRIMARY KEY, name VARCHAR, source_file VARCHAR,
        generated_at VARCHAR, generator VARCHAR, schema_version VARCHAR, stats JSON
    );
    CREATE TABLE nodes (
        id VARCHAR PRIMARY KEY, type VARCHAR NOT NULL, label VARCHAR, tag VARCHAR,
        kind VARCHAR, name VARCHAR, status VARCHAR, dexpi_class VARCHAR,
        sheets VARCHAR[], confidence DOUBLE, attributes JSON, provenance JSON
    );
    CREATE TABLE edges (
        id VARCHAR PRIMARY KEY, source VARCHAR NOT NULL, target VARCHAR NOT NULL,
        type VARCHAR NOT NULL, confidence DOUBLE, derived BOOLEAN, sheet VARCHAR,
        attributes JSON
    );
    CREATE TABLE issues (
        severity VARCHAR, code VARCHAR, message VARCHAR, node VARCHAR, sheet VARCHAR, page INTEGER
    );
    """
    
    FLOW_VIEW_SQL = """
    CREATE VIEW flow_edges AS
    SELECT source, target, sheet, attributes FROM edges
    WHERE type = 'FLOWS_TO' {pfd_filter};
    """
    
    VIEWS_SQL = """
    CREATE VIEW equipment AS
    SELECT id, tag, kind, name, status, sheets, confidence FROM nodes WHERE type = 'Equipment';
    
    CREATE VIEW equipment_flow AS
    SELECT e.source AS from_id, s.label AS from_label, e.target AS to_id, t.label AS to_label,
           e.attributes->>'via_lines' AS via_lines, e.attributes->>'via_items' AS via_items
    FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target
    WHERE e.type = 'FEEDS' {feeds_filter};
    
    CREATE VIEW instruments AS
    SELECT n.id, n.tag, n.attributes->>'variable' AS variable, n.attributes->>'role' AS role,
           n.attributes->>'location' AS location, n.name,
           m.target AS measures_id, mt.label AS measures_label, mt.type AS measures_type,
           l.target AS loop_id, n.sheets, n.confidence
    FROM nodes n
    LEFT JOIN edges m ON m.source = n.id AND m.type = 'MEASURES'
    LEFT JOIN nodes mt ON mt.id = m.target
    LEFT JOIN edges l ON l.source = n.id AND l.type = 'MEMBER_OF'
    WHERE n.type = 'Instrument';
    
    CREATE VIEW loops AS
    SELECT id, label AS loop, kind AS loop_type, attributes->>'variable' AS variable,
           attributes->'sensors' AS sensors, attributes->'controllers' AS controllers,
           attributes->'final_elements' AS final_elements,
           attributes->>'description' AS description, sheets
    FROM nodes WHERE type = 'ControlLoop';
    
    CREATE VIEW signals AS
    SELECT e.source, e.target, e.attributes->>'signal_type' AS signal_type,
           e.attributes->>'purpose' AS purpose, e.sheet
    FROM edges e WHERE e.type = 'SIGNAL_TO';
    
    CREATE MACRO downstream(start_id) AS TABLE
    WITH RECURSIVE walk(id) AS (
        SELECT target FROM flow_edges WHERE source = start_id
        UNION
        SELECT e.target FROM flow_edges e JOIN walk w ON e.source = w.id
    )
    SELECT n.id, n.type, n.label, n.kind, n.name FROM walk JOIN nodes n ON n.id = walk.id;
    
    CREATE MACRO upstream(start_id) AS TABLE
    WITH RECURSIVE walk(id) AS (
        SELECT source FROM flow_edges WHERE target = start_id
        UNION
        SELECT e.source FROM flow_edges e JOIN walk w ON e.target = w.id
    )
    SELECT n.id, n.type, n.label, n.kind, n.name FROM walk JOIN nodes n ON n.id = walk.id;
    """
    
    
    def write_duckdb(kg: KnowledgeGraph, path: Path) -> None:
        import duckdb
    
        path = Path(path)
        if path.exists():
            path.unlink()
        con = duckdb.connect(str(path))
        try:
            con.execute(SCHEMA_SQL)
            con.execute("INSERT INTO plant VALUES (?, ?, ?, ?, ?, ?, ?)",
                        [kg.plant.id, kg.plant.name, kg.plant.source_file, kg.generated_at,
                         kg.generator, kg.schema_version, json.dumps(kg.stats)])
            con.executemany(
                "INSERT INTO nodes VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
                [[n.id, n.type, n.label, n.tag, n.kind, n.name, n.status, n.dexpi_class, n.sheets,
                  n.confidence, json.dumps(n.attributes),
                  json.dumps([p.model_dump(exclude_none=True) for p in n.provenance])]
                 for n in kg.nodes])
            con.executemany(
                "INSERT INTO edges VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
                [[e.id, e.source, e.target, e.type, e.confidence, e.derived, e.sheet,
                  json.dumps(e.attributes)] for e in kg.edges])
            if kg.issues:
                con.executemany(
                    "INSERT INTO issues VALUES (?, ?, ?, ?, ?, ?)",
                    [[i.get("severity"), i.get("code"), i.get("message"), i.get("node"),
                      i.get("sheet"), i.get("page")] for i in kg.issues])
            pfd_filter = ("AND coalesce(attributes->>'drawing_type', '') <> 'pfd'"
                          if has_pid_sheets(kg) else "")
            con.execute(FLOW_VIEW_SQL.format(pfd_filter=pfd_filter))
            con.execute(VIEWS_SQL.replace("{feeds_filter}", "AND coalesce(e.attributes->>'view', '') <> 'pfd'"
                                          if pfd_filter else ""))
        finally:
            con.close()
    
    
    def has_pid_sheets(kg: KnowledgeGraph) -> bool:
        sheets = [n for n in kg.nodes if n.type == "Sheet"]
        return any(n.attributes.get("drawing_type", "pid") != "pfd" for n in sheets) and         any(n.attributes.get("drawing_type") == "pfd" for n in sheets)
    
    
    class Graph:
        """In-memory view of a knowledge graph for traversal queries."""
    
        def __init__(self, kg: KnowledgeGraph, include_pfd: bool = False):
            """``include_pfd``: when a plant has P&IDs, PFD-sheet flow is skipped in
            traversals by default because the P&IDs show the same flow in detail."""
            self.kg = kg
            self.nodes = {n.id: n for n in kg.nodes}
            self.out: dict[str, list] = defaultdict(list)
            self.inn: dict[str, list] = defaultdict(list)
            skip_pfd = not include_pfd and has_pid_sheets(kg)
            for e in kg.edges:
                if skip_pfd and e.attributes.get("drawing_type") == "pfd":
                    continue
                if skip_pfd and e.type == "FEEDS" and e.attributes.get("view") == "pfd":
                    continue
                self.out[e.source].append(e)
                self.inn[e.target].append(e)
            self._by_tag = {}
            for n in kg.nodes:
                for key in {n.id, n.tag, n.label}:
                    if key:
                        self._by_tag.setdefault(key.upper(), n.id)
    
        @classmethod
        def load(cls, path: Path, include_pfd: bool = False) -> "Graph":
            path = Path(path)
            if path.suffix == ".duckdb":
                path = path.with_name("graph.json")
            return cls(KnowledgeGraph.model_validate_json(path.read_text(encoding="utf-8")), include_pfd)
    
        def find(self, ref: str) -> Optional[str]:
            from .isa import normalize_tag
    
            if ref in self.nodes:
                return ref
            for key in (ref.upper(), normalize_tag(ref)):
                if key and key in self._by_tag:
                    return self._by_tag[key]
            # Untagged items: a unique local id ('P12.OXIDANT_TOTE') or a unique name.
            r = ref.upper()
            hits = [n.id for n in self.kg.nodes if n.id.upper().endswith("." + r)]
            if len(hits) != 1:
                hits = [n.id for n in self.kg.nodes if n.name and n.name.upper() == r]
            return hits[0] if len(hits) == 1 else None
    
        def walk(self, start: str, direction: str = "down", edge_types=("FLOWS_TO",),
                 max_depth: Optional[int] = None) -> list[tuple[str, int]]:
            """Breadth-first traversal returning (node id, hops) in visiting order."""
            adj = self.out if direction == "down" else self.inn
            seen = {start: 0}
            order: list[tuple[str, int]] = []
            q = deque([start])
            while q:
                cur = q.popleft()
                d = seen[cur]
                if max_depth is not None and d >= max_depth:
                    continue
                for e in adj[cur]:
                    if e.type not in edge_types:
                        continue
                    nxt = e.target if direction == "down" else e.source
                    if nxt not in seen:
                        seen[nxt] = d + 1
                        order.append((nxt, d + 1))
                        q.append(nxt)
            return order
    
        def path(self, a: str, b: str, edge_types=("FLOWS_TO",)) -> Optional[list[str]]:
            prev: dict[str, Optional[str]] = {a: None}
            q = deque([a])
            while q:
                cur = q.popleft()
                if cur == b:
                    out = []
                    while cur is not None:
                        out.append(cur)
                        cur = prev[cur]
                    return out[::-1]
                for e in self.out[cur]:
                    if e.type in edge_types and e.target not in prev:
                        prev[e.target] = cur
                        q.append(e.target)
            return None
    
        def neighbors(self, nid: str) -> dict[str, list[dict[str, Any]]]:
            rel: dict[str, list[dict[str, Any]]] = defaultdict(list)
            for e in self.out[nid]:
                rel[f"{e.type} ->"].append({"id": e.target, "label": self.nodes[e.target].label,
                                            **({"attributes": e.attributes} if e.attributes else {})})
            for e in self.inn[nid]:
                rel[f"<- {e.type}"].append({"id": e.source, "label": self.nodes[e.source].label,
                                            **({"attributes": e.attributes} if e.attributes else {})})
            return dict(rel)
    
  • skills/pid-to-kg/scripts/pidkg/summary.py7.6 KB24a99daf7421upstream ↗
    """Compact Markdown summary of a knowledge graph, written for LLM context windows."""
    
    from __future__ import annotations
    
    from collections import defaultdict
    from typing import Optional
    
    from .loops import sensed_points
    from .models import KnowledgeGraph
    from .store import Graph
    
    
    def _esc(s: Optional[str]) -> str:
        return (s or "").replace("|", "\\|").replace("\n", " ")
    
    
    def _flow_order(g: Graph) -> list[str]:
        """Equipment/boundaries in reading order: sources first, then breadth-first along FEEDS."""
        feeds_in: dict[str, int] = defaultdict(int)
        terminals = [n.id for n in g.kg.nodes if n.type in ("Equipment", "Boundary")]
        for e in g.kg.edges:
            if e.type == "FEEDS":
                feeds_in[e.target] += 1
        starts = [t for t in terminals if feeds_in[t] == 0]
        starts.sort(key=lambda t: (g.nodes[t].type != "Boundary", g.nodes[t].sheets[:1], t))
        order: list[str] = []
        seen: set[str] = set()
        for s in starts + terminals:
            if s in seen:
                continue
            queue = [s]
            while queue:
                cur = queue.pop(0)
                if cur in seen:
                    continue
                seen.add(cur)
                order.append(cur)
                queue.extend(e.target for e in g.out[cur] if e.type == "FEEDS")
        return order
    
    
    def write_summary(kg: KnowledgeGraph) -> str:
        g = Graph(kg)
        nodes = g.nodes
        sheets = [n for n in kg.nodes if n.type == "Sheet"]
        sheets.sort(key=lambda n: n.attributes.get("page", 0))
        st = kg.stats
        out: list[str] = []
        w = out.append
    
        w(f"# {kg.plant.name}: process knowledge graph summary")
        w("")
        w(f"Built from `{kg.plant.source_file or 'P&IDs'}` ({len(sheets)} sheets) by {kg.generator} "
          f"on {kg.generated_at[:10]}. The graph has {st.get('nodes', 0)} nodes and "
          f"{st.get('edges', 0)} edges; this summary covers equipment-level flow, instruments and "
          "control loops. Query `graph.duckdb` or `graph.json` for pipe-segment and valve detail.")
        w("")
        nbt = st.get("nodes_by_type", {})
        w("| Equipment | Instruments | Valves & fittings | Pipe segments | Control loops | Off-page links |")
        w("|---|---|---|---|---|---|")
        w(f"| {nbt.get('Equipment', 0)} | {nbt.get('Instrument', 0)} | {nbt.get('Component', 0)} | "
          f"{nbt.get('PipeSegment', 0)} | {nbt.get('ControlLoop', 0)} | {nbt.get('OffPageConnector', 0)} |")
        w("")
    
        w("## Sheets")
        w("")
        w("| Page | Drawing | Title | Type |")
        w("|---|---|---|---|")
        for s in sheets:
            a = s.attributes
            w(f"| {a.get('page', '')} | {_esc(a.get('drawing_number'))} | {_esc(a.get('title'))} | "
              f"{a.get('drawing_type', '')} |")
        w("")
    
        w("## Process flow (equipment to equipment)")
        w("")
        w("Each line reads *upstream → downstream*, with the pipe lines and in-line items between them. "
          "Pipes, valves and instruments are collapsed; use `kg_query.py path` for the full route.")
        w("")
        drains: dict[str, set[str]] = defaultdict(set)
        for nid in _flow_order(g):
            feeds = [e for e in g.out[nid] if e.type == "FEEDS"]
            for e in [e for e in feeds if _is_drain(nodes[e.target])]:
                drains[nodes[e.target].label].add(nodes[nid].label)
            feeds = [e for e in feeds if not _is_drain(nodes[e.target])]
            if not feeds:
                continue
            n = nodes[nid]
            head = f"**{n.label}**" + (f" ({_esc(n.name)})" if n.name and n.name != n.label else "")
            for e in feeds:
                t = nodes[e.target]
                via = []
                if e.attributes.get("via_lines"):
                    via.append("lines " + ", ".join(e.attributes["via_lines"]))
                if e.attributes.get("via_items"):
                    via.append("through " + ", ".join(e.attributes["via_items"]))
                if e.attributes.get("off_page"):
                    via.append("crosses sheets")
                tname = f" ({_esc(t.name)})" if t.name and t.name != t.label else ""
                w(f"- {head} → **{t.label}**{tname}" + (f": {'; '.join(via)}" if via else ""))
        if drains:
            w("")
            w("Drains, vents and overflows (not process flow):")
            for d, srcs in sorted(drains.items()):
                w(f"- {d} ← {', '.join(sorted(srcs))}")
        w("")
    
        w("## Equipment")
        w("")
        w("| Tag | Kind | Description | Fed by | Feeds | Status | Sheets |")
        w("|---|---|---|---|---|---|---|")
        for n in sorted((n for n in kg.nodes if n.type == "Equipment"), key=lambda n: n.label):
            ups = sorted({nodes[e.source].label for e in g.inn[n.id] if e.type == "FEEDS"})
            downs = sorted({nodes[e.target].label for e in g.out[n.id] if e.type == "FEEDS"})
            w(f"| {n.label} | {n.kind or ''} | {_esc(n.name)} | {', '.join(ups)} | {', '.join(downs)} | "
              f"{n.status or ''} | {', '.join(_sheet_labels(n.sheets, nodes))} |")
        w("")
    
        w("## Instruments")
        w("")
        w("Location is where the instrument senses the process. Pipe locations read "
          "`line (upstream item -> downstream item)`.")
        w("")
        w("| Tag | Variable | Function | Mounted | Senses | Loop |")
        w("|---|---|---|---|---|---|")
        for n in sorted((n for n in kg.nodes if n.type == "Instrument"), key=lambda n: n.label):
            a = n.attributes
            senses = sensed_points(n.id, nodes, kg.edges)
            loops = [nodes[e.target].label for e in g.out[n.id] if e.type == "MEMBER_OF"]
            w(f"| {n.label} | {a.get('variable', '')} | {a.get('role', '')} | {a.get('location', '')} | "
              f"{_esc('; '.join(senses))} | {', '.join(loops)} |")
        w("")
    
        loops = [n for n in kg.nodes if n.type == "ControlLoop"]
        w("## Control loops")
        w("")
        if not loops:
            w("No control loops were identified.")
        else:
            w("| Loop | Type | Sensor(s) | Controller(s) | Final element(s) | Description |")
            w("|---|---|---|---|---|---|")
            lab = lambda ids: ", ".join(nodes[i].label for i in ids if i in nodes)  # noqa: E731
            for n in sorted(loops, key=lambda n: n.label):
                a = n.attributes
                w(f"| {n.label} | {n.kind} | {lab(a.get('sensors', []))} | {lab(a.get('controllers', []))} | "
                  f"{lab(a.get('final_elements', []))} | {_esc(a.get('description'))} |")
        w("")
    
        opcs = [n for n in kg.nodes if n.type == "OffPageConnector"]
        if opcs:
            w("## Off-page connections")
            w("")
            w("| Sheet | Connector | Direction | Other drawing | Linked to |")
            w("|---|---|---|---|---|")
            for n in sorted(opcs, key=lambda n: n.id):
                a = n.attributes
                link = a.get("linked_to") or a.get("linked_from") or a.get("linked_item") or "unmatched"
                if link in nodes:
                    link = f"{nodes[link].label} ({', '.join(_sheet_labels(nodes[link].sheets, nodes))})"
                w(f"| {', '.join(_sheet_labels(n.sheets, nodes))} | {_esc(n.label)} | {a.get('direction', '')} | "
                  f"{_esc(a.get('other_drawing'))} | {_esc(link)} |")
            w("")
    
        sev = st.get("issues_by_severity", {})
        w("## Data quality")
        w("")
        w(f"{sev.get('error', 0)} errors, {sev.get('warning', 0)} warnings and {sev.get('info', 0)} "
          "informational items; see `qa_report.md`. Every node carries `confidence` and the sheet "
          "region it was read from (`provenance`). This graph was machine-extracted from drawings "
          "and must be verified by a qualified engineer before use in design, operations or safety work.")
        return "\n".join(out) + "\n"
    
    
    _DRAIN_WORDS = ("DRAIN", "VENT", "OVERFLOW", "SUMP")
    
    
    def _is_drain(n) -> bool:
        return n.type == "Boundary" and any(w in (n.label or "").upper() for w in _DRAIN_WORDS[:3])
    
    
    def _sheet_labels(sheet_ids: list[str], nodes) -> list[str]:
        return [nodes[s].label if s in nodes else s for s in sheet_ids]
    
  • skills/pid-to-kg/scripts/probe_pdf.py3.5 KBc3343eda29a7upstream ↗
    #!/usr/bin/env python3
    """Classify every page of a P&ID PDF and start a work directory.
    
    Each page is classified as vector_text, vector_outlined, raster,
    raster_with_text, hybrid or blank, which decides how it is rendered and
    which hints are available. Writes <work>/manifest.json.
    
    Usage:
        python probe_pdf.py drawings.pdf --work work/my-plant --plant-name "My Plant"
        python probe_pdf.py permit.pdf --work work/my-plant --pages 47-60   # only the P&ID pages
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import re
    import sys
    from pathlib import Path
    
    sys.path.insert(0, str(Path(__file__).resolve().parent))
    
    from pidkg.pdf import probe_pdf  # noqa: E402
    from render_pages import parse_pages  # noqa: E402
    
    
    def main() -> int:
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument("pdf", type=Path)
        ap.add_argument("--work", type=Path, required=True, help="work directory for this plant")
        ap.add_argument("--plant-name", help="human-readable plant name (default: PDF file name)")
        ap.add_argument("--plant-id", help="short id (default: slug of the plant name)")
        ap.add_argument("--pages", help="pages that hold the P&IDs, e.g. '47-60' or '50-51,62-63' "
                                        "(default: all). Later steps use only these pages.")
        args = ap.parse_args()
    
        if not args.pdf.exists():
            print(f"error: {args.pdf} not found", file=sys.stderr)
            return 2
        name = args.plant_name or args.pdf.stem
        plant_id = args.plant_id or re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-")
        probes = probe_pdf(str(args.pdf))
    
        args.work.mkdir(parents=True, exist_ok=True)
        manifest_path = args.work / "manifest.json"
        previous = {}
        if manifest_path.exists():
            old = json.loads(manifest_path.read_text(encoding="utf-8"))
            previous = {p["page"]: p for p in old.get("pages", [])}
        pages = []
        for p in probes:
            d = p.to_dict()
            # Keep a rotation that was already confirmed for this page.
            if previous.get(p.page, {}).get("rotation") is not None:
                d["rotation"] = previous[p.page]["rotation"]
                d["rotation_source"] = previous[p.page].get("rotation_source")
            pages.append(d)
        selected = parse_pages(args.pages, len(probes)) if args.pages else None
        if args.pages and not selected:
            print(f"error: --pages {args.pages} selects no page of a {len(probes)}-page PDF", file=sys.stderr)
            return 2
        manifest = {"source_pdf": str(args.pdf.resolve()), "plant_id": plant_id, "plant_name": name,
                    "pages": pages}
        if selected:
            manifest["selected_pages"] = selected
        manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")
    
        shown = [p for p in probes if not selected or p.page in selected]
        print(f"{args.pdf.name}: {len(probes)} pages -> {manifest_path}")
        if selected:
            print(f"using {len(selected)} selected pages: {args.pages}")
        print(f"{'page':>4}  {'class':<17} {'chars':>6} {'paths':>6} {'img cov':>7}  notes")
        for p in shown:
            print(f"{p.page:>4}  {p.page_class:<17} {p.text_chars:>6} {p.path_objects:>6} "
                  f"{p.max_image_coverage:>7.2f}  {'; '.join(p.notes)}")
        classes = sorted({p.page_class for p in probes})
        print("\nRoutes:")
        for c in classes:
            print(f"  {c}: {next(p.route for p in probes if p.page_class == c)}")
        return 0
    
    
    if __name__ == "__main__":
        sys.exit(main())
    
  • skills/pid-to-kg/scripts/render_pages.py8.2 KBf1722ca0f502upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/pid-to-kg/scripts/requirements.txt283 Be71cc9e5c434upstream ↗
    pypdfium2>=4.30
    pillow>=10.0
    pydantic>=2.6
    duckdb>=1.1
    networkx>=3.2
    # Optional but recommended: OCR hints and automatic page orientation.
    # The OCR package supports Python 3.10-3.12 only; on newer Python the skill runs without it.
    rapidocr-onnxruntime>=1.3; python_version < "3.13"
    
  • skills/pid-to-kg/references/extraction_procedure.md15 KB2c609c62fd86upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/pid-to-kg/references/isa_5_1.md5.8 KB1addb63fff8cupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/pid-to-kg/references/ontology.md7.6 KB26823553594dupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/pid-to-kg/references/schema/extraction.schema.json25 KBd62ea8a88fbfupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/pid-to-kg/references/schema/graph.schema.json6.2 KBb7b1ab2d094dupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • skills/pid-to-kg/references/symbol_guide.md4.3 KB95a2c84d7828upstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/scottduncanai-pid-to-knowledge-graph-skills-pid-to-kg.md

// REPO STATS

260 stars
1 open issues
Last commit: 7d ago