⏳ This skill is pending AI review.
Scores will appear once the review pipeline completes.
pid-to-kg
Converts P&IDs (piping and instrumentation diagrams) and PFDs in PDF form, either vector or scanned, into a process knowledge graph covering equipment connectivity (upstream/downstream), instrument locations and measured variables, and control loops. Output is JSON plus a DuckDB file plus an LLM-ready Markdown summary. Use when the user wants to digitize, extract, analyze or build a knowledge graph or context layer from P&IDs, or asks what is upstream/downstream of a piece of equipment, which sensors measure what, or how a plant's control loops are structured.
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
Not yet listed on ClawHub or SkillsMP
// README
P&ID to Knowledge Graph
By Scott Duncan. Reach me on LinkedIn or at scottduncanai.com.
Turn P&ID drawings into a process knowledge graph that an LLM can reason over.
pid-to-kg is a Claude skill that reads piping and instrumentation diagrams (P&IDs) and PFDs from PDF and builds a knowledge graph of the process. The PDFs can be digital (exported from CAD) or scanned. The graph captures:
- Process flow: what is upstream and downstream of every tank, pump, valve and pipe segment.
- Measurement: which instrument measures which variable, and exactly where it sits in the flow.
- Control: control loops from sensor through controller to final element, including I/P relays, cascades, interlocks and VFD drives.
The output is meant to be one layer of a chemical-manufacturing context layer. An LLM application can query it to answer questions such as "what feeds this clarifier?", "which transmitters sit downstream of P-101?" or "what does FIC-0502 manipulate?" from the drawings themselves.
Quick start
No programming is needed. You need Claude Code (the desktop app or the terminal version) and Python 3.10 or newer. Python 3.12 is the best choice: on 3.13 and later the skill still works, but without the text-recognition (OCR) cross-check and automatic page orientation.
1. Download the skill. On this page, click the green Code button, then Download ZIP. Unzip the file.
2. Install it. In the unzipped folder, open skills and copy the pid-to-kg folder into your personal Claude skills folder:
| System | Skills folder |
|---|---|
| Windows | C:\Users\<your user name>\.claude\skills\ |
| macOS | /Users/<your user name>/.claude/skills/ |
If the skills folder doesn't exist yet, create it. The .claude folder is hidden by default: in Windows File Explorer use View → Show → Hidden items, and in macOS Finder press Cmd + Shift + . to show it.
3. Add your P&IDs. Make a new folder for the project and put your P&ID PDF in it. Open that folder in Claude Code: in the desktop app, start a new Code session and choose the folder.
No P&IDs at hand? My public chemical manufacturing datasets repo lists public P&ID sets from real permitted facilities, with the page numbers where the drawings are. Both digital and scanned sets are included.
4. Ask Claude to convert them. For example:
Convert
haile.pdfinto a knowledge graph. The P&IDs are on pages 47–60.
The skill loads by itself; there is no need to name it. The first time, Claude installs the Python packages it needs, and it asks your permission before running commands. It then reads every sheet in detail, so a large drawing set takes a while. Claude may ask a question along the way, such as which revision to use when a PDF holds two versions of the same drawing. When it is done, it reports what it found and what remains uncertain.
5. Check the graph by asking questions. The best test is to ask about things you can verify on the drawings:
Name all the equipment upstream of the clarifier.
What sensor measures the flow into TK-101, and where exactly is it?
What does FIC-0502 control, and what valve does it move?
The results are saved in work/<plant name>/kg/ inside your project folder. summary.md is a readable overview of the whole plant, and qa_report.md lists anything Claude could not confirm.
How it works
Claude reads the drawings; Python scripts do everything else. Recognising symbols and tracing lines needs visual judgement, so that part is Claude's. Preparing pages, building the graph and checking it are routine, so tested scripts do them the same way every time.
1. Preparing the pages
A script first looks at every page and works out what kind of drawing it is. Digital and scanned P&IDs need different handling:
| Page type | How to recognise it | What the skill does |
|---|---|---|
| Digital, searchable text | Exported from CAD; you can select the text in a PDF viewer | Renders the page straight from the vector data, so lines stay sharp at any zoom. The exact text in the PDF is used to cross-check the tags Claude reads. |
| Digital, outlined text | Looks digital, but the text can't be selected: every letter was drawn as shapes. Common in CAD exports. | Renders from the vector data at high resolution. Since there is no text to pull out, OCR (text recognition) reads the tags for the cross-check. |
| Scanned | Each page is a picture of a paper drawing | Takes the scanned image at its original resolution, enlarges it up to 2.5×, and boosts the contrast so faded lines and small text stand out. OCR reads the tags for the cross-check. Low-resolution scans are flagged, because small text may be lost. |
| Scanned with OCR text | A scan with a searchable text layer added | Handled like a scan. The added text layer is only used as a hint, since it often contains errors. |
Then, for every page:
- Orientation: the page is turned so the text is upright, since drawings are often stored sideways in a PDF.
- Resolution: chosen from the size of the drawing's text, so the smallest tags stay legible.
- Tiles: the sheet is cut into overlapping tiles, and blank tiles (empty margins, border lines) are skipped.
2. Claude reads each sheet
- Claude reads the whole sheet first, then every tile up close.
- It records equipment, pipe runs, valves, instruments, signal lines and off-page connectors.
- Each item gets the place on the sheet where it was found and a confidence level. Anything unreadable is marked as such, never guessed.
- Each sheet is saved to its own file before the next one starts, so large drawing sets don't overload Claude.
3. Building and checking the knowledge graph
The scripts turn Claude's readings into one connected knowledge graph:
- Process flow: each pipe is split into segments between its valves and fittings, and off-page connectors link the sheets, so flow can be traced upstream or downstream across the whole plant.
- Measurement: every instrument is attached to the exact pipe segment or vessel it measures.
- Control: loops are assembled from the instrument tags (per ISA-5.1) and the signal lines, from sensor to controller to valve or drive.
A QA report then lists anything doubtful: a sensor with no location, an off-page connector with no partner, a tag OCR saw that Claude didn't record. Claude re-checks each one on the drawing and rebuilds until the report is clean.
4. Asking questions
summary.mdis a readable overview to give an LLM as context.graph.jsonholds the full graph for any tool to read.graph.duckdbis a small database file for precise queries. Nothing needs a server.- Every item records the sheet it came from, so any answer can be traced back to the drawing.
Outputs
| File | What it is |
|---|---|
summary.md | A readable overview of the plant: equipment-level flow, equipment, instruments with their locations, control loops and off-page links. This is the file to give an LLM as context. |
qa_report.md | Errors, warnings and notes, each pointing to the part of the drawing to re-check |
graph.json | The complete graph (schema: graph.schema.json) |
graph.duckdb | The same graph as a DuckDB database file, for SQL queries, including recursive upstream and downstream searches |
For larger projects, the graph can also be exported to Neo4j or to GraphML (for yEd, Gephi or NetworkX). Ask Claude to do it.
The node and edge types are described in [`refere
// HOW IT'S BUILT
// FILES
28 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.
skills/pid-to-kg/SKILL.md9.3 KB352f8b8fd5f2upstream ↗
--- name: pid-to-kg description: Converts P&IDs (piping and instrumentation diagrams) and PFDs in PDF form, either vector or scanned, into a process knowledge graph covering equipment connectivity (upstream/downstream), instrument locations and measured variables, and control loops. Output is JSON plus a DuckDB file plus an LLM-ready Markdown summary. Use when the user wants to digitize, extract, analyze or build a knowledge graph or context layer from P&IDs, or asks what is upstream/downstream of a piece of equipment, which sensors measure what, or how a plant's control loops are structured. --- # P&ID to knowledge graph Work is split between deterministic scripts and you: - **Scripts** classify pages (vector vs scanned), render them into readable tiles, collect OCR/text hints, then build, check, store and query the graph. - **You** read the drawings and write one structured extraction file per sheet. Symbol recognition and line tracing are visual reasoning, and that is the part scripts cannot do well. All scripts are in `scripts/` next to this file. Call them by their full path (the commands below abbreviate it as `scripts/`), and keep the work directory in the user's current folder, e.g. `work/<plant>`. **Setup.** Many users are engineers, not programmers, so handle setup yourself. The scripts need Python 3.10+. Before the first run, install the dependencies with `pip install -r <skill dir>/scripts/requirements.txt` (use a virtual environment if the user's Python is shared with other work). If Python is missing, tell the user to install it from python.org and stop there. ## Ground rules - **Only the P&ID pages.** Drawings often sit inside a long permit or design report. If the user names the pages, pass them with `--pages`. If not, and the PDF has more than a few pages, probe it, look at the page overviews or ask the user, then re-run the probe with `--pages` so later steps skip reports, legends and non-P&ID drawings. - **One plant, one work directory, one graph.** Never put sheets from different plants in the same work directory: tags would merge across unrelated plants. If the user supplies several plants, run the whole workflow separately for each. - **One configuration per graph.** If the PDF contains two revisions of the same drawing (same title or area, overlapping tags, e.g. an as-built and an older draft), do not merge them. Ask the user which revision to use, extract only that one, and record the excluded pages and the reason in the report. QA flags likely duplicates as `possible_duplicate_revision`. Record each sheet's `issue_status` from the title block or stamps. - **Record only what is drawn.** Never invent tags, connections or loops to make the graph look complete. Anything unclear goes in with a lower `confidence`, or into `unreadable`. - **The QA loop is not optional.** Rebuild until there are no errors, and re-check every warning on the drawing. ## Workflow Copy this checklist into your notes and tick it off: ``` - [ ] 1. Probe: python scripts/probe_pdf.py <pdf> --work <work> --plant-name "<name>" [--pages 47-60] - [ ] 2. Render: python scripts/render_pages.py <work> - [ ] 3. Orient: open each pages/page_NN/overview_grid.png; fix rotation if text is not upright - [ ] 4. Hints: python scripts/make_hints.py <work> - [ ] 5. Extract: for each sheet, write <work>/extract/page_NN.json (references/extraction_procedure.md) - [ ] 6. Build: python scripts/build_kg.py <work> - [ ] 7. QA loop: fix every error, re-check every warning, rebuild (repeat) - [ ] 8. Report: summarize the results and open issues for the user ``` ### 1. Probe `probe_pdf.py` assigns each page a class and an extraction route, and writes `<work>/manifest.json`. With `--pages`, the later steps render and read only those pages. | Class | What it means | Route | |---|---|---| | `vector_text` | Vector drawing with a text layer | Render from vectors; exact text from the text layer as hints | | `vector_outlined` | Vector drawing, text drawn as outlines (no text layer) | High-resolution vector render; OCR hints | | `raster` | Scanned image | Native-resolution render, upscaled and contrast-enhanced; OCR hints | | `raster_with_text` | Scan with an OCR text layer | As raster; text layer as (untrusted) hints | | `hybrid` | Vectors plus large images | Both routes | Report the probe result to the user in one line, e.g. "14 vector sheets with outlined text." Low scan resolution is flagged in the notes. Expect to record more items in `unreadable` on those sheets. ### 2–3. Render and check orientation `render_pages.py` writes the following for each page: - `overview.png` - `overview_grid.png`, which shows the tile names - overlapping tiles in `tiles/` - `tiles.json`, which records which tiles are blank Orientation is detected with OCR when it is installed. Always confirm it on `overview_grid.png`. If the main drawing text is not upright, re-render that page with an explicit rotation, given clockwise: `render_pages.py <work> --pages N --rotation 90`. Title blocks are sometimes rotated differently from the drawing. Orient by the drawing labels, not by the title block. ### 4. Hints `make_hints.py` writes `hints/page_NN.json`. Each file holds candidate tags with page positions, taken from the text layer or from OCR. They are only hints: OCR misreads, and it misses things. You don't need to read the hint files during extraction. QA uses them to flag tags you may have missed. If OCR is not installed the step is skipped; note this in the final report. ### 5. Extract each sheet This is the core of the job. Follow **[references/extraction_procedure.md](references/extraction_procedure.md)** exactly. It covers: - the reading order: overview, then every non-blank tile, then zoomed crops - how to model pipe runs, tees, taps, signals and off-page connectors - confidence calibration - a self-check list to run before saving Reference material, to read when needed: - [references/isa_5_1.md](references/isa_5_1.md): instrument letter codes, bubble symbols, signal line types - [references/symbol_guide.md](references/symbol_guide.md): how equipment and valve symbols map to `kind` values - [references/ontology.md](references/ontology.md): the graph's node and edge types and how they are derived - `references/schema/extraction.schema.json`: the exact JSON schema of a page file Write `extract/page_NN.json` and save it before starting the next sheet, so a long job never holds more than one sheet in context. With many sheets, and if your environment supports subagents, you can extract sheets in parallel. Give each subagent the procedure file and its own page directory. ### 6–7. Build and the QA loop `build_kg.py` validates every page file and stops with a precise message on schema errors. It then writes these files to `<work>/kg/`: - `graph.json` - `graph.duckdb` - `summary.md` - `qa_report.md` Work through `qa_report.md`: - **Errors**, such as `unresolved_reference`, are mistakes in the page files. Fix them and rebuild. - **Warnings** name the tiles to re-check. Look at those tiles again, or use `crop.py` for a zoomed view, and correct the extraction if the drawing shows something different. Common warnings: - `instrument_not_located`: a sensor with no `measures` - `equipment_not_connected` - `unmatched_off_page` - `possible_missed_tags`: an OCR tag that is not in your extraction - `tiles_not_reviewed` - A warning that turns out correct on the drawing is fine to leave. Examples: an off-page connector whose other drawing was not supplied, or an OCR misread. List these in your report. Stop when there are no errors and every warning has been checked. ```bash python scripts/crop.py <work> --page 5 --bbox 0.40 0.20 0.55 0.35 # zoomed view of a region ``` ### 8. Report Tell the user: - where the outputs are - the headline counts (equipment, instruments, loops) - two or three example flow paths and loops taken from `summary.md` - what remains uncertain Point to `qa_report.md` for the details. State plainly that the graph is machine-extracted and needs engineering review before it is used for design, operations or safety. ## Using the graph - `summary.md` is the compact, token-efficient context for an LLM: equipment-level flow, an instrument table with locations, and a loop table. - `graph.json` is the canonical graph, which any tool can read. - `graph.duckdb` is for SQL. Its views are `equipment`, `equipment_flow`, `instruments`, `loops` and `signals`, and its table macros are `upstream(id)` and `downstream(id)`. Use `kg_query.py` to answer questions; add `--json` when another tool will consume the output: ```bash python scripts/kg_query.py <work>/kg downstream 660-TK-001 --equipment python scripts/kg_query.py <work>/kg upstream FV-0502 python scripts/kg_query.py <work>/kg path 660-TK-001 660-TK-004 python scripts/kg_query.py <work>/kg instruments --on 660-PP-011 python scripts/kg_query.py <work>/kg loop FIC-0502 python scripts/kg_query.py <work>/kg sql "SELECT * FROM equipment_flow" ``` To measure extraction quality on a sheet, hand-check it into a small gold file (format in `scripts/eval_kg.py`) and run `python scripts/eval_kg.py <work>/kg gold.json`: it reports tag, flow and loop precision/recall and instrument placement. To move a graph into other tools: `python scripts/export.py <work>/kg graphml` produces a file for yEd, Gephi or NetworkX, and `python scripts/export.py <work>/kg neo4j` produces CSVs plus a Cypher import script.skills/pid-to-kg/scripts/build_kg.py4.5 KB9de11c3fe63fupstream ↗
#!/usr/bin/env python3 """Build the knowledge graph for one plant from its page extractions. Reads <work>/extract/page_*.json (validated against the extraction schema), merges items across sheets by tag, expands pipe runs into segments, links off-page connectors, infers control loops and equipment-level flow, runs QA, and writes to <work>/kg/: graph.json canonical knowledge graph (node-link JSON) graph.duckdb the same graph as DuckDB tables, views and traversal macros summary.md compact LLM-ready summary of flow, instruments and loops qa_report.md issues to fix or re-check, with the tiles to look at Usage: python build_kg.py work/my-plant python build_kg.py --write-schemas # regenerate references/schema/*.json """ from __future__ import annotations import argparse import json import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) from pidkg.build import BuildError, build_graph, load_extractions # noqa: E402 from pidkg.models import KnowledgeGraph, PageExtraction # noqa: E402 from pidkg.qa import run_qa, write_report # noqa: E402 from pidkg.store import write_duckdb # noqa: E402 from pidkg.summary import write_summary # noqa: E402 SCHEMA_DIR = Path(__file__).resolve().parent.parent / "references" / "schema" def write_schemas() -> None: SCHEMA_DIR.mkdir(parents=True, exist_ok=True) for name, model in (("extraction.schema.json", PageExtraction), ("graph.schema.json", KnowledgeGraph)): schema = model.model_json_schema(by_alias=True) schema["$schema"] = "https://json-schema.org/draft/2020-12/schema" (SCHEMA_DIR / name).write_text(json.dumps(schema, indent=2) + "\n", encoding="utf-8") print(f"wrote {SCHEMA_DIR / name}") def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("work", type=Path, nargs="?") ap.add_argument("--out", type=Path, help="output directory (default: <work>/kg)") ap.add_argument("--plant-name") ap.add_argument("--plant-id") ap.add_argument("--pages", help="only build from these pages, e.g. '1-2' (default: all extracted)") ap.add_argument("--write-schemas", action="store_true") args = ap.parse_args() if hasattr(sys.stdout, "reconfigure"): sys.stdout.reconfigure(encoding="utf-8") if args.write_schemas: write_schemas() return 0 if not args.work: ap.error("work directory is required") manifest = {} mpath = args.work / "manifest.json" if mpath.exists(): manifest = json.loads(mpath.read_text(encoding="utf-8")) plant_name = args.plant_name or manifest.get("plant_name") or args.work.name plant_id = args.plant_id or manifest.get("plant_id") or args.work.name source = Path(manifest["source_pdf"]).name if manifest.get("source_pdf") else None try: pages = load_extractions(args.work / "extract") except BuildError as e: print(f"error: {e}", file=sys.stderr) return 1 if args.pages: from render_pages import parse_pages keep = set(parse_pages(args.pages, max(p.page for p in pages))) pages = [p for p in pages if p.page in keep] if not pages: print(f"error: no extracted pages in {args.pages}", file=sys.stderr) return 1 kg = build_graph(pages, plant_id, plant_name, source) kg.issues.extend(run_qa(kg, args.work)) sev: dict[str, int] = {} for i in kg.issues: sev[i["severity"]] = sev.get(i["severity"], 0) + 1 kg.stats["issues_by_severity"] = sev out = args.out or (args.work / "kg") out.mkdir(parents=True, exist_ok=True) (out / "graph.json").write_text(kg.model_dump_json(indent=1, exclude_none=True), encoding="utf-8") write_duckdb(kg, out / "graph.duckdb") (out / "summary.md").write_text(write_summary(kg), encoding="utf-8") write_report(kg, kg.issues, out / "qa_report.md", args.work) s = kg.stats print(f"{plant_name}: {s['nodes']} nodes, {s['edges']} edges -> {out}") print(" nodes: " + ", ".join(f"{k} {v}" for k, v in s["nodes_by_type"].items())) print(" edges: " + ", ".join(f"{k} {v}" for k, v in s["edges_by_type"].items())) if s.get("loops_by_type"): print(" loops: " + ", ".join(f"{k} {v}" for k, v in s["loops_by_type"].items())) print(f" issues: {sev.get('error', 0)} errors, {sev.get('warning', 0)} warnings, " f"{sev.get('info', 0)} info (see qa_report.md)") return 1 if sev.get("error") else 0 if __name__ == "__main__": sys.exit(main())skills/pid-to-kg/scripts/crop.py1.8 KBcb6e1a7f8cbeupstream ↗
#!/usr/bin/env python3 """Cut a zoomed view of a page region for a closer look. The bbox is in normalized page coordinates (0..1, origin top-left), the same system used for tiles and for ``bbox`` in extractions. The crop is upscaled so its long edge is --size pixels. Usage: python crop.py work/my-plant --page 5 --bbox 0.40 0.20 0.55 0.35 """ from __future__ import annotations import argparse import sys from pathlib import Path def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("work", type=Path) ap.add_argument("--page", type=int, required=True) ap.add_argument("--bbox", type=float, nargs=4, required=True, metavar=("X0", "Y0", "X1", "Y1")) ap.add_argument("--size", type=int, default=1400, help="long edge of the output in px") ap.add_argument("--out", type=Path) a = ap.parse_args() from PIL import Image page_dir = a.work / "pages" / f"page_{a.page:02d}" src = page_dir / "page.png" if not src.exists(): print(f"error: {src} not found; run render_pages.py first", file=sys.stderr) return 2 img = Image.open(src) x0, y0, x1, y1 = a.bbox box = (round(min(x0, x1) * img.width), round(min(y0, y1) * img.height), round(max(x0, x1) * img.width), round(max(y0, y1) * img.height)) crop = img.crop(box) scale = a.size / max(crop.size) crop = crop.resize((max(1, round(crop.width * scale)), max(1, round(crop.height * scale))), Image.LANCZOS) out = a.out or page_dir / "crops" / f"crop_{x0:.2f}_{y0:.2f}_{x1:.2f}_{y1:.2f}.png" out.parent.mkdir(parents=True, exist_ok=True) crop.save(out) print(out) return 0 if __name__ == "__main__": sys.exit(main())skills/pid-to-kg/scripts/eval_kg.py7.5 KB282d8a37aeaeupstream ↗
#!/usr/bin/env python3 """Score a built graph against a hand-checked gold annotation. Gold file (JSON), usually covering one or a few sheets: { "sheets": ["660-PI-203"], # drawing numbers the gold covers "tags": ["660-TK-004", "FIT-0502", ...], # every tagged item on those sheets "feeds": [["660-EQ-002", "660-PP-011"], ...], # equipment-level flow (upstream, downstream) "instrument_at": {"FIT-0502": "660-PP-011"}, # nearest equipment to where it senses "loops": {"F-0502": ["FE-0502", "FIT-0502"]}, # loop key -> member tags "feeds_scope": "within_sheets" # optional: "all" (default), "within_sheets" # (both ends on gold sheets), "equipment" (both ends equipment) } Reports precision/recall/F1 for tags and flow, instrument placement accuracy, and loop recall with member overlap. Only graph items on the gold sheets are scored, so a partial gold set gives a fair spot check. Usage: python eval_kg.py work/my-plant/kg gold.json [--json] """ from __future__ import annotations import argparse import json import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) from pidkg.isa import normalize_tag # noqa: E402 from pidkg.store import Graph # noqa: E402 def prf(pred: set, gold: set) -> dict: tp = len(pred & gold) p = tp / len(pred) if pred else 0.0 r = tp / len(gold) if gold else 0.0 f = 2 * p * r / (p + r) if p + r else 0.0 return {"precision": round(p, 3), "recall": round(r, 3), "f1": round(f, 3), "tp": tp, "false_positives": sorted(map(str, pred - gold)), "missed": sorted(map(str, gold - pred))} def near_equipment(g: Graph, nid: str, hops: int = 60) -> set[str]: """Equipment directly measured, or nearest up/downstream of a measured pipe or item.""" n = g.nodes[nid] if n.type == "Equipment": return {n.tag or n.label} found: set[str] = set() for direction in ("up", "down"): best = None for other, d in g.walk(nid, direction, max_depth=hops): if g.nodes[other].type == "Equipment": best = (d, g.nodes[other].tag or g.nodes[other].label) break if best: found.add(best[1]) return found def evaluate(g: Graph, gold: dict) -> dict: sheets = {f"SHEET:{normalize_tag(s)}" for s in gold.get("sheets", [])} on_sheet = {n.id for n in g.kg.nodes if not sheets or set(n.sheets) & sheets} def find(tag: str): """Prefer the node with this tag that is drawn on the gold sheets.""" t = normalize_tag(tag) hits = [n.id for n in g.kg.nodes if n.tag == t and n.id in on_sheet] return hits[0] if hits else g.find(tag) pred_tags = {n.tag for n in g.kg.nodes if n.tag and n.id in on_sheet and n.type in ("Equipment", "Instrument", "Component")} gold_tags = {normalize_tag(t) for t in gold.get("tags", [])} res: dict = {"tags": prf(pred_tags, gold_tags)} scope = gold.get("feeds_scope", "all") feeds = [e for es in g.out.values() for e in es if e.type == "FEEDS"] # honours the PFD/P&ID view if scope == "within_sheets": feeds = [e for e in feeds if e.source in on_sheet and e.target in on_sheet] else: feeds = [e for e in feeds if e.source in on_sheet or e.target in on_sheet] if scope in ("equipment", "within_sheets"): feeds = [e for e in feeds if g.nodes[e.source].type == "Equipment" and g.nodes[e.target].type == "Equipment"] pred_feeds = {(g.nodes[e.source].tag or g.nodes[e.source].label, g.nodes[e.target].tag or g.nodes[e.target].label) for e in feeds} gold_feeds = {(normalize_tag(a) or a, normalize_tag(b) or b) for a, b in gold.get("feeds", [])} norm_pred = {(normalize_tag(a) or a, normalize_tag(b) or b) for a, b in pred_feeds} res["flow"] = prf(norm_pred, gold_feeds) placed, wrong = 0, [] for tag, eq in gold.get("instrument_at", {}).items(): nid = find(tag) if nid is None: wrong.append({"tag": tag, "expected": eq, "got": "missing"}) continue targets = [e.target for e in g.out[nid] if e.type == "MEASURES"] if g.nodes[nid].attributes.get("inline"): targets.append(nid) got = set().union(*(near_equipment(g, t) for t in targets)) if targets else set() if normalize_tag(eq) in {normalize_tag(x.split("@")[0]) for x in got}: placed += 1 else: wrong.append({"tag": tag, "expected": eq, "got": sorted(got) or "unlocated"}) n_inst = len(gold.get("instrument_at", {})) res["instrument_placement"] = {"accuracy": round(placed / n_inst, 3) if n_inst else None, "correct": placed, "total": n_inst, "errors": wrong} loops_found, overlaps, loop_detail = 0, [], [] for key, members in gold.get("loops", {}).items(): lid = next((n.id for n in g.kg.nodes if n.type == "ControlLoop" and n.id in on_sheet | {n.id} and n.label.split("@")[0] == key and set(n.sheets) & (sheets or set(n.sheets))), None) gm = {normalize_tag(m) for m in members} if lid is None: loop_detail.append({"loop": key, "found": False}) continue loops_found += 1 pm = {g.nodes[e.source].tag or g.nodes[e.source].label for e in g.inn[lid] if e.type == "MEMBER_OF"} jac = len(pm & gm) / len(pm | gm) if pm | gm else 1.0 overlaps.append(jac) loop_detail.append({"loop": key, "found": True, "member_jaccard": round(jac, 3), "missing_members": sorted(gm - pm), "extra_members": sorted(pm - gm)}) n_loops = len(gold.get("loops", {})) res["loops"] = {"recall": round(loops_found / n_loops, 3) if n_loops else None, "mean_member_jaccard": round(sum(overlaps) / len(overlaps), 3) if overlaps else None, "detail": loop_detail} return res def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("kg", type=Path) ap.add_argument("gold", type=Path) ap.add_argument("--json", action="store_true") a = ap.parse_args() if hasattr(sys.stdout, "reconfigure"): sys.stdout.reconfigure(encoding="utf-8") g = Graph.load(a.kg / "graph.json" if a.kg.is_dir() else a.kg) res = evaluate(g, json.loads(a.gold.read_text(encoding="utf-8"))) if a.json: print(json.dumps(res, indent=1)) return 0 t, f, ip, lp = res["tags"], res["flow"], res["instrument_placement"], res["loops"] print(f"Tags: P {t['precision']:.2f} R {t['recall']:.2f} F1 {t['f1']:.2f}" f" missed: {', '.join(t['missed']) or '-'} extra: {', '.join(t['false_positives']) or '-'}") print(f"Flow: P {f['precision']:.2f} R {f['recall']:.2f} F1 {f['f1']:.2f}" f" missed: {f['missed'] or '-'} extra: {f['false_positives'] or '-'}") if ip["total"]: print(f"Instrument placement: {ip['correct']}/{ip['total']}" + "".join(f"\n {w['tag']}: expected {w['expected']}, got {w['got']}" for w in ip["errors"])) if lp["recall"] is not None: print(f"Loops: recall {lp['recall']:.2f}, mean member overlap {lp['mean_member_jaccard']}") for d in lp["detail"]: if not d["found"] or d.get("missing_members") or d.get("extra_members"): print(f" {d}") return 0 if __name__ == "__main__": sys.exit(main())skills/pid-to-kg/scripts/export.py4.7 KB24d5caa75ea6upstream ↗
#!/usr/bin/env python3 """Export a knowledge graph to GraphML or a Neo4j import bundle. python export.py work/my-plant/kg graphml # -> graph.graphml (yEd, Gephi, NetworkX) python export.py work/my-plant/kg neo4j # -> neo4j/nodes.csv, edges.csv, import.cypher Node types become Neo4j labels and edge types become relationship types. Nested attributes are flattened to scalar properties where possible and kept whole as a JSON string in ``attributes_json``. """ from __future__ import annotations import argparse import csv import json import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) from pidkg.store import Graph # noqa: E402 def _scalar(v): if v is None or isinstance(v, (str, int, float, bool)): return v return json.dumps(v) def _node_props(n) -> dict: props = {"id": n.id, "type": n.type, "label": n.label, "tag": n.tag, "kind": n.kind, "name": n.name, "status": n.status, "dexpi_class": n.dexpi_class, "sheets": ";".join(n.sheets), "confidence": n.confidence, "attributes_json": json.dumps(n.attributes)} for k, v in n.attributes.items(): props.setdefault(f"a_{k}", _scalar(v)) return {k: v for k, v in props.items() if v is not None} def export_graphml(g: Graph, path: Path) -> None: import networkx as nx G = nx.MultiDiGraph(plant=g.kg.plant.name) for n in g.kg.nodes: G.add_node(n.id, **_node_props(n)) for e in g.kg.edges: attrs = {"type": e.type, "confidence": e.confidence, "derived": e.derived, "attributes_json": json.dumps(e.attributes)} if e.sheet: attrs["sheet"] = e.sheet G.add_edge(e.source, e.target, key=e.id, **attrs) nx.write_graphml(G, path) def export_neo4j(g: Graph, out: Path) -> None: out.mkdir(parents=True, exist_ok=True) keys = ["id", "type", "label", "tag", "kind", "name", "status", "dexpi_class", "sheets", "confidence", "attributes_json"] with open(out / "nodes.csv", "w", newline="", encoding="utf-8") as f: w = csv.DictWriter(f, fieldnames=keys, extrasaction="ignore") w.writeheader() for n in g.kg.nodes: w.writerow(_node_props(n)) with open(out / "edges.csv", "w", newline="", encoding="utf-8") as f: w = csv.writer(f) w.writerow(["id", "source", "target", "type", "confidence", "derived", "sheet", "attributes_json"]) for e in g.kg.edges: w.writerow([e.id, e.source, e.target, e.type, e.confidence, e.derived, e.sheet or "", json.dumps(e.attributes)]) node_types = sorted({n.type for n in g.kg.nodes}) edge_types = sorted({e.type for e in g.kg.edges}) plant = g.kg.plant.id lines = [ f"// Neo4j import for plant '{g.kg.plant.name}'. Copy nodes.csv and edges.csv into the", "// Neo4j import directory, then run this script (e.g. with cypher-shell).", "CREATE CONSTRAINT pid_node_id IF NOT EXISTS FOR (n:PidNode) REQUIRE (n.plant, n.id) IS UNIQUE;", "", "LOAD CSV WITH HEADERS FROM 'file:///nodes.csv' AS row", f"MERGE (n:PidNode {{plant: '{plant}', id: row.id}})", "SET n += row, n.confidence = toFloat(row.confidence);", "", ] for t in node_types: lines.append(f"MATCH (n:PidNode {{plant: '{plant}', type: '{t}'}}) SET n:{t};") lines.append("") for t in edge_types: lines += [ "LOAD CSV WITH HEADERS FROM 'file:///edges.csv' AS row", f"WITH row WHERE row.type = '{t}'", f"MATCH (a:PidNode {{plant: '{plant}', id: row.source}}), (b:PidNode {{plant: '{plant}', id: row.target}})", f"MERGE (a)-[r:{t} {{id: row.id}}]->(b)", "SET r.confidence = toFloat(row.confidence), r.derived = row.derived = 'True',", " r.sheet = row.sheet, r.attributes_json = row.attributes_json;", "", ] (out / "import.cypher").write_text("\n".join(lines), encoding="utf-8") def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("kg", type=Path, help="kg/ directory or graph.json") ap.add_argument("format", choices=["graphml", "neo4j"]) a = ap.parse_args() if hasattr(sys.stdout, "reconfigure"): sys.stdout.reconfigure(encoding="utf-8") json_path = a.kg / "graph.json" if a.kg.is_dir() else a.kg g = Graph.load(json_path) if a.format == "graphml": target = json_path.with_name("graph.graphml") export_graphml(g, target) else: target = json_path.with_name("neo4j") export_neo4j(g, target) print(f"wrote {target}") return 0 if __name__ == "__main__": sys.exit(main())skills/pid-to-kg/scripts/kg_query.py8.8 KBd27d262fad34upstream ↗
#!/usr/bin/env python3 """Query a P&ID knowledge graph (for people, and as a tool for LLM agents). Usage: python kg_query.py KG stats python kg_query.py KG describe 660-TK-004 python kg_query.py KG upstream FV-0502 [--equipment] [--depth 3] python kg_query.py KG downstream 660-TK-001 [--equipment] python kg_query.py KG path 660-TK-001 660-TK-004 python kg_query.py KG loops [--type control] python kg_query.py KG loop F-0502 python kg_query.py KG instruments [--on 660-PP-011] [--variable flow] python kg_query.py KG sql "SELECT * FROM equipment_flow" KG is a kg/ directory, graph.json or graph.duckdb. Add --json for machine-readable output. Item references accept ids, tags or labels; tags are matched loosely (FIT0502 = FIT-0502). """ from __future__ import annotations import argparse import json import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) from pidkg.loops import point_label, sensed_points # noqa: E402 from pidkg.store import Graph # noqa: E402 TERMINAL = ("Equipment", "Boundary") def _resolve_paths(kg: Path) -> tuple[Path, Path]: if kg.is_dir(): return kg / "graph.json", kg / "graph.duckdb" if kg.suffix == ".duckdb": return kg.with_name("graph.json"), kg return kg, kg.with_name("graph.duckdb") def _node_brief(g: Graph, nid: str) -> dict: n = g.nodes[nid] d = {"id": n.id, "type": n.type, "label": n.label} if n.kind: d["kind"] = n.kind if n.name: d["name"] = n.name return d def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("kg", type=Path) ap.add_argument("command", choices=["stats", "describe", "upstream", "downstream", "path", "loops", "loop", "instruments", "sql"]) ap.add_argument("args", nargs="*") ap.add_argument("--equipment", action="store_true", help="only equipment and boundaries") ap.add_argument("--depth", type=int) ap.add_argument("--type", help="loop type filter") ap.add_argument("--on", help="instruments measuring this item (or anything on it)") ap.add_argument("--variable", help="instrument variable filter, e.g. flow") ap.add_argument("--include-pfd", action="store_true", help="also traverse PFD-sheet flow when the plant has P&IDs (default: P&ID detail only)") ap.add_argument("--json", action="store_true") a = ap.parse_args() if hasattr(sys.stdout, "reconfigure"): sys.stdout.reconfigure(encoding="utf-8") json_path, db_path = _resolve_paths(a.kg) if a.command == "sql": import duckdb con = duckdb.connect(str(db_path), read_only=True) rel = con.sql(" ".join(a.args)) if a.json: cols = rel.columns print(json.dumps([dict(zip(cols, r)) for r in rel.fetchall()], indent=1, default=str)) else: rel.show(max_width=200, max_rows=200) return 0 g = Graph.load(json_path, include_pfd=a.include_pfd) def need(n: int) -> list[str]: if len(a.args) < n: ap.error(f"'{a.command}' needs {n} item reference(s)") ids = [] for ref in a.args[:n]: nid = g.find(ref) if nid is None: print(f"error: '{ref}' not found in the graph", file=sys.stderr) sys.exit(2) ids.append(nid) return ids result: object if a.command == "stats": result = {"plant": g.kg.plant.model_dump(), **g.kg.stats} elif a.command == "describe": (nid,) = need(1) n = g.nodes[nid] result = {**n.model_dump(exclude_none=True, exclude={"provenance"}), "relations": g.neighbors(nid)} if n.type == "PipeSegment": result["location"] = point_label(nid, g.nodes, g.kg.edges) elif a.command in ("upstream", "downstream"): (nid,) = need(1) hits = g.walk(nid, "down" if a.command == "downstream" else "up", max_depth=a.depth) rows = [dict(_node_brief(g, h), hops=d) for h, d in hits if not a.equipment or g.nodes[h].type in TERMINAL] result = {"from": _node_brief(g, nid), "direction": a.command, "items": rows} elif a.command == "path": x, y = need(2) p = g.path(x, y) or g.path(y, x) result = {"path": [_node_brief(g, i) for i in p]} if p else {"path": None, "note": "no directed flow path between these items"} elif a.command == "loops": loops = [n for n in g.kg.nodes if n.type == "ControlLoop" and (not a.type or n.kind == a.type)] result = [{"id": n.id, "loop": n.label, "type": n.kind, "description": n.attributes.get("description")} for n in loops] elif a.command == "loop": ref = a.args[0] if a.args else ap.error("'loop' needs a loop id") nid = g.find(ref) or g.find(f"LOOP:{ref}") if nid is None or g.nodes[nid].type != "ControlLoop": # Accept a member tag, e.g. 'loop FIC-0502'. member = g.find(ref) nid = next((e.target for e in g.out.get(member, []) if e.type == "MEMBER_OF"), None) \ if member else None if nid is None: print(f"error: loop '{ref}' not found", file=sys.stderr) return 2 n = g.nodes[nid] members = [dict(_node_brief(g, e.source), role=e.attributes.get("role")) for e in g.inn[nid] if e.type == "MEMBER_OF"] result = {**n.model_dump(exclude_none=True, exclude={"provenance"}), "members": members} else: # instruments target_ids: set[str] | None = None if a.on: t = g.find(a.on) if t is None: print(f"error: '{a.on}' not found", file=sys.stderr) return 2 # The item plus its own piping: everything reachable up- and downstream # before the next piece of equipment. target_ids = {t} | _local_piping(g, t) rows = [] for n in g.kg.nodes: if n.type != "Instrument": continue if a.variable and a.variable.lower() not in (n.attributes.get("variable") or "").lower(): continue meas = {e.target for e in g.out[n.id] if e.type == "MEASURES"} if target_ids is not None and not ((meas | {n.id}) & target_ids): continue rows.append({"tag": n.label, "variable": n.attributes.get("variable"), "role": n.attributes.get("role"), "location": n.attributes.get("location"), "measures": sensed_points(n.id, g.nodes, g.kg.edges), "loop": [g.nodes[e.target].label for e in g.out[n.id] if e.type == "MEMBER_OF"]}) result = rows if a.json: print(json.dumps(result, indent=1, default=str)) else: _print_text(result) return 0 def _local_piping(g: Graph, start: str, limit: int = 8) -> set[str]: """Segments and fittings within ``limit`` hops of an item, stopping at equipment.""" found: set[str] = set() for direction in ("down", "up"): frontier = [start] while frontier and len(found) < 500: nxt = [] for cur in frontier: edges = g.out[cur] if direction == "down" else g.inn[cur] for e in edges: if e.type != "FLOWS_TO": continue other = e.target if direction == "down" else e.source if other in found or other == start or g.nodes[other].type in TERMINAL: continue found.add(other) nxt.append(other) frontier = nxt limit -= 1 if limit <= 0: break return found def _print_text(obj, indent: int = 0) -> None: pad = " " * indent if isinstance(obj, list): if not obj: print(pad + "(none)") for item in obj: if isinstance(item, dict): print(pad + "- " + ", ".join(f"{k}: {_fmt(v)}" for k, v in item.items())) else: print(pad + f"- {item}") elif isinstance(obj, dict): for k, v in obj.items(): if isinstance(v, (dict, list)) and v and any(isinstance(x, (dict, list)) for x in (v.values() if isinstance(v, dict) else v)): print(f"{pad}{k}:") _print_text(v, indent + 1) else: print(f"{pad}{k}: {_fmt(v)}") else: print(pad + str(obj)) def _fmt(v) -> str: if isinstance(v, list): return "[" + ", ".join(_fmt(x) for x in v) + "]" if isinstance(v, dict): return "{" + ", ".join(f"{k}: {_fmt(x)}" for k, x in v.items()) + "}" return str(v) if __name__ == "__main__": sys.exit(main())skills/pid-to-kg/scripts/make_hints.py4.0 KBc45e830a7b63upstream ↗
#!/usr/bin/env python3 """Collect deterministic text hints for each rendered page. Hints are candidate tags with page positions, gathered without the LLM: * text layer (vector_text, raster_with_text, hybrid pages): exact text and positions straight from the PDF; * OCR (vector_outlined, raster, hybrid pages; needs rapidocr-onnxruntime): each non-blank tile is OCR'd and mapped back to page coordinates. ISA bubbles split over two lines ('FIT' over '0502') are joined. QA compares these candidates with the extraction to flag tags that may have been missed. Writes <work>/hints/page_NN.json. Usage: python make_hints.py work/my-plant [--pages 1-3] [--no-ocr] """ from __future__ import annotations import argparse import json import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) from pidkg import ocr # noqa: E402 from pidkg.pdf import text_runs # noqa: E402 from render_pages import parse_pages # noqa: E402 TEXT_LAYER_CLASSES = {"vector_text", "raster_with_text", "hybrid"} OCR_CLASSES = {"vector_outlined", "raster", "hybrid"} def tiles_for(bbox: list[float], tiles: list[dict]) -> list[str]: cx, cy = (bbox[0] + bbox[2]) / 2, (bbox[1] + bbox[3]) / 2 return [t["name"] for t in tiles if t["norm"][0] <= cx <= t["norm"][2] and t["norm"][1] <= cy <= t["norm"][3]] def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("work", type=Path) ap.add_argument("--pages") ap.add_argument("--no-ocr", action="store_true") args = ap.parse_args() if hasattr(sys.stdout, "reconfigure"): sys.stdout.reconfigure(encoding="utf-8", line_buffering=True) from PIL import Image manifest = json.loads((args.work / "manifest.json").read_text(encoding="utf-8")) use_ocr = not args.no_ocr and ocr.available() if not args.no_ocr and not use_ocr: print(f"note: {ocr.maybe_import_error()}") (args.work / "hints").mkdir(exist_ok=True) for page_no in parse_pages(args.pages, len(manifest["pages"]), manifest.get("selected_pages")): info = manifest["pages"][page_no - 1] cls = info["page_class"] page_dir = args.work / "pages" / f"page_{page_no:02d}" tiles_file = page_dir / "tiles.json" if not tiles_file.exists(): print(f"page {page_no}: not rendered yet (run render_pages.py), skipped") continue meta = json.loads(tiles_file.read_text(encoding="utf-8")) runs: list[dict] = [] sources: list[str] = [] if cls in TEXT_LAYER_CLASSES: runs += [dict(r, source="text_layer") for r in text_runs(manifest["source_pdf"], page_no, meta["rotation"])] sources.append("text_layer") if cls in OCR_CLASSES and use_ocr: for t in meta["tiles"]: if t.get("blank"): continue img = Image.open(page_dir / "tiles" / f"{t['name']}.png") for r in ocr.ocr_image(img): if r["conf"] < 0.5 or not r["text"]: continue runs.append({"text": r["text"], "conf": round(r["conf"], 3), "source": "ocr", "bbox": ocr.to_norm(r["bbox"], t["px"], meta["width"], meta["height"])}) sources.append("ocr") runs = ocr.dedupe_runs(runs) runs = ocr.join_bubbles(runs) cands = ocr.tag_candidates(runs) for c in cands: c["tiles"] = tiles_for(c["bbox"], meta["tiles"]) out = {"page": page_no, "source": "+".join(sources) or "none", "page_class": cls, "tag_candidates": cands, "runs": runs} (args.work / "hints" / f"page_{page_no:02d}.json").write_text( json.dumps(out, indent=1), encoding="utf-8") print(f"page {page_no}: {len(runs)} text runs, {len(cands)} tag candidates ({out['source']})") return 0 if __name__ == "__main__": sys.exit(main())skills/pid-to-kg/scripts/pidkg/__init__.py153 Bcdc430366f41upstream ↗
"""pidkg: build process knowledge graphs from P&ID extractions.""" __version__ = "0.1.0" EXTRACTION_SCHEMA_VERSION = "1.0" GRAPH_SCHEMA_VERSION = "1.0"skills/pid-to-kg/scripts/pidkg/build.py44 KB7ecaf83f96f3upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/pid-to-kg/scripts/pidkg/isa.py4.9 KB3ea93fc81e20upstream ↗
"""Tag normalization and ISA-5.1 instrument tag parsing.""" from __future__ import annotations import re from dataclasses import dataclass, field from typing import Optional from .ontology import ISA_FUNCTIONS, ISA_MEASURED, ISA_VARIABLE_MODIFIERS _DASHES = dict.fromkeys(map(ord, "‐‑‒–—−"), "-") _COMPACT = re.compile(r"^([A-Z]{1,6})(\d{1,6}[A-Z]{0,2})$") def normalize_tag(raw: Optional[str]) -> Optional[str]: """Normalize a printed tag so the same item read twice compares equal. ``"fit 0502"``, ``"FIT-0502"``, ``"FIT_0502"`` and ``"FIT0502"`` all become ``"FIT-0502"``. Area prefixes and suffixes are preserved (``"660 TK 004"`` -> ``"660-TK-004"``). """ if raw is None: return None s = str(raw).translate(_DASHES).strip().strip("\"'").upper() if not s: return None s = re.sub(r"[\s_/]*-[\s_/]*", "-", s) s = re.sub(r"[\s_]+", "-", s) s = re.sub(r"-{2,}", "-", s).strip("-") m = _COMPACT.match(s) if m: s = f"{m.group(1)}-{m.group(2)}" return s def normalize_label(raw: Optional[str]) -> str: """Normalize free text (boundary names, off-page labels) for matching.""" s = (raw or "").translate(_DASHES).upper() s = re.sub(r"[^A-Z0-9#\-\. ]+", " ", s) return re.sub(r"\s+", " ", s).strip() _ISA_TAG = re.compile( r"^(?:(?P<prefix>[0-9][0-9A-Z]*)-)?" r"(?P<letters>[A-Z]{1,6})-" r"(?P<number>[A-Z]?\d{1,5})(?P<suffix>[A-Z]{0,2})" r"(?:-(?P<ext>[A-Z0-9]{1,4}))?$" ) _RELIEF = {"PSV", "PRV", "PSE", "TSV", "TSE", "VRV", "PVRV", "PVSV"} ROLE_ORDER = ( "relief_device", "hand_switch", "final_element", "relay", "controller", "transmitter", "switch", "element", "gauge", "indicator", "recorder", "totalizer", "alarm", "other", ) @dataclass class IsaTag: tag: str prefix: Optional[str] letters: str number: str suffix: str variable_code: str variable: str modifiers: list[str] = field(default_factory=list) functions: list[str] = field(default_factory=list) role: str = "other" @property def loop_key(self) -> str: base = f"{self.variable_code[0]}-{self.number}" return f"{self.prefix}-{base}" if self.prefix else base def parse_isa_tag(tag: Optional[str]) -> Optional[IsaTag]: """Parse an ISA-5.1 style instrument tag. Returns None if it does not fit.""" t = normalize_tag(tag) if not t: return None m = _ISA_TAG.match(t) if not m: return None letters = m.group("letters") first, rest = letters[0], letters[1:] if first not in ISA_MEASURED: return None variable_code = first modifiers: list[str] = [] state = None # Open/closed state suffix on position and event tags: ZSO, ZSC, ZIO, ZIC, XSO, XSC. if first in "XYZ" and len(rest) >= 2 and rest[-1] in "OC" and rest[-2] in "IS": state = "open" if rest[-1] == "O" else "closed" rest = rest[:-1] if letters in _RELIEF: functions = ["safety", "valve"] role = "relief_device" variable_code = first else: if rest and rest[0] in ISA_VARIABLE_MODIFIERS and len(rest) > 1: variable_code = first + rest[0] modifiers.append(ISA_VARIABLE_MODIFIERS[rest[0]]) rest = rest[1:] functions = [ISA_FUNCTIONS.get(ch, ch) for ch in rest] if state: functions.append(state) role = _role(rest) if first == "H" and role in ("switch", "controller", "indicator", "other"): role = "hand_switch" # operator input (HS, HIC), not a process sensor variable = ISA_MEASURED[first] if modifiers: variable = f"{modifiers[0]} {variable}" return IsaTag( tag=t, prefix=m.group("prefix"), letters=letters, number=m.group("number"), suffix=m.group("suffix") or "", variable_code=variable_code, variable=variable, modifiers=modifiers, functions=functions, role=role, ) def _role(fn: str) -> str: if fn.endswith("V") or fn.endswith("Z"): return "final_element" if "Y" in fn: return "relay" if "C" in fn or fn.endswith("K"): return "controller" if "T" in fn: return "transmitter" if "S" in fn: return "switch" if "E" in fn: return "element" if "G" in fn: return "gauge" if "I" in fn: return "indicator" if "R" in fn: return "recorder" if "Q" in fn: return "totalizer" if "A" in fn: return "alarm" return "other" # Roles that sense the process directly and therefore should have a MEASURES edge. SENSING_ROLES = {"transmitter", "switch", "element", "gauge"} # Candidate tags in free text (OCR or vector text). Deliberately loose; used only # to suggest items the extraction may have missed. TAG_CANDIDATE = re.compile( r"\b(?:\d{2,4}[- ]?)?[A-Z]{1,5}[- ]?\d{2,5}[A-Z]?\b" )skills/pid-to-kg/scripts/pidkg/loops.py15 KBf95338ee95acupstream ↗
"""Control-loop inference. Loops are inferred in two steps: 1. **Tag grouping (ISA-5.1).** Instruments and actuated components sharing a measured-variable letter and loop number (FT-101, FIC-101, FV-101) form a loop ``F-101``. An explicit ``loop`` on an item overrides this. 2. **Signal tracing.** Signal lines refine the groups: a controller's output is followed (through relays such as I/P converters) to the final element it moves, even when the numbers differ; controller-to-controller signals become cascades; and sensors wired into a controller from another loop number are added as inputs. """ from __future__ import annotations from collections import defaultdict from typing import TYPE_CHECKING, Optional from .models import Node from .ontology import ACTUATED_COMPONENT_KINDS if TYPE_CHECKING: # pragma: no cover from .build import GraphBuilder SENSOR_ROLES = {"transmitter", "element", "switch", "gauge", "indicator", "recorder", "totalizer", "alarm"} READOUT_ROLES = {"indicator", "recorder", "totalizer", "alarm"} PANEL_LOCATIONS = {"main_panel", "behind_panel", "dcs", "plc", "scada", "sis", "computer"} def _member_role(n: Node) -> str: """Role within a loop: panel indicators and alarms are read-outs, not sensors.""" r = _role(n) if r in READOUT_ROLES and n.attributes.get("location") in PANEL_LOCATIONS: return "readout" return r def loop_variable(key: str) -> Optional[str]: """Measured variable named by a loop key such as 'F-101', '660-F-0502' or 'S-1@M702'.""" from .ontology import ISA_MEASURED, ISA_VARIABLE_MODIFIERS core = key.split("@")[0].split("#")[0] parts = core.split("-") code = parts[-2] if len(parts) >= 2 else "" if not code or code[0] not in ISA_MEASURED: return None var = ISA_MEASURED[code[0]] if len(code) > 1 and code[1] in ISA_VARIABLE_MODIFIERS: var = f"{ISA_VARIABLE_MODIFIERS[code[1]]} {var}" return var def _role(n: Node) -> str: if n.type == "Component": if n.kind == "relief_valve" or n.attributes.get("role") == "relief_device": return "relief_device" return "final_element" if n.type == "Equipment": return "final_element" return n.attributes.get("role", "other") def _is_final(n: Node) -> bool: if n.type == "Component": return n.kind in ACTUATED_COMPONENT_KINDS or n.attributes.get("role") == "final_element" \ or n.attributes.get("actuator") not in (None, "manual", "none", "self") if n.type == "Equipment": return True # e.g. a pump motor / VFD driven by a speed controller return n.attributes.get("role") == "final_element" def derive_loops(b: "GraphBuilder") -> None: nodes = b.nodes sig_out: dict[str, list[str]] = defaultdict(list) sig_in: dict[str, list[str]] = defaultdict(list) sig_purpose: dict[tuple[str, str], str] = {} for e in b.edges: if e.type == "SIGNAL_TO": sig_out[e.source].append(e.target) sig_in[e.target].append(e.source) sig_purpose[(e.source, e.target)] = e.attributes.get("purpose", "unknown") # 1. Group by loop key. groups: dict[str, dict[str, str]] = defaultdict(dict) # loop key -> {node id: role} for n in nodes.values(): key = n.attributes.get("loop_key") if key and n.type in ("Instrument", "Component"): groups[key][n.id] = _member_role(n) # 2. Refine with signals. cascades: set[tuple[str, str]] = set() loop_of = {nid: key for key, members in groups.items() for nid in members} for key, members in list(groups.items()): for cid in [m for m, r in members.items() if r == "controller"]: for target, via in _trace_outputs(cid, sig_out, nodes): tn = nodes[target] if tn.type == "Instrument" and tn.attributes.get("role") == "controller": other = loop_of.get(target) if other and other != key: cascades.add((key, other)) continue if _is_final(tn): members.setdefault(target, "final_element") for r in via: members.setdefault(r, "relay") b._edge(cid, target, "ACTUATES", derived=True, via=[nodes[r].label for r in via] or None) for src in sig_in.get(cid, []): sn = nodes[src] if sn.type == "Instrument" and sn.attributes.get("role") in SENSOR_ROLES: members.setdefault(src, "sensor_input") # Sensors or hand switches wired straight to a final element (interlocks, # on/off logic in a PLC that is not drawn as a controller). for sid in [m for m, r in members.items() if r in ("switch", "hand_switch", "transmitter", "alarm", "other")]: for target, via in _trace_outputs(sid, sig_out, nodes): if _is_final(nodes[target]) and target not in members: members[target] = "final_element" b._edge(sid, target, "ACTUATES", derived=True, via=[nodes[r].label for r in via] or None) # 3. Emit loop nodes for meaningful groups. loop_ids: dict[str, str] = {} for key, members in sorted(groups.items()): roles = set(members.values()) if len(members) < 2 and "controller" not in roles: continue lid = f"LOOP:{key}" loop_ids[key] = lid loop_type = _loop_type(members, nodes, sig_purpose) variable = loop_variable(key) or next((nodes[m].attributes.get("variable") for m in members if nodes[m].attributes.get("variable")), None) sheets: list[str] = [] for m in members: for s in nodes[m].sheets: if s not in sheets: sheets.append(s) sensors = [m for m, r in members.items() if r in SENSOR_ROLES or r == "sensor_input"] controllers = [m for m, r in members.items() if r == "controller"] finals = [m for m, r in members.items() if r == "final_element"] measured_at = _measured_points(sensors, b) manipulated_at = _manipulated_points(finals, b) node = Node( id=lid, type="ControlLoop", label=key, kind=loop_type, sheets=sheets, confidence=min(nodes[m].confidence for m in members), dexpi_class="InstrumentationLoopFunction", attributes={k: v for k, v in dict( variable=variable, loop_type=loop_type, sensors=sensors, controllers=controllers, final_elements=finals, relays=[m for m, r in members.items() if r == "relay"], readouts=[m for m, r in members.items() if r == "readout"], measured_at=measured_at, manipulated_at=manipulated_at).items() if v}) node.attributes["description"] = _describe(node, b) b._upsert(node) for m, role in members.items(): b._edge(m, lid, "MEMBER_OF", derived=True, role=role) if loop_type == "interlock": for s in sensors: for f in finals: b._edge(s, f, "TRIPS", derived=True, loop=lid) for master, slave in sorted(cascades): if master in loop_ids and slave in loop_ids: b._edge(loop_ids[master], loop_ids[slave], "CASCADES_TO", derived=True) b.nodes[loop_ids[master]].attributes.setdefault("cascade_slaves", []).append(loop_ids[slave]) b.nodes[loop_ids[slave]].attributes["cascade_master"] = loop_ids[master] for lid in loop_ids.values(): ln = b.nodes[lid] if ln.attributes.get("cascade_slaves") or ln.attributes.get("cascade_master"): ln.attributes["description"] = _describe(ln, b) # 4. QA on loop completeness. for key, lid in loop_ids.items(): ln = b.nodes[lid] a = ln.attributes if ln.kind == "control": if not a.get("final_elements") and not a.get("cascade_slaves"): b.issue("warning", "loop_no_final_element", f"control loop {key} has a controller but no final element", node=lid) if not a.get("sensors"): # Common for speed/position controllers that only take an operator setpoint. b.issue("info", "loop_no_sensor", f"control loop {key} has a controller but no sensor", node=lid) for c in a.get("controllers", []): if not sig_in.get(c) and not sig_out.get(c): b.issue("info", "loop_no_signals", f"controller {c} has no signal lines; loop {key} built from tag numbers only", node=c) def _trace_outputs(start: str, sig_out: dict[str, list[str]], nodes: dict[str, Node], max_depth: int = 4) -> list[tuple[str, list[str]]]: """Follow signals from ``start`` through relays; return (endpoint, relays) pairs.""" results: list[tuple[str, list[str]]] = [] stack = [(t, []) for t in sig_out.get(start, [])] seen = {start} while stack: nid, via = stack.pop() if nid in seen or len(via) > max_depth: continue seen.add(nid) n = nodes[nid] if n.type == "Instrument" and n.attributes.get("role") in ("relay", "other") and sig_out.get(nid): for t in sig_out[nid]: stack.append((t, via + [nid])) continue results.append((nid, via)) return results def _loop_type(members: dict[str, str], nodes: dict[str, Node], sig_purpose: dict[tuple[str, str], str]) -> str: roles = set(members.values()) if any(p == "trip" for (a, _), p in sig_purpose.items() if a in members): return "interlock" if "controller" in roles: return "control" variables = {nodes[m].attributes.get("variable") for m in members} if "final_element" in roles: if "hand" in variables or "hand_switch" in roles: return "manual" process_switch = any(r in ("switch", "alarm") and nodes[m].attributes.get("variable") not in ("event/state", "unclassified", "position", "hand") for m, r in members.items()) if process_switch: return "interlock" if roles & {"transmitter", "element", "sensor_input"}: # Sensor and final element share a loop but no controller is drawn: # the control function lives in a PLC/DCS not shown on the sheet. return "control" return "on_off" return "monitoring" def sensed_points(nid: str, nodes: dict[str, Node], edges: list) -> list[str]: """Where an instrument senses the process, as human-readable labels.""" if nodes[nid].attributes.get("inline"): return [point_label(nid, nodes, edges)] return [point_label(e.target, nodes, edges) for e in edges if e.type == "MEASURES" and e.source == nid] def _measured_points(sensors: list[str], b: "GraphBuilder") -> list[str]: out: list[str] = [] for s in sensors: for lbl in sensed_points(s, b.nodes, b.edges): if lbl not in out: out.append(lbl) return out def _manipulated_points(finals: list[str], b: "GraphBuilder") -> list[str]: out: list[str] = [] for f in finals: n = b.nodes[f] if n.type == "Equipment": continue lines = {b.nodes[e.source].attributes.get("line_number") for e in b.edges if e.type == "FLOWS_TO" and e.target == f and b.nodes[e.source].type == "PipeSegment"} lines.discard(None) if lines: out.append(f"{n.label} on {', '.join(sorted(lines))}") return out def point_label(nid: str, nodes: dict[str, Node], edges: list) -> str: """Human-readable process location. An item gives its label; a pipe segment or an in-line instrument gives ``line (nearest upstream item -> nearest downstream item)``. """ n = nodes[nid] inline = n.type == "Instrument" and n.attributes.get("inline") if n.type != "PipeSegment" and not inline: return n.label up = _nearest(nid, nodes, edges, upstream=True) down = _nearest(nid, nodes, edges, upstream=False) line = n.attributes.get("line_number") if not line: line = next((nodes[e.source].attributes.get("line_number") for e in edges if e.type == "FLOWS_TO" and e.target == nid and nodes[e.source].type == "PipeSegment"), None) where = f"{line or 'line'} ({up or '?'} -> {down or '?'})" return f"in-line on {where}" if inline else where def _nearest(start: str, nodes: dict[str, Node], edges: list, upstream: bool, limit: int = 50) -> Optional[str]: """Nearest equipment/component/boundary label upstream or downstream of a segment.""" adj: dict[str, list[str]] = defaultdict(list) for e in edges: if e.type == "FLOWS_TO": if upstream: adj[e.target].append(e.source) else: adj[e.source].append(e.target) frontier, seen = [start], {start} for _ in range(limit): nxt = [] for nid in frontier: for other in adj[nid]: if other in seen: continue seen.add(other) if nodes[other].type in ("Equipment", "Boundary", "Component", "Instrument", "OffPageConnector"): return nodes[other].label nxt.append(other) if not nxt: return None frontier = nxt return None def _describe(loop: Node, b: "GraphBuilder") -> str: a = loop.attributes lbl = lambda ids: ", ".join(b.nodes[i].label for i in ids) # noqa: E731 var = a.get("variable") or "process" parts = [f"{loop.kind.replace('_', ' ').capitalize()} loop {loop.label} ({var})."] if a.get("sensors"): where = f" ({'; '.join(a['measured_at'])})" if a.get("measured_at") else "" parts.append(f"Sensed by {lbl(a['sensors'])}{where}.") if a.get("controllers"): locs = {b.nodes[c].attributes.get('location', 'unknown') for c in a["controllers"]} parts.append(f"Controlled by {lbl(a['controllers'])} ({', '.join(sorted(locs))}).") if a.get("final_elements"): placed = {m.split(" on ")[0]: m for m in a.get("manipulated_at", [])} acts = [placed.get(b.nodes[f].label, b.nodes[f].label) for f in a["final_elements"]] parts.append(f"Acts on {'; '.join(acts)}.") if a.get("cascade_slaves"): parts.append(f"Cascade master: sets the setpoint of {lbl(a['cascade_slaves'])}.") if a.get("cascade_master"): parts.append(f"Cascade slave of {b.nodes[a['cascade_master']].label}.") return " ".join(parts)skills/pid-to-kg/scripts/pidkg/models.py9.5 KB99bc109f63b0upstream ↗
"""Pydantic models for page extractions and the knowledge graph. Two schemas live here: * ``PageExtraction`` is what Claude writes for every P&ID sheet after reading it visually (``extract/page_NN.json``). It mirrors how an engineer reads a drawing: items on the sheet, the pipe runs between them, and the signal lines between instruments. * ``KnowledgeGraph`` is the canonical output (``graph.json``) produced by ``build_kg.py``. It is a typed property graph in node-link form. JSON Schemas for both are generated into ``references/schema/`` by ``python scripts/build_kg.py --write-schemas``. """ from __future__ import annotations from typing import Any, Literal, Optional, Union from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator from . import EXTRACTION_SCHEMA_VERSION, GRAPH_SCHEMA_VERSION BBox = list[float] # normalized [x0, y0, x1, y1] on the upright page, 0..1 class _Strict(BaseModel): model_config = ConfigDict(extra="forbid", populate_by_name=True) def _check_bbox(v: Optional[BBox]) -> Optional[BBox]: if v is None: return v if len(v) != 4: raise ValueError("bbox must be [x0, y0, x1, y1]") if any(c < -0.01 or c > 1.01 for c in v): raise ValueError("bbox must be normalized to 0..1 of the page") x0, y0, x1, y1 = v return [min(x0, x1), min(y0, y1), max(x0, x1), max(y0, y1)] # ========================================================================== # Page extraction (written by Claude, one file per sheet) # ========================================================================== class ProcessRef(_Strict): """A point in the process: an item, or a position on a pipe run. Use ``item`` for equipment, components and in-line instruments. Use ``line`` (plus optional ``after``) for a tap or tee on a pipe run, where ``after`` is the path item immediately upstream of the tap. """ item: Optional[str] = None line: Optional[str] = None after: Optional[str] = None @model_validator(mode="after") def _one_target(self) -> "ProcessRef": if bool(self.item) == bool(self.line): raise ValueError("ProcessRef needs exactly one of 'item' or 'line'") if self.after and not self.line: raise ValueError("'after' is only valid together with 'line'") return self Endpoint = Union[str, ProcessRef] class _Item(_Strict): id: str = Field(description="Unique within the page. Use the tag when the item has one.") tag: Optional[str] = Field(None, description="Tag exactly as printed, e.g. 'FIT-0502'.") name: Optional[str] = Field(None, description="Descriptive text printed with the item.") status: Optional[Literal["new", "existing", "future", "demolish", "unknown"]] = None confidence: float = Field(0.9, ge=0.0, le=1.0) bbox: Optional[BBox] = None notes: Optional[str] = None attributes: dict[str, Any] = Field(default_factory=dict) @field_validator("bbox") @classmethod def _bbox_ok(cls, v: Optional[BBox]) -> Optional[BBox]: return _check_bbox(v) class Equipment(_Item): kind: str = Field(description="Vocabulary in references/ontology.md, e.g. 'tank', 'centrifugal_pump'.") part_of: Optional[str] = Field(None, description="Id of the equipment this is mounted on (agitator on a tank, drive on a pump).") class Component(_Item): kind: str = Field(description="Vocabulary in references/ontology.md, e.g. 'gate_valve', 'control_valve'.") actuator: Optional[ Literal["manual", "pneumatic", "electric", "hydraulic", "solenoid", "self", "none"] ] = None fail_position: Optional[Literal["FO", "FC", "FL", "FI"]] = None normal_position: Optional[Literal["open", "closed"]] = None size: Optional[str] = None loop: Optional[str] = Field(None, description="Override the ISA loop this item belongs to.") class Instrument(_Item): tag: Optional[str] = Field(None, description="Printed tag. Leave empty for untagged gauges and give 'function'.") function: Optional[str] = Field(None, description="ISA letters for an untagged instrument, e.g. 'PI' for a bare pressure gauge.") location: str = Field("unknown", description="field | local_panel | main_panel | behind_panel | dcs | plc | scada | sis | computer | unknown") measures: Optional[ProcessRef] = Field(None, description="Where the instrument senses the process.") loop: Optional[str] = Field(None, description="Override the ISA loop this instrument belongs to.") @model_validator(mode="after") def _tag_or_function(self) -> "Instrument": if not self.tag and not self.function: raise ValueError("an instrument needs a 'tag', or a 'function' (e.g. 'PI') if untagged") return self class Line(_Strict): id: str = Field(description="Unique within the page, e.g. 'L1'.") line_number: Optional[str] = None size: Optional[str] = None service: Optional[str] = Field(None, description="Fluid/service code or description.") from_: Endpoint = Field(alias="from") to: Endpoint path: list[str] = Field(default_factory=list, description="Ids of in-line items in flow order.") flow_direction: Literal["shown", "inferred"] = "shown" status: Optional[Literal["new", "existing", "future", "demolish", "unknown"]] = None confidence: float = Field(0.85, ge=0.0, le=1.0) notes: Optional[str] = None class Signal(_Strict): from_: str = Field(alias="from") to: str type: str = Field("unknown", description="electrical | pneumatic | software | hydraulic | capillary | wireless | mechanical | unknown") purpose: Literal["control", "measurement", "trip", "alarm", "status", "unknown"] = "unknown" confidence: float = Field(0.85, ge=0.0, le=1.0) notes: Optional[str] = None class OffPageConnector(_Item): direction: Literal["in", "out"] other_drawing: Optional[str] = Field(None, description="Drawing number the flow continues on/comes from.") other_item: Optional[str] = Field(None, description="Item named at the far end, e.g. '660-TK-004'.") line_number: Optional[str] = None class Boundary(_Item): direction: Literal["source", "sink", "area"] = Field( description="source/sink for streams entering or leaving; 'area' for a room or zone that " "area instruments (gas detectors, room temperature) measure") class SheetInfo(_Strict): drawing_number: Optional[str] = None title: Optional[str] = None revision: Optional[str] = None drawing_type: Literal["pid", "pfd", "other"] = "pid" issue_status: Optional[str] = Field(None, description="As printed: 'as-built', 'issued for construction', 'draft', 'superseded'...") rotation: Optional[int] = None legend_notes: list[str] = Field(default_factory=list) class PageExtraction(_Strict): schema_version: str = EXTRACTION_SCHEMA_VERSION page: int = Field(ge=1, description="1-based page number in the input PDF.") sheet: SheetInfo = Field(default_factory=SheetInfo) equipment: list[Equipment] = Field(default_factory=list) components: list[Component] = Field(default_factory=list) instruments: list[Instrument] = Field(default_factory=list) lines: list[Line] = Field(default_factory=list) signals: list[Signal] = Field(default_factory=list) off_page_connectors: list[OffPageConnector] = Field(default_factory=list) boundaries: list[Boundary] = Field(default_factory=list) tiles_reviewed: list[str] = Field(default_factory=list) unreadable: list[str] = Field(default_factory=list, description="Things seen but not legible.") @model_validator(mode="after") def _unique_ids(self) -> "PageExtraction": seen: dict[str, str] = {} groups = { "equipment": self.equipment, "components": self.components, "instruments": self.instruments, "off_page_connectors": self.off_page_connectors, "boundaries": self.boundaries, "lines": self.lines, } for group, items in groups.items(): for it in items: if it.id in seen: raise ValueError(f"duplicate id '{it.id}' in {group} and {seen[it.id]}") seen[it.id] = group return self # ========================================================================== # Knowledge graph (written by build_kg.py) # ========================================================================== class Provenance(_Strict): sheet: str page: int bbox: Optional[BBox] = None source: str = "vision" local_id: Optional[str] = None class Node(_Strict): id: str type: str label: str tag: Optional[str] = None kind: Optional[str] = None name: Optional[str] = None status: Optional[str] = None dexpi_class: Optional[str] = None sheets: list[str] = Field(default_factory=list) confidence: float = 1.0 attributes: dict[str, Any] = Field(default_factory=dict) provenance: list[Provenance] = Field(default_factory=list) class Edge(_Strict): id: str source: str target: str type: str confidence: float = 1.0 derived: bool = False attributes: dict[str, Any] = Field(default_factory=dict) sheet: Optional[str] = None class PlantInfo(_Strict): id: str name: str source_file: Optional[str] = None sheets: list[str] = Field(default_factory=list) class KnowledgeGraph(_Strict): schema_version: str = GRAPH_SCHEMA_VERSION generator: str generated_at: str plant: PlantInfo nodes: list[Node] edges: list[Edge] stats: dict[str, Any] = Field(default_factory=dict) issues: list[dict[str, Any]] = Field(default_factory=list)skills/pid-to-kg/scripts/pidkg/ocr.py6.5 KB9ba3ecc29a6eupstream ↗
"""Optional OCR (RapidOCR, ONNX runtime; pip-installable, no system binaries). OCR is used for two things only: * detecting page orientation, and * producing *hints*: candidate tags with positions that QA compares against Claude's visual extraction to flag possible misses. It never writes to the graph directly. """ from __future__ import annotations import re from typing import Optional from .isa import normalize_tag, parse_isa_tag _engine = None EQUIPMENT_LIKE = re.compile(r"^(\d{2,4}-)?[A-Z]{2,4}-\d{2,5}[A-Z]?$") _LETTERS = re.compile(r"^[A-Z]{1,5}$") _DIGITS = re.compile(r"^\d{2,5}[A-Z]?$") def available() -> bool: try: import rapidocr_onnxruntime # noqa: F401 return True except ImportError: return False def _get_engine(): global _engine if _engine is None: from rapidocr_onnxruntime import RapidOCR _engine = RapidOCR() return _engine def ocr_image(img) -> list[dict]: """OCR a PIL image. Returns [{'text', 'bbox' (pixels x0,y0,x1,y1), 'conf'}].""" import numpy as np arr = np.array(img.convert("RGB")) result, _ = _get_engine()(arr, use_cls=False) out = [] for box, text, conf in result or []: xs = [p[0] for p in box] ys = [p[1] for p in box] out.append({"text": str(text).strip(), "bbox": [min(xs), min(ys), max(xs), max(ys)], "conf": float(conf)}) return out def _upright_boxes(results: list[dict]) -> list[dict]: """Confident, horizontal, alphanumeric OCR boxes. Only horizontal boxes count: the recognizer silently turns tall crops, so sideways text would otherwise read in every orientation. """ keep = [] for r in results: t = re.sub(r"[^A-Za-z0-9]", "", r["text"]) x0, y0, x1, y1 = r["bbox"] if (x1 - x0) > 1.2 * (y1 - y0) and len(t) >= 2 and r["conf"] >= 0.6: keep.append(r) return keep def detect_rotation(img, rotations=(0, 90, 180, 270)) -> tuple[int, dict[int, float], Optional[float]]: """Find the clockwise rotation that makes the text upright, and measure text size. Returns (rotation, score per rotation, 25th-percentile text box height in pixels of ``img`` at that rotation, or None if no text was found). """ scores: dict[int, float] = {} heights: dict[int, list[float]] = {} for rot in rotations: # PIL rotates counter-clockwise; negate to match the clockwise convention. test = img.rotate(-rot, expand=True) if rot else img boxes = _upright_boxes(ocr_image(test)) scores[rot] = round(sum(r["conf"] * len(re.sub(r"[^A-Za-z0-9]", "", r["text"])) for r in boxes), 1) heights[rot] = sorted(r["bbox"][3] - r["bbox"][1] for r in boxes) best = max(scores, key=scores.get) hs = heights[best] p25 = hs[len(hs) // 4] if len(hs) >= 8 else None return best, scores, p25 def join_bubbles(runs: list[dict]) -> list[dict]: """Join ISA bubble text split over two lines ('FIT' above '0502') into one run. ``runs`` use normalized page bboxes. Joined runs are added; originals kept. """ letters = [r for r in runs if _LETTERS.match(r["text"].replace(" ", ""))] digits = [r for r in runs if _DIGITS.match(r["text"].replace(" ", ""))] joined = [] for a in letters: ax0, ay0, ax1, ay1 = a["bbox"] h = ay1 - ay0 acx = (ax0 + ax1) / 2 best, best_gap = None, None for d in digits: dx0, dy0, dx1, dy1 = d["bbox"] dcx = (dx0 + dx1) / 2 gap = dy0 - ay1 width = max(ax1 - ax0, dx1 - dx0) if -0.5 * h <= gap <= 1.5 * h and abs(acx - dcx) <= 0.6 * width: if best_gap is None or gap < best_gap: best, best_gap = d, gap if best is not None: text = f"{a['text'].replace(' ', '')}-{best['text'].replace(' ', '')}" bb = [min(ax0, best["bbox"][0]), ay0, max(ax1, best["bbox"][2]), best["bbox"][3]] joined.append({"text": text, "bbox": [round(v, 4) for v in bb], "conf": round(min(a.get("conf", 1), best.get("conf", 1)), 3), "joined": True}) return runs + joined def tag_candidates(runs: list[dict]) -> list[dict]: """Pick tag-like strings from text runs, de-duplicated by normalized tag.""" seen: dict[str, dict] = {} for r in runs: tokens = [t for t in re.split(r"[\s,;()]+", r["text"]) if t] # Tags are often printed with a space ('FIT 0502'): also try adjacent pairs. pairs = [f"{a}-{b}" for a, b in zip(tokens, tokens[1:]) if _LETTERS.match(a) and _DIGITS.match(b)] for token in tokens + pairs: t = normalize_tag(token) if not t or len(t) < 4: continue isa = parse_isa_tag(t) eq = bool(EQUIPMENT_LIKE.match(t)) if not isa and not eq: continue if t not in seen or r.get("conf", 0) > seen[t]["conf"]: seen[t] = {"text": t, "bbox": r["bbox"], "conf": r.get("conf", 1.0), "isa": bool(isa), "equipment_like": eq} return sorted(seen.values(), key=lambda c: (c["bbox"][1], c["bbox"][0])) def dedupe_runs(runs: list[dict], tol: float = 0.004) -> list[dict]: """Drop runs seen twice in overlapping tiles (same text, nearly same position).""" kept: list[dict] = [] for r in sorted(runs, key=lambda r: -r.get("conf", 0)): cx = (r["bbox"][0] + r["bbox"][2]) / 2 cy = (r["bbox"][1] + r["bbox"][3]) / 2 dup = False for k in kept: if k["text"] != r["text"]: continue kx = (k["bbox"][0] + k["bbox"][2]) / 2 ky = (k["bbox"][1] + k["bbox"][3]) / 2 if abs(kx - cx) < tol and abs(ky - cy) < tol: dup = True break if not dup: kept.append(r) return kept def to_norm(bbox_px: list[float], tile_px: list[int], page_w: int, page_h: int, tile_scale: float = 1.0) -> list[float]: """Convert a bbox in tile pixels to normalized page coordinates.""" x0, y0, x1, y1 = (v / tile_scale for v in bbox_px) return [round((tile_px[0] + x0) / page_w, 4), round((tile_px[1] + y0) / page_h, 4), round((tile_px[0] + x1) / page_w, 4), round((tile_px[1] + y1) / page_h, 4)] def maybe_import_error() -> Optional[str]: if available(): return None return ("RapidOCR is not installed; OCR hints and automatic rotation are skipped. " "Install with: pip install rapidocr-onnxruntime")skills/pid-to-kg/scripts/pidkg/ontology.py7.8 KBef58d2c49585upstream ↗
"""Controlled vocabularies for the P&ID knowledge graph. Vocabularies are deliberately permissive: an unknown ``kind`` is accepted and reported by QA rather than rejected, because real drawings always contain something the vocabulary did not anticipate. ``DEXPI_CLASS`` maps each kind to the closest DEXPI (ISO 15926-based P&ID exchange standard) class name. The mapping is informative: it helps users align the graph with DEXPI tooling but is not a certified DEXPI export. """ from __future__ import annotations # -------------------------------------------------------------------------- # Node and edge types # -------------------------------------------------------------------------- NODE_TYPES = ( "Plant", "Sheet", "Equipment", "PipeSegment", "Component", "Instrument", "ControlLoop", "OffPageConnector", "Boundary", ) EDGE_TYPES = ( # Physical process connectivity (directed in the direction of flow). "FLOWS_TO", # Derived equipment-to-equipment flow, collapsing pipes and fittings. "FEEDS", # Instrument -> the equipment, pipe segment or component it senses. "MEASURES", # Instrument/control signal (directed from sender to receiver). "SIGNAL_TO", # Controller or logic -> the final element it moves (valve, VFD/motor). "ACTUATES", # Instrument/component -> ControlLoop, with a ``role`` attribute. "MEMBER_OF", # ControlLoop -> ControlLoop (master -> slave in a cascade). "CASCADES_TO", # Safety/interlock action: initiator -> the item it trips. "TRIPS", # Sheet -> Plant; mounted equipment -> its parent (agitator -> tank). "PART_OF", ) # Edge types followed when tracing upstream/downstream process flow. FLOW_EDGE_TYPES = ("FLOWS_TO",) # -------------------------------------------------------------------------- # Equipment # -------------------------------------------------------------------------- EQUIPMENT_DEXPI = { "tank": "Tank", "vessel": "Vessel", "pressure_vessel": "PressureVessel", "pump": "Pump", "centrifugal_pump": "CentrifugalPump", "positive_displacement_pump": "ReciprocatingPump", "metering_pump": "ReciprocatingPump", "slurry_pump": "CentrifugalPump", "sump_pump": "CentrifugalPump", "compressor": "Compressor", "blower": "Blower", "fan": "Fan", "heat_exchanger": "HeatExchanger", "cooler": "HeatExchanger", "heater": "HeatExchanger", "boiler": "Boiler", "furnace": "Furnace", "column": "Column", "reactor": "Vessel", "agitated_tank": "Tank", "agitator": "Agitator", "mixer": "Mixer", "static_mixer": "StaticMixer", "filter": "Filter", "filter_press": "Filter", "membrane_filter": "Filter", "strainer_unit": "Filter", "screen": "Sieve", "cyclone": "CycloneSeparator", "hydrocyclone": "CycloneSeparator", "separator": "Separator", "clarifier": "Separator", "thickener": "Separator", "flotation_cell": "Separator", "centrifuge": "Centrifuge", "dryer": "Dryer", "mill": "Mill", "crusher": "Crusher", "conveyor": "Conveyor", "feeder": "Feeder", "hopper": "Silo", "bin": "Silo", "silo": "Silo", "sump": "Tank", "pond": "Tank", "box": "Tank", "electrowinning_cell": "Vessel", "kiln": "Furnace", "scrubber": "Column", "ion_exchange": "Column", "uv_unit": "Equipment", "sampler": "Equipment", "package_unit": "Equipment", "motor": "ElectricMotor", "other": "Equipment", } EQUIPMENT_KINDS = tuple(EQUIPMENT_DEXPI) # -------------------------------------------------------------------------- # In-line piping components # -------------------------------------------------------------------------- COMPONENT_DEXPI = { "manual_valve": "OperatedValve", "gate_valve": "GateValve", "globe_valve": "GlobeValve", "ball_valve": "BallValve", "butterfly_valve": "ButterflyValve", "plug_valve": "PlugValve", "knife_gate_valve": "GateValve", "diaphragm_valve": "DiaphragmValve", "pinch_valve": "OperatedValve", "needle_valve": "NeedleValve", "three_way_valve": "OperatedValve", "check_valve": "CheckValve", "backflow_preventer": "CheckValve", "control_valve": "OperatedValve", "on_off_valve": "OperatedValve", "solenoid_valve": "OperatedValve", "motor_operated_valve": "OperatedValve", "pressure_regulator": "OperatedValve", "relief_valve": "SafetyValveOrFitting", "rupture_disc": "RuptureDisc", "vacuum_breaker": "SafetyValveOrFitting", "strainer": "Strainer", "filter": "Filter", "reducer": "PipeReducer", "blind": "Blind", "spectacle_blind": "SpectacleBlind", "flexible_hose": "Hose", "expansion_joint": "Compensator", "sight_glass": "SightGlass", "steam_trap": "SteamTrap", "flame_arrestor": "FlameArrestor", "sample_point": "PipingComponent", "drain": "PipingComponent", "vent": "PipingComponent", "quick_connect": "PipingComponent", "tee": "PipeTee", "other": "PipingComponent", } COMPONENT_KINDS = tuple(COMPONENT_DEXPI) # Components that can be the final element of a control or trip loop. ACTUATED_COMPONENT_KINDS = { "control_valve", "on_off_valve", "solenoid_valve", "motor_operated_valve", "three_way_valve", } # -------------------------------------------------------------------------- # Instruments # -------------------------------------------------------------------------- INSTRUMENT_LOCATIONS = ( "field", # plain circle "local_panel", # circle with line, dashed or in a local/field panel "main_panel", # circle with solid line: main control room panel "behind_panel", # circle with dashed line "dcs", # circle in square: shared display / DCS "plc", # diamond in square "scada", "sis", # safety instrumented system "computer", # hexagon "unknown", ) SIGNAL_TYPES = ( "electrical", # dashed line "pneumatic", # line with double cross-hatches "software", # line with circles / data link "hydraulic", "capillary", # line with x's "wireless", # zig-zag / wave "mechanical", "unknown", ) STATUSES = ("new", "existing", "future", "demolish", "unknown") # ISA-5.1 first letters (measured or initiating variable). ISA_MEASURED = { "A": "analysis", "B": "burner/flame", "C": "conductivity", # user's choice; conductivity is the most common "D": "density", # user's choice "E": "voltage", "F": "flow", "G": "gauging/position", # user's choice "H": "hand", "I": "current", "J": "power", "K": "time", "L": "level", "M": "moisture", # user's choice "N": "user-defined", "O": "user-defined", "P": "pressure", "Q": "quantity", "R": "radiation", "S": "speed/frequency", "T": "temperature", "U": "multivariable", "V": "vibration", "W": "weight/force", "X": "unclassified", "Y": "event/state", "Z": "position", } # Variable modifiers that may follow the first letter. ISA_VARIABLE_MODIFIERS = { "D": "differential", "F": "ratio", "J": "scan", "K": "rate of change", "Q": "integrate/totalize", } # Succeeding letters (readout/passive or output functions). ISA_FUNCTIONS = { "A": "alarm", "B": "user's choice", "C": "control", "D": "differential", "E": "primary element", "G": "glass/gauge", "H": "high", "I": "indicate", "K": "control station", "L": "light/low", "M": "middle", "N": "user's choice", "O": "orifice/restriction", "P": "test point", "Q": "totalize", "R": "record", "S": "switch", "T": "transmit", "U": "multifunction", "V": "valve/damper", "W": "well", "X": "unclassified", "Y": "relay/compute/convert", "Z": "driver/actuator", }skills/pid-to-kg/scripts/pidkg/pdf.py8.3 KB305445be2e91upstream ↗
"""PDF inspection and rendering (pypdfium2: Apache-2.0/BSD, no system dependencies).""" from __future__ import annotations import math from dataclasses import asdict, dataclass, field from typing import Optional import pypdfium2 as pdfium import pypdfium2.raw as pdfium_c # Page classes and how each is extracted. ROUTES = { "vector_text": "Vector drawing with a text layer: render from vectors, take exact text and " "positions from the text layer as hints, then read visually.", "vector_outlined": "Vector drawing whose text was converted to outlines (no text layer): " "render from vectors at high resolution, OCR the render for hints, then " "read visually.", "raster": "Scanned image: render at the scan's native resolution, upscale and enhance " "contrast, OCR for hints, then read visually.", "raster_with_text": "Scanned image with an OCR text layer: as raster, and use the existing " "text layer as (untrusted) hints.", "hybrid": "Mix of vector content and large images: render from vectors at high " "resolution and use both text layer and OCR hints.", "blank": "No meaningful content.", } MIN_TEXT_CHARS = 50 MIN_VECTOR_PATHS = 200 BIG_IMAGE_COVERAGE = 0.5 @dataclass class PageProbe: page: int width_pt: float height_pt: float pdf_rotation: int text_chars: int path_objects: int text_objects: int image_objects: int max_image_coverage: float image_px: Optional[list[int]] effective_image_dpi: Optional[float] page_class: str route: str = "" rotation: Optional[int] = None notes: list[str] = field(default_factory=list) def to_dict(self) -> dict: return asdict(self) def probe_pdf(path: str) -> list[PageProbe]: pdf = pdfium.PdfDocument(path) try: return [probe_page(pdf, i) for i in range(len(pdf))] finally: pdf.close() def probe_page(pdf: "pdfium.PdfDocument", index: int) -> PageProbe: page = pdf[index] w, h = page.get_size() area = max(w * h, 1.0) tp = page.get_textpage() text = tp.get_text_range() or "" chars = sum(1 for ch in text if not ch.isspace()) counts = {pdfium_c.FPDF_PAGEOBJ_PATH: 0, pdfium_c.FPDF_PAGEOBJ_TEXT: 0, pdfium_c.FPDF_PAGEOBJ_IMAGE: 0} best_cov, best_px, best_dpi = 0.0, None, None for obj in page.get_objects(max_depth=8): if obj.type in counts: counts[obj.type] += 1 if obj.type == pdfium_c.FPDF_PAGEOBJ_IMAGE: l, b, r, t = obj.get_bounds() cov = max(0.0, (r - l) * (t - b)) / area if cov > best_cov: best_cov = cov try: px = list(obj.get_px_size()) best_px = px best_dpi = round(72.0 * max(px) / max(r - l, t - b, 1e-6), 1) except Exception: # pragma: no cover - exotic image types best_px, best_dpi = None, None paths = counts[pdfium_c.FPDF_PAGEOBJ_PATH] has_text = chars >= MIN_TEXT_CHARS has_vec = paths >= MIN_VECTOR_PATHS big_img = best_cov >= BIG_IMAGE_COVERAGE if big_img and not has_vec: cls = "raster_with_text" if has_text else "raster" elif big_img and has_vec: cls = "hybrid" elif has_text: cls = "vector_text" elif has_vec: cls = "vector_outlined" elif paths or counts[pdfium_c.FPDF_PAGEOBJ_IMAGE]: cls = "vector_outlined" if paths else "raster" else: cls = "blank" probe = PageProbe( page=index + 1, width_pt=round(w, 1), height_pt=round(h, 1), pdf_rotation=page.get_rotation(), text_chars=chars, path_objects=paths, text_objects=counts[pdfium_c.FPDF_PAGEOBJ_TEXT], image_objects=counts[pdfium_c.FPDF_PAGEOBJ_IMAGE], max_image_coverage=round(best_cov, 3), image_px=best_px, effective_image_dpi=best_dpi, page_class=cls, route=ROUTES[cls]) if cls.startswith("raster") and best_dpi and best_dpi < 200: probe.notes.append(f"low scan resolution (~{best_dpi:.0f} dpi): small text may be " "illegible; record unreadable items in 'unreadable'") if max(w, h) < 1000: probe.notes.append("page is smaller than a typical drawing sheet; the drawing was " "probably reduced, so render at high resolution") return probe def render_page(pdf_path: str, page_no: int, page_class: str, rotation: int = 0, vector_long_edge: int = 5000, raster_long_edge: int = 4000, max_upscale: float = 2.5): """Render a page to a PIL image following the route for its class.""" from PIL import Image, ImageOps pdf = pdfium.PdfDocument(pdf_path) try: page = pdf[page_no - 1] w, h = page.get_size() if page_class.startswith("raster"): probe = probe_page(pdf, page_no - 1) native = (probe.effective_image_dpi or 200) / 72.0 img = page.render(scale=native, rotation=rotation, grayscale=True).to_pil() long_edge = max(img.size) factor = min(max_upscale, max(1.0, raster_long_edge / long_edge)) if factor > 1.0: img = img.resize((round(img.width * factor), round(img.height * factor)), Image.LANCZOS) img = ImageOps.autocontrast(img.convert("L"), cutoff=0.5) return img scale = vector_long_edge / max(w, h) return page.render(scale=scale, rotation=rotation).to_pil().convert("RGB") finally: pdf.close() def tile_grid(width: int, height: int, tile: int = 1400, overlap: int = 200) -> list[dict]: """Evenly spaced overlapping tiles covering the image.""" def spans(total: int) -> list[tuple[int, int]]: if total <= tile: return [(0, total)] n = math.ceil((total - overlap) / (tile - overlap)) step = (total - tile) / (n - 1) return [(round(i * step), round(i * step) + tile) for i in range(n)] tiles = [] for r, (y0, y1) in enumerate(spans(height), 1): for c, (x0, x1) in enumerate(spans(width), 1): tiles.append({"name": f"r{r}c{c}", "row": r, "col": c, "px": [x0, y0, x1, y1], "norm": [round(x0 / width, 4), round(y0 / height, 4), round(x1 / width, 4), round(y1 / height, 4)]}) return tiles def ink_fraction(img, threshold: int = 200) -> float: """Fraction of pixels darker than ``threshold``, ignoring frame lines. Rows and columns that are dark across most of the tile (the drawing border, title-block rules) are not content, so they are excluded; a tile holding only a border line then counts as blank. """ g = img.convert("L") try: import numpy as np except ImportError: # pragma: no cover - numpy ships with the OCR extra hist = g.histogram() return sum(hist[:threshold]) / max(1, g.width * g.height) dark = np.asarray(g) < threshold rows = dark.mean(axis=1) > 0.6 cols = dark.mean(axis=0) > 0.6 dark[rows, :] = False dark[:, cols] = False return float(dark.mean()) def rotate_norm_bbox(b: list[float], rotation: int) -> list[float]: """Map a normalized bbox on the unrotated page onto the page rotated clockwise.""" x0, y0, x1, y1 = b pts = [(x0, y0), (x1, y1)] if rotation == 90: pts = [(1 - y, x) for x, y in pts] elif rotation == 180: pts = [(1 - x, 1 - y) for x, y in pts] elif rotation == 270: pts = [(y, 1 - x) for x, y in pts] (a, b2), (c, d) = pts return [round(min(a, c), 4), round(min(b2, d), 4), round(max(a, c), 4), round(max(b2, d), 4)] def text_runs(pdf_path: str, page_no: int, rotation: int = 0) -> list[dict]: """Text-layer runs with normalized bboxes on the (rotated) page.""" pdf = pdfium.PdfDocument(pdf_path) try: page = pdf[page_no - 1] w, h = page.get_size() tp = page.get_textpage() runs = [] for i in range(tp.count_rects()): l, b, r, t = tp.get_rect(i) txt = (tp.get_text_bounded(l, b, r, t) or "").strip() if not txt: continue norm = [l / w, 1 - t / h, r / w, 1 - b / h] runs.append({"text": txt, "bbox": rotate_norm_bbox(norm, rotation), "conf": 1.0}) return runs finally: pdf.close()skills/pid-to-kg/scripts/pidkg/qa.py12 KB24bf8fa4eaa2upstream ↗
"""Quality checks on a built graph, and the QA report Claude uses for its second pass.""" from __future__ import annotations import json import re from collections import defaultdict from pathlib import Path from typing import Any, Optional from .isa import SENSING_ROLES, normalize_tag, parse_isa_tag from .models import KnowledgeGraph LOW_CONFIDENCE = 0.6 _MONTHS = re.compile(r"^(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)[A-Z]*-\d") def run_qa(kg: KnowledgeGraph, work_dir: Optional[Path] = None) -> list[dict[str, Any]]: """Return graph-level issues (build issues are already in ``kg.issues``).""" issues: list[dict[str, Any]] = [] nodes = {n.id: n for n in kg.nodes} flow_deg: dict[str, int] = defaultdict(int) any_deg: dict[str, int] = defaultdict(int) measured: set[str] = set() for e in kg.edges: any_deg[e.source] += 1 any_deg[e.target] += 1 if e.type == "FLOWS_TO": flow_deg[e.source] += 1 flow_deg[e.target] += 1 elif e.type == "MEASURES": measured.add(e.source) def add(sev: str, code: str, msg: str, node=None) -> None: n = nodes.get(node) if node else None bbox = next((p.bbox for p in n.provenance if p.bbox), None) if n else None sheet = n.sheets[0] if n and n.sheets else None page = n.provenance[0].page if n and n.provenance else None issues.append({k: v for k, v in dict(severity=sev, code=code, message=msg, node=node, sheet=sheet, page=page, bbox=bbox).items() if v is not None}) for n in kg.nodes: if n.type == "Equipment" and flow_deg[n.id] == 0 and not n.attributes.get("part_of"): add("warning", "equipment_not_connected", f"equipment {n.label} has no process connections", n.id) elif n.type == "Component" and flow_deg[n.id] == 0: add("warning", "component_not_connected", f"component {n.label} is not on any line (add it to a line 'path')", n.id) elif n.type == "Instrument": role = n.attributes.get("role") loc = n.attributes.get("location", "unknown") if role in SENSING_ROLES and loc in ("field", "unknown") and n.id not in measured \ and not n.attributes.get("inline"): add("warning", "instrument_not_located", f"{n.label} ({role}) senses the process but has no 'measures' location", n.id) if any_deg[n.id] == 0: add("info", "isolated_instrument", f"{n.label} has no connections at all", n.id) if n.type not in ("Plant", "Sheet") and n.confidence < LOW_CONFIDENCE: add("info", "low_confidence", f"{n.label} was read with confidence {n.confidence:.2f}", n.id) # Tags that differ only in zero padding ('660-TK-06' vs '660-TK-006'). padded: dict[str, list[str]] = defaultdict(list) for n in kg.nodes: if n.tag and n.type == "Equipment": # instrument loop numbers legitimately vary padded[re.sub(r"(?<![0-9])0+(?=[0-9])", "", n.tag.split("@")[0].split("#")[0])].append(n.id) for ids in padded.values(): tags = sorted({nodes[i].tag for i in ids}) if len(tags) > 1: add("warning", "near_duplicate_tags", f"tags {', '.join(tags)} differ only in zero padding; probably one item " "drawn with inconsistent tags (check the sheets)", ids[0]) issues.extend(_revision_overlap(kg)) inferred = sum(1 for e in kg.edges if e.type == "FLOWS_TO" and e.attributes.get("flow_direction") == "inferred") if inferred: issues.append({"severity": "info", "code": "inferred_flow_direction", "message": f"{inferred} flow edges have inferred (not drawn) direction"}) if work_dir: issues.extend(_coverage_issues(kg, Path(work_dir))) issues.extend(_hint_issues(kg, Path(work_dir))) return issues def _revision_overlap(kg: KnowledgeGraph) -> list[dict[str, Any]]: """Two sheets sharing most of their tags are probably two revisions of one drawing.""" tags: dict[str, set[str]] = defaultdict(set) for n in kg.nodes: if n.tag and n.type in ("Equipment", "Instrument", "Component"): for s in {p.sheet for p in n.provenance}: tags[s].add(n.tag) out = [] sheets = sorted(t for t in tags if len(tags[t]) >= 5) for i, a in enumerate(sheets): for b in sheets[i + 1:]: shared = tags[a] & tags[b] frac = len(shared) / min(len(tags[a]), len(tags[b])) if frac >= 0.5: out.append({"severity": "warning", "code": "possible_duplicate_revision", "sheet": b, "message": f"{a} and {b} share {len(shared)} tags ({frac:.0%} of the smaller sheet); " "if they are two revisions of the same drawing, keep only the current one " "in this graph and ask the user what to do with the other"}) return out def _coverage_issues(kg: KnowledgeGraph, work: Path) -> list[dict[str, Any]]: out = [] for n in kg.nodes: if n.type != "Sheet": continue page = n.attributes.get("page") tiles_file = work / "pages" / f"page_{page:02d}" / "tiles.json" if not tiles_file.exists(): continue tiles = json.loads(tiles_file.read_text(encoding="utf-8")) expected = {t["name"] for t in tiles["tiles"] if not t.get("blank")} reviewed = set(n.attributes.get("tiles_reviewed", [])) missing = sorted(expected - reviewed) if missing: out.append({"severity": "warning", "code": "tiles_not_reviewed", "sheet": n.id, "page": page, "message": f"{len(missing)} non-blank tiles not listed in " f"tiles_reviewed: {', '.join(missing)}"}) return out def _hint_issues(kg: KnowledgeGraph, work: Path) -> list[dict[str, Any]]: """Compare tags found by OCR / the text layer with tags in the graph.""" out = [] known = {n.tag for n in kg.nodes if n.tag} | {n.id for n in kg.nodes} known_core = {_core(t) for t in known if t} # Drawing numbers (title blocks, off-page flags) look like tags but are not items. drawings = {_core(normalize_tag(v)) for n in kg.nodes for v in (n.attributes.get("drawing_number"), n.attributes.get("other_drawing")) if v} # Text already captured in names/attributes (off-page 'other_item', tie-in labels...). blob = _core(normalize_tag(" ".join( f"{n.name or ''} {n.label} {json.dumps(n.attributes)}" for n in kg.nodes)) or "") by_number: dict[str, set[str]] = defaultdict(set) for t in known: p = parse_isa_tag(t) if t else None if p: by_number[p.number].add(p.letters) for n in kg.nodes: if n.type != "Sheet": continue page = n.attributes.get("page") hint_file = work / "hints" / f"page_{page:02d}.json" if not hint_file.exists(): continue hints = json.loads(hint_file.read_text(encoding="utf-8")) missed = [] for cand in hints.get("tag_candidates", []): t = normalize_tag(cand["text"]) if not t or t in known: continue core = _core(t) if core in known_core or core in drawings or _MONTHS.match(t): continue # OCR fragments cut at a tile edge ('0-BX-002', '660-TK-00'). if any(core in k or k in core for k in known_core if len(k) >= 4) or core in blob: continue isa = parse_isa_tag(t) if not isa and not cand.get("equipment_like"): continue # Likely OCR misread of a known tag: same number, letters one edit apart. if isa and any(_close(isa.letters, l) for l in by_number.get(isa.number, ())): continue missed.append(cand) if missed: listed = ", ".join(f"{c['text']}" for c in missed[:40]) out.append({"severity": "warning", "code": "possible_missed_tags", "sheet": n.id, "page": page, "count": len(missed), "message": f"{len(missed)} tags found by {hints.get('source', 'hints')} are not " f"in the graph: {listed}", "candidates": missed}) return out def _edit1(a: str, b: str) -> bool: """True if a and b differ by at most one substitution, insertion or deletion.""" if abs(len(a) - len(b)) > 1: return False if len(a) == len(b): return sum(x != y for x, y in zip(a, b)) <= 1 if len(a) > len(b): a, b = b, a return any(a == b[:i] + b[i + 1:] for i in range(len(b))) def _core(tag: str) -> str: """Tag without separators, so 'FIT-0502' and 'FIT0502' match.""" return tag.replace("-", "") def tiles_for_bbox(work: Path, page: int, bbox: Optional[list[float]]) -> list[str]: if not bbox: return [] f = work / "pages" / f"page_{page:02d}" / "tiles.json" if not f.exists(): return [] tiles = json.loads(f.read_text(encoding="utf-8"))["tiles"] x0, y0, x1, y1 = bbox cx, cy = (x0 + x1) / 2, (y0 + y1) / 2 hits = [t["name"] for t in tiles if t["norm"][0] <= cx <= t["norm"][2] and t["norm"][1] <= cy <= t["norm"][3]] return hits[:2] def write_report(kg: KnowledgeGraph, issues: list[dict[str, Any]], path: Path, work_dir: Optional[Path] = None) -> None: sev_order = {"error": 0, "warning": 1, "info": 2} nodes = {n.id: n for n in kg.nodes} for i in issues: # fill page/bbox from the node's provenance when missing n = nodes.get(i.get("node", "")) if n and n.provenance: i.setdefault("page", n.provenance[0].page) if not i.get("bbox"): bb = next((p.bbox for p in n.provenance if p.bbox), None) if bb: i["bbox"] = bb issues = sorted(issues, key=lambda i: (sev_order.get(i["severity"], 3), i.get("page") or 0, i["code"])) counts: dict[str, int] = defaultdict(int) by_code: dict[tuple[str, str], int] = defaultdict(int) for i in issues: counts[i["severity"]] += 1 by_code[(i["severity"], i["code"])] += 1 lines = [f"# QA report: {kg.plant.name}", "", f"Generated {kg.generated_at} by {kg.generator}.", "", f"**{counts.get('error', 0)} errors, {counts.get('warning', 0)} warnings, " f"{counts.get('info', 0)} info.**", "", "Errors must be fixed in the page extraction files before the graph is trusted. " "Warnings are things to re-check on the drawing (the *Re-check* column names the tiles). " "Info items are for the record.", "", "| Severity | Code | Count |", "|---|---|---|"] for (sev, code), c in sorted(by_code.items(), key=lambda kv: (sev_order.get(kv[0][0], 3), kv[0][1])): lines.append(f"| {sev} | `{code}` | {c} |") lines += ["", "## Details", "", "| Severity | Page | Code | Item | Message | Re-check |", "|---|---|---|---|---|---|"] for i in issues: tiles = tiles_for_bbox(work_dir, i["page"], i.get("bbox")) if work_dir and i.get("page") else [] msg = i["message"].replace("|", "\\|") lines.append(f"| {i['severity']} | {i.get('page', '')} | `{i['code']}` | {i.get('node', '')} | " f"{msg} | {', '.join(tiles)} |") path.write_text("\n".join(lines) + "\n", encoding="utf-8") def _close(a: str, b: str) -> bool: """Letter codes close enough to be an OCR misread (same first letter, <=2 slips).""" if _edit1(a, b): return True return len(a) == len(b) and a[:1] == b[:1] and sum(x != y for x, y in zip(a, b)) <= 2skills/pid-to-kg/scripts/pidkg/store.py9.4 KB233c319230d6upstream ↗
"""DuckDB storage and graph traversal. The DuckDB file is a query layer over ``graph.json``: two tables (``nodes``, ``edges``), convenience views, and table macros for flow traversal, e.g.:: SELECT * FROM downstream('660-TK-001'); SELECT * FROM upstream('FV-0502'); SELECT * FROM instruments WHERE variable = 'flow'; """ from __future__ import annotations import json from collections import defaultdict, deque from pathlib import Path from typing import Any, Optional from .models import KnowledgeGraph SCHEMA_SQL = """ CREATE TABLE plant ( id VARCHAR PRIMARY KEY, name VARCHAR, source_file VARCHAR, generated_at VARCHAR, generator VARCHAR, schema_version VARCHAR, stats JSON ); CREATE TABLE nodes ( id VARCHAR PRIMARY KEY, type VARCHAR NOT NULL, label VARCHAR, tag VARCHAR, kind VARCHAR, name VARCHAR, status VARCHAR, dexpi_class VARCHAR, sheets VARCHAR[], confidence DOUBLE, attributes JSON, provenance JSON ); CREATE TABLE edges ( id VARCHAR PRIMARY KEY, source VARCHAR NOT NULL, target VARCHAR NOT NULL, type VARCHAR NOT NULL, confidence DOUBLE, derived BOOLEAN, sheet VARCHAR, attributes JSON ); CREATE TABLE issues ( severity VARCHAR, code VARCHAR, message VARCHAR, node VARCHAR, sheet VARCHAR, page INTEGER ); """ FLOW_VIEW_SQL = """ CREATE VIEW flow_edges AS SELECT source, target, sheet, attributes FROM edges WHERE type = 'FLOWS_TO' {pfd_filter}; """ VIEWS_SQL = """ CREATE VIEW equipment AS SELECT id, tag, kind, name, status, sheets, confidence FROM nodes WHERE type = 'Equipment'; CREATE VIEW equipment_flow AS SELECT e.source AS from_id, s.label AS from_label, e.target AS to_id, t.label AS to_label, e.attributes->>'via_lines' AS via_lines, e.attributes->>'via_items' AS via_items FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target WHERE e.type = 'FEEDS' {feeds_filter}; CREATE VIEW instruments AS SELECT n.id, n.tag, n.attributes->>'variable' AS variable, n.attributes->>'role' AS role, n.attributes->>'location' AS location, n.name, m.target AS measures_id, mt.label AS measures_label, mt.type AS measures_type, l.target AS loop_id, n.sheets, n.confidence FROM nodes n LEFT JOIN edges m ON m.source = n.id AND m.type = 'MEASURES' LEFT JOIN nodes mt ON mt.id = m.target LEFT JOIN edges l ON l.source = n.id AND l.type = 'MEMBER_OF' WHERE n.type = 'Instrument'; CREATE VIEW loops AS SELECT id, label AS loop, kind AS loop_type, attributes->>'variable' AS variable, attributes->'sensors' AS sensors, attributes->'controllers' AS controllers, attributes->'final_elements' AS final_elements, attributes->>'description' AS description, sheets FROM nodes WHERE type = 'ControlLoop'; CREATE VIEW signals AS SELECT e.source, e.target, e.attributes->>'signal_type' AS signal_type, e.attributes->>'purpose' AS purpose, e.sheet FROM edges e WHERE e.type = 'SIGNAL_TO'; CREATE MACRO downstream(start_id) AS TABLE WITH RECURSIVE walk(id) AS ( SELECT target FROM flow_edges WHERE source = start_id UNION SELECT e.target FROM flow_edges e JOIN walk w ON e.source = w.id ) SELECT n.id, n.type, n.label, n.kind, n.name FROM walk JOIN nodes n ON n.id = walk.id; CREATE MACRO upstream(start_id) AS TABLE WITH RECURSIVE walk(id) AS ( SELECT source FROM flow_edges WHERE target = start_id UNION SELECT e.source FROM flow_edges e JOIN walk w ON e.target = w.id ) SELECT n.id, n.type, n.label, n.kind, n.name FROM walk JOIN nodes n ON n.id = walk.id; """ def write_duckdb(kg: KnowledgeGraph, path: Path) -> None: import duckdb path = Path(path) if path.exists(): path.unlink() con = duckdb.connect(str(path)) try: con.execute(SCHEMA_SQL) con.execute("INSERT INTO plant VALUES (?, ?, ?, ?, ?, ?, ?)", [kg.plant.id, kg.plant.name, kg.plant.source_file, kg.generated_at, kg.generator, kg.schema_version, json.dumps(kg.stats)]) con.executemany( "INSERT INTO nodes VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", [[n.id, n.type, n.label, n.tag, n.kind, n.name, n.status, n.dexpi_class, n.sheets, n.confidence, json.dumps(n.attributes), json.dumps([p.model_dump(exclude_none=True) for p in n.provenance])] for n in kg.nodes]) con.executemany( "INSERT INTO edges VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [[e.id, e.source, e.target, e.type, e.confidence, e.derived, e.sheet, json.dumps(e.attributes)] for e in kg.edges]) if kg.issues: con.executemany( "INSERT INTO issues VALUES (?, ?, ?, ?, ?, ?)", [[i.get("severity"), i.get("code"), i.get("message"), i.get("node"), i.get("sheet"), i.get("page")] for i in kg.issues]) pfd_filter = ("AND coalesce(attributes->>'drawing_type', '') <> 'pfd'" if has_pid_sheets(kg) else "") con.execute(FLOW_VIEW_SQL.format(pfd_filter=pfd_filter)) con.execute(VIEWS_SQL.replace("{feeds_filter}", "AND coalesce(e.attributes->>'view', '') <> 'pfd'" if pfd_filter else "")) finally: con.close() def has_pid_sheets(kg: KnowledgeGraph) -> bool: sheets = [n for n in kg.nodes if n.type == "Sheet"] return any(n.attributes.get("drawing_type", "pid") != "pfd" for n in sheets) and any(n.attributes.get("drawing_type") == "pfd" for n in sheets) class Graph: """In-memory view of a knowledge graph for traversal queries.""" def __init__(self, kg: KnowledgeGraph, include_pfd: bool = False): """``include_pfd``: when a plant has P&IDs, PFD-sheet flow is skipped in traversals by default because the P&IDs show the same flow in detail.""" self.kg = kg self.nodes = {n.id: n for n in kg.nodes} self.out: dict[str, list] = defaultdict(list) self.inn: dict[str, list] = defaultdict(list) skip_pfd = not include_pfd and has_pid_sheets(kg) for e in kg.edges: if skip_pfd and e.attributes.get("drawing_type") == "pfd": continue if skip_pfd and e.type == "FEEDS" and e.attributes.get("view") == "pfd": continue self.out[e.source].append(e) self.inn[e.target].append(e) self._by_tag = {} for n in kg.nodes: for key in {n.id, n.tag, n.label}: if key: self._by_tag.setdefault(key.upper(), n.id) @classmethod def load(cls, path: Path, include_pfd: bool = False) -> "Graph": path = Path(path) if path.suffix == ".duckdb": path = path.with_name("graph.json") return cls(KnowledgeGraph.model_validate_json(path.read_text(encoding="utf-8")), include_pfd) def find(self, ref: str) -> Optional[str]: from .isa import normalize_tag if ref in self.nodes: return ref for key in (ref.upper(), normalize_tag(ref)): if key and key in self._by_tag: return self._by_tag[key] # Untagged items: a unique local id ('P12.OXIDANT_TOTE') or a unique name. r = ref.upper() hits = [n.id for n in self.kg.nodes if n.id.upper().endswith("." + r)] if len(hits) != 1: hits = [n.id for n in self.kg.nodes if n.name and n.name.upper() == r] return hits[0] if len(hits) == 1 else None def walk(self, start: str, direction: str = "down", edge_types=("FLOWS_TO",), max_depth: Optional[int] = None) -> list[tuple[str, int]]: """Breadth-first traversal returning (node id, hops) in visiting order.""" adj = self.out if direction == "down" else self.inn seen = {start: 0} order: list[tuple[str, int]] = [] q = deque([start]) while q: cur = q.popleft() d = seen[cur] if max_depth is not None and d >= max_depth: continue for e in adj[cur]: if e.type not in edge_types: continue nxt = e.target if direction == "down" else e.source if nxt not in seen: seen[nxt] = d + 1 order.append((nxt, d + 1)) q.append(nxt) return order def path(self, a: str, b: str, edge_types=("FLOWS_TO",)) -> Optional[list[str]]: prev: dict[str, Optional[str]] = {a: None} q = deque([a]) while q: cur = q.popleft() if cur == b: out = [] while cur is not None: out.append(cur) cur = prev[cur] return out[::-1] for e in self.out[cur]: if e.type in edge_types and e.target not in prev: prev[e.target] = cur q.append(e.target) return None def neighbors(self, nid: str) -> dict[str, list[dict[str, Any]]]: rel: dict[str, list[dict[str, Any]]] = defaultdict(list) for e in self.out[nid]: rel[f"{e.type} ->"].append({"id": e.target, "label": self.nodes[e.target].label, **({"attributes": e.attributes} if e.attributes else {})}) for e in self.inn[nid]: rel[f"<- {e.type}"].append({"id": e.source, "label": self.nodes[e.source].label, **({"attributes": e.attributes} if e.attributes else {})}) return dict(rel)skills/pid-to-kg/scripts/pidkg/summary.py7.6 KB24a99daf7421upstream ↗
"""Compact Markdown summary of a knowledge graph, written for LLM context windows.""" from __future__ import annotations from collections import defaultdict from typing import Optional from .loops import sensed_points from .models import KnowledgeGraph from .store import Graph def _esc(s: Optional[str]) -> str: return (s or "").replace("|", "\\|").replace("\n", " ") def _flow_order(g: Graph) -> list[str]: """Equipment/boundaries in reading order: sources first, then breadth-first along FEEDS.""" feeds_in: dict[str, int] = defaultdict(int) terminals = [n.id for n in g.kg.nodes if n.type in ("Equipment", "Boundary")] for e in g.kg.edges: if e.type == "FEEDS": feeds_in[e.target] += 1 starts = [t for t in terminals if feeds_in[t] == 0] starts.sort(key=lambda t: (g.nodes[t].type != "Boundary", g.nodes[t].sheets[:1], t)) order: list[str] = [] seen: set[str] = set() for s in starts + terminals: if s in seen: continue queue = [s] while queue: cur = queue.pop(0) if cur in seen: continue seen.add(cur) order.append(cur) queue.extend(e.target for e in g.out[cur] if e.type == "FEEDS") return order def write_summary(kg: KnowledgeGraph) -> str: g = Graph(kg) nodes = g.nodes sheets = [n for n in kg.nodes if n.type == "Sheet"] sheets.sort(key=lambda n: n.attributes.get("page", 0)) st = kg.stats out: list[str] = [] w = out.append w(f"# {kg.plant.name}: process knowledge graph summary") w("") w(f"Built from `{kg.plant.source_file or 'P&IDs'}` ({len(sheets)} sheets) by {kg.generator} " f"on {kg.generated_at[:10]}. The graph has {st.get('nodes', 0)} nodes and " f"{st.get('edges', 0)} edges; this summary covers equipment-level flow, instruments and " "control loops. Query `graph.duckdb` or `graph.json` for pipe-segment and valve detail.") w("") nbt = st.get("nodes_by_type", {}) w("| Equipment | Instruments | Valves & fittings | Pipe segments | Control loops | Off-page links |") w("|---|---|---|---|---|---|") w(f"| {nbt.get('Equipment', 0)} | {nbt.get('Instrument', 0)} | {nbt.get('Component', 0)} | " f"{nbt.get('PipeSegment', 0)} | {nbt.get('ControlLoop', 0)} | {nbt.get('OffPageConnector', 0)} |") w("") w("## Sheets") w("") w("| Page | Drawing | Title | Type |") w("|---|---|---|---|") for s in sheets: a = s.attributes w(f"| {a.get('page', '')} | {_esc(a.get('drawing_number'))} | {_esc(a.get('title'))} | " f"{a.get('drawing_type', '')} |") w("") w("## Process flow (equipment to equipment)") w("") w("Each line reads *upstream → downstream*, with the pipe lines and in-line items between them. " "Pipes, valves and instruments are collapsed; use `kg_query.py path` for the full route.") w("") drains: dict[str, set[str]] = defaultdict(set) for nid in _flow_order(g): feeds = [e for e in g.out[nid] if e.type == "FEEDS"] for e in [e for e in feeds if _is_drain(nodes[e.target])]: drains[nodes[e.target].label].add(nodes[nid].label) feeds = [e for e in feeds if not _is_drain(nodes[e.target])] if not feeds: continue n = nodes[nid] head = f"**{n.label}**" + (f" ({_esc(n.name)})" if n.name and n.name != n.label else "") for e in feeds: t = nodes[e.target] via = [] if e.attributes.get("via_lines"): via.append("lines " + ", ".join(e.attributes["via_lines"])) if e.attributes.get("via_items"): via.append("through " + ", ".join(e.attributes["via_items"])) if e.attributes.get("off_page"): via.append("crosses sheets") tname = f" ({_esc(t.name)})" if t.name and t.name != t.label else "" w(f"- {head} → **{t.label}**{tname}" + (f": {'; '.join(via)}" if via else "")) if drains: w("") w("Drains, vents and overflows (not process flow):") for d, srcs in sorted(drains.items()): w(f"- {d} ← {', '.join(sorted(srcs))}") w("") w("## Equipment") w("") w("| Tag | Kind | Description | Fed by | Feeds | Status | Sheets |") w("|---|---|---|---|---|---|---|") for n in sorted((n for n in kg.nodes if n.type == "Equipment"), key=lambda n: n.label): ups = sorted({nodes[e.source].label for e in g.inn[n.id] if e.type == "FEEDS"}) downs = sorted({nodes[e.target].label for e in g.out[n.id] if e.type == "FEEDS"}) w(f"| {n.label} | {n.kind or ''} | {_esc(n.name)} | {', '.join(ups)} | {', '.join(downs)} | " f"{n.status or ''} | {', '.join(_sheet_labels(n.sheets, nodes))} |") w("") w("## Instruments") w("") w("Location is where the instrument senses the process. Pipe locations read " "`line (upstream item -> downstream item)`.") w("") w("| Tag | Variable | Function | Mounted | Senses | Loop |") w("|---|---|---|---|---|---|") for n in sorted((n for n in kg.nodes if n.type == "Instrument"), key=lambda n: n.label): a = n.attributes senses = sensed_points(n.id, nodes, kg.edges) loops = [nodes[e.target].label for e in g.out[n.id] if e.type == "MEMBER_OF"] w(f"| {n.label} | {a.get('variable', '')} | {a.get('role', '')} | {a.get('location', '')} | " f"{_esc('; '.join(senses))} | {', '.join(loops)} |") w("") loops = [n for n in kg.nodes if n.type == "ControlLoop"] w("## Control loops") w("") if not loops: w("No control loops were identified.") else: w("| Loop | Type | Sensor(s) | Controller(s) | Final element(s) | Description |") w("|---|---|---|---|---|---|") lab = lambda ids: ", ".join(nodes[i].label for i in ids if i in nodes) # noqa: E731 for n in sorted(loops, key=lambda n: n.label): a = n.attributes w(f"| {n.label} | {n.kind} | {lab(a.get('sensors', []))} | {lab(a.get('controllers', []))} | " f"{lab(a.get('final_elements', []))} | {_esc(a.get('description'))} |") w("") opcs = [n for n in kg.nodes if n.type == "OffPageConnector"] if opcs: w("## Off-page connections") w("") w("| Sheet | Connector | Direction | Other drawing | Linked to |") w("|---|---|---|---|---|") for n in sorted(opcs, key=lambda n: n.id): a = n.attributes link = a.get("linked_to") or a.get("linked_from") or a.get("linked_item") or "unmatched" if link in nodes: link = f"{nodes[link].label} ({', '.join(_sheet_labels(nodes[link].sheets, nodes))})" w(f"| {', '.join(_sheet_labels(n.sheets, nodes))} | {_esc(n.label)} | {a.get('direction', '')} | " f"{_esc(a.get('other_drawing'))} | {_esc(link)} |") w("") sev = st.get("issues_by_severity", {}) w("## Data quality") w("") w(f"{sev.get('error', 0)} errors, {sev.get('warning', 0)} warnings and {sev.get('info', 0)} " "informational items; see `qa_report.md`. Every node carries `confidence` and the sheet " "region it was read from (`provenance`). This graph was machine-extracted from drawings " "and must be verified by a qualified engineer before use in design, operations or safety work.") return "\n".join(out) + "\n" _DRAIN_WORDS = ("DRAIN", "VENT", "OVERFLOW", "SUMP") def _is_drain(n) -> bool: return n.type == "Boundary" and any(w in (n.label or "").upper() for w in _DRAIN_WORDS[:3]) def _sheet_labels(sheet_ids: list[str], nodes) -> list[str]: return [nodes[s].label if s in nodes else s for s in sheet_ids]skills/pid-to-kg/scripts/probe_pdf.py3.5 KBc3343eda29a7upstream ↗
#!/usr/bin/env python3 """Classify every page of a P&ID PDF and start a work directory. Each page is classified as vector_text, vector_outlined, raster, raster_with_text, hybrid or blank, which decides how it is rendered and which hints are available. Writes <work>/manifest.json. Usage: python probe_pdf.py drawings.pdf --work work/my-plant --plant-name "My Plant" python probe_pdf.py permit.pdf --work work/my-plant --pages 47-60 # only the P&ID pages """ from __future__ import annotations import argparse import json import re import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) from pidkg.pdf import probe_pdf # noqa: E402 from render_pages import parse_pages # noqa: E402 def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("pdf", type=Path) ap.add_argument("--work", type=Path, required=True, help="work directory for this plant") ap.add_argument("--plant-name", help="human-readable plant name (default: PDF file name)") ap.add_argument("--plant-id", help="short id (default: slug of the plant name)") ap.add_argument("--pages", help="pages that hold the P&IDs, e.g. '47-60' or '50-51,62-63' " "(default: all). Later steps use only these pages.") args = ap.parse_args() if not args.pdf.exists(): print(f"error: {args.pdf} not found", file=sys.stderr) return 2 name = args.plant_name or args.pdf.stem plant_id = args.plant_id or re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-") probes = probe_pdf(str(args.pdf)) args.work.mkdir(parents=True, exist_ok=True) manifest_path = args.work / "manifest.json" previous = {} if manifest_path.exists(): old = json.loads(manifest_path.read_text(encoding="utf-8")) previous = {p["page"]: p for p in old.get("pages", [])} pages = [] for p in probes: d = p.to_dict() # Keep a rotation that was already confirmed for this page. if previous.get(p.page, {}).get("rotation") is not None: d["rotation"] = previous[p.page]["rotation"] d["rotation_source"] = previous[p.page].get("rotation_source") pages.append(d) selected = parse_pages(args.pages, len(probes)) if args.pages else None if args.pages and not selected: print(f"error: --pages {args.pages} selects no page of a {len(probes)}-page PDF", file=sys.stderr) return 2 manifest = {"source_pdf": str(args.pdf.resolve()), "plant_id": plant_id, "plant_name": name, "pages": pages} if selected: manifest["selected_pages"] = selected manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8") shown = [p for p in probes if not selected or p.page in selected] print(f"{args.pdf.name}: {len(probes)} pages -> {manifest_path}") if selected: print(f"using {len(selected)} selected pages: {args.pages}") print(f"{'page':>4} {'class':<17} {'chars':>6} {'paths':>6} {'img cov':>7} notes") for p in shown: print(f"{p.page:>4} {p.page_class:<17} {p.text_chars:>6} {p.path_objects:>6} " f"{p.max_image_coverage:>7.2f} {'; '.join(p.notes)}") classes = sorted({p.page_class for p in probes}) print("\nRoutes:") for c in classes: print(f" {c}: {next(p.route for p in probes if p.page_class == c)}") return 0 if __name__ == "__main__": sys.exit(main())skills/pid-to-kg/scripts/render_pages.py8.2 KBf1722ca0f502upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/pid-to-kg/scripts/requirements.txt283 Be71cc9e5c434upstream ↗
pypdfium2>=4.30 pillow>=10.0 pydantic>=2.6 duckdb>=1.1 networkx>=3.2 # Optional but recommended: OCR hints and automatic page orientation. # The OCR package supports Python 3.10-3.12 only; on newer Python the skill runs without it. rapidocr-onnxruntime>=1.3; python_version < "3.13"skills/pid-to-kg/references/extraction_procedure.md15 KB2c609c62fd86upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/pid-to-kg/references/isa_5_1.md5.8 KB1addb63fff8cupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/pid-to-kg/references/ontology.md7.6 KB26823553594dupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/pid-to-kg/references/schema/extraction.schema.json25 KBd62ea8a88fbfupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/pid-to-kg/references/schema/graph.schema.json6.2 KBb7b1ab2d094dupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
skills/pid-to-kg/references/symbol_guide.md4.3 KB95a2c84d7828upstream ↗
Contents are not inlined in the catalog. Open the upstream file.
README.mdupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
Plain-text copy, no login: /skill-bundles/scottduncanai-pid-to-knowledge-graph-skills-pid-to-kg.md