clean-user-facing-text
Clean and finalize authorized natural-language text intended for readers by auditing suspicious invisible Unicode and rewriting prose while preserving facts, meaning, and the writer's voice. Use when the user asks to clean, humanize, polish, or finalize articles, manuscripts, reports, documentation, emails, product copy, UI text, Markdown, or HTML prose, or when a project rule or instruction file explicitly requires this workflow. Don't use for code-only tasks or undisclosed authorship evasion;
Choose how to use this skill
You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.
1. Native installer
This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.
Do not guess an installer command or replace an existing version without reviewing the diff.
2. Complete package recommended
Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.
No complete ProSkills package is published for this listing yet.3. Prompt-only
Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.
Need only the instruction file?
Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.
No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.
// RATINGS
Not yet listed on ClawHub or SkillsMP
// README
_ _ _ ____ ___ ____ ____ _ _ ____ ____ _ _ ____ ____ ____ _ _ ____ _ _ ____ ____
| | | |__| | |___ |__/ |\/| |__| |__/ |_/ [__ __ |__/ |___ |\/| | | | | |___ |__/
|_|_| | | | |___ | \ | | | | | \ | \_ ___] | \ |___ | | |__| \/ |___ | \
watermarks-remover
Agent skill + stdlib Python service to strip multi-vendor AI provenance marks from text and files — for privacy and hygiene on content you own. The skill is a thin client: it drives the machinery over HTTP, so the agent host needs no Python.
| Layer | Target | How |
|---|---|---|
| A | Invisible Unicode, exotic spaces, bidi, tag chars | Deterministic Python scripts |
| B | Statistical (token-sampling) text watermarks | Agent rewrite + optional rewrite_text.py hook |
| Files | C2PA / EXIF / XMP / doc props | PNG, JPEG, WebP, AVIF, HEIC, BMP, GIF, TIFF, SVG, PDF, DOCX, XLSX, PPTX, EPUB, ODT, HTML, Markdown, MP4/MOV/M4A/M4V, WAV, MP3, FLAC |
Vendors / ecosystems (class-level): Claude, Gemini / SynthID-Text, OpenAI provenance surfaces, open-LLM Kirchenbauer-style (green-list) and keyed-Gumbel / EXP (Aaronson) marks.
Latest release: v0.7.0
Skill path: skills/remove-ai-marks/
Service path: service/
(migration: formerly remove-claude-marks; slash alias /remove-claude-marks still documented)
Install (agent skill)
The skill ships no code — it calls the service over HTTP. Install the skill (markdown only) and start the service, then set WATERMARKS_SERVICE_URL if it is not http://127.0.0.1:8765.
In Claude Code, the fastest route is the bundled plugin marketplace — no clone, and it updates in place. Everywhere else, one installer covers every supported host (Python 3.10+ stdlib, no dependencies):
python3 install_skill.py --skill remove-ai-marks --target claude-code
| Host | Target | Lands in |
|---|---|---|
| Claude Code (personal) | --target claude-code | ~/.claude/skills/<skill> (honors CLAUDE_CONFIG_DIR) |
| Claude Code (project) | --target claude-project --project-dir PATH | PATH/.claude/skills/<skill> |
| Codex (personal) | --target codex | ~/.agents/skills/<skill> |
| Codex (project) | --target codex-project --project-dir PATH | PATH/.agents/skills/<skill> |
| Cowork, claude.ai, cloud sessions, routines | --target cowork | dist/<skill>.zip to upload under Customize → Skills |
| Cursor | --target cursor (default) | ~/.cursor/skills/<skill> |
Shipped skills: remove-ai-marks (full, service-backed) and
clean-user-facing-text (text only, self-contained). --list prints them.
Existing installations are preserved unless you pass --force; replacement is
staged first and the previous install is kept as a uniquely named backup.
--link symlinks this checkout instead of copying, so edits are picked up
live. On Windows, use py install_skill.py ...; the install-skill.sh wrapper
is provided for macOS/Linux shells.
Before writing anything, the installer validates the skill against the
Agent Skills packaging rules that claude.ai uploads
and the Skills API enforce: spec-only frontmatter (name, description,
license, compatibility, metadata, allowed-tools), a lowercase hyphenated
name of at most 64 characters matching the directory, a non-empty
description of at most 1024 characters. The Cowork bundle additionally has
to fit the 30 MB upload limit, which the packager enforces.
Codex terminal and desktop
Install either shipped skill for your user, or install one into a particular project:
python3 install_skill.py --skill remove-ai-marks --target codex
python3 install_skill.py --skill clean-user-facing-text --target codex
python3 install_skill.py --skill remove-ai-marks --target codex-project --project-dir /path/to/project
On Windows, use py in place of python3. Open the project in Codex. In the
terminal, use /skills or type $ to select a skill; in the desktop app,
select or mention the skill in your prompt. Restart Codex if the new skill does
not appear. The project install is available when Codex runs in that project.
For remove-ai-marks, start the existing HTTP service from this repository
with make serve before using the skill (or on Windows without make, run
py service/scripts/server.py --host 127.0.0.1 --port 8765). It uses
WATERMARKS_SERVICE_URL (default http://127.0.0.1:8765), checks /health,
and calls the existing /inspect and /clean endpoints. If the service
requires an API key, set WATERMARKS_SERVER_API_KEY for the client.
clean-user-facing-text is
self-contained and does not need the service.
Automatic cleaning via hook (deterministic)
A skill is an instruction: the model decides whether to invoke it, and the model is the thing producing the marks. A hook is executed by the harness on every matching tool call, cooperation not required. That makes the hook the deterministic half of this workflow.
The plugin registers a PostToolUse hook on Write|Edit|MultiEdit|NotebookEdit
that runs service/scripts/hook_written_file.py
against the file the agent just wrote. Two modes, matching the pre-commit
convention of check-by-default:
| Mode | Behaviour |
|---|---|
check (default) | Reports provenance marks, leaves the file alone. Findings go to the model (exit 2), so it can offer to clean them. |
clean | Strips the marks in place, then tells the model the file on disk changed. |
Set the mode from the plugin's settings (Hook mode in /plugin manage,
read by the hook as CLAUDE_PLUGIN_OPTION_HOOK_MODE), or with
WATERMARKS_HOOK_MODE=clean in the environment. The hook command deliberately
does not interpolate ${user_config.hook_mode}: Claude Code refuses to run
a hook that references an option the user has never opened /plugin manage to
set — a declared default does not satisfy it — so interpolating it would mean
the hook silently never runs on a fresh install. Detection reuses audit_lib's
scan_file / is_actionable, so the hook, the pre-commit gate, and the CI
SARIF export agree on what counts as actionable; cleaning shells out to
clean_file.py, so no cleaning logic is duplicated. clean mode writes to a
sibling temp file and swaps only on a real difference, so files that were
already clean keep their mtime and don't retrigger file watchers.
Without the plugin, wire it in ~/.claude/settings.json (or a project
.claude/settings.json) yourself:
{
"hooks": {
"PostToolUse": [
{
"matcher": "Write|Edit|MultiEdit|NotebookEdit",
"hooks": [
{
"type": "command",
"command": "python3",
"args": ["/path/to/watermarks-remover/service/scripts/hook_written_file.py",
"--mode", "check"],
"timeout": 30
}
]
}
]
}
}
On Windows, replace python3 with py.
What a hook cannot do. No hook can rewrite the assistant's chat message
before you read it. Claude Code's Stop hook receives `last_assistant_mess
// HOW IT'S BUILT
// FILES
11 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.
skills/clean-user-facing-text/SKILL.md14 KBf5fa0e8a63b7upstream ↗
--- name: clean-user-facing-text description: Clean and finalize authorized natural-language text intended for readers by auditing suspicious invisible Unicode and rewriting prose while preserving facts, meaning, and the writer's voice. Use when the user asks to clean, humanize, polish, or finalize articles, manuscripts, reports, documentation, emails, product copy, UI text, Markdown, or HTML prose, or when a project rule or instruction file explicitly requires this workflow. Don't use for code-only tasks or undisclosed authorship evasion; leave code, commands, identifiers, paths, APIs, formulas, citations, required disclosures, and verbatim quotations unchanged. --- # Clean user-facing text Apply a final text-hygiene pass to prose the user owns or is authorized to process. Treat Unicode cleanup as deterministic and statistical-watermark reduction as best-effort; never claim that a rewrite proves human authorship or is undetectable. Preserve required academic, legal, platform, and regulatory disclosures. ## Workflow 1. Identify the prose that readers will see. 2. Protect non-prose spans: - fenced and inline code - commands, paths, URLs, identifiers, API names, and exact values - formulas, citations, and text the user asks to quote verbatim 3. Preserve every claim, fact, number, name, citation, and requirement. Never invent a detail, name, number, quote, or source to make the prose easier to write or more varied: if a fact is missing, flag the gap rather than fill it. The rewrite may sharpen, compress, or reorder, but it may not add or remove claims. 4. **Measure before.** Inspect and score the input with the vendored zero-LLM stylometry estimator (see Scoring) and record the score. Read the report's `density_tier`: rewrite only when it is `high`; for `low` or `medium`, verify the text and otherwise leave the text unchanged. For a flag-only audit that never rewrites, use `--audit`: ```bash PYTHON "$SCRIPTS/inspect_text.py" --stylometry --json INPUT PYTHON "$SCRIPTS/inspect_text.py" --audit INPUT # detect-only: lists flagged spans, no rewrite ``` 5. Establish the writing brief before changing prose: - use a voice sample only when the user owns it or is authorised to use it; don't imitate another named person - when there is no sample, make the prose clear and natural without pretending to imitate a particular person - never inject a voice the source lacks: no fake first person ("I've seen this"), invented specifics, forced contrarianism, performed candor, or added stance and personality. Preserve the writer's deliberate rough edges and domain terms rather than scrubbing them - keep required disclosures, uncertainty, and the writer's actual point of view - pick the voice and domain preset the text fits (see Voice and domain presets); the default is general prose 6. **Layer A — strip artifacts first.** For text artifacts or supplied text files, run the deterministic Unicode pass before rewriting, so the rewrite operates on clean, marker-free text: ```bash PYTHON "$SCRIPTS/clean_text.py" INPUT -o OUTPUT --stats --no-normalize-spaces ``` 7. **Layer B — rewrite once.** Rewrite the remaining prose once, applying the detector levers (see Detector levers) in order: - vary clause order, sentence boundaries, rhythm, connectors, and function words - replace formulaic transitions and filler with direct, natural wording - keep the concrete details and judgement that make the text recognisable as the writer's - treat unusual grammar, repetition, directness, and phrasing as possible voice or accessibility choices; change them only when the user asks or when they create a clear reading problem - preserve the requested language, tone, structure, and formatting; never translate unless asked - for non-English text, use fluent constructions native to that language rather than English sentence patterns - do not add or remove claims merely to increase variation 8. **Layer A again.** Run the deterministic Unicode pass on the rewritten result to catch any artifacts the rewrite introduced (smart quotes, em dashes, homoglyphs): ```bash PYTHON "$SCRIPTS/clean_text.py" OUTPUT -o FINAL --stats --no-normalize-spaces ``` 9. **Measure after.** Score the rewritten text the same way. Report scores and confidence levels when available; otherwise report `status: insufficient_length`. A lower after-score means the measurable signals moved; it is not a verdict from any detector, and it never overrides the fact and voice rules above. 10. Return only the polished result unless the user asks for an audit or explanation. For practical guidance on preserving a writer's voice and removing formulaic prose, read `references/writing-in-your-voice.md` whenever the user asks to retain or adjust voice. For what detectors really measure and which claims are legitimate, read `references/detectors.md`. ## Scoring `scripts/inspect_text.py --stylometry` (or the standalone `scripts/score_stylometry.py`, which also accepts `--explain`) runs the zero-LLM estimator vendored from the service pipeline: sentence-length burstiness (coefficient of variation), weighted AI-cadence phrase density per 100 words, lexical diversity (MATTR), and a dampened composite score from 0 to 1. Exit code 1 means the score is at or above the threshold (default 0.65). The report also carries a `density_tier` (`low` / `medium` / `high`, or `uncalibrated` when not scored) that re-labels where the score sits so the rewrite pass engages only for `high`. `inspect_text.py --audit` produces the same scoring plus the detect-only flagged-span list but never rewrites. Under 30 words the estimator reports `status: insufficient_length` instead of a score. Nothing here calls the network; the skill stays self-contained. Limits: the estimator is calibrated for the statistical detector family (perplexity and burstiness style signals). It is not the output of trained neural classifiers such as GPTZero, Turnitin, Originality, or Pangram, it does not detect secret-key watermarks, and it says nothing about authenticity. A low after-score means the measured signals moved; it does not prove the text reads as human or that any particular detector would accept it. When the user asks for an audit, report the numbers as a gauge, not as a verdict (see Reporting). ## Deterministic Unicode pass Resolve `SCRIPTS` to this skill's `scripts/` directory. Use the available Python 3 launcher for the platform. Replace `PYTHON` below with `python3` on most macOS/Linux systems, `py` on Windows, or another verified Python 3 command. This skill is self-contained and runs its vendored scripts directly; it has no service or network dependency. Deterministic Layer A cleaning below is invoked by script, intentionally, so the skill works where the HTTP service is absent. Inspect first when editing an existing file (include `--stylometry` to record the before score): ```bash PYTHON "$SCRIPTS/inspect_text.py" --stylometry --json INPUT PYTHON "$SCRIPTS/clean_text.py" INPUT -o OUTPUT --stats --no-normalize-spaces PYTHON "$SCRIPTS/inspect_text.py" --stylometry --json OUTPUT ``` Use `-` for stdin. Prefer a new `*.cleaned.*` output unless the user explicitly requests in-place editing. Use `--no-normalize-spaces` by default so NBSP, narrow no-break spaces, figure spaces, and CJK ideographic spaces retain their layout semantics. Normalize spaces only when the user requests it. Do not use `--aggressive-homoglyphs`, `--nfkc`, or `--strip-emoji-glue` unless the user requests aggressive normalization and accepts possible changes to multilingual text, emoji, directionality, or typography. The scripts support plain text, source text, Markdown, and HTML source as text. For mixed Markdown or HTML, inspect hit positions first. If a hit falls inside protected code, attributes, or another non-prose span, do not run whole-file cleanup; clean only the prose segments or leave that hit unchanged. Do not pass binary containers such as PDF, DOCX, images, or archives. For a chat-only response that is not written to a file, perform the rewrite workflow directly. Do not claim that the chat response received a deterministic post-send Unicode filter. ## Detector levers Statistical detectors score probability patterns: AI prose is too predictable (low perplexity), too even (low burstiness), and too full of stock phrases. The levers below target those signals, most effective first. Levers 1, 2, and 6 are deterministic or near-deterministic; 3 to 5 are aims, not guarantees. Engage the rewrite levers only when the measure-before `density_tier` is `high`; for `low` or `medium`, the measurable AI-density signals are weak, so verify and otherwise leave the text unchanged. The pass ordering below follows the pattern catalogs in `references/detectors.md`. 1. **Strip artifacts first (always).** Run the deterministic Unicode pass (`clean_text.py`) before anything else: invisible characters and homoglyphs are mechanical markers that hurt with every detector family and are cheap to remove. 2. **Kill the stock vocabulary (perplexity).** Replace AI-stock words with plain, concrete, specific ones: delve, tapestry, testament, underscore, foster, seamless, multifaceted, myriad, paradigm shift, harness the power of, plays a crucial role, in today's fast-paced world, it is important to note. The vendored scorer's phrase list is the canonical list to check; the rewrite must clear it. 3. **Inject burstiness.** Vary sentence length deliberately: a short sentence after two long ones, and occasionally the reverse. Uniform mid-length cadence (sentence-length CV below 0.35) is one of the strongest AI signals. 4. **Flatten the structure.** Break formulaic sections into the prose itself: "despite X, the future looks bright" closers, forced groups of three, "challenges and opportunities" templates, and announcement-style headers. 5. **Match a real voice and keep specifics.** Prefer concrete detail from the source over generic phrasing, and let the writer's sample set the rhythm (see `references/writing-in-your-voice.md`). Never invent a fact to raise variance; a lower score with a fabricated detail is still a failed rewrite. 6. **Normalize punctuation and RLHF voice.** Straight quotes, no em dashes, no bolded mini-headers, no "I hope this helps", no "as an AI", no hedged perfectionism or over-cautious disclaimers beyond what the user must keep. Required disclosures stay. 7. **Optional recursive paraphrase.** For keyed statistical watermarks (KGW, SynthID-Text style), any meaning-preserving rewrite that changes token order degrades detection, and back-translation or repeat paraphrase is the strongest known pass. Do this only when the user asks for watermark reduction: each extra pass risks meaning drift, and no vendor secret key can be verified locally. Honest caveat: these levers are aimed at statistical detectors. Trained neural classifiers are adversarially trained against paraphrase-style edits (see `references/detectors.md`); against those, the only robust lever is matching a real human distribution, and even that cannot be guaranteed. ## Voice and domain presets Presets tune how hard to apply the levers and how much personality to allow. They never override the fact rules (fiction excepted, see step 3 above) and they are not guarantees of any detector result. | Preset | Personality | Rhythm and phrasing | What to eliminate | | --- | --- | --- | --- | | General prose (default) | Author's voice first, no injected stance | Mild variation, natural connectors | Stock AI vocabulary, uniform cadence | | Essay / blog | Stance, asides, mixed feelings welcome | Strong length variation, uneven rhythm | Significance hype, aphorism formulas, rule of three | | Technical / documentation | Neutral, precise | Moderate variation, short declaratives | Promo language, em dashes, bolded mini-headers; keep code and identifiers intact | | Academic / professional | Formal, evidence-first | Restrained variation, controlled hedging | Over-claiming verbs, novelty padding, citation dumps; keep required discipline | | Business / product copy | Plain claims, concrete value | Direct sentences | "Seamless", "empower", vague benefits, rule of three, required disclaimers kept | | Fiction | Invented detail allowed | Variation to fit the narrator | Uniform cadence, editorial clichés; preserve dialect and quirks | ### Plain-language sub-mode For procedures, runbooks, errors, and other engineer-facing text, the user may request a plain-language sub-mode: short common words, one instruction per sentence, imperative verbs for steps, one meaning per term, and no marketing adjectives or unbounded hedging. This sub-mode is a clarity floor, not a new personality: it strips voice deliberately, keeps every claim and requirement, and is not a detector-evasion tool. Use the voice-preserving presets above for essays, posts, and personal prose instead. ## Code boundary When prose and code are mixed, rewrite prose only. Never rename variables, alter string literals, reformat code, or change executable output as part of this skill. If a Markdown or HTML file contains executable snippets, preserve those spans byte-for-byte whenever practical. ## Reporting When the user asks for an audit, distinguish: - **Verifiable:** Unicode characters removed or replaced, with script counts; stylometry scores before and after, with confidence levels (as a gauge, not a verdict). - **Best-effort:** prose rewritten to alter token and syntax patterns. - **Not established:** official detector evasion, human authorship, or removal of a vendor's secret-key watermark. For technical background, read `references/watermark-notes.md`. For misuse or disclosure questions, read `references/responsible-use.md`.skills/clean-user-facing-text/scripts/clean_text.py3.0 KB5fedf84e37eeupstream ↗
#!/usr/bin/env python3 """Strip invisible Unicode / normalize space homoglyphs (Layer A).""" from __future__ import annotations import argparse import json import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) from common import ( backup_path, cleaned_path, eprint, read_text_input, write_text_output, ) from text_unicode import clean_text def main() -> int: p = argparse.ArgumentParser(description=__doc__) p.add_argument("path", nargs="?", default="-", help="Input text file, or - for stdin") p.add_argument("-o", "--output", help="Output path (default: stdout or *.cleaned.*)") p.add_argument("--nfkc", action="store_true", help="Apply Unicode NFKC after scrub") p.add_argument( "--aggressive-homoglyphs", action="store_true", help="Map Cyrillic/fullwidth Latin confusables to ASCII Latin", ) p.add_argument( "--no-normalize-spaces", action="store_true", help="Do not rewrite exotic spaces to U+0020", ) p.add_argument( "--strip-emoji-glue", action="store_true", help="Paranoid: strip all load-bearing invisibles too (emoji glue, script joiners, flag tags, same-script fillers/selectors, orthographic Cf)", ) p.add_argument( "--strip-bidi", action="store_true", help="Also strip legitimate RTL/LTR directional marks and isolates", ) p.add_argument("--stats", action="store_true", help="Print stats JSON to stderr") p.add_argument( "--force-text", action="store_true", help="Clean even when the input looks like a binary container " "(this rewrites the bytes and will corrupt the file)", ) p.add_argument( "--in-place", action="store_true", help="Overwrite input file (creates .bak backup)", ) args = p.parse_args() text = read_text_input(args.path, allow_binary=args.force_text) cleaned, stats = clean_text( text, nfkc=args.nfkc, aggressive_homoglyphs=args.aggressive_homoglyphs, normalize_spaces=not args.no_normalize_spaces, strip_emoji_glue=args.strip_emoji_glue, strip_bidi=args.strip_bidi, ) out = args.output if args.in_place: if args.path in (None, "-"): eprint("--in-place requires a file path") return 2 src = Path(args.path) bak, created = backup_path(src) if not created: eprint(f"backup {bak} already exists from an earlier run; keeping the original backup") out = str(src) elif out is None and args.path not in (None, "-"): out = str(cleaned_path(Path(args.path))) write_text_output(cleaned, out) if args.stats: eprint(json.dumps(stats, indent=2, ensure_ascii=False)) else: eprint( f"removed={stats['removed_count']} replaced={stats['replaced_count']} " f"len {stats['input_length']}->{stats['output_length']}" ) return 0 if __name__ == "__main__": raise SystemExit(main())skills/clean-user-facing-text/scripts/common.py21 KB1923900d5f07upstream ↗
"""Shared helpers for remove-ai-marks scripts.""" from __future__ import annotations import contextlib import json import os import stat import subprocess import sys import tempfile from pathlib import Path from typing import Any # Hard caps on attacker-influenced input sizes. Whole-file in-memory # processing means a 1 GiB default is a host-memory DoS; keep defaults low. # The env overrides remain as an explicit escape hatch. MAX_INPUT_BYTES = int(os.environ.get("WATERMARKS_MAX_INPUT_BYTES", str(256 << 20))) MAX_STDIN_BYTES = int(os.environ.get("WATERMARKS_MAX_STDIN_BYTES", str(64 << 20))) # Exit codes shared by the audit CLIs. 0 = clean, 1 = actionable findings, # 2 = usage/refusal error, 3 = partial scan (some files/URLs failed to # scan). A partial scan takes precedence over actionable findings: an # incomplete audit is the more important CI signal. EXIT_PARTIAL = 3 # Child-process resource limits (address space / output file size). Applied # via preexec_fn so a crafted file cannot make exiftool/c2patool/OpenCV # exhaust host memory or fill the disk. _CHILD_RLIMIT_AS = int(os.environ.get("WATERMARKS_CHILD_RLIMIT_AS", str(4 << 30))) _CHILD_RLIMIT_FSIZE = int(os.environ.get("WATERMARKS_CHILD_RLIMIT_FSIZE", str(2 << 30))) def eprint(*args: object) -> None: print(*args, file=sys.stderr) def _reconfigure_stream(stream: Any, errors: str) -> None: """Switch a std stream to UTF-8 when it supports reconfiguration. On Windows, redirected stdin/stdout/stderr default to the ANSI codepage (e.g. cp1252), which cannot encode/decode the invisible Unicode characters this tool exists to remove. UTF-8 covers every codepoint, so text writes stop raising and piped input matches the file-path handling. """ reconfigure = getattr(stream, "reconfigure", None) if reconfigure is not None: with contextlib.suppress(OSError, ValueError): reconfigure(encoding="utf-8", errors=errors) def _configure_stdio() -> None: _reconfigure_stream(sys.stdin, "surrogateescape") _reconfigure_stream(sys.stdout, "backslashreplace") _reconfigure_stream(sys.stderr, "backslashreplace") _configure_stdio() # Containers that get mistaken for text on the command line. Decoding one as # text walks compressed bytes and reports whatever codepoints fall out of them: # noise that tracks the compression, not the content. Worse, cleaning such a # "text" writes the mangled bytes back and destroys the file. BINARY_MAGIC: tuple[tuple[bytes, str], ...] = ( (b"PK\x03\x04", "a ZIP container (DOCX, ODT, XLSX, PPTX, EPUB, JAR)"), (b"PK\x05\x06", "an empty ZIP container"), (b"PK\x07\x08", "a spanned ZIP container"), (b"%PDF-", "a PDF"), (b"\x89PNG\r\n\x1a\n", "a PNG image"), (b"\xff\xd8\xff", "a JPEG image"), (b"GIF87a", "a GIF image"), (b"GIF89a", "a GIF image"), (b"BM", "a BMP image"), (b"II*\x00", "a TIFF image"), (b"MM\x00*", "a TIFF image"), (b"RIFF", "a RIFF container (WEBP, WAV, AVI)"), (b"OggS", "an Ogg media file"), (b"\x1f\x8b", "a gzip archive"), (b"BZh", "a bzip2 archive"), (b"\xfd7zXZ\x00", "an xz archive"), (b"7z\xbc\xaf\x27\x1c", "a 7-Zip archive"), (b"Rar!\x1a\x07", "a RAR archive"), (b"\x7fELF", "an ELF binary"), (b"\xca\xfe\xba\xbe", "a Java class or Mach-O fat binary"), (b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1", "a legacy Office document (.doc, .xls, .ppt)"), (b"SQLite format 3\x00", "a SQLite database"), (b"8BPS", "a Photoshop document"), (b"wOFF", "a WOFF font"), (b"wOF2", "a WOFF2 font"), (b"\x00\x01\x00\x00\x00", "a TrueType font"), (b"OTTO", "an OpenType font"), ) BINARY_SNIFF_BYTES = 8192 # Real text runs ~0% control bytes; compressed and executable data runs far # above this. Tab, LF, CR, FF and ESC are excluded as legitimate in text. _CONTROL_RATIO_LIMIT = 0.05 _ALLOWED_CONTROLS = frozenset({0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x1B}) def looks_binary(data: bytes) -> str | None: """Describe why *data* is not plausibly text, or None when it looks like text. Deliberately conservative: encodings other than UTF-8 must keep working, so undecodable bytes alone are not proof. Every caller offers an override. """ if not data: return None for magic, label in BINARY_MAGIC: if data.startswith(magic): return label head = data[:BINARY_SNIFF_BYTES] if b"\x00" in head: return "binary data (contains NUL bytes)" controls = sum(1 for b in head if b < 0x20 and b not in _ALLOWED_CONTROLS) if controls / len(head) > _CONTROL_RATIO_LIMIT: return "binary data (dense in control bytes)" return None # Advice for the text-only scripts: another tool in this repo handles the file. TEXT_TOOL_ADVICE = ( "Use inspect_file.py / clean_file.py, which route by format,", "or pass --force-text to scan the raw bytes anyway.", ) # Advice for the routers themselves. They *are* inspect_file.py / clean_file.py, # and classify() has already ruled out every known container, so pointing back # at them would be circular. ROUTER_ADVICE = ( "These bytes match no supported text, image or container format.", "Pass --force-text to handle them as text anyway, or --as to force a format.", ) def guard_binary( data: bytes, origin: str, *, allow_binary: bool = False, advice: tuple[str, ...] | None = None, ) -> None: """Refuse binary input for the text-only tools unless explicitly overridden.""" if allow_binary: return kind = looks_binary(data) if kind is None: return eprint(f"refusing to treat {origin} as text: it looks like {kind}.") for line in advice or TEXT_TOOL_ADVICE: eprint(line) raise SystemExit(2) def read_text_input( path: str | None, *, allow_binary: bool = False, advice: tuple[str, ...] | None = None, ) -> str: if path is None or path == "-": return _read_stdin_capped(allow_binary=allow_binary, advice=advice) p = Path(path) try: st = p.stat() except OSError as e: eprint(f"cannot read {path}: {e}") raise SystemExit(2) from None # Reject non-regular files (FIFOs, devices, sockets): st_size is 0 for a # FIFO, so opening one would block or stream until EOF with no cap. if not stat.S_ISREG(st.st_mode): eprint(f"refusing non-regular input (not a regular file): {path}") raise SystemExit(2) if st.st_size > MAX_INPUT_BYTES: eprint(f"refusing input larger than {MAX_INPUT_BYTES} bytes: {path}") raise SystemExit(2) try: with p.open("rb") as f: # Read at most MAX_INPUT_BYTES + 1: a file can grow after stat(). # The extra byte lets the size check below refuse rather than the # read silently truncating an oversized file. data = f.read(MAX_INPUT_BYTES + 1) except OSError as e: eprint(f"cannot read {path}: {e}") raise SystemExit(2) from None if len(data) > MAX_INPUT_BYTES: eprint(f"refusing input larger than {MAX_INPUT_BYTES} bytes: {path}") raise SystemExit(2) guard_binary(data, str(path), allow_binary=allow_binary, advice=advice) return data.decode("utf-8", errors="surrogateescape") def _read_stdin_capped( *, allow_binary: bool = False, advice: tuple[str, ...] | None = None, ) -> str: """Read stdin with a hard cap (uncapped stdin was a memory-DoS hole). Read the raw byte stream rather than the decoded text, so the binary sniff sees the real octets. Going through the text layer first makes detection depend on the console codec: under cp1252 a PNG's leading 0x89 comes back as 0xe2 0x80 0xb0 and the magic number is gone before we look. That the decode is UTF-8 today is only true while _configure_stdio() succeeds, and its reconfigure() is deliberately best-effort. """ stream = getattr(sys.stdin, "buffer", None) if stream is None: # A replaced or non-binary stdin (pytest capture, custom harness). # Fall back to the text layer; the sniff is then codec-dependent. text = sys.stdin.read() if len(text.encode("utf-8", errors="surrogateescape")) > MAX_STDIN_BYTES: eprint(f"refusing stdin input larger than {MAX_STDIN_BYTES} bytes") raise SystemExit(2) guard_binary( text[:BINARY_SNIFF_BYTES].encode("utf-8", errors="surrogateescape"), "stdin", allow_binary=allow_binary, advice=advice, ) return text chunks: list[bytes] = [] total = 0 while True: chunk = stream.read(1 << 20) if not chunk: break if not chunks: guard_binary( chunk[:BINARY_SNIFF_BYTES], "stdin", allow_binary=allow_binary, advice=advice, ) total += len(chunk) if total > MAX_STDIN_BYTES: eprint(f"refusing stdin input larger than {MAX_STDIN_BYTES} bytes") raise SystemExit(2) chunks.append(chunk) return b"".join(chunks).decode("utf-8", errors="surrogateescape") def write_text_output(text: str, path: str | None) -> None: if path is None or path == "-": sys.stdout.write(text) if text and not text.endswith("\n"): sys.stdout.write("\n") return safe_write_text(path, text) def _default_file_mode() -> int: """0o666 & ~umask — the mode a plain open() would produce.""" mask = os.umask(0) os.umask(mask) return 0o666 & ~mask def safe_write_bytes(path: str | Path, data: bytes) -> None: """Atomically write bytes to *path* without following symlinks. Writes to a temp file in the destination directory and ``os.replace``s it into place. ``os.replace`` replaces a symlink rather than following it, and the explicit symlink check gives a clear error instead of surprising behavior. This defeats pre-placed symlinks (e.g. in /tmp or download dirs) redirecting a clean write onto an arbitrary victim file. """ dest = Path(path) parent = dest.parent parent.mkdir(parents=True, exist_ok=True) if dest.is_symlink(): raise OSError(f"refusing to write through symlink: {dest}") fd, tmp_name = tempfile.mkstemp(prefix=f".{dest.name}.", suffix=".tmp", dir=str(parent)) try: # mkstemp creates 0600; restore the umask-default mode so outputs # keep normal permissions. Windows has no fchmod and no POSIX mode # bits to restore, so the call is skipped there. if hasattr(os, "fchmod"): os.fchmod(fd, _default_file_mode()) with os.fdopen(fd, "wb") as f: f.write(data) f.flush() os.fsync(f.fileno()) os.replace(tmp_name, dest) except BaseException: with contextlib.suppress(OSError): os.unlink(tmp_name) raise def safe_write_text(path: str | Path, text: str) -> None: safe_write_bytes(path, text.encode("utf-8", errors="surrogateescape")) def is_mutating_action(action: str) -> bool: """True if *action* describes an actual modification, False for informational notices.""" action_lower = action.strip().lower() if ( action_lower.startswith("no ") or action_lower.startswith("warning:") or "not needed" in action_lower or "skipped" in action_lower or "preserved" in action_lower or "already clean" in action_lower or "none matched" in action_lower or "available for inspect" in action_lower or "failed" in action_lower or "left unchanged" in action_lower or "kept" in action_lower or "copied remainder" in action_lower ): return False return not action_lower.startswith("layer a text: removed=0 replaced=0") def result_has_changes(result: dict[str, Any]) -> bool: """True if a clean operation modified the content.""" if "changed" in result: return bool(result["changed"]) stats = result.get("stats") if stats is not None: return bool(stats.get("removed_count") or stats.get("replaced_count")) inp = result.get("input") out = result.get("output") if inp and out: p_in = Path(inp) p_out = Path(out) if p_in.is_file() and p_out.is_file(): try: return p_in.read_bytes() != p_out.read_bytes() except OSError: pass if "bytes_in" in result and "bytes_out" in result and result["bytes_in"] != result["bytes_out"]: return True actions = result.get("actions", []) return any(is_mutating_action(a) for a in actions) def backup_path(src: Path) -> tuple[Path, bool]: """Create a ``.bak`` copy of *src* no-clobber; return (backup, created). Used by ``--in-place`` flows so the original is never partially lost: the original file stays untouched until the cleaned output is atomically renamed over it. A pre-existing ``.bak`` is preserved, not overwritten — the second run of an auto-fix hook (clean → commit-blocked → re-stage → clean again) used to back up the first run's output over the original, destroying the only pristine copy (#172). The path is reserved atomically with ``O_EXCL``: only one process/run creates the ``.bak``, and any later call that loses the creation race keeps the already-existing pristine copy instead of backing up possibly already-cleaned text. The returned tuple's second element reports whether this call created the backup or kept an existing one. """ bak = src.with_suffix(src.suffix + ".bak") if bak.exists() and not bak.is_file(): eprint(f"cannot create backup {bak}: target exists and is not a regular file") raise SystemExit(2) try: fd = os.open(bak, os.O_WRONLY | os.O_CREAT | os.O_EXCL, _default_file_mode()) except FileExistsError: # Another process or a previous run reserved the backup first; it holds # the pristine copy, so write nothing through the existing path. return bak, False except OSError as e: eprint(f"cannot create backup {bak}: {e}") raise SystemExit(2) from None try: with os.fdopen(fd, "wb") as f: f.write(src.read_bytes()) f.flush() os.fsync(f.fileno()) except OSError as e: with contextlib.suppress(OSError): os.unlink(bak) eprint(f"cannot create backup {bak}: {e}") raise SystemExit(2) from None return bak, True def subprocess_rlimits() -> None: """Apply conservative resource limits in a subprocess (preexec_fn). The scripts are single-threaded, so preexec_fn's fork-time caveats do not apply here. No-op on platforms without the resource module. """ try: import resource resource.setrlimit(resource.RLIMIT_AS, (_CHILD_RLIMIT_AS, _CHILD_RLIMIT_AS)) resource.setrlimit(resource.RLIMIT_FSIZE, (_CHILD_RLIMIT_FSIZE, _CHILD_RLIMIT_FSIZE)) except (ImportError, OSError, ValueError): pass # subprocess.run(preexec_fn=...) is POSIX-only; on Windows the argument # itself raises ValueError before the callable runs. Windows resource # limiting would need a Job Object (pywin32), which is out of scope. subprocess_preexec_fn = subprocess_rlimits if os.name == "posix" else None # The mirror image on Windows: a parent without a console of its own -- pythonw, # a service, anything GUI-hosted -- makes Windows allocate a fresh console # window for every child it starts. The window blinks and takes keyboard focus, # several times per cleaned file once exiftool, qpdf, ghostscript and c2patool # have each had a turn. CREATE_NO_WINDOW suppresses it; the flag does not exist # on POSIX, where 0 is the no-op value subprocess already expects. subprocess_creationflags = subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0 def emit_json(data: Any) -> None: json.dump(data, sys.stdout, indent=2, ensure_ascii=False) sys.stdout.write("\n") CONFIDENCE_LEVELS = ( "confirmed", "probable", "informational", "likely_false_positive", ) def classify_finding_confidence(finding: str) -> str: """Classify a scanner finding by confidence. The four buckets are a heuristic mapping of *how strong* a finding is: - confirmed: a recognized provenance structure (C2PA/JUMBF manifest, or a parsed field such as digitalSourceType / trainedAlgorithmicMedia). - probable: an AI/vendor marker found inside a recognized metadata structure, but not a fully parsed provenance claim. - informational: context-only notes (CMS generators, presence of an XMP packet or customXml parts, unsupported/partial inspection). - likely_false_positive: raw whole-file byte scans that can collide with compressed image/stream data. The mapping is intentionally conservative; a scanner finding is a signal, not a verdict. """ t = finding.lower() # Layer A space homoglyphs are weaker context than invisible carriers, and # text_hit_confidence() already says so on the text path. Container formats # (markdown/HTML) are classified here instead, so without this the very same # non-breaking space reads "informational" in a .txt and "probable" in a .md, # and only the .md makes the file actionable. if t.startswith("layer-a") and t.endswith("(space)"): return "informational" if any( s in t for s in ( "c2patool reports", "c2pa-related manifest", "png chunk c2", "png chunk cabx", "png chunk jumb", "png chunk jumd", "jpeg app11 segment", "digital_source_type", "digitalsourcetype", "trainedalgorithmicmedia", "compositewithtrainedalgorithmicmedia", "softwareagent", "web-extension vendor property", ) ): return "confirmed" if t.startswith("info:") or any( s in t for s in ( "cms generator", "customxml parts", "xmp packet present", "unsupported", "not fully inspected", "format not", "svg <metadata> present", "not a valid", "truncated chunk", "bad segment length", "svg decode note", ) ): return "informational" if "byte-scan" in t: return "likely_false_positive" if any( s in t for s in ( "ai:", "marker:", "meta:", "frontmatter", "json-ld", "attr:", "png ", "jpeg app", "exif", "xmp", "interesting", "pdf-structured", "layer-a", "web-extension property", "web-extension reference", ) ): return "probable" return "informational" def c2patool_probe_note(tools: dict[str, Any]) -> str | None: """Describe an inconclusive c2patool run, or None when it answered. c2patool exits non-zero both for an asset with no manifest and for a binary that failed to run, so a caller that only reads `has_manifest` cannot tell "this asset is clean" from "the probe never ran". Reporting the second as the first is the dangerous direction: `has_c2pa: False` beside a dead probe reads as a clean bill of health on the one check a user would trust. The wording deliberately avoids the substring "c2patool reports", which classify_finding_confidence() maps to `confirmed`; "not fully inspected" puts it in the `informational` bucket instead. """ ct = tools.get("c2patool") or {} # An explicitly recorded unavailable entry (available is False) means the # probe was attempted but could not run; report it so callers do not read # "not checked" as "clean". A tool with no entry at all is simply absent, # which the `tools` dict already reflects and is not a finding. if ct.get("available") is False: detail = ct.get("error") or "not found" return f"c2patool unavailable ({detail}); C2PA not fully inspected by this tool" if not ct.get("ok", True): detail = ct.get("error") or "no usable verdict" return f"c2patool probe inconclusive ({detail}); C2PA not fully inspected by this tool" return None def cleaned_path(src: Path, suffix: str = ".cleaned") -> Path: """path/to/file.ext -> path/to/file.cleaned.ext""" return src.with_name(f"{src.stem}{suffix}{src.suffix}") def which(cmd: str) -> str | None: from shutil import which as _which return _which(cmd) def safe_arg(path: str) -> str: """Guard paths passed to option-parsing CLIs (exiftool, c2patool). A filename starting with '-' would otherwise be interpreted as an option (e.g. exiftool's -@argfile), turning a crafted filename into argv injection. """ if path.startswith("-"): # './' also resolves correctly on Windows (Win32 accepts '/' as a # path separator), so no platform branch is needed here. return "./" + path return pathskills/clean-user-facing-text/scripts/inspect_text.py4.4 KB2c3d430cf8a6upstream ↗
#!/usr/bin/env python3 """Inspect text for invisible Unicode / space homoglyphs (Layer A) and optional stylometry.""" from __future__ import annotations import argparse import sys from pathlib import Path from typing import Any # Allow running as script from any cwd sys.path.insert(0, str(Path(__file__).resolve().parent)) from common import emit_json, read_text_input from score_stylometry import print_human_stylometry_report, score_text_stylometry from text_unicode import human_report, inspect_text def main() -> int: p = argparse.ArgumentParser(description=__doc__) p.add_argument("path", nargs="?", default="-", help="Text file path, or - for stdin") p.add_argument("--json", action="store_true", help="JSON report") p.add_argument( "--aggressive", action="store_true", help="Also flag Latin confusable / fullwidth lookalikes", ) p.add_argument( "--strip-emoji-glue", action="store_true", help="Paranoid: flag all load-bearing invisibles too (emoji glue, script joiners, flag tags, same-script fillers/selectors, orthographic Cf)", ) p.add_argument( "--stylometry", action="store_true", help="Also run zero-LLM statistical and stylometric AI cadence scoring", ) p.add_argument( "--threshold", type=float, default=0.65, help="Score threshold for --stylometry exit code (default: 0.65)", ) p.add_argument( "--force-text", action="store_true", help="Scan even when the input looks like a binary container", ) p.add_argument( "--audit", action="store_true", help="Detect-only audit: flag Unicode and stylometric spans without rewriting " "and emit a structured span list (implies --stylometry)", ) args = p.parse_args() if args.audit: args.stylometry = True # read_text_input raises SystemExit(2) on unreadable/oversized/non-regular input. text = read_text_input(args.path, allow_binary=args.force_text) report = inspect_text( text, aggressive=args.aggressive, strip_emoji_glue=args.strip_emoji_glue, ) stylometry_report = None if args.stylometry: input_label = "<stdin>" if args.path == "-" else args.path stylometry_report = score_text_stylometry(text, path=input_label) if args.audit: flagged: list[dict[str, Any]] = [] for hit in report.to_dict()["hits"]: flagged.append( { "detector": "unicode", "kind": hit["kind"], "label": hit["label"], "count": hit["count"], "sample_offsets": hit["sample_offsets"], "severity": hit["confidence"], } ) density_tier = "uncalibrated" if stylometry_report: density_tier = stylometry_report.density_tier for m in stylometry_report.matched_markers: flagged.append( { "detector": "stylometry", "phrase": m["phrase"], "count": m["count"], "weight": m["weight"], "samples": m.get("samples", []), "spans": m.get("spans", []), "severity": density_tier, } ) audit: dict[str, Any] = { "path": "<stdin>" if args.path == "-" else args.path, "density_tier": density_tier, "flagged_count": len(flagged), "flagged": flagged, } if stylometry_report: audit["stylometry"] = stylometry_report.to_dict() emit_json(audit) elif args.json: data = report.to_dict() if stylometry_report: data["stylometry"] = stylometry_report.to_dict() emit_json(data) else: print(human_report(report)) if stylometry_report: print("\n" + "=" * 50 + "\n") print_human_stylometry_report(stylometry_report, explain=True) exit_code = 0 if report.suspicious_total > 0: exit_code = 1 if ( stylometry_report and stylometry_report.status == "ok" and stylometry_report.score is not None and stylometry_report.score >= args.threshold ): exit_code = 1 return exit_code if __name__ == "__main__": raise SystemExit(main())skills/clean-user-facing-text/scripts/score_stylometry.py19 KBc29a7777b2d0upstream ↗
#!/usr/bin/env python3 """Zero-LLM statistical & stylometric AI-text detector. Evaluates text for high-frequency AI cadence markers, sentence-length burstiness variance, lexical diversity (MATTR), and structural uniformity. Stdlib-only, no PyTorch or external model dependencies. Exit codes: 0 clean (score < threshold) 1 suspicious / AI stylometric signals detected (score >= threshold) 2 bad input (missing file, binary data, bad args) """ from __future__ import annotations import argparse import math import re import sys from collections import Counter from dataclasses import dataclass, field from pathlib import Path from typing import Any SCRIPTS_DIR = Path(__file__).resolve().parent sys.path.insert(0, str(SCRIPTS_DIR)) from common import ( # noqa: E402 classify_finding_confidence, emit_json, read_text_input, ) DEFAULT_THRESHOLD = 0.65 MIN_SAMPLE_WORDS = 30 FULL_WEIGHT_WORDS = 100 # High-frequency formulaic transition markers, hedging verbs, and structural # boilerplate commonly overrepresented in AI-generated text across frontier LLMs. AI_PHRASE_PATTERNS: tuple[tuple[str, str, float], ...] = ( # (regex_pattern, human_label, weight) (r"\bdelve(?:s|d)?\s+into\b", "delve into", 1.2), (r"\ba\s+testament\s+to\b", "a testament to", 1.1), (r"\brich\s+tapestry(?:\s+of)?\b", "rich tapestry", 1.3), (r"\bplays?\s+a\s+(?:pivotal|crucial|vital|key)\s+role\b", "plays a pivotal/crucial role", 1.0), ( r"\bin\s+(?:today'?s|the)\s+" r"(?:(?:fast-paced|ever-evolving|digital|rapidly\s+changing)\s+)*" r"(?:world|landscape|era|environment)\b", "in today's fast-paced world/landscape", 1.4, ), ( r"\bit\s+is\s+(?:important|essential|crucial|worth\s+noting)\s+to\s+" r"(?:note|remember|consider|highlight)\b", "it is important/crucial to note", 0.9, ), ( r"\bnot\s+only\b[\w\s,]+\bbut\s+(?:also\s+)?(?:serves\s+to|acts\s+as|highlights)\b", "not only ... but also serves to", 0.8, ), ( r"\bserve(?:s|d)?\s+as\s+a\s+(?:beacon|reminder|catalyst|cornerstone)\b", "serves as a beacon/catalyst/cornerstone", 1.1, ), ( r"\bunderscore(?:s|d)?\s+the\s+(?:importance|need|significance)\b", "underscores the importance/need", 0.9, ), ( r"\bfoster(?:s|ing|ed)?\s+a\s+(?:sense|culture|deeper\s+understanding)\b", "fosters a sense/culture", 0.9, ), ( r"\bseamlessly\s+(?:integrates?|integrated|blends?|combine[sd]?)\b", "seamlessly integrates/blends", 1.0, ), ( r"\bnavigat(?:e|ing|es|ed)\s+the\s+(?:complexities|intricacies|nuances)\b", "navigating the complexities/nuances", 1.0, ), (r"\bmultifaceted\s+(?:nature|approach|landscape)\b", "multifaceted nature/approach", 1.0), (r"\bharness(?:ing|ed|es)?\s+the\s+power\s+of\b", "harnessing the power of", 1.0), (r"\ba\s+myriad\s+of\b", "a myriad of", 0.8), (r"\bparadigm\s+shift\b", "paradigm shift", 0.9), (r"\bholistic\s+(?:approach|view|perspective)\b", "holistic approach/perspective", 0.9), (r"\bin\s+conclusion\b[,\s]", "in conclusion", 0.8), (r"\bto\s+summarize\b[,\s]", "to summarize", 0.8), (r"\bultimately\b[,\s]", "ultimately,", 0.6), (r"\bfurthermore\b[,\s]", "furthermore,", 0.6), (r"\bmoreover\b[,\s]", "moreover,", 0.6), (r"\bas\s+an\s+ai\b", "as an AI", 1.5), (r"\bi\s+hope\s+this\s+helps\b", "I hope this helps", 1.2), # Patterns folded in from blader/humanizer (Wikipedia "Signs of AI writing") # and conorbronsdon/avoid-ai-writing's replacement table. These are the # regex-detectable subset; the full prose category list lives in # skill references/detectors.md. Weights stay moderate: this is a gauge, # not a verdict, so a single hit should not dominate the composite. (r"\bstands?\s+as\s+a\s+testament\b", "stands as a testament to", 1.1), ( r"\bmark(?:s|ing)?\s+an?\s+" r"(?:indelible|pivotal|significant|new)\s+(?:moment|chapter|milestone)\b", "marking a pivotal moment/chapter", 1.0, ), ( r"\b(?:reflecting|symbolizing|showcasing|underscoring)\s+(?:the|a|its)\b", "shallow -ing analysis (reflecting/symbolizing/showcasing)", 0.9, ), (r"\b(?:nestled|vibrant|breathtaking)\b", "sales language (nestled/vibrant)", 0.8), (r"\b(?:game[- ]changer|game-changing)\b", "game-changer", 0.9), (r"\b(?:unparalleled|unprecedented)\b", "unparalleled/unprecedented", 0.7), (r"\b(?:world-class|state-of-the-art|cutting-edge)\b", "world-class/state-of-the-art", 0.8), (r"\b(?:revolutionary|groundbreaking)\b", "revolutionary/groundbreaking", 0.6), (r"\b(?:leverag(?:e|ing|ed|es)|utiliz(?:e|ing|ed|es))\b", "leverage/utilize", 0.8), (r"\bboasts?\b", "boasts (copula avoidance)", 0.6), (r"\bit['’]?s\s+not\s+just\b", "it's not just X, it's Y", 0.8), (r"\b(?:not\s+just\b.{0,60}\bbut\s+also\b)", "not just X but also Y", 0.8), (r"\bdive(?:s|d)?\s+into\b", "dive into", 0.7), (r"\blet['’]?s\s+(?:dive\s+in|get\s+started)\b", "let's dive in", 0.7), (r"\bin\s+order\s+to\b", "in order to", 0.6), (r"\bdue\s+to\s+the\s+fact\s+that\b", "due to the fact that", 0.8), (r"\bit(?:['’]s| is)\s+worth\s+noting\s+that\b", "it is worth noting that", 0.8), (r"\b(?:needless\s+to\s+say|it\s+goes\s+without\s+saying)\b", "needless to say", 0.8), (r"\bthe\s+future\s+looks\s+bright\b", "the future looks bright", 0.9), ( r"\b(?:sure\s+thing!?|great\s+question!?|happy\s+to\s+help)\b", "assistant chatter (sure thing/great question)", 0.6, ), ) RE_SENTENCE_SPLIT = re.compile(r"(?<=[.!?])\s+(?=[A-Z0-9\"'(\[])") RE_WORDS = re.compile(r"\b[\w'-]+\b", re.UNICODE) @dataclass class MarkerMatch: phrase: str count: int weight: float samples: list[str] = field(default_factory=list) spans: list[tuple[int, int]] = field(default_factory=list) # start/end char offsets def to_dict(self) -> dict[str, Any]: return { "phrase": self.phrase, "count": self.count, "weight": self.weight, "samples": self.samples, "spans": [list(s) for s in self.spans], } @dataclass class StylometryReport: path: str word_count: int sentence_count: int burstiness_cv: float | None lexical_diversity: float ai_ngram_density: float matched_markers: list[dict[str, Any]] score: float | None confidence_level: str | None # CLEAN | LOW | MEDIUM | HIGH, or None when uncalibrated density_tier: str # low | medium | high, or uncalibrated when not scored status: str # ok | insufficient_length findings: list[str] = field(default_factory=list) notes: list[str] = field(default_factory=list) def to_dict(self) -> dict[str, Any]: return { "path": self.path, "word_count": self.word_count, "sentence_count": self.sentence_count, "burstiness_cv": round(self.burstiness_cv, 4) if self.burstiness_cv is not None else None, "lexical_diversity": round(self.lexical_diversity, 4), "ai_ngram_density": round(self.ai_ngram_density, 4), "matched_markers": self.matched_markers, "score": round(self.score, 4) if self.score is not None else None, "confidence_level": self.confidence_level, "density_tier": self.density_tier, "status": self.status, "findings": self.findings, "findings_confidence": [classify_finding_confidence(f) for f in self.findings], "notes": self.notes, } def extract_sentences(text: str) -> list[str]: """Split text into sentences while ignoring blank lines and code block markers.""" clean_lines = [] in_code_block = False for line in text.splitlines(): trimmed = line.strip() if trimmed.startswith("```"): in_code_block = not in_code_block continue if in_code_block or not trimmed: continue clean_lines.append(trimmed) raw_text = "\n".join(clean_lines) if not raw_text.strip(): return [] chunks = re.split(r"(?<=[.!?])\s+|\n+", raw_text) sentences = [s.strip() for s in chunks if s.strip()] return sentences def extract_words(text: str) -> list[str]: """Extract normalized alphanumeric word tokens.""" return [w.lower() for w in RE_WORDS.findall(text)] def compute_burstiness(sentences: list[str]) -> tuple[float, float, float | None]: """Compute mean sentence word length, standard deviation, and coefficient of variation (CV). CV is ``None`` when it cannot be measured (no sentences, or fewer than two with words): a one-or-zero-sample CV is undefined, and reporting it as 0.0 made the caller's tiering read "perfectly uniform" — the strongest LLM-likeness signal — for text whose body yielded no sentences at all (#132). """ if not sentences: return 0.0, 0.0, None lengths = [len(extract_words(s)) for s in sentences] lengths = [L for L in lengths if L > 0] if len(lengths) < 2: mean_len = float(lengths[0]) if lengths else 0.0 return mean_len, 0.0, None mean_len = sum(lengths) / len(lengths) variance = sum((x - mean_len) ** 2 for x in lengths) / (len(lengths) - 1) std_dev = math.sqrt(variance) cv = (std_dev / mean_len) if mean_len > 0 else 0.0 return mean_len, std_dev, cv def compute_mattr(words: list[str], window_size: int = 50) -> float: """Compute Moving-Average Type-Token Ratio (MATTR) across sliding windows.""" n = len(words) if n == 0: return 0.0 if n <= window_size: return len(set(words)) / n total_ttr = 0.0 num_windows = n - window_size + 1 # Sliding window counts current_window = Counter(words[:window_size]) total_ttr += len(current_window) / window_size for i in range(1, num_windows): leaving_word = words[i - 1] entering_word = words[i + window_size - 1] current_window[leaving_word] -= 1 if current_window[leaving_word] == 0: del current_window[leaving_word] current_window[entering_word] += 1 total_ttr += len(current_window) / window_size return total_ttr / num_windows def scan_ai_phrases(text: str) -> list[MarkerMatch]: """Find and tally high-frequency AI cadence phrases.""" matches: list[MarkerMatch] = [] for pattern, label, weight in AI_PHRASE_PATTERNS: found_spans = [] offsets: list[tuple[int, int]] = [] for m in re.finditer(pattern, text, re.IGNORECASE): found_spans.append(m.group(0)) offsets.append((m.start(), m.end())) if found_spans: matches.append( MarkerMatch( phrase=label, count=len(found_spans), weight=weight, samples=found_spans[:3], spans=offsets[:10], ) ) return matches def classify_density(score: float | None) -> str: """Map a composite score to an edit-decision tier. ``high`` means the text sits at or above the suspicious threshold, so the skill's rewrite pass should engage; ``medium``/``low`` mean the measurable AI-density signals are weaker and the skill should mainly verify rather than rewrite. ``uncalibrated`` is returned for samples too short to score. """ if score is None: return "uncalibrated" if score >= DEFAULT_THRESHOLD: return "high" if score >= 0.40: return "medium" return "low" def score_text_stylometry(text: str, path: str = "<text>") -> StylometryReport: """Run full multi-dimensional stylometric analysis and return a structured report.""" words = extract_words(text) word_count = len(words) sentences = extract_sentences(text) sentence_count = len(sentences) findings: list[str] = [] notes: list[str] = [] # 1. Length Guard if word_count < MIN_SAMPLE_WORDS: marker_matches = scan_ai_phrases(text) for m in marker_matches: findings.append(f"AI phrase marker '{m.phrase}' found ({m.count}x)") notes.append( f"Sample contains {word_count} words; statistical stylometry is " f"uncalibrated below {MIN_SAMPLE_WORDS} words, so no score is reported" ) return StylometryReport( path=path, word_count=word_count, sentence_count=sentence_count, burstiness_cv=None, lexical_diversity=compute_mattr(words), ai_ngram_density=0.0, matched_markers=[m.to_dict() for m in marker_matches], score=None, confidence_level=None, density_tier="uncalibrated", status="insufficient_length", findings=findings, notes=notes, ) # 2. Metric Calculations _, _, cv = compute_burstiness(sentences) mattr = compute_mattr(words) marker_matches = scan_ai_phrases(text) # N-gram density: weighted marker instances per 100 words total_marker_weight = sum(m.count * m.weight for m in marker_matches) ngram_density = (total_marker_weight / (word_count / 100.0)) if word_count > 0 else 0.0 # 3. Component Sub-scores (0.0 to 1.0) # Burstiness subscore: low CV (<0.35) is strongly characteristic of LLMs; # high CV (>0.60) is human burstiness_score: float | None if cv is None: # Fewer than two parseable sentences: burstiness is unmeasurable, not # maximally LLM-like. The composite renormalizes over the remaining # components instead of crediting the strongest signal (#132). burstiness_score = None elif cv < 0.25: burstiness_score = 0.95 elif cv < 0.35: burstiness_score = 0.80 elif cv < 0.45: burstiness_score = 0.50 elif cv < 0.55: burstiness_score = 0.25 else: burstiness_score = 0.05 # N-gram subscore: >1.5 weighted matches per 100 words is very high if ngram_density >= 2.0: ngram_score = 1.0 elif ngram_density >= 1.0: ngram_score = 0.75 elif ngram_density >= 0.5: ngram_score = 0.45 elif ngram_density > 0: ngram_score = 0.20 else: ngram_score = 0.0 # Lexical uniformity subscore: LLMs cluster tightly around MATTR 0.65-0.78 for 50-word windows diversity_score = 0.4 if 0.68 <= mattr <= 0.76 else 0.1 # 4. Composite Scoring & Small-Sample Dampening if burstiness_score is None: notes.append( "Sentence burstiness unavailable (fewer than 2 parsed sentences — " "e.g. body wrapped in a code fence); composite renormalized over " "AI-phrase density and lexical diversity" ) raw_composite = ((ngram_score * 0.45) + (diversity_score * 0.10)) / 0.55 else: raw_composite = (burstiness_score * 0.45) + (ngram_score * 0.45) + (diversity_score * 0.10) # Dampening factor: scales smoothly from 0.4 at MIN_SAMPLE_WORDS up to 1.0 at FULL_WEIGHT_WORDS if word_count < FULL_WEIGHT_WORDS: dampener = 0.4 + 0.6 * ( (word_count - MIN_SAMPLE_WORDS) / (FULL_WEIGHT_WORDS - MIN_SAMPLE_WORDS) ) notes.append( f"Sample word count ({word_count}) is in calibration range " f"({MIN_SAMPLE_WORDS}-{FULL_WEIGHT_WORDS}); score dampened by factor {dampener:.2f}" ) else: dampener = 1.0 final_score = min(1.0, max(0.0, raw_composite * dampener)) # 5. Classify Findings & Confidence Tiers if marker_matches: for m in marker_matches: findings.append(f"AI cadence phrase '{m.phrase}' ({m.count}x)") if cv is not None and cv < 0.35 and sentence_count >= 3: findings.append(f"Unnaturally uniform sentence cadence (CV={cv:.2f} < 0.35)") if ngram_density >= 1.0: findings.append(f"Elevated AI formulaic transition density ({ngram_density:.2f}/100w)") if final_score >= 0.75: confidence = "HIGH" elif final_score >= 0.50: confidence = "MEDIUM" elif final_score >= 0.25: confidence = "LOW" else: confidence = "CLEAN" return StylometryReport( path=path, word_count=word_count, sentence_count=sentence_count, burstiness_cv=cv, lexical_diversity=mattr, ai_ngram_density=ngram_density, matched_markers=[m.to_dict() for m in marker_matches], score=final_score, confidence_level=confidence, density_tier=classify_density(final_score), status="ok", findings=findings, notes=notes, ) def print_human_stylometry_report(report: StylometryReport, explain: bool = False) -> None: """Print clean human-readable output to stdout.""" print(f"=== Stylometric AI-Text Report: {report.path} ===") print("Gauge only: measures lexical and cadence signals, not authorship.") print(f"Status: {report.status}") if report.confidence_level is not None: print(f"Confidence Level: {report.confidence_level}") else: print("Confidence Level: n/a") if report.score is not None: print(f"Stylometry Score: {report.score * 100:.1f}% (score: {report.score:.3f})") else: print("Stylometry Score: n/a (sample too short to score)") print(f"Density Tier: {report.density_tier}") print(f"Word Count: {report.word_count}") print(f"Sentence Count: {report.sentence_count}") cv_display = ( f"{report.burstiness_cv:.3f}" if report.burstiness_cv is not None else "n/a (fewer than 2 sentences)" ) print(f"Sentence CV: {cv_display}") print(f"Lexical Diversity: {report.lexical_diversity:.3f} (MATTR)") print(f"AI Marker Density: {report.ai_ngram_density:.3f} / 100 words") if report.findings: print("\nFindings:") for f in report.findings: print(f" - {f}") if explain and report.matched_markers: print("\nMatched Phrases Detail:") for m in report.matched_markers: print(f" * {m['phrase']} (occurrences: {m['count']}, weight: {m['weight']})") if m.get("samples"): print(f' sample: "{m["samples"][0]}"') if report.notes: print("\nNotes:") for n in report.notes: print(f" * {n}") def main() -> int: p = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter, ) p.add_argument("path", nargs="?", default="-", help="Text file to score ('-' for stdin)") p.add_argument( "--threshold", type=float, default=DEFAULT_THRESHOLD, help=f"Score threshold to trigger exit code 1 (default: {DEFAULT_THRESHOLD})", ) p.add_argument("--json", action="store_true", help="Emit JSON output") p.add_argument( "--explain", action="store_true", help="Include detailed matched phrase breakdown" ) args = p.parse_args() # read_text_input raises SystemExit(2) on unreadable/oversized/non-regular input. text = read_text_input(args.path) input_label = "<stdin>" if args.path == "-" else args.path report = score_text_stylometry(text, path=input_label) if args.json: emit_json(report.to_dict()) else: print_human_stylometry_report(report, explain=args.explain) if report.status == "ok" and report.score is not None and report.score >= args.threshold: return 1 return 0 if __name__ == "__main__": sys.exit(main())skills/clean-user-facing-text/scripts/text_unicode.py25 KBe1b406bdc544upstream ↗
"""Layer A: invisible Unicode / homoglyph space detection and cleaning.""" from __future__ import annotations import unicodedata from collections import Counter from dataclasses import dataclass, field from difflib import SequenceMatcher # Format / invisible controls commonly used for steganography or broken pastes. STRIP_CODEPOINTS: frozenset[int] = frozenset( { 0x00AD, # soft hyphen 0x034F, # combining grapheme joiner 0x061C, # Arabic letter mark 0x115F, # Hangul choseong filler 0x1160, # Hangul jungseong filler 0x17B4, # Khmer vowel inherent AQ 0x17B5, # Khmer vowel inherent AA 0x180B, # Mongolian free variation selector-1 0x180C, 0x180D, 0x180E, # Mongolian vowel separator 0x180F, # Mongolian free variation selector-4 (Unicode 14) 0x200B, # zero width space 0x200C, # zero width non-joiner 0x200D, # zero width joiner 0x200E, # LRM 0x200F, # RLM 0x202A, # LRE 0x202B, # RLE 0x202C, # PDF 0x202D, # LRO 0x202E, # RLO 0x2060, # word joiner 0x2061, # function application 0x2062, # invisible times 0x2063, # invisible separator 0x2064, # invisible plus 0x2066, # LRI 0x2067, # RLI 0x2068, # FSI 0x2069, # PDI 0x206A, # inhibit symmetric swapping 0x206B, 0x206C, 0x206D, 0x206E, 0x206F, 0xFEFF, # BOM / ZWNBSP 0xFE00, # variation selectors 0xFE01, 0xFE02, 0xFE03, 0xFE04, 0xFE05, 0xFE06, 0xFE07, 0xFE08, 0xFE09, 0xFE0A, 0xFE0B, 0xFE0C, 0xFE0D, 0xFE0E, 0xFE0F, 0x3164, # Hangul filler (blank compatibility jamo) 0xFFA0, # halfwidth Hangul filler 0xFFF9, # interlinear annotation 0xFFFA, 0xFFFB, } ) # Spaces that look like (or substitute for) U+0020. SPACE_HOMOGLYPHS: dict[int, str] = { 0x00A0: " ", # no-break space 0x1680: " ", # Ogham space mark 0x2000: " ", # en quad 0x2001: " ", # em quad 0x2002: " ", # en space 0x2003: " ", # em space 0x2004: " ", # three-per-em space 0x2005: " ", # four-per-em space 0x2006: " ", # six-per-em space 0x2007: " ", # figure space 0x2008: " ", # punctuation space 0x2009: " ", # thin space 0x200A: " ", # hair space 0x202F: " ", # narrow no-break space 0x205F: " ", # medium mathematical space 0x3000: " ", # ideographic space } # Optional confusable Latin lookalikes (aggressive mode only). LATIN_CONFUSABLES: dict[int, str] = { 0x0410: "A", # Cyrillic 0x0412: "B", 0x0415: "E", 0x041A: "K", 0x041C: "M", 0x041D: "H", 0x041E: "O", 0x0420: "P", 0x0421: "C", 0x0422: "T", 0x0425: "X", 0x0430: "a", 0x0435: "e", 0x043E: "o", 0x0440: "p", 0x0441: "c", 0x0443: "y", 0x0445: "x", 0x0456: "i", 0xFF21: "A", # fullwidth 0xFF22: "B", 0xFF23: "C", 0xFF24: "D", 0xFF25: "E", 0xFF26: "F", 0xFF27: "G", 0xFF28: "H", 0xFF29: "I", 0xFF2A: "J", 0xFF2B: "K", 0xFF2C: "L", 0xFF2D: "M", 0xFF2E: "N", 0xFF2F: "O", 0xFF30: "P", 0xFF31: "Q", 0xFF32: "R", 0xFF33: "S", 0xFF34: "T", 0xFF35: "U", 0xFF36: "V", 0xFF37: "W", 0xFF38: "X", 0xFF39: "Y", 0xFF3A: "Z", 0xFF41: "a", 0xFF42: "b", 0xFF43: "c", 0xFF44: "d", 0xFF45: "e", 0xFF46: "f", 0xFF47: "g", 0xFF48: "h", 0xFF49: "i", 0xFF4A: "j", 0xFF4B: "k", 0xFF4C: "l", 0xFF4D: "m", 0xFF4E: "n", 0xFF4F: "o", 0xFF50: "p", 0xFF51: "q", 0xFF52: "r", 0xFF53: "s", 0xFF54: "t", 0xFF55: "u", 0xFF56: "v", 0xFF57: "w", 0xFF58: "x", 0xFF59: "y", 0xFF5A: "z", } # Variation selectors beyond FE0x (VS17-VS256 in Supplementary Special-purpose) _VS_SUPPLEMENT = range(0xE0100, 0xE01F0) # Unassigned code points with Other_Default_Ignorable_Code_Point=Yes: reserved # for future default-ignorable characters, so conformant renderers display them # invisibly today and normalisation preserves them. They have no legitimate use # in interchange text (conformance clause C7), which makes them ideal covert # carriers. Kept as explicit ranges, never a category-Cn rule: unicodedata is # pinned per Python build, so a Cn rule would destroy freshly assigned real # characters. Re-check these ranges on Unicode version bumps: assignment turns # a strip entry into a potential preserve-in-context case, exactly as happened # when U+180F became Mongolian FVS4 in Unicode 14. _RESERVED_IGNORABLE_CPS: frozenset[int] = frozenset({0x2065, 0xE0000}) _RESERVED_IGNORABLE_RANGES: tuple[range, ...] = ( range(0xFFF0, 0xFFF9), range(0xE0080, 0xE0100), range(0xE01F0, 0xE1000), ) def _is_reserved_ignorable(cp: int) -> bool: if cp in _RESERVED_IGNORABLE_CPS: return True return any(cp in r for r in _RESERVED_IGNORABLE_RANGES) # The 66 Unicode noncharacters: U+FDD0..U+FDEF plus U+nFFFE/U+nFFFF at the # end of every plane. Permanently reserved for internal use and prohibited in # interchange text (TUS 23.7), so any occurrence in interchange is contraband. # Rendered as nothing or tofu, preserved by normalisation, and permanently # unassignable, so stripping them carries no future-Unicode risk. def _is_noncharacter(cp: int) -> bool: return 0xFDD0 <= cp <= 0xFDEF or (cp & 0xFFFE) == 0xFFFE # Bidi / directional format controls (subset of strip set, finer inspect labels) _BIDI_CPS: frozenset[int] = frozenset( { 0x061C, 0x200E, 0x200F, 0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069, } ) # Directional marks and isolates are legitimate in mixed RTL/LTR prose. Inspect # them, but preserve them during the default clean. Paired LRE/RLE embeddings # (see _valid_bidi_embedding_indices) are preserved too; overrides and # unpaired embeddings remain destructive by default because they can reorder # unrelated spans. _PRESERVABLE_BIDI_CPS: frozenset[int] = frozenset( { 0x061C, 0x200E, 0x200F, 0x2066, 0x2067, 0x2068, 0x2069, } ) # Visible-layout format controls: Egyptian hieroglyph quadrat controls, # Duployan shorthand overlap/step controls, and musical beam/tie/slur/phrase # controls are Cf but visibly govern how their script renders. Next to their # own script they are document body, not carriers; floating between unrelated # text they stay strip-class. Context ranges include each script's own block # (and the controls themselves) so control sequences survive intact. _LAYOUT_CF_CONTROLS: tuple[tuple[range, range], ...] = ( (range(0x13430, 0x13440), range(0x13000, 0x14400)), # Egyptian hieroglyphs (range(0x1BCA0, 0x1BCA4), range(0x1BC00, 0x1BCA4)), # Duployan shorthand (range(0x1D173, 0x1D17B), range(0x1D100, 0x1D200)), # musical symbols ) def _layout_cf_script(cp: int) -> range | None: for controls, script in _LAYOUT_CF_CONTROLS: if cp in controls: return script return None # Zero-width family (common edit-based carriers) _ZW_FAMILY: frozenset[int] = frozenset({0x200B, 0x200C, 0x200D, 0x2060, 0xFEFF, 0x180E}) def _is_private_use(cp: int) -> bool: """BMP and supplementary private-use planes (Co: no portable meaning).""" return 0xE000 <= cp <= 0xF8FF or 0xF0000 <= cp <= 0xFFFFD or 0x100000 <= cp <= 0x10FFFD def _is_strip_cp(cp: int) -> bool: if cp in STRIP_CODEPOINTS: return True if cp in _VS_SUPPLEMENT: return True # Tag characters used in some stego schemes (U+E0001-U+E007F) if 0xE0001 <= cp <= 0xE007F: return True if _is_noncharacter(cp): return True if _is_reserved_ignorable(cp): return True return bool(_is_private_use(cp)) def _strip_kind(cp: int) -> str: """Finer-grained inspect kind for strip-class codepoints.""" if 0xE0001 <= cp <= 0xE007F: return "tag_chars" if _is_noncharacter(cp): return "noncharacter" if _is_reserved_ignorable(cp): return "reserved_ignorable" if cp in _VS_SUPPLEMENT or 0xFE00 <= cp <= 0xFE0F or cp in _MONGOLIAN_FVS: return "variation_selector" if cp in _BIDI_CPS: return "bidi" if cp in _ZW_FAMILY: return "zwj_family" if _is_private_use(cp): return "private_use" return "strip" # Emoji presentation glue: zero-width joiner and text/emoji variation # selectors. These are invisible carriers when free-floating, but after an # emoji base they are part of the visible sequence (⚖️, 👨👩👧, ❤️🔥) and # stripping them visibly alters the text. EMOJI_GLUE_CODEPOINTS: frozenset[int] = frozenset({0x200D, 0xFE0E, 0xFE0F}) def _is_emoji_glue(cp: int) -> bool: return cp in EMOJI_GLUE_CODEPOINTS def _is_emoji_base(cp: int) -> bool: """Return True for characters that can start or continue an emoji sequence.""" if 0x1F000 <= cp <= 0x1FAFF: return True if 0x2190 <= cp <= 0x25FF: # arrows, technical symbols, enclosed symbols return True if 0x2600 <= cp <= 0x27BF: # misc symbols / dingbats / arrows return True if 0x2B00 <= cp <= 0x2BFF: # misc symbols and arrows return True # Emoji=Yes singletons outside the ranges above (general punctuation, # letterlike symbols, supplemental arrows-B): !!, !?, i, curved up/down # arrows. Without these a VS16 after them is stripped, visibly turning # the emoji presentation back into the text glyph. if cp in (0x203C, 0x2049, 0x2139, 0x2934, 0x2935): return True if cp in (0x00A9, 0x00AE, 0x2122, 0x3030, 0x303D, 0x3297, 0x3299): return True # keycap bases return cp in (0x0023, 0x002A) or 0x0030 <= cp <= 0x0039 # ZWNJ/ZWJ are orthographic inside complex scripts (Persian میروم, Devanagari # क्ष); flag emoji are an emoji base followed by tag chars (🏴); and a # handful of Cf codepoints are normal Arabic/Syriac orthography, not carriers. # So are Mongolian free variation selectors (choose a glyph of the preceding # letter), Khmer inherent vowels (invisible but phonemic), and Hangul fillers # (hold a jamo slot in a partial syllable). Each is only meaningful directly # after a base from its own script; isolated instances are contraband. _SCRIPT_JOINERS: frozenset[int] = frozenset({0x200C, 0x200D}) _TAG_RANGE = range(0xE0020, 0xE0080) _ORTHOGRAPHIC_CF: frozenset[int] = frozenset( {0x0600, 0x0601, 0x0602, 0x0603, 0x0604, 0x0605, 0x06DD, 0x070F, 0x08E2, 0x110BD, 0x110CD} ) _MONGOLIAN_FVS: frozenset[int] = frozenset({0x180B, 0x180C, 0x180D, 0x180F}) _KHMER_VOWELS: frozenset[int] = frozenset({0x17B4, 0x17B5}) _HANGUL_FILLERS: frozenset[int] = frozenset({0x115F, 0x1160, 0x3164, 0xFFA0}) _SCRIPT_GLUE: frozenset[int] = _MONGOLIAN_FVS | _KHMER_VOWELS | _HANGUL_FILLERS def _joining_script(cp: int) -> str | None: """Return a broad script group where ZWJ/ZWNJ can be orthographic.""" for start, end, name in ( (0x0600, 0x08FF, "arabic"), (0x0900, 0x0DFF, "indic"), (0x0F00, 0x109F, "south-asian"), (0x1780, 0x17FF, "khmer"), (0x1800, 0x18AF, "mongolian"), ): if start <= cp <= end and unicodedata.category(chr(cp))[0] in ("L", "M"): return name return None def _is_cjk_ideograph(cp: int) -> bool: return ( 0x3400 <= cp <= 0x4DBF or 0x4E00 <= cp <= 0x9FFF or 0xF900 <= cp <= 0xFAFF or 0x20000 <= cp <= 0x323AF ) def _is_mongolian_base(cp: int) -> bool: return 0x1800 <= cp <= 0x18AF def _is_variation_selector(cp: int) -> bool: return cp in _VS_SUPPLEMENT or 0xFE00 <= cp <= 0xFE0F or cp in _MONGOLIAN_FVS def _valid_flag_tag_indices(text: str) -> set[int]: """Indices in complete subdivision-flag tag sequences.""" valid: set[int] = set() i = 0 while i < len(text): if ord(text[i]) != 0x1F3F4: # waving black flag i += 1 continue j = i + 1 while j < len(text) and 0xE0020 <= ord(text[j]) <= 0xE007E: j += 1 if j > i + 1 and j < len(text) and ord(text[j]) == 0xE007F: valid.update(range(i + 1, j + 1)) i = j + 1 else: i += 1 return valid def _valid_bidi_embedding_indices(text: str) -> set[int]: """Indices belonging to complete LRE/RLE ... PDF pairs, excluding overrides.""" valid: set[int] = set() stack: list[tuple[int, int]] = [] for index, char in enumerate(text): cp = ord(char) if cp in (0x202A, 0x202B, 0x202D, 0x202E): stack.append((cp, index)) elif cp == 0x202C: if not stack: continue opener, opener_index = stack.pop() if opener in (0x202A, 0x202B): valid.update((opener_index, index)) return valid def _is_mongolian_letter(cp: int) -> bool: return 0x1800 <= cp <= 0x18AF and unicodedata.category(chr(cp))[0] == "L" def _is_khmer_letter(cp: int) -> bool: return 0x1780 <= cp <= 0x17FF and unicodedata.category(chr(cp))[0] == "L" def _is_hangul_jamo(cp: int) -> bool: # Conjoining jamo plus the compatibility and halfwidth presentation forms, # so each filler can follow letters of its own form (U+115F/U+1160 after # conjoining jamo, U+3164 after compatibility jamo, U+FFA0 after halfwidth). return ( 0x1100 <= cp <= 0x11FF or 0xA960 <= cp <= 0xA97C # Hangul Jamo Extended-A or 0xD7B0 <= cp <= 0xD7C6 # Hangul Jamo Extended-B or 0x3131 <= cp <= 0x318E # Hangul Compatibility Jamo (incl. U+3164) or 0xFFA1 <= cp <= 0xFFDC # halfwidth Hangul jamo letters ) def _is_glue(cp: int) -> bool: """Load-bearing invisible char: emoji glue, script joiner, flag tag char, or same-script filler/selector (Mongolian FVS, Khmer vowel, Hangul filler).""" return ( _is_emoji_glue(cp) or _is_variation_selector(cp) or cp in _SCRIPT_JOINERS or cp in _TAG_RANGE or cp in _SCRIPT_GLUE ) def _decide( ch: str, prev_kept: str | None, prev_input: str | None, next_input: str | None, *, valid_flag_tag: bool, valid_bidi_embedding: bool, normalize_spaces: bool, treat_confusables: bool, strip_emoji_glue: bool, strip_bidi: bool, ) -> tuple[str, str, str | None]: """Classify one input char for both inspect and clean. Returns ``(action, out_char, kind)`` where action is ``keep``, ``strip`` or ``replace``; out_char is the surviving character for keep/replace; and kind is the inspect classification (None when not suspicious). """ cp = ord(ch) if valid_bidi_embedding and not strip_bidi: return ("keep", ch, None) if cp in _PRESERVABLE_BIDI_CPS and not strip_bidi: return ("keep", ch, None) if prev_input is not None and not strip_emoji_glue: prev_cp = ord(prev_input) if cp in _VS_SUPPLEMENT and _is_cjk_ideograph(prev_cp): return ("keep", ch, None) if cp in _MONGOLIAN_FVS and _is_mongolian_base(prev_cp): return ("keep", ch, None) if 0xFE00 <= cp <= 0xFE0D and _is_cjk_ideograph(prev_cp): return ("keep", ch, None) if _is_emoji_glue(cp) and not strip_emoji_glue: if cp in (0xFE0E, 0xFE0F) and prev_input is not None and _is_emoji_base(ord(prev_input)): return ("keep", ch, None) if ( cp == 0x200D and prev_kept is not None and next_input is not None and _is_emoji_base(ord(prev_kept)) and _is_emoji_base(ord(next_input)) ): return ("keep", ch, None) if not strip_emoji_glue: if cp in _SCRIPT_JOINERS and prev_input is not None and next_input is not None: prev_script = _joining_script(ord(prev_input)) next_script = _joining_script(ord(next_input)) if prev_script is not None and prev_script == next_script: return ("keep", ch, None) if cp in _TAG_RANGE and valid_flag_tag: return ("keep", ch, None) if cp in _MONGOLIAN_FVS and prev_kept is not None and _is_mongolian_letter(ord(prev_kept)): return ("keep", ch, None) if cp in _KHMER_VOWELS and prev_kept is not None and _is_khmer_letter(ord(prev_kept)): return ("keep", ch, None) if cp in _HANGUL_FILLERS and prev_kept is not None and _is_hangul_jamo(ord(prev_kept)): return ("keep", ch, None) if cp in _ORTHOGRAPHIC_CF: return ("keep", ch, None) script = _layout_cf_script(cp) if script is not None and ( (prev_input is not None and ord(prev_input) in script) or (next_input is not None and ord(next_input) in script) ): return ("keep", ch, None) if _is_strip_cp(cp): return ("strip", "", _strip_kind(cp)) if normalize_spaces and cp in SPACE_HOMOGLYPHS: return ("replace", SPACE_HOMOGLYPHS[cp], "space") if treat_confusables and cp in LATIN_CONFUSABLES: return ("replace", LATIN_CONFUSABLES[cp], "confusable") if unicodedata.category(ch) == "Cf" and cp not in SPACE_HOMOGLYPHS: return ("strip", "", "other_cf") return ("keep", ch, None) def _char_label(ch: str) -> str: cp = ord(ch) name = unicodedata.name(ch, "UNKNOWN") cat = unicodedata.category(ch) return f"U+{cp:04X} {name} ({cat})" def _hit_confidence(kind: str) -> str: """Layer A hits are edit-based carriers; space homoglyphs are weaker context.""" return "informational" if kind == "space" else "probable" @dataclass class CharHit: codepoint: int char: str label: str count: int kind: str # strip | bidi | tag_chars | variation_selector | zwj_family | private_use | noncharacter | reserved_ignorable | space | confusable | other_cf samples: list[int] = field(default_factory=list) # character offsets @dataclass class TextInspectReport: length: int suspicious_total: int hits: list[CharHit] notes: list[str] = field(default_factory=list) def to_dict(self) -> dict: return { "length": self.length, "suspicious_total": self.suspicious_total, "hits": [ { "codepoint": f"U+{h.codepoint:04X}", "label": h.label, "count": h.count, "kind": h.kind, "confidence": _hit_confidence(h.kind), "sample_offsets": h.samples[:10], } for h in self.hits ], "notes": self.notes, } def inspect_text( text: str, *, aggressive: bool = False, strip_emoji_glue: bool = False, ) -> TextInspectReport: buckets: dict[tuple[int, str], list[int]] = {} prev_kept: str | None = None valid_flag_tags = _valid_flag_tag_indices(text) valid_bidi_embeddings = _valid_bidi_embedding_indices(text) for i, ch in enumerate(text): action, out_char, kind = _decide( ch, prev_kept, text[i - 1] if i > 0 else None, text[i + 1] if i + 1 < len(text) else None, valid_flag_tag=i in valid_flag_tags, valid_bidi_embedding=i in valid_bidi_embeddings, normalize_spaces=True, treat_confusables=aggressive, strip_emoji_glue=strip_emoji_glue, strip_bidi=True, ) if kind is None: # Kept; glue (emoji/script joiner/tag) does not advance the # "previous kept" base so ZWJ chains and flag runs stay bound. if not _is_glue(ord(ch)): prev_kept = out_char continue key = (ord(ch), kind) buckets.setdefault(key, []).append(i) if action == "replace": prev_kept = out_char # strip: prev_kept unchanged hits: list[CharHit] = [] total = 0 for (cp, kind), offsets in sorted(buckets.items(), key=lambda x: (-len(x[1]), x[0][0])): ch = chr(cp) hits.append( CharHit( codepoint=cp, char=ch, label=_char_label(ch), count=len(offsets), kind=kind, samples=offsets[:10], ) ) total += len(offsets) notes = [ "Layer A only: invisible/format Unicode and space homoglyphs (edit-based carriers).", "Statistical (token-sampling) watermarks are not detectable here; use Layer B rewrite.", "Inspect kinds: strip, bidi, tag_chars, variation_selector, zwj_family, private_use, space, confusable, other_cf.", "Load-bearing invisibles are preserved by default during cleaning: emoji glue, CJK/Mongolian variation selectors, script joiners, complete flag tag sequences, same-script fillers/selectors (Mongolian FVS, Khmer inherent vowels, Hangul jamo fillers), RTL directional marks/paired embeddings, orthographic Arabic/Syriac Cf marks, and visible-layout format controls next to their own script (Egyptian hieroglyph quadrat, Duployan shorthand, musical beaming). Inspection still reports bidi controls. Use explicit strip flags only after review.", ] if not hits: notes.append( "No deterministic Layer A (invisible Unicode/format) carriers detected; " "statistical and pixel-domain marks are out of scope here." ) return TextInspectReport(length=len(text), suspicious_total=total, hits=hits, notes=notes) def clean_text( text: str, *, nfkc: bool = False, aggressive_homoglyphs: bool = False, normalize_spaces: bool = True, strip_emoji_glue: bool = False, strip_bidi: bool = False, ) -> tuple[str, dict]: """Return cleaned text and a stats dict.""" removed: Counter[str] = Counter() replaced: Counter[str] = Counter() out_chars: list[str] = [] prev_kept: str | None = None valid_flag_tags = _valid_flag_tag_indices(text) valid_bidi_embeddings = _valid_bidi_embedding_indices(text) for i, ch in enumerate(text): action, out_char, _kind = _decide( ch, prev_kept, text[i - 1] if i > 0 else None, text[i + 1] if i + 1 < len(text) else None, valid_flag_tag=i in valid_flag_tags, valid_bidi_embedding=i in valid_bidi_embeddings, normalize_spaces=normalize_spaces, treat_confusables=aggressive_homoglyphs, strip_emoji_glue=strip_emoji_glue, strip_bidi=strip_bidi, ) if action == "keep": out_chars.append(out_char) # Glue (emoji/script joiner/tag) does not advance the "previous # kept" base, so ZWJ chains (❤️🔥) and flag runs stay bound. if not _is_glue(ord(ch)): prev_kept = out_char elif action == "replace": out_chars.append(out_char) replaced[_char_label(ch)] += 1 prev_kept = out_char else: # strip removed[_char_label(ch)] += 1 # prev_kept unchanged result = "".join(out_chars) nfkc_changed = False if nfkc: before = result result = unicodedata.normalize("NFKC", result) if result != before: nfkc_changed = True changed_inputs = sum( end - start for operation, start, end, _new_start, _new_end in SequenceMatcher( None, before, result, autojunk=False ).get_opcodes() if operation != "equal" ) replaced["NFKC_normalize"] += changed_inputs or 1 # Collapse runs of spaces only if we introduced space replacements? Keep conservative: no. stats = { "input_length": len(text), "output_length": len(result), "removed": dict(removed), "replaced": dict(replaced), "removed_count": sum(removed.values()), "replaced_count": sum(replaced.values()), "nfkc_changed": nfkc_changed, } return result, stats def human_report(report: TextInspectReport) -> str: lines = [ f"Length: {report.length} chars", f"Suspicious: {report.suspicious_total}", ] if report.hits: lines.append("Hits:") for h in report.hits: lines.append( f" [{h.kind}/{_hit_confidence(h.kind)}] {h.label} x{h.count} @ {h.samples[:5]}" ) for n in report.notes: lines.append(f"Note: {n}") return "\n".join(lines)skills/clean-user-facing-text/references/detectors.md8.1 KBf467abe8791bupstream ↗
# AI text detectors: what exists and what is legitimate to claim Background for the scoring and levers in `SKILL.md`. Current as of mid-2026. The short version: detectors are uncalibrated proxies, and a humanizer can honestly report that measurable signals moved. It cannot honestly claim a verdict, and none of it applies to content the user is not authorized to clean (see `responsible-use.md`). ## Two mechanically different families A pass that beats one family is often invisible to the other, so any claim must say which family it refers to. **Family A: statistical / zero-shot detectors.** They score the text's probability profile under a reference model, with no training on labeled data. - Perplexity: AI text has uniformly low perplexity because LLMs emit the most probable next token. Countered by genuinely raising perplexity through specific, concrete diction. - Burstiness: the variance of perplexity and sentence length across a document. AI text is too even. Countered by varied sentence and paragraph lengths. - DetectGPT (Mitchell et al., 2023): perturbs the text and checks whether the original sits at a local maximum of model log-probability. Countered by rebuilding sentence logic, not just swapping words. - Binoculars (Hans et al., 2024, arXiv:2401.12070): ratio of an observer LLM's log-perplexity to cross-perplexity between two LLMs. High accuracy and notably robust to styled noise and simple adversarial edits: the high-perplexity choices must be coherent human choices. **Family B: trained neural classifiers.** A transformer fine-tuned on millions of human-vs-AI samples learns the LLM fingerprint directly. - Pangram (arXiv:2402.14873): tokenize, embed, classify; hardened with hard-negative mining and synthetic mirrors that include generic humanizer/paraphraser output, so paraphrase-style edits are trained in as the AI class. - GPTZero (3.2b and successors): end-to-end deep classifiers plus sentence-level and mixed-content detection, with explicit shields against paraphrasing and homoglyph attacks. - Originality, Turnitin, Copyleaks: same family; specifics undisclosed. Against Family B, perplexity and burstiness tricks and synonym swaps are weak. The levers that matter are matching a real human distribution (voice matching against a sample the user owns) and, when the user can run the actual detector, iterating with detector feedback. ## Watermarking is a separate axis Watermarks are inserted at generation time by the model owner; they are not a property of "AI-ness" the way perplexity is. - Kirchenbauer green/red list (2023): hash the preceding token to bias a green subset; detection counts the green-token surplus. - SynthID-Text (Google DeepMind, 2024): tournament sampling from the same red-green family, less perceptible. - Both key on the preceding token, so any meaning-preserving rewrite that changes token order degrades detection against that specific detector: paraphrasing, edits, and back-translation all work. Substantial rewriting can reduce detection by a given detector, but watermark absence cannot be verified without the applicable detector and key, so no rewrite removes a watermark with certainty. Vendor status notes: Google retired SynthID text watermarking on the Generative Language API (mid-2026), so current API output is no longer watermarked. Anthropic's text-watermark detection API is not public, so no local tool can verify a Claude watermark. ## What the vendored scorer measures `score_stylometry.py` is a zero-LLM estimator: sentence-length burstiness CV, weighted AI-cadence phrase density per 100 words, MATTR lexical diversity, and a dampened composite (0-1) with the same thresholds as the service pipeline. It deliberately covers Family-A signals only. It does **not** measure: trained classifier output, secret-key watermarks, semantic quality, fact accuracy, or authenticity. Also note the small-sample guard: below 30 words it reports `insufficient_length` and no score; below 100 words the score is dampened. Treat score movement as evidence about lexical and cadence signals, never as a verdict. Every report also carries a `density_tier` (`low` / `medium` / `high`, or `uncalibrated` when not scored). It is only a re-label of where the composite score sits: `high` means at or above the suspicious threshold, so the rewrite pass should engage. `low` / `medium` mean the measurable AI-density signals are weaker, so the skill should mostly verify rather than rewrite. It is not a proof of authorship either way. ## Pattern categories folded in The scorer's phrase list (44 entries) and the prose levers below draw on two public humanizer catalogs, both derived from Wikipedia's "Signs of AI writing" field guide: - **blader/humanizer** — 35 numbered rewrite patterns grouped as: inflated importance and legacy; name-dropping to prove importance; shallow `-ing` analysis; sales language; vague sources; formulaic "challenges and outlook"; overused AI words; copula avoidance ("serves as", "boasts"); "not X but Y" and clipped endings; forced groups of three; repeated openings; false "from X to Y" ranges; passive voice; em/en dashes; bold/bold mini-heading overuse; title-case headings; emojis; curly quotes; excessive hyphenated pairs; fake-deeper-truth and fake-candid openings; announcing the next point; forced punchlines; formulaic sayings; leftover chatbot text; knowledge-limit disclaimers; overly agreeable tone; filler and hedging; generic positive endings. - **conorbronsdon/avoid-ai-writing** — a 100+ word-replacement table in three density tiers (always flag, flag in clusters, flag by density) plus a detect-only audit mode and voice profiles. The tiering is the reason the scorer flags common-but-normal words (`leverage`, `utilize`, `robust`, `unprecedented`) only at low weight: its job is a gauge, not a proof. The regex-detectable subset is baked into `AI_PHRASE_PATTERNS`; the categories that resist regex (rule of three, repeated openings, forced punchlines, em-dash overuse) remain prose levers and are handled by the rewrite pass. ## Legitimate claims versus not Legitimate to report, honestly labeled: - Before and after scores from the vendored estimator, with its confidence level and findings ("the lexical and cadence signals moved"). - What changed: phrase list cleared, burstiness CV moved, artifacts removed. - A detect-only audit (`inspect_text.py --audit`): a structured list of flagged Unicode and stylometric spans with a per-span severity and a density tier. This flags without rewriting and makes no authorship claim. - What the user must verify themselves: semantic equality, fact preservation, voice, required disclosures. Not legitimate to claim: - "Passed GPTZero / Turnitin / Originality" without the user running that exact detector on the output, and even then only as that detector's own report. - "Undetectable", "no AI trace", or "certified human". - Removal of a vendor secret-key watermark (impossible to verify locally). - That a rewrite proves human authorship. For high-stakes cases, the only honest workflow is detector feedback: the user runs the actual detector(s) they care about, and only claims what those reports show. The rewrite itself should aim at genuine, coherent human prose first; scoring is a check that the measurable signals moved, not a substitute for quality. ## Sources - Mitchell et al., DetectGPT (2023). - Hans et al., Binoculars (arXiv:2401.12070, 2024). - Pangram detector notes (arXiv:2402.14873, 2024). - Sadasivan et al., "Can AI-Generated Text be Reliably Detected?" (arXiv:2303.11156, 2023): paraphrase attacks break watermarking, neural, and zero-shot detectors; recursive paraphrasing defeats even watermark and retrieval-based defenses; the paper also establishes a theoretical impossibility result, which is why all of the above is the honest limit. - Kirchenbauer et al., watermarking (2023); Google DeepMind SynthID-Text (2024). - Pattern catalogs folded in (both MIT): blader/humanizer (35 Wikipedia-derived rewrite patterns) and conorbronsdon/avoid-ai-writing (density-tiered word-replacement table and detect-only audit mode). Both trace back to Wikipedia's "Signs of AI writing" field guide.skills/clean-user-facing-text/references/responsible-use.md536 Bd51c326638f8upstream ↗
# Responsible use Use this skill for content the user owns or is authorized to process, including privacy, publishing hygiene, accessibility, and research. Do not present cleaned text as proof that no AI assistance occurred. Do not help misrepresent authorship or bypass disclosure requirements in academic, legal, platform, or regulatory settings. When intent is ambiguous, perform ordinary writing and Unicode hygiene without making detector-evasion claims. When disclosure is required, preserve or add the appropriate disclosure.skills/clean-user-facing-text/references/watermark-notes.md1.2 KB59e919ca79d0upstream ↗
# Text watermark notes ## Deterministic marks Invisible Unicode, bidirectional controls, tag characters, exotic spaces, and selected confusables can carry machine-readable signals or cause broken copy, search, and diffs. `clean_text.py` removes or normalizes known carriers and reports exact counts. By default it preserves contextual characters used by emoji, joining scripts, Mongolian selectors, Khmer inherent vowels, Hangul fillers, and Arabic/Syriac orthography. The lightweight workflow also disables space normalization by default so multilingual typography remains intact. Aggressive flags can damage intentional text and should remain opt-in. ## Statistical marks Token-sampling watermarks such as green-list or tournament-sampling schemes live in word choice and token sequences rather than metadata. A substantial rewrite can weaken such signals by changing syntax and vocabulary. This is best-effort: - no bundled detector has the vendor's secret key - a rewrite generated by the same provider may introduce a new signal - short or predictable text provides little statistical evidence either way - detector behavior can change independently of this skill Never describe a successful rewrite as certified, undetectable, or proof of human authorship.skills/clean-user-facing-text/references/writing-in-your-voice.md2.5 KBe413c1747f34upstream ↗
# Writing in your voice A good rewrite should leave the writer sounding more like themselves, not like a generic editor. Keep their opinion, level of formality, regional spelling, and the details that show they know what they are talking about. ## Start with a brief Before rewriting, identify what the writer is trying to say and who needs to read it. Use a writing sample only when the user owns it or is authorised to use it. Don't imitate another named person. An approved sample can guide sentence length, word choice, humour, and formality. It's a reference point, not a licence to invent personal stories or feelings the writer didn't provide. If there is no sample, use plain language and a natural rhythm. Don't claim to write in a named person's voice from a guess. Keep the language and regional spelling the writer used unless they ask for a change. For example, retain Australian spelling in an Australian document. ## Make the prose sound lived-in Prefer the specific subject of the sentence over vague commentary about it. Cut filler that does not add information. Let sentence length move around when the subject calls for it: a longer explanation can sit beside a short, direct sentence. Keep a writer's useful rough edges. A dry joke, a firm judgement, or an unusual turn of phrase may be the part that makes the piece feel like theirs. Clean up confusion, repetition, and accidental clutter. Don't sand every opinion down until it says nothing. Treat unusual grammar, repetition, directness, and phrasing as possible voice or accessibility choices. Change them only when the writer asks, or when they create a clear reading problem. ## Watch for template prose Remove or rewrite phrases that announce a point without making it, such as "In today's fast-paced world", "It is important to note", or "This article will explore". Be wary of grand claims about significance, empty conclusions, and stock transitions between paragraphs. Avoid a steady run of identical sentence shapes. Break up mechanical lists of adjectives and filler phrases that merely repeat the sentence before them. ## Keep the boundary clear Don't add facts, citations, experience, emotion, or certainty. Don't remove a required disclosure, attribution, warning, or qualification. Preserve quoted text, code, names, numbers, and technical identifiers exactly where the main skill requires it. This pass can make authorised prose clearer and more personal. It can't prove who wrote the original text or establish that a detector will reach any result.README.mdupstream ↗
Contents are not inlined in the catalog. Open the upstream file.
Plain-text copy, no login: /skill-bundles/guillaumemeyer-watermarks-remover-skills-clean-user-facing-text.md