slop-grading

slop-grader

Rule-based slop grader for text files, powered by Jev. Runs every rule against every line in parallel. No skimming, no missed lines.

• How it works
• Features
• FAQ
• Quick Start
• Rulesets
• CLI Reference
• Output Formats
• Development
• Changelog
• Contributing
• Roadmap

How it works

slop-grader runs as a two-step loop: grade text with the CLI, then paste the output to your AI agent to plan the improvements.

1. Grade the document

Run slop-grader on a document like examples/slop.md:

npx @lukstei/slop-grader@latest -r no-ai-slop examples/slop.md

2. Fix with an AI agent

Pass the output to your AI agent:

• The agent distinguishes real violations from false positives and generates concrete replacements (Example with Gemini 3.8 Flash):

  Line 1 — banned_word

  • Original: # 🚀 The Ultimate Paradigm Shift in Modern Data Architecture
  • Fix: # Modern Data Architecture
  • Reason: Removes the banned phrase "paradigm shift" and decorative emoji.

  Line 7 — binary_contrast + faux_insight + colon_reveal

  • Original: What most people get wrong about databases is simple: it's not about speed, it's about trust.
  • Fix: Database design balances speed and trust.
  • Reason: Removes rhetorical framing and fake insight.

• After your review the plan is applied to produce an improved document.

Features

• Parallel exhaustive grading: Checks every rule against every line independently. No skimming.
• Incremental line caching: Re-evaluates only edited lines; unchanged text resolves from cache with zero API calls. Toggle with --no-cache.
• System One efficiency: Typed probabilities via Jev without text generation. Thousands of checks for cents.
• Dynamic batching: Groups lines to token limits to minimize API calls. See evaluation details.
• Context window padding: Evaluates line rules with $\pm 10$ lines of surrounding context. Boundary lines and sparse edits keep full visibility without paying for extra questions.
• Line and document scope: Flags line patterns and rates whole documents on qualitative rubrics.
• Plain Markdown rulesets: Write rules in Markdown; validate offline with --check.
• Built-in rulesets: Ready-to-use rules for AI writing patterns, document scores, tech docs, and grammar.
• Agent and CI ready: Terminal output for agent fix plans; structured JSON for pipelines.
• Multi-provider: Works with TypeSafe AI and OpenRouter out of the box.

FAQ

Evaluation separates line-level checks (spotting specific patterns or phrases) from document-level checks (evaluating tone or overall structure).

Documents have hundreds of lines, but the number of rules is fixed. Sending one API request per line would mean hundreds of calls. Instead, slop-grader dynamically groups lines into batches sized to fit the model's context budget (up to 255 lines per batch) and evaluates each rule across its batch in a single call. Batches include up to 10 lines of surrounding document context padding so boundary lines and sparse edits retain neighboring visibility for cross-line checks. Every batch response is verified for complete answer-to-question parity; any dropped questions halt execution immediately without caching incomplete results.

Document rules run in a single request across the entire text.

Line rules often depend on nearby text: pronoun references, synonym repetition across sentences, or transitions between paragraphs. Evaluating a line in isolation makes these checks impossible.

slop-grader pads each evaluated line with up to 10 lines of surrounding text ($\pm 10$ lines):

1. Region clustering: Lines slated for evaluation are grouped into continuous regions. Targets within 21 lines of each other merge into a single region so the model sees uninterrupted text without repeated lines.
2. Structured omission: When distant lines or sparse edits share a batch, gaps between regions are marked with ...| [omitted lines], preserving uniform line structure for the model.
3. Zero extra question cost: Padding lines exist strictly as non-evaluated context in the prompt state. Questions are generated only for active evaluation targets, so context lines add no question fees and don't count toward rule quotas.

This ensures single-line edits during incremental cache runs retain their full document surroundings rather than floating in isolation.

Line evaluations are cached by the hash of the line's content and the rule's criteria.

When you edit a document and run slop-grader again:

1. Every line is matched against the local cache for each rule.
2. Unchanged lines resolve immediately from the cache with zero API calls and zero cost.
3. Only new or modified lines are batched and sent to the model.
4. If a rule's instructions or criteria change, its cache invalidates automatically.

This makes repeated runs on large files or during editing loops nearly instantaneous. Pass --no-cache to bypass the cache, or --cache-dir <dir> to customize its location. See docs/CACHE.md for technical details on storage layout, hashing, and eviction.

Standard generative LLMs evaluate an entire document in a single prompt against a list of rules. On longer texts, they skip lines, miss rules, and report wrong line numbers.

Running a separate check for every line and rule with a generative LLM is impractical. A 300-line draft tested against 10 rules would require 3,000 text-generation requests, which is slow and expensive.

slop-grader uses a System One model (Jev). System One models answer discrete semantic questions with typed probabilities without generating text. Because these judgments return numbers instead of prose tokens, slop-grader can test every line against every rule separately and in parallel.

Cost scales with the number of rules and non-