auditing-data-and-ground-truth

5 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

• skills/auditing-data-and-ground-truth/SKILL.md3.7 KB89cd2e2c044dupstream ↗

  ---
  name: auditing-data-and-ground-truth
  description: Audit datasets, joins, labels, and ground truth before analysis or modeling. Use when data meaning, row grain, time semantics, source-of-truth reliability, or label construction may invalidate conclusions; not for general model evaluation after the evidence base is already trusted.
  ---
  
  # Audit Data and Ground Truth
  
  Determine what the data actually represents and whether its labels can support the intended claim. The outcome is an evidence-backed readiness judgment, not a generic profile report.
  
  ## Anchor the audit
  
  State the intended decision, entity, row grain, target, prediction moment, outcome window, and population. If these are not known, surface that uncertainty before interpreting columns or labels.
  
  Preserve source data unchanged. Work from read-only inputs or documented snapshots, and keep every correction, exclusion, or derived label traceable to its source.
  
  ## Inspect in risk order
  
  1. **Source and lineage:** Identify origin, owner, extraction time, filters, transformations, versions, and competing sources of truth.
  2. **Structure:** Measure row and entity counts, schema, types, key uniqueness, duplicates, missingness, ranges, and category cardinality. Report both counts and rates where scale matters.
  3. **Meaning:** Confirm units, code sets, null semantics, sentinel values, event meanings, and whether one row means what the analysis assumes.
  4. **Relationships:** Test key uniqueness on each side before joining. State expected and observed cardinality, unmatched rates, row multiplication, and aggregation effects.
  5. **Time:** Separate event time, record time, update time, and availability time. Check impossible orderings, late arrivals, backfills, and future knowledge.
  6. **Ground truth:** Trace each label from operational event to encoded value. Verify annotator or system provenance, adjudication, label maturity, class definitions, coverage, disagreement, and missing-label behavior.
  7. **Reconciliation:** Compare raw records with independent records or a stratified sample. Do not let agreement with a downstream table validate the upstream process that created both.
  
  Read [references/audit-playbook.md](references/audit-playbook.md) when selecting concrete tests or grading the severity of a finding.
  
  ## Protect semantics
  
  - Do not silently interpret missing as negative, absence as zero, or the latest value as historically available.
  - Do not deduplicate until the real-world uniqueness rule is established.
  - Do not repair values in place; produce a derived correction with a reason and impact count.
  - Do not accept labels merely because they are named `target`, `truth`, `gold`, or `final`.
  - Do not average away disagreements that reveal a definition or annotation problem.
  
  Within already authorized data, systems, and compute, run read-only or reversible tests and document the evidence. Obtain approval before choosing a new source of truth, changing label policy, excluding a material population, applying irreversible corrections, accessing a new or sensitive source, incurring material cost, involving an external reviewer, or sharing records outside the approved boundary.
  
  ## Issue a readiness judgment
  
  Use [assets/data-and-ground-truth-audit.md](assets/data-and-ground-truth-audit.md) for the deliverable. For every material finding, record evidence, affected scope, consequence, and recommended treatment. End with one status:
  
  - **ready:** evidence supports the intended use;
  - **ready with conditions:** use is supportable only under named constraints;
  - **not ready:** a known defect can change the conclusion;
  - **unknown:** critical evidence is missing or cannot be independently checked.
  
  Never upgrade `unknown` to `ready` because no defect was observed.
  

• skills/auditing-data-and-ground-truth/references/audit-playbook.md4.0 KBd08400b15188upstream ↗

  # Data and ground-truth audit playbook
  
  Select checks according to the intended decision and failure risk. Do not run a large catalog without explaining what each check can falsify.
  
  ## Grain and identity
  
  - Compare total rows, distinct candidate keys, and distinct real-world entities.
  - Test key uniqueness within the time or source boundaries where uniqueness is claimed.
  - Sample duplicates and classify exact replay, legitimate repeated event, correction, or unresolved collision.
  - Check whether aggregations change the target population or create unequal observation periods.
  
  High-risk signal: the modeled row does not map one-to-one to the decision unit.
  
  ## Joins and coverage
  
  For each join, record expected cardinality and measure:
  
  - unmatched keys on both sides;
  - duplicate keys per side;
  - row counts before and after;
  - entity counts before and after;
  - multiplication-factor distribution;
  - whether missing matches cluster by source, time, or outcome.
  
  High-risk signal: join survival or multiplication depends on the target or a protected or operationally important slice.
  
  ## Missingness and encoded absence
  
  Separate unavailable, not applicable, not collected, failed extraction, suppressed, and true zero. Compare missingness by source, time, target, and population slice. Inspect sentinel values and default-filled fields before converting types.
  
  High-risk signal: missingness is downstream of the outcome or a processing path unavailable in production.
  
  ## Time and historical truth
  
  Distinguish:
  
  - event time: when reality occurred;
  - effective time: when a value became valid;
  - record time: when the system stored it;
  - availability time: when the model or analyst could retrieve it;
  - update time: when a correction or backfill occurred.
  
  Check negative durations, impossible sequences, time-zone changes, late arrivals, backfills, snapshot reconstruction, and use of current attributes for historical cases.
  
  High-risk signal: a historically evaluated row contains a corrected or enriched value that was unavailable then.
  
  ## Ground-truth provenance
  
  Trace label construction as a chain:
  
  `real-world event -> source record -> extraction -> rule or annotation -> adjudication -> encoded label`
  
  At each link, identify owner, timestamp, version, exclusions, and failure behavior. Measure coverage and unresolved labels. For human labels, measure agreement on a shared sample and examine disagreements by class and slice; agreement alone does not establish correctness. For delayed outcomes, establish a maturity cutoff and treat immature examples separately.
  
  High-risk signals include using treatment or follow-up as proof of the original condition, treating missing follow-up as a negative, circularly validating a label against a derivative of itself, and allowing unresolved cases into a forced binary target.
  
  ## Independent reconciliation
  
  Within the authorized access boundary, choose samples that can reveal systematic failure: random, edge cases, positives, negatives, missing labels, high-loss cases, and relevant sources or time periods. Compare against an approved operational record, original document, or independent reviewer that does not inherit the same transformation path. If that evidence requires new access or external sharing, specify the needed reconciliation and obtain authority before retrieving or transmitting records.
  
  Record the sampling method, reviewer, evidence accessed, mismatch taxonomy, and estimated affected scope. Keep examples privacy-safe in public artifacts.
  
  ## Severity and readiness
  
  Classify a finding by consequence, not cosmetic size:
  
  - **critical:** can reverse the target, split, metric, or primary conclusion;
  - **major:** materially biases a population, time period, or operational slice;
  - **moderate:** limits interpretation or reproducibility but has a bounded workaround;
  - **minor:** does not change the conclusion and is straightforward to correct.
  
  An unresolved critical or major finding normally means `not ready` or `unknown`. A condition is acceptable only when it is explicit, enforced, and reflected in the claimed scope.
  

• skills/auditing-data-and-ground-truth/assets/data-and-ground-truth-audit.md2.5 KB68b0ab72ff42upstream ↗

  # Data and ground-truth audit
  
  ## Intended use
  
  | Field | Definition | Confidence and evidence |
  |---|---|---|
  | Decision or claim |  |  |
  | Entity and row grain |  |  |
  | Target |  |  |
  | Prediction or observation time |  |  |
  | Outcome window |  |  |
  | Population and exclusions |  |  |
  
  ## Source and lineage inventory
  
  | Source | Owner | Version or extract time | Upstream filters or transformations | Expected grain and keys | Independent evidence |
  |---|---|---|---|---|---|
  |  |  |  |  |  |  |
  
  Raw preservation location or snapshot identifiers:
  
  ## Structural evidence
  
  | Check | Expected | Observed | Scope | Result | Evidence artifact |
  |---|---|---|---|---|---|
  | Row and entity counts |  |  |  |  |  |
  | Key uniqueness |  |  |  |  |  |
  | Duplicate classes |  |  |  |  |  |
  | Missingness semantics |  |  |  |  |  |
  | Value ranges and units |  |  |  |  |  |
  
  ## Join evidence
  
  | Join | Expected cardinality | Unmatched left/right | Rows before/after | Entity effect | Multiplication risk | Disposition |
  |---|---|---|---|---|---|---|
  |  |  |  |  |  |  |  |
  
  ## Time evidence
  
  | Field or event | Event time | Record time | Availability time | Backfill behavior | Risk or test result |
  |---|---|---|---|---|---|
  |  |  |  |  |  |  |
  
  ## Ground-truth evidence
  
  - Real-world event represented:
  - Label construction and version:
  - Provenance and owner:
  - Maturity rule:
  - Missing or unresolved-label policy:
  - Coverage by class, source, time, and relevant slice:
  - Agreement or independent reconciliation method:
  - Known ambiguity:
  
  ### Human review, when applicable
  
  - Reviewer roles and relevant qualifications:
  - Written instructions and label version:
  - Blinding to predictions or prior labels:
  - Individual decisions preserved at:
  - Raw and chance-adjusted agreement, when appropriate:
  - Disagreement categories and rate:
  - Adjudicator and independence:
  - Adjudication rule and output:
  - Unresolved or inherently ambiguous cases:
  
  | Sample or stratum | Sample method | Compared evidence | Agreement or mismatch | Mismatch class | Estimated affected scope |
  |---|---|---|---|---|---|
  |  |  |  |  |  |  |
  
  ## Findings and dispositions
  
  | ID | Severity | Finding | Direct evidence | Affected scope | Consequence | Treatment | Owner or approval needed | Residual risk |
  |---|---|---|---|---|---|---|---|---|
  | F-01 |  |  |  |  |  |  |  |  |
  
  ## Readiness judgment
  
  Status: **unknown / not ready / ready with conditions / ready**
  
  Supported uses:
  
  Unsupported uses:
  
  Required conditions:
  
  Evidence still missing:
  
  Next falsifying or reconciliation check:
  

• skills/auditing-data-and-ground-truth/agents/openai.yaml250 B7cfcfe30bd27upstream ↗

  interface:
    display_name: "Audit Data and Ground Truth"
    short_description: "Verify data meaning, joins, labels, and lineage"
    default_prompt: "Use $auditing-data-and-ground-truth to audit this dataset and its labels before I rely on the result."
  

• README.mdupstream ↗

  Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/aiopshwang-data-analysis-ml-agent-skills-skills-auditing-data-and-ground-truth.md