# data-update

Adopt this skill: https://proskills.md/skills/other/sixian-li-plain-backtest-agents-skills-data-update

Page: https://proskills.md/skills/other/sixian-li-plain-backtest-agents-skills-data-update
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: .agents/skills/data-update/SKILL.md
size: 6356
sha: abe24ba21fe23e52a7d19b503427fd46e98eee9c
upstream: https://github.com/sixian-li/plain-backtest/blob/main/.agents/skills/data-update/SKILL.md

````text
---
name: data-update
description: Operate and assess the Quant workspace market-data layer through the tested data-update CLI. Use when checking QQQ/SPY or constituent freshness, reviewing supported versus missing data capabilities, diagnosing quality, running a SPY-current-member shadow update, inspecting update reports, or previewing/importing purchased CSV/ZIP/JSON/XLSX/XLS data without silently changing approved datasets.
---

# Data Update

Use the workspace CLI as the single execution path. Keep update logic in the tested project code; do not reimplement provider calls in the skill.

## Distributed project

This skill is versioned at `.agents/skills/data-update/` in this repository. Locate the root from the current checkout; do not depend on a user-level skill or the author’s original workspace. Read `README.md` and `backtest/docs/release/scope.md` first. The distribution contains canonical data and experiment definitions, not the complete source archives or historical runs.

For the bundled snapshot, begin with `backtest/.venv/bin/python backtest/scripts/release_data.py status` and `check`. These are offline and read-only. The live CLI below is for optional data operations; its source/archive audit may report omitted historical files. Do not rebuild canonical data implicitly. Data reuse is authorized under `data/LICENSE`; quality-failed and pending-review products keep their original quality gates.

## Locate and inspect

1. Locate the Quant root containing `catalog.md`, `log.md`, `data/`, and `backtest/`.
2. Read `data/data_update_registry.json` before any mutating operation.
3. Run commands from the Quant root with:

```bash
backtest/.venv/bin/python backtest/scripts/data_update.py <command>
```

4. Never display API keys. The CLI resolves them from environment variables or macOS Keychain.

Read [contracts.md](references/contracts.md) when interpreting statuses, running a full update, or importing purchased files.

## Route the request

### Check the last update or health

Run:

```bash
backtest/.venv/bin/python backtest/scripts/data_update.py status
backtest/.venv/bin/python backtest/scripts/data_update.py check
```

Use `check --deep` only for an explicit full audit, after material data work, or before enabling promotion. It verifies large raw files and runs the full test suite, so expect it to take longer.

Report separately:

- approved database date;
- latest completed XNYS session;
- current-member coverage;
- tracked QQQ/SPY/VOO dates and whether their shadow update exists;
- capability boundaries, especially production promotion, Nasdaq-100 point-in-time membership, and hedge-asset coverage;
- latest source-validation and shadow-update run;
- warnings versus blocking failures;
- whether production writes and scheduling are enabled.

### Manually update

Run an isolated shadow update:

```bash
backtest/.venv/bin/python backtest/scripts/data_update.py shadow-update
```

For a fast diagnostic, pass `--symbols AAPL,FERG,BRK.B,BF.B --tiingo all`. For a requested full current-universe run, omit `--symbols` and `--limit`; warn that Twelve Data's free 8-credit/minute limit makes 503 symbols take roughly 63 minutes for the single adjusted-price request.

Treat exit code 2 as “review required,” not automatically as a crash. Open the `report_json` referenced by `data/processed/updates/sp500_shadow/latest.json` and explain the exact symbol statuses.

For a campaign rerun after code-only fixes, use immutable provider archives instead of spending primary-source credits again. `--replay-twelve-run RUN_ID` reuses a complete Twelve Data archive; `--replay-tiingo-runs RUN_ID[,RUN_ID...]` reuses available Tiingo payloads and fetches only missing required symbols. Never use replay to stand in for a new completed XNYS session.

If a live run stops after writing only some Twelve Data batches, preserve that raw run and resume into a new run ID:

```bash
backtest/.venv/bin/python backtest/scripts/data_update.py shadow-update \
  --resume-twelve-run PARTIAL_RUN_ID \
  --run-id NEW_RUN_ID
```

Resume reuses valid archived symbols, fetches only missing or stale symbols, and never modifies the source archive. Inspect `resume_source_issues` in the new report. Do not call a partial transport failure a completed campaign day.

If one archived symbol contains a transient latest-session value that later fails the independent cross-check, preserve the archive and refetch only that selected symbol while resuming:

```bash
backtest/.venv/bin/python backtest/scripts/data_update.py shadow-update \
  --resume-twelve-run SOURCE_RUN_ID \
  --refresh-symbols CBOE \
  --replay-tiingo-runs SOURCE_RUN_ID \
  --run-id NEW_RUN_ID
```

Use `--refresh-symbols` only with `--resume-twelve-run`. Require the replacement payload to pass the unchanged candidate and cross-source gates; do not use this option to overwrite or hide the original failed evidence.

Do not edit `data/processed/daily/equities/`. While `production_writes_enabled=false`, no command may promote candidates into the approved database.

### Import newly purchased data

Require the exact source path. Always preview first:

```bash
backtest/.venv/bin/python backtest/scripts/data_update.py import-purchased \
  --source /exact/external/path --deep
```

Summarize file count, bytes, root SHA256, formats, and inspection failures. Apply only after the user explicitly authorizes copying that exact source and supplies or confirms provider and acquisition date:

```bash
backtest/.venv/bin/python backtest/scripts/data_update.py import-purchased \
  --source /exact/external/path \
  --provider provider-name \
  --acquired-date YYYY-MM-DD \
  --label optional-label \
  --apply
```

Applied files remain immutable under `data/raw/purchased/` with status `pending_review`. Never infer permission to merge them into approved prices, rebuild historical membership, or delete the external source.

## Finish safely

1. Inspect the generated machine report and human-readable report when present.
2. Run `check`; use `check --deep` after an applied purchase import or material implementation change.
3. If material workspace files changed, also follow `quant-tidy` for `catalog.md`, `log.md`, audit, and tests.
4. State exactly what was updated, what stayed shadow/pending, the last covered session, and any user action required.
````

## File: .agents/skills/data-update/references/contracts.md
size: 4849
sha: 310f4229e17296bbb7f3dfca3f7c38f40cbc3aae
upstream: https://github.com/sixian-li/plain-backtest/blob/main/.agents/skills/data-update/references/contracts.md

```text
# Data-update contracts

## Authoritative paths

- Operations contract: `data/data_update_registry.json`
- Historical baseline contract: `data/sp500_history_registry.json`
- Current dated universe: `data/processed/universes/sp500/current_constituents.csv`
- Approved current/history prices: `data/processed/daily/equities/`
- Shadow updates: `data/processed/updates/sp500_shadow/`
- Archived public responses: `data/raw/public/market_data_updates/`
- Purchased intake registry: `data/purchased_import_registry.json`
- Immutable purchased intake: `data/raw/purchased/`
- Nasdaq-100 candidate contract: `data/nasdaq100_history_registry.json`
- Nasdaq-100 pending-review products: `data/processed/universes/nasdaq100/pending_review/`

## Capability boundary

Treat `data/data_update_registry.json.capabilities` as authoritative. The S&P 500 point-in-time baseline, current-member shadow update, provider replay, and purchased intake are ready. Nasdaq-100 point-in-time membership is implemented as `candidate_pending_review`, not approved: use its frozen manifest and hashes only for explicitly authorized exploratory research, preserve the one-XNYS-session membership lag, and block strategy promotion while provenance, authorization, identity, adjustment, and early-history reviews remain open. Approved-price promotion and the scheduler remain disabled; QQQ/SPY shadow updates and a hedge-asset universe are not implemented. Report these boundaries explicitly instead of implying that the current SPY-constituent updater or Nasdaq candidate covers every strategy input.

## Status meanings

- `ready_candidate`: overlap, date coverage, and adjusted-return continuity passed; still shadow-only.
- `ready_bootstrap_candidate`: dated official identity plus independent security evidence allow a post-baseline shadow candidate; it is not production approval and pre-baseline history is not spliced.
- `bootstrap_needs_identity_review`: public prices exist for a missing current constituent, but security identity/history must be approved before any merge.
- `review`: one or more date, overlap, freshness, or cross-source gates need inspection.
- `fetch_or_analysis_failed`: the provider or parser failed for that symbol; other symbols may still be valid.
- `pending_review`: a purchased batch was copied and hash-verified but is not an approved input.
- `candidate_pending_review`: a reproducible derived candidate passed machine parsing and hash gates but still has unresolved human-review fields. It may support explicitly authorized exploratory research with frozen provenance; it is not an approved production input and cannot support promotion claims.

## Non-negotiable boundaries

- A shadow candidate is not an approved database row.
- A successful Tiingo/Twelve comparison does not prove historical security identity.
- Price level differences caused by different adjustment bases require an anchor; compare overlapping daily returns before scaling new rows.
- A historical adjustment-basis jump requires registered public corporate-action evidence, a stable recent return tail, and same-run Tiingo coverage; an undocumented jump blocks the run.
- Volume is warning-only because provider semantics differ materially.
- Never fill missing bars, splice ticker histories, or merge purchased files silently.
- Never expose credentials in argv, reports, source files, or chat.

## Free-tier behavior

- Twelve Data: 8 credits/minute and 800/day; one time-series symbol consumes one credit. The CLI waits across minute windows and uses a single adjusted-price request per symbol.
- Tiingo: 500 unique symbols/month, 50 requests/hour, and 1,000/day. The configured 480-symbol monthly budget leaves headroom; the monthly offset rotates the otherwise omitted current constituents over time.
- Eastmoney: opportunistic validator only until repeated transport-stability runs pass.

## Five-session campaign

- A qualifying day covers all 503 dated current equities, has no primary-provider failure, produces only allowed shadow statuses, passes at least 30 Tiingo cross-checks, and writes zero production rows.
- Every bootstrap candidate and every symbol with a documented historical adjustment-basis warning must be included in that day's Tiingo sample.
- Progress is the latest consecutive XNYS-session streak. Weekends and exchange holidays do not break it; a missing or failed trading session restarts the streak.

## Purchased intake phases

1. Preview external source, inventory hashes, and inspect supported containers.
2. Explicitly apply a reviewed source into immutable raw storage.
3. Review provider schema, ticker identity, adjustment semantics, coverage, and licensing.
4. Build a separate candidate transformation and compare against approved overlap.
5. Promote only through a separately implemented, tested, explicitly authorized gate.
```

## File: .agents/skills/data-update/agents/openai.yaml
size: 236
sha: 1a09d11a83ae9fa10555ff739aba2d752650aa79
upstream: https://github.com/sixian-li/plain-backtest/blob/main/.agents/skills/data-update/agents/openai.yaml

```text
interface:
  display_name: "Data Update"
  short_description: "Check and safely operate Quant market-data workflows"
  default_prompt: "Use $data-update to check Quant data freshness, capability gaps, and run the safe update workflow."
```

## File: README.md
upstream: https://github.com/sixian-li/plain-backtest/blob/main/README.md

````text
# PlainBacktest

**English** | [中文](README.zh-CN.md)

**Say It Simply, Test It Properly.**

Describe your trading strategy in everyday language. PlainBacktest works with your coding agent to turn it into explicit rules, run the backtest, and reconcile orders and daily account balances against an independent ledger. Explicit execution timing, recorded data and code hashes, and reproducible reports make the calculation open to inspection.

Source code and bundled data: [Sixian-Li/plainbacktest](https://github.com/Sixian-Li/plainbacktest).

```text
Strategy in natural language → Agent + Skill clarify the rules → Freeze the experiment
→ Data checks → PyBroker execution + independent ledger → Report and research record
```

Your agent interprets the strategy and implements its rules; the Python framework handles reproducible calculation and validation. The repository includes three Skills, curated local data, research code, and 83 experiment definitions. Start with an offline example, then use your agent to continue the research.

## Run your first example

Clone the repository and run the following commands. The verified environment is macOS with Python 3.13. Installing dependencies requires an internet connection; running the example needs no market-data API, credentials, or live data.

```bash
git clone https://github.com/Sixian-Li/plainbacktest.git
cd plainbacktest
python3.13 -m venv backtest/.venv
backtest/.venv/bin/python -m pip install -r backtest/requirements.lock
backtest/.venv/bin/python backtest/scripts/release_data.py check
backtest/.venv/bin/python backtest/scripts/quickstart.py
```

Open the `outputs/example_…/report.html` path printed at the end. Each execution creates a new directory containing the input data, frozen rules, daily account state, orders, metrics, code and data hashes, and reconciliation results.

The default example uses QQQ from 2016-01-01 through 2026-08-04. It holds QQQ when SMA20 is above SMA100 and stays in cash otherwise. Signals are confirmed at the close and executed at the next session's open, with a cost of 5 bps per side. Initial capital is $100,000; fractional shares are allowed, with no leverage. The example uses the framework's PyBroker execution engine, independent ledger, and v5 report template.

Change the parameters or choose another approved symbol:

```bash
backtest/.venv/bin/python backtest/scripts/quickstart.py --symbol SPY --fast 50 --slow 200
```

This exploratory example checks installation, execution, and reconciliation. It does not mark existing experiments as validated or replace parameter robustness and out-of-sample analysis. Moving averages use earlier history for warmup. Adjusted OHLC prices approximate total returns without modeling individual cash dividend payments.

## Research with natural language

Open this directory with an agent that supports project Skills. Have it read [AGENTS.md](AGENTS.md), then describe a strategy. For example:

> Use quant-backtest to study QQQ with an SMA20/SMA100 rule. If the fast SMA is above the slow SMA at the close, hold a fully invested position from the next session's open; otherwise, exit to cash. Define the data range, costs, warmup, and benchmark before implementing the strategy, reconciling the ledgers, and explaining the results.

The project overview and quickstart are available in English and Chinese. Detailed research documents and report text are currently mainly in Chinese.

All three Skills are versioned with the repository, each with a `SKILL.md` entry point:

| Skill | Purpose |
| --- | --- |
| [quant-backtest](.agents/skills/quant-backtest/SKILL.md) | Turn a strategy description into an experiment, execution, independent reconciliation, report, and research record |
| [data-update](.agents/skills/data-update/SKILL.md) | Inspect data snapshots and manage isolated updates and candidate review when needed |
| [quant-tidy](.agents/skills/quant-tidy/SKILL.md) | Maintain directories, experiment lineage, and workspace structure |

These Skills are part of the project. Agents that support `.agents/skills` can discover them directly; other agents can read the relevant `SKILL.md` and its references. They do not depend on the author's global Skill installation. See the [research architecture](backtest/docs/architecture.md) for Git provenance, run lifecycles, validation gates, and worktree conventions used in formal research. The example also works from a downloaded source ZIP.

## What's included

- `backtest/quantkit/`, `scripts/`, and `tests/`: execution, ledgers, strategies, metrics, data handling, and validation.
- `backtest/report_templates/`: self-contained Plotly reports. The default v5 template explains the strategy before presenting results.
- `backtest/experiments/`: 83 definitions and their lineage across three research programs: DER, ROT, and TIM.
- `data/`: 1,303 canonical daily-price CSVs, trading calendars, membership tables, quality evidence, and two original Nasdaq-100 ZIP archives. See the [data documentation](data/README.md).
- `research/`: indicator and market-observation tools without strategy account ledgers.

This is an independent distribution snapshot. Historical runs and large collections of old reports are not bundled, and the active/latest run pointers have been cleared. Historical findings from the source workspace have not been revalidated here. Some experiments require earlier training artifacts or original archives that are not included; see the [distribution scope](backtest/docs/release/scope.md). The source revision and data hashes at the time of copying are recorded in [source_snapshot.json](backtest/docs/release/source_snapshot.json).

This version uses the bundled data by default. `yfinance` is a transitive dependency of PyBroker, but it has not been integrated as this project's default data provider. The quickstart does not call Yahoo. The existing shadow-update tools are optional advanced features.

## Validation and optional labs

```bash
backtest/.venv/bin/python -m pip check
backtest/.venv/bin/python backtest/scripts/release_data.py check
cd backtest
.venv/bin/python -m unittest discover -s tests -t . -v
.venv/bin/python -m scripts.build_research_catalog --check
.venv/bin/python -m scripts.audit_workspace --workspace ..
```

Tests that require original archives absent from this distribution report an explicit skip, not a pass. See the [verification record](backtest/docs/release/verification.md) for results. The distributed test suite does not implicitly rebuild the bundled canonical data.

Two optional Streamlit labs are included. Launch either from the project root:

```bash
backtest/.venv/bin/python -m streamlit run backtest/dual_sma_lab/app.py
backtest/.venv/bin/python -m streamlit run research/indicator_lab/app.py
```

Browser interaction checks and PDF export also require Node.js and Chrome/Chromium. Set `QUANT_CHROME_PATH` to use a specific browser executable. The standard example and Python tests do not require them. Other operating systems have not yet been fully verified.

## Licensing

Original project code, Skills, and documentation are licensed under [MIT](LICENSE). The bundled data under `data/` is licensed under [CC BY 4.0](data/LICENSE); attribution details are in the [data documentation](data/README.md). The data rights holder has confirmed ownership and authorized this license.

Third-party dependencies retain their own licenses. In particular, PyBroker 1.2.12 uses **Apache 2.0 with Commons Clause**, which includes a restriction on selling the software as defined in that license. The dependency stack is therefore not entirely MIT-licensed. See [third-party notices](THIRD_PARTY_NOTICES.md) for the original terms.
````
