⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

version unknown

ar-io-gateway-operator

ar-io@ar-io⭐ 130 stars· last commit 5d ago· 7 open issues

Operate any AR.IO node deployment — architecture, daily ops, diagnostics, and recurring pitfalls that apply to every operator. Use whenever the user asks about gateway health, indexing lag, ClickHouse pipeline, GraphQL routing, ArNS resolution, data retrieval and caching, bundle unbundling, webhook fan-out, container restarts, image rebuilds, disk pressure, HTTPSig overhead, or anything operationally generic to running an AR.IO node — even when they don't name a specific service ("is it healthy?", "why is it slow?", "did the build pick up?", "how full is the disk?"). For deployment-specific knowledge (domains, repo paths, sidecar config, this-machine numbers), pair this with the deployment-overlay skill (e.g. `vilenarios-gateway`) if one exists.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐⭐⭐ 130 on GitHubGitHub ↗

Popular

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

AR.IO Gateway Node

codecov protocol.land Ask DeepWiki

Getting Started

Dev Workflow

Install dependencies

yarn install

Initialize the SQLite DB

yarn db:migrate up

Run lint

yarn lint:check

Run tests

yarn test

Run the service

With defaults:

yarn start

Starting at an arbitrary block (only works immediately after initial DB migration):

START_HEIGHT=800000 yarn start

Operating with Claude Code

This repo ships skills under .claude/skills/ that Claude Code (claude.ai/code) auto-loads when invoked from the repo root:

  • ar-io-gateway-operator — operator runbook covering the ANS-104 pipeline, ClickHouse, ArNS resolution, the observer/cranker, and common pitfalls. Includes scripts/health-check for a one-screen gateway health snapshot.
  • release — version bumps, image SHA pinning, tag and GitHub-release creation.
  • testing — picks the right test layer (unit / property / e2e / auto-verify / parquet integration / load) for a given change.

The skills are plain Markdown — readable as runbooks even if you don't use Claude Code.

Dev Docs

docs/INDEX.md is the index of the documents in docs/.

Schema (WIP)

  • [Bundle schema]

Docker

Standalone AR.IO Node

You can run the ar.io gateway as a standalone docker container:

docker build . -t ar-io-core:latest
docker run -p 4000:4000 -v ar-io-data:/app/data ar-io-core:latest

To run with a specified start height (sets height on first run only):

docker run -e START_HEIGHT=800000 -v $PWD/data/:/app/data ar-io-core:latest

Envoy & AR.IO Node

You can also run [Envoy] alongside an ar.io node via [Docker Compose]. Envoy proxies the Arweave node routes the gateway does not serve itself (such as POST /tx) to Arweave nodes: by default, public peers it discovers through ARWEAVE_PEER_DNS_RECORDS (peers.arweave.xyz); with ENABLE_ARWEAVE_PEER_EDS=false, the static TRUSTED_NODE_HOST (default arweave.net) and FALLBACK_NODE_HOST. With ARWEAVE_POST_DRY_RUN=true, Envoy sends POST /tx to the gateway instead, which answers it itself and broadcasts nothing.

docker compose up --build

Once running, requests can be directed to Envoy server at localhost:3000.

Run a Turbo Bundler as a Sidecar

You can run a [Turbo] [ANS-104] data item bundler as a sidecar to the ar.io gateway service. This will allow the deployed system to accept data items and bundle them into a single transaction before submitting them to the network. The bundler's APIs will be reachable at the /bundler/ path. For more information on its APIs, you can access docs at /bundler/api-docs/.

Note: A local bundler configured to integrate with an ar.io node relies upon GraphQL indexing of recently bundled and uploaded data to manage its pipeline operations. The ar.io node should have its indexes synced up to Arweave's current block height minus 18 blocks before starting up the bundler's services stack.

Bundling services are most easily managed via an independent docker compose file whose services share their network with that of the core services docker compose stack. This allows you to spin the services up when your core service is prepared to integrate with it, or down whenever you want without affecting your core services stack.

To get started, supply the required environment variables in an environment variables file (e.g. .env.bundler) for the integration, most notably:

  • BUNDLER_ARWEAVE_WALLET: a stringified JWK wallet used for uploading bundles to Arweave.
  • ALLOW_LISTED_ADDRESSES: a comma-separated list of allowed uploader wallet addresses (normalized). See Managing Bundler Access for more permissioning options.

See the .env.bundler.example file for other important configuration options, including settings for serving bundler-uploaded data items from your gateway as soon as they are uploaded.

Once environment variables are set, run docker compose with the bundler-specific compose file.

docker compose --env-file ./.env.bundler --file docker-compose.bundler.yaml up

Now, the bundler service will be running alongside the ar.io gateway. Your gateway will now accept data items at <your gateway url>/bundler/tx 🚀

Managing Bundler Access

By default, the bundler will only accept data items uploaded by data item signers whose normalized wallet addresses are in the ALLOW_LISTED_ADDRESSES list. But the following other permissioning configuration schemes are possible:

SchemeALLOW_LISTED_ADDRESSESSKIP_BALANCE_CHECKSALLOW_LISTED_SIGNATURE_TYPESPAYMENT_SERVICE_BASE_URL
Allow specific walletscomma-separated normalized wallet addressesfalseEMPTY or suppliedEMPTY
Allow specific chainsEMPTY or suppliedfalsearbundles sigtype intEMPTY
Allow alln/atruen/an/a
Allow noneEMPTYfalseEMPTYEMPTY
Allow payersEMPTY or suppliedfalseEMPTY or suppliedyour payment svc url

Configuration

When running via docker compose, it will read a .env file in the project root directory and use the environment variables set there.

GraphQL Pass-Through

Add the following to your .env file to proxy GraphQL to another server while using the ar.io gateway to serve data (using arweave.net GraphQL as an example):

GRAPHQL_HOST=arweave.net
GRAPHQL_PORT=443

Unbundling

The ar.io gateway supports unbundling and indexing [ANS-104] bundle data. To enable this add the following environment variables to your .env file:

ANS104_UNBUNDLE_FILTER="<filter string>"
ANS104_INDEX_FILTER="<filter string>"

ANS104_UNBUNDLE_FILTER determines which TXs and data items (in the case of nested bundles) are unbundled, and ANS104_INDEX_FILTER determines which data items within a bundle get indexed.

The following types of filters are supported:

{ "never": true } # the default
{ "always": true }
{ "attributes": { "owner_address": <owner address>, ... }}
{ "tags": [{ "name": <utf8 tag name>, "value": <utf8 tag value> }, { "name": <utf8 tag name> }, ...]}
{ "and": [ <nested filter>, ... ]}
{ "or": [ <nested filter>, ... ]}
{ "not": [ <nested filter>, ... ]}

Place an ANS-104 bundle at the start of the queue for unbundling and indexing on your gateway:

curl -X POST -H "Authorization: Bearer <ADMIN_KEY>" \
  -H "Content-Type: application/json" \
  "http://<HOST>:<PORT>/ar-io/admin/queue-tx" \
  -d '{ "id": "<ID>" }'

Note: ANS-104 indexing support is currently experimental. It has been tested successfully with small sets of bundles (using filters), but you may still encounter problems with it when indexing larger sets of transactions.

For detailed information about filter types, operators, and advanced examples, see Filter Documentation.

Root Transaction Index (CD

// HOW IT'S BUILT

// FILES

2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • .claude/skills/ar-io-gateway-operator/SKILL.md7.9 KBupstream ↗
    ---
    name: ar-io-gateway-operator
    description: Operate any AR.IO node deployment — architecture, daily ops, diagnostics, and recurring pitfalls that apply to every operator. Use whenever the user asks about gateway health, indexing lag, ClickHouse pipeline, GraphQL routing, ArNS resolution, data retrieval and caching, bundle unbundling, webhook fan-out, container restarts, image rebuilds, disk pressure, HTTPSig overhead, or anything operationally generic to running an AR.IO node — even when they don't name a specific service ("is it healthy?", "why is it slow?", "did the build pick up?", "how full is the disk?"). For deployment-specific knowledge (domains, repo paths, sidecar config, this-machine numbers), pair this with the deployment-overlay skill (e.g. `vilenarios-gateway`) if one exists.
    ---
    
    # AR.IO Gateway Operator (generic)
    
    You are operating an AR.IO node — an Arweave gateway that indexes, caches, serves, and signs blockchain data. Deployments share the same architecture but differ in domains, paths, and which sidecars are present. **Run `scripts/health-check` first** every session: it prints a one-screen snapshot you can compare against the prior state before deciding to act.
    
    If a deployment-specific overlay skill is loaded (e.g. `vilenarios-gateway`), it provides the local repo paths, domains, and which sidecars exist. Treat the overlay as authoritative for those facts.
    
    ## Run first: `scripts/health-check`
    
    ```bash
    # from the ar-io-node repo root
    .claude/skills/ar-io-gateway-operator/scripts/health-check
    ```
    
    What it checks and why:
    
    | Section | What "good" looks like | If wrong, jump to |
    |---|---|---|
    | containers | All core ar-io-node containers `Up` | "Bring everything up" |
    | disk | `/` < 80%, `/data` (if separate) has headroom | "Disk pressure" |
    | gateway healthcheck | `{"status":"ok",...}` | "When something breaks" |
    | indexer height vs chain head | lag ≤ 5 blocks | "Indexing falling behind" |
    | ClickHouse rows | If streaming enabled: `new_blocks > 0` and growing | "Streaming tables empty" |
    | core errors last 5 min | < ~10 (operational noise: peer fetches, aborts) | grep deeper |
    | image SHAs | If on a dev branch, must differ from `:latest` digest | "Building from source" |
    
    The script only covers the core ar-io-node containers defined in this repo's docker-compose. Sidecar checks belong in a deployment-overlay skill that calls this script then adds its own checks on top.
    
    ## How the gateway works (the model an operator needs)
    
    Knowing the data flows tells you where to look when symptoms appear. Source of truth is `src/system.ts` (DI wiring) and `docs/`; this section captures the operationally-load-bearing facts.
    
    ### Request flow
    
    ```text
    client → (optional edge: nginx/CDN) → envoy :3000 → core :4000 → response with X-AR-IO-* trust headers
                                                                  ↘ webhook fan-out → optional sidecars
    ```
    
    Every served data response carries trust headers (`X-AR-IO-Verified`, `X-AR-IO-Trusted`, `X-AR-IO-Stable`, `X-AR-IO-Hops`, plus `X-Arweave-*` tag passthrough). Those headers are what HTTPSig signs — see `src/middleware/httpsig.ts` `TRIGGER_HEADERS`. Endpoints with no trust trigger (`/ar-io/info`, `/ar-io/healthcheck`, `/graphql` errors) are deliberately not signed.
    
    ### Data retrieval — composite source with fallback chain
    
    Configurable via `.env`. Defaults:
    - `ON_DEMAND_RETRIEVAL_ORDER=trusted-gateways,ar-io-network,chunks-offset-aware,tx-data` — what `/raw/<id>` and `/<id>` walk through on a cache miss
    - `BACKGROUND_RETRIEVAL_ORDER=chunks` — used by the unbundler when fetching bundle bodies
    
    Each source's failures are logged with `class` so you can grep (`GatewaysDataSource`, `TxChunksDataSource`, `ChunksDataSource`, etc.). Persistent failures from one tier are normal — the chain is designed to fall through. Look for `Failed to fetch chunk data from any source` / `all sources failed` as the *real* failure signal, not individual tier warnings.
    
    Three response headers worth knowing: `X-Cache: HIT|MISS` (gateway-side contiguous cache), `X-AR-IO-Hops` (how many AR.IO peers were traversed before this gateway served), `X-AR-IO-Trusted` / `X-AR-IO-Verified` (whether the data has been hash-verified against the chain).
    
    ### Indexing pipeline
    
    Two write paths run side-by-side:
    
    **1. SQLite primary indexer (always on)** — `BlockImporter` worker imports blocks one at a time into `new_blocks`/`new_transactions` (the unstable head, ~last 18 blocks), and a separate process promotes them to `stable_blocks`/`stable_transactions` once they're past the reorg horizon. ANS-104 bundle unbundling is gated by `ANS104_UNBUNDLE_FILTER` (often narrow). Database access is in a worker thread (`StandaloneSqlite`) — never call SQLite synchronously from the main thread.
    
    **2. ClickHouse pipeline (optional, `--profile clickhouse`)** — Stable rows are exported to parquet then imported by `clickhouse-auto-import` (hourly cycles) into `staging_*` and finally `transactions`. **Streaming pipeline** (PR #699 / `clickhouse-streaming-unstable-head` branch) mirrors the unstable-head SQLite writes directly into ClickHouse `new_blocks` / `new_transactions` so GraphQL can query the head without waiting for the parquet round-trip. Activated by `CLICKHOUSE_STREAMING_ENABLED=true`. Reorgs trigger bounded `ALTER TABLE ... DELETE WHERE` on the `new_*` tables.
    
    ### GraphQL routing
    
    `/graphql` reads from up to three legs that get merged: stable `transactions` table in CH, streaming `new_transactions` in CH, and the SQLite `new_*` fallback. With `CLICKHOUSE_GQL_SKIP_SQLITE_READS=true`, the SQLite leg is dropped to a tight-timeout fallback only. With `GRAPHQL_ON_DEMAND_RESOLUTION_ENABLED=true` (default), `transaction(id)` queries that miss the local DB will fall back to extracting from ANS-104 bundle binaries on demand — single-ID only, capped by `GRAPHQL_ON_DEMAND_RESOLUTION_MAX_CONCURRENT`.
    
    ### ANS-104 unbundling pipeline
    
    Whenever a bundle TX matches `ANS104_UNBUNDLE_FILTER`, it walks a four-queue chain. Knowing the chain is essential for diagnosing throughput regressions — a single queue at cap can mask everything downstream of it.
    
    ```text
    bundleDataImporter ─→ ans104Unbundler ─→ ans104DataIndexer ─→ dataItemIndexer ─→ SQLite writer
       download workers      parser workers     post-parse buffer     pre-write buffer    single-writer
       (ANS104_DOWNLOAD_…)   (ANS104_UNBUNDLE_)  (ANS104_DATA_…_SIZE)  (DATA_ITEM_…_SIZE)
    ```
    
    Tune knobs (all env-overridable):
    
    | Knob | Default | What it controls |
    |---|---:|---|
    | `ANS104_DOWNLOAD_WORKERS` | 5 | Parallel bundle downloads. Network-bound; overprovisioning is cheap |
    | `ANS104_UNBUNDLE_WORKERS` | 1 | Parallel parser worker_threads. CPU + network bound |
    | `DATA_ITEM_INDEXER_WORKER_COUNT` | 1 | Fastq concurrency for `indexDataItem`; SQLite is single-writer so gains are bounded |
    | `MAX_DATA_ITEM_QUEUE_SIZE` | 100000 | **Soft gate**: `shouldUnbundle()` returns false above this on either indexer queue |
    | `DATA_ITEM_INDEXER_QUEUE_SIZE` | 500000 | Hard cap on `dataItemIndexer`; items past this get dropped silently |
    | `ANS104_DATA_INDEXER_QUEUE_SIZE` | 500000 | Hard cap on `ans104DataIndexer` |
    
    **The buffer between soft gate and hard cap is where data loss happens.** With defaults the gate engages at 100 K but the hard cap is 500 K — a 5× buffer. Under heavy load, in-flight parser workers can flood items into the queue *after* the gate engages but *before* the indexer drains, exceeding the hard cap and silently dropping items. Symptom: `data_items_dropped_total{queue_name="dataItemIndexer"}` climbs while `bundles_unbundled_total` keeps increasing — bundles look "indexed" but their items aren't searchable.
    
    Mitigations:
    - Tighten `MAX_DATA_ITEM_QUEUE_SIZE` so the gate engages well before the hard cap (e.g., 25 000 with 500 000 hard caps, giving a 20× buffer).
    - Raise the hard cap if memory permits (each item slot ≈ 1–2 KB).
    - Don't add parser workers when `data_items_dropped_total` is non-zero — m
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/ar-io-ar-io-node-claude-skills-ar-io-gateway-operator.md

// REPO STATS

130 stars
7 open issues
Last commit: 5d ago