# ar-io-gateway-operator

Adopt this skill: https://proskills.md/skills/other/ar-io-ar-io-node-claude-skills-ar-io-gateway-operator

Page: https://proskills.md/skills/other/ar-io-ar-io-node-claude-skills-ar-io-gateway-operator
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: .claude/skills/ar-io-gateway-operator/SKILL.md
size: 8074
upstream: https://github.com/ar-io/ar-io-node/blob/develop/.claude/skills/ar-io-gateway-operator/SKILL.md

````text
---
name: ar-io-gateway-operator
description: Operate any AR.IO node deployment — architecture, daily ops, diagnostics, and recurring pitfalls that apply to every operator. Use whenever the user asks about gateway health, indexing lag, ClickHouse pipeline, GraphQL routing, ArNS resolution, data retrieval and caching, bundle unbundling, webhook fan-out, container restarts, image rebuilds, disk pressure, HTTPSig overhead, or anything operationally generic to running an AR.IO node — even when they don't name a specific service ("is it healthy?", "why is it slow?", "did the build pick up?", "how full is the disk?"). For deployment-specific knowledge (domains, repo paths, sidecar config, this-machine numbers), pair this with the deployment-overlay skill (e.g. `vilenarios-gateway`) if one exists.
---

# AR.IO Gateway Operator (generic)

You are operating an AR.IO node — an Arweave gateway that indexes, caches, serves, and signs blockchain data. Deployments share the same architecture but differ in domains, paths, and which sidecars are present. **Run `scripts/health-check` first** every session: it prints a one-screen snapshot you can compare against the prior state before deciding to act.

If a deployment-specific overlay skill is loaded (e.g. `vilenarios-gateway`), it provides the local repo paths, domains, and which sidecars exist. Treat the overlay as authoritative for those facts.

## Run first: `scripts/health-check`

```bash
# from the ar-io-node repo root
.claude/skills/ar-io-gateway-operator/scripts/health-check
```

What it checks and why:

| Section | What "good" looks like | If wrong, jump to |
|---|---|---|
| containers | All core ar-io-node containers `Up` | "Bring everything up" |
| disk | `/` < 80%, `/data` (if separate) has headroom | "Disk pressure" |
| gateway healthcheck | `{"status":"ok",...}` | "When something breaks" |
| indexer height vs chain head | lag ≤ 5 blocks | "Indexing falling behind" |
| ClickHouse rows | If streaming enabled: `new_blocks > 0` and growing | "Streaming tables empty" |
| core errors last 5 min | < ~10 (operational noise: peer fetches, aborts) | grep deeper |
| image SHAs | If on a dev branch, must differ from `:latest` digest | "Building from source" |

The script only covers the core ar-io-node containers defined in this repo's docker-compose. Sidecar checks belong in a deployment-overlay skill that calls this script then adds its own checks on top.

## How the gateway works (the model an operator needs)

Knowing the data flows tells you where to look when symptoms appear. Source of truth is `src/system.ts` (DI wiring) and `docs/`; this section captures the operationally-load-bearing facts.

### Request flow

```text
client → (optional edge: nginx/CDN) → envoy :3000 → core :4000 → response with X-AR-IO-* trust headers
                                                              ↘ webhook fan-out → optional sidecars
```

Every served data response carries trust headers (`X-AR-IO-Verified`, `X-AR-IO-Trusted`, `X-AR-IO-Stable`, `X-AR-IO-Hops`, plus `X-Arweave-*` tag passthrough). Those headers are what HTTPSig signs — see `src/middleware/httpsig.ts` `TRIGGER_HEADERS`. Endpoints with no trust trigger (`/ar-io/info`, `/ar-io/healthcheck`, `/graphql` errors) are deliberately not signed.

### Data retrieval — composite source with fallback chain

Configurable via `.env`. Defaults:
- `ON_DEMAND_RETRIEVAL_ORDER=trusted-gateways,ar-io-network,chunks-offset-aware,tx-data` — what `/raw/<id>` and `/<id>` walk through on a cache miss
- `BACKGROUND_RETRIEVAL_ORDER=chunks` — used by the unbundler when fetching bundle bodies

Each source's failures are logged with `class` so you can grep (`GatewaysDataSource`, `TxChunksDataSource`, `ChunksDataSource`, etc.). Persistent failures from one tier are normal — the chain is designed to fall through. Look for `Failed to fetch chunk data from any source` / `all sources failed` as the *real* failure signal, not individual tier warnings.

Three response headers worth knowing: `X-Cache: HIT|MISS` (gateway-side contiguous cache), `X-AR-IO-Hops` (how many AR.IO peers were traversed before this gateway served), `X-AR-IO-Trusted` / `X-AR-IO-Verified` (whether the data has been hash-verified against the chain).

### Indexing pipeline

Two write paths run side-by-side:

**1. SQLite primary indexer (always on)** — `BlockImporter` worker imports blocks one at a time into `new_blocks`/`new_transactions` (the unstable head, ~last 18 blocks), and a separate process promotes them to `stable_blocks`/`stable_transactions` once they're past the reorg horizon. ANS-104 bundle unbundling is gated by `ANS104_UNBUNDLE_FILTER` (often narrow). Database access is in a worker thread (`StandaloneSqlite`) — never call SQLite synchronously from the main thread.

**2. ClickHouse pipeline (optional, `--profile clickhouse`)** — Stable rows are exported to parquet then imported by `clickhouse-auto-import` (hourly cycles) into `staging_*` and finally `transactions`. **Streaming pipeline** (PR #699 / `clickhouse-streaming-unstable-head` branch) mirrors the unstable-head SQLite writes directly into ClickHouse `new_blocks` / `new_transactions` so GraphQL can query the head without waiting for the parquet round-trip. Activated by `CLICKHOUSE_STREAMING_ENABLED=true`. Reorgs trigger bounded `ALTER TABLE ... DELETE WHERE` on the `new_*` tables.

### GraphQL routing

`/graphql` reads from up to three legs that get merged: stable `transactions` table in CH, streaming `new_transactions` in CH, and the SQLite `new_*` fallback. With `CLICKHOUSE_GQL_SKIP_SQLITE_READS=true`, the SQLite leg is dropped to a tight-timeout fallback only. With `GRAPHQL_ON_DEMAND_RESOLUTION_ENABLED=true` (default), `transaction(id)` queries that miss the local DB will fall back to extracting from ANS-104 bundle binaries on demand — single-ID only, capped by `GRAPHQL_ON_DEMAND_RESOLUTION_MAX_CONCURRENT`.

### ANS-104 unbundling pipeline

Whenever a bundle TX matches `ANS104_UNBUNDLE_FILTER`, it walks a four-queue chain. Knowing the chain is essential for diagnosing throughput regressions — a single queue at cap can mask everything downstream of it.

```text
bundleDataImporter ─→ ans104Unbundler ─→ ans104DataIndexer ─→ dataItemIndexer ─→ SQLite writer
   download workers      parser workers     post-parse buffer     pre-write buffer    single-writer
   (ANS104_DOWNLOAD_…)   (ANS104_UNBUNDLE_)  (ANS104_DATA_…_SIZE)  (DATA_ITEM_…_SIZE)
```

Tune knobs (all env-overridable):

| Knob | Default | What it controls |
|---|---:|---|
| `ANS104_DOWNLOAD_WORKERS` | 5 | Parallel bundle downloads. Network-bound; overprovisioning is cheap |
| `ANS104_UNBUNDLE_WORKERS` | 1 | Parallel parser worker_threads. CPU + network bound |
| `DATA_ITEM_INDEXER_WORKER_COUNT` | 1 | Fastq concurrency for `indexDataItem`; SQLite is single-writer so gains are bounded |
| `MAX_DATA_ITEM_QUEUE_SIZE` | 100000 | **Soft gate**: `shouldUnbundle()` returns false above this on either indexer queue |
| `DATA_ITEM_INDEXER_QUEUE_SIZE` | 500000 | Hard cap on `dataItemIndexer`; items past this get dropped silently |
| `ANS104_DATA_INDEXER_QUEUE_SIZE` | 500000 | Hard cap on `ans104DataIndexer` |

**The buffer between soft gate and hard cap is where data loss happens.** With defaults the gate engages at 100 K but the hard cap is 500 K — a 5× buffer. Under heavy load, in-flight parser workers can flood items into the queue *after* the gate engages but *before* the indexer drains, exceeding the hard cap and silently dropping items. Symptom: `data_items_dropped_total{queue_name="dataItemIndexer"}` climbs while `bundles_unbundled_total` keeps increasing — bundles look "indexed" but their items aren't searchable.

Mitigations:
- Tighten `MAX_DATA_ITEM_QUEUE_SIZE` so the gate engages well before the hard cap (e.g., 25 000 with 500 000 hard caps, giving a 20× buffer).
- Raise the hard cap if memory permits (each item slot ≈ 1–2 KB).
- Don't add parser workers when `data_items_dropped_total` is non-zero — m
````

## File: README.md
upstream: https://github.com/ar-io/ar-io-node/blob/develop/README.md

````text
# AR.IO Gateway Node

[![codecov](https://codecov.io/gh/ar-io/ar-io-node/graph/badge.svg?token=F3BJ7W74HY)](https://codecov.io/gh/ar-io/ar-io-node)
[![protocol.land](https://arweave.net/eZp8gOeR8Yl_cyH9jJToaCrt2He1PHr0pR4o-mHbEcY)](https://protocol.land/#/repository/713c1b6f-86c8-493e-b2e6-6cb231862b93)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ar-io/ar-io-node)

## Getting Started

- [Linux Installation Instructions](./docs/linux-setup.md)
- [Windows Installation Instructions](./docs/windows-setup.md)

## Dev Workflow

### Install dependencies

`yarn install`

### Initialize the SQLite DB

`yarn db:migrate up`

### Run lint

`yarn lint:check`

### Run tests

`yarn test`

### Run the service

With defaults:

`yarn start`

Starting at an arbitrary block (only works immediately after initial DB
migration):

`START_HEIGHT=800000 yarn start`

### Operating with Claude Code

This repo ships skills under `.claude/skills/` that Claude Code
([claude.ai/code](https://claude.ai/code)) auto-loads when invoked from the
repo root:

- **`ar-io-gateway-operator`** — operator runbook covering the ANS-104
  pipeline, ClickHouse, ArNS resolution, the observer/cranker, and common
  pitfalls. Includes `scripts/health-check` for a one-screen gateway
  health snapshot.
- **`release`** — version bumps, image SHA pinning, tag and GitHub-release
  creation.
- **`testing`** — picks the right test layer (unit / property / e2e /
  auto-verify / parquet integration / load) for a given change.

The skills are plain Markdown — readable as runbooks even if you don't use
Claude Code.

## Dev Docs

[docs/INDEX.md](docs/INDEX.md) is the index of the documents in `docs/`.

### Schema (WIP)

- [Bundle schema]

## Docker

### Standalone AR.IO Node

You can run the ar.io gateway as a standalone docker container:

```shell
docker build . -t ar-io-core:latest
docker run -p 4000:4000 -v ar-io-data:/app/data ar-io-core:latest
```

To run with a specified start height (sets height on first run only):

```shell
docker run -e START_HEIGHT=800000 -v $PWD/data/:/app/data ar-io-core:latest
```

### Envoy & AR.IO Node

You can also run [Envoy] alongside an `ar.io` node via [Docker Compose]. Envoy
proxies the Arweave node routes the gateway does not serve itself (such as
`POST /tx`) to Arweave nodes: by default, public peers it discovers through
`ARWEAVE_PEER_DNS_RECORDS` (`peers.arweave.xyz`); with
`ENABLE_ARWEAVE_PEER_EDS=false`, the static `TRUSTED_NODE_HOST` (default
`arweave.net`) and `FALLBACK_NODE_HOST`. With `ARWEAVE_POST_DRY_RUN=true`,
Envoy sends `POST /tx` to the gateway instead, which answers it itself and
broadcasts nothing.

```shell
docker compose up --build
```

Once running, requests can be directed to Envoy server at `localhost:3000`.

### Run a Turbo Bundler as a Sidecar

You can run a [Turbo] [ANS-104] data item bundler as a sidecar to the ar.io gateway service. This will allow the deployed system to accept data items and bundle them into a single transaction before submitting them to the network. The bundler's APIs will be reachable at the `/bundler/` path. For more information on its APIs, you can access docs at `/bundler/api-docs/`.

Note: A local bundler configured to integrate with an ar.io node relies upon GraphQL indexing of recently bundled and uploaded data to manage its pipeline operations. The ar.io node should have its indexes synced up to Arweave's current block height minus 18 blocks before starting up the bundler's services stack.

Bundling services are most easily managed via an independent docker compose file whose services share their network with that of the core services docker compose stack. This allows you to spin the services up when your core service is prepared to integrate with it, or down whenever you want without affecting your core services stack.

To get started, supply the required environment variables in an environment variables file (e.g. `.env.bundler`) for the integration, most notably:

- `BUNDLER_ARWEAVE_WALLET`: a stringified JWK wallet used for uploading bundles to Arweave.
- `ALLOW_LISTED_ADDRESSES`: a comma-separated list of allowed uploader wallet addresses (normalized). See [Managing Bundler Access](#managing-bundler-access) for more permissioning options.

See the `.env.bundler.example` file for other important configuration options, including settings for serving bundler-uploaded data items from your gateway as soon as they are uploaded.

Once environment variables are set, run docker compose with the bundler-specific compose file.

```shell
docker compose --env-file ./.env.bundler --file docker-compose.bundler.yaml up
```

Now, the bundler service will be running alongside the ar.io gateway. Your gateway will now accept data items at `<your gateway url>/bundler/tx` 🚀

#### Managing Bundler Access

By default, the bundler will only accept data items uploaded by data item signers whose normalized wallet addresses are in the `ALLOW_LISTED_ADDRESSES` list. But the following other permissioning configuration schemes are possible:

| Scheme                 | ALLOW_LISTED_ADDRESSES                      | SKIP_BALANCE_CHECKS | ALLOW_LISTED_SIGNATURE_TYPES | PAYMENT_SERVICE_BASE_URL |
| ---------------------- | ------------------------------------------- | ------------------- | ---------------------------- | ------------------------ |
| Allow specific wallets | comma-separated normalized wallet addresses | false               | EMPTY or supplied            | EMPTY                    |
| Allow specific chains  | EMPTY or supplied                           | false               | arbundles sigtype int        | EMPTY                    |
| Allow all              | n/a                                         | true                | n/a                          | n/a                      |
| Allow none             | EMPTY                                       | false               | EMPTY                        | EMPTY                    |
| Allow payers           | EMPTY or supplied                           | false               | EMPTY or supplied            | your payment svc url     |

## Configuration

When running via `docker compose`, it will read a `.env` file in the project root
directory and use the environment variables set there.

### GraphQL Pass-Through

Add the following to your `.env` file to proxy GraphQL to another server while
using the ar.io gateway to serve data (using arweave.net GraphQL as an example):

```
GRAPHQL_HOST=arweave.net
GRAPHQL_PORT=443
```

### Unbundling
The ar.io gateway supports unbundling and indexing [ANS-104] bundle data. To
enable this add the following environment variables to your `.env` file:

```
ANS104_UNBUNDLE_FILTER="<filter string>"
ANS104_INDEX_FILTER="<filter string>"
```

`ANS104_UNBUNDLE_FILTER` determines which TXs and data items (in the case of
nested bundles) are unbundled, and `ANS104_INDEX_FILTER` determines which data
items within a bundle get indexed.

The following types of filters are supported:

```
{ "never": true } # the default
{ "always": true }
{ "attributes": { "owner_address": <owner address>, ... }}
{ "tags": [{ "name": <utf8 tag name>, "value": <utf8 tag value> }, { "name": <utf8 tag name> }, ...]}
{ "and": [ <nested filter>, ... ]}
{ "or": [ <nested filter>, ... ]}
{ "not": [ <nested filter>, ... ]}

```

Place an ANS-104 bundle at the start of the queue for unbundling and indexing
on your gateway:

```
curl -X POST -H "Authorization: Bearer <ADMIN_KEY>" \
  -H "Content-Type: application/json" \
  "http://<HOST>:<PORT>/ar-io/admin/queue-tx" \
  -d '{ "id": "<ID>" }'

```

Note: ANS-104 indexing support is currently experimental. It has been tested
successfully with small sets of bundles (using filters), but you may still
encounter problems with it when indexing larger sets of transactions.

For detailed information about filter types, operators, and advanced examples, see [Filter Documentation](docs/filters.md).

### Root Transaction Index (CD
````
