# arcgis-to-portaljs

Adopt this skill: https://proskills.md/skills/other/datopian-portaljs-skills-arcgis-to-portaljs

Page: https://proskills.md/skills/other/datopian-portaljs-skills-arcgis-to-portaljs
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: skills/arcgis-to-portaljs/SKILL.md
size: 8007
upstream: https://github.com/datopian/portaljs/blob/main/skills/arcgis-to-portaljs/SKILL.md

````text
---
name: arcgis-to-portaljs
description: Migrate a whole ArcGIS Hub site into a PortalJS Arc portal end-to-end. Harvests the Hub /data.json (DCAT-US) inventory, exports every FeatureService layer through the ArcGIS REST query API with resultOffset paging, converts each to the serverless dual tier (PMTiles render + GeoParquet query) with tabular items to Parquet, pushes everything to Cloudflare R2 via Git LFS, appends dual-tier datasets.json entries, and writes a source-vs-derived parity report. Use to move a City or sector ArcGIS Hub open-data portal onto PortalJS with no server-side compute.
allowed-tools: Read, Write, Edit, Bash(ogr2ogr:*), Bash(ogrinfo:*), Bash(tippecanoe:*), Bash(duckdb:*), Bash(git:*), Bash(curl:*), Bash(jq:*), Bash(npx:*), Bash(node:*), Bash(mkdir:*), Bash(cp:*), Bash(wc:*), Bash(command:*), WebFetch
version: 0.1.0
author: Datopian <hello@datopian.com>
license: MIT
compatibility: Claude Code with PortalJS portals (Next.js 14, React 18/19, Node 18+). Requires native GDAL, tippecanoe, duckdb, and jq on the operator's machine (macOS or Linux/WSL). Runs from any project via the plugin, a personal install, or a portaljs clone.
tags:
  - portaljs
  - data-portal
  - migration
  - arcgis
  - geospatial
  - pmtiles
---

# ArcGIS Hub → PortalJS

## Overview

Migrate an entire **ArcGIS Hub** open-data site into a **PortalJS Arc** portal in one pass.
Every Hub site is machine-readable — a DCAT-US catalog at `/data.json`, with every dataset
backed by an ArcGIS REST FeatureService — so migration is a **harvest → export → convert →
publish → verify** pipeline that runs almost fully automated on the operator's machine, no
server-side compute. The tooling is the reusable `arcgis-to-portaljs` migrator: input is one
Hub URL, output is a ready-to-deploy PortalJS catalog plus a parity report.

The skill is an orchestrator: it reuses the DCAT-US harvest from `portaljs-migrate`, the
`ogr2ogr`/`tippecanoe`/`duckdb` dual-tier conversion from `portaljs-add-geo`, and the bulk
Git-LFS → R2 push from `portaljs-migrate`. Its novel parts are the FeatureService REST export
loop (paged features, not just a link) and the source-vs-derived parity report.

## Prerequisites

- A scaffolded PortalJS portal whose template ships `components/MapPreview.tsx` and
  `components/GeoQuery.tsx` (PR #1647 or later). Run `portaljs-new-portal` first if none.
- Native CLIs: **GDAL** (`ogr2ogr`, `ogrinfo`), **tippecanoe**, **duckdb** (with `spatial`),
  and **jq**. macOS: `brew install gdal tippecanoe duckdb jq`; Debian/Ubuntu:
  `apt-get install gdal-bin duckdb jq` plus tippecanoe (apt or build from source); Windows via
  WSL. The skill hard-stops with the install hint if any is missing.
- Arc credentials for the Git-LFS → R2 push (the token `portaljs-deploy` resolves), or an OSS
  self-hosted Giftless.

## Instructions

The canonical, full step-by-step workflow is
[`.claude/commands/arcgis-to-portaljs.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/arcgis-to-portaljs.md)
— the single source of truth. Read and follow it when executing. Summary:

1. Gather input — Hub URL, portal directory, project slug, optional flags (`--limit`,
   `--only`, `--dry-run`, `--namespace-mode`). Interview if missing; never dead-end.
2. Check native tools (`ogr2ogr`, `tippecanoe`, `duckdb` + `spatial`, `jq`). Any missing →
   print the per-OS install and stop.
3. Validate the portal directory and confirm the geo showcase components exist.
4. Harvest the Hub `/data.json` (reuse the `portaljs-migrate` DCAT-US map) and classify each
   item: vector (FeatureService), table, or non-data (web map / 3D / imagery → skipped).
   Under `--namespace-mode owner`, resolve namespaces through a publisher-normalization
   table with title-prefix fallback for broken `{{source}}` publishers (multi-publisher
   Hubs ship dirty publisher labels). Dedup near-duplicate hosted-`view` layers — but only
   after a mandatory live record-count check on BOTH twins: equal ⇒ dedup (keep the source
   layer, log the pair); different ⇒ keep both as distinct datasets. Consolidate per-year
   dataset series into one year-partitioned Parquet with legacy per-year view entries. Enrich
   from the AGOL item: sanitized metadata (license/description/dates), cleaned display title
   (`cleanTitle` — raw title still drives the slug), `category` (item categories → meaningful
   theme → keyword mapping), and a `thumbnail` snapshot into `public/thumbnails/`.
5. Export each vector layer through the ArcGIS REST `query` API with `resultOffset` paging
   (`f=geojson`, `outSR=4326`); fall back to keyset paging on transfer limits; accept a
   customer File Geodatabase dump for very large layers.
6. Convert each layer to the dual tier via the `portaljs-add-geo` recipe (PMTiles +
   GeoParquet); tabular items to Parquet. Preserve the native-CRS original.
7. Publish — bulk Git-LFS track + one push to R2 through Giftless, then append dual-tier
   `datasets.json` entries (upsert on `(namespace, slug)`).
8. Write `arcgis-parity-report.md` — record count, extent, attribute schema, and geometry
   validity, source vs derived, per dataset, plus the migrated/skipped/failed accounting.
9. Report the inventory, migrated datasets, R2 push, and parity summary.

## Output

- **Created:** `data/<namespace>/<slug>.pmtiles`, `.parquet`, and the original per vector
  dataset (all LFS-tracked → R2); Parquet + original per table; `arcgis-parity-report.md`.
- **Modified:** `datasets.json` (one dual-tier entry per vector dataset, one resource entry
  per table); `.gitattributes` (LFS tracking).
- **Verified:** the parity report compares each derived artifact to the live FeatureService.
- **Result:** `/@<namespace>/<slug>` renders `<MapPreview>` + `<GeoQuery>` for each vector
  dataset with no page edits; the catalog lists everything migrated.

## Error Handling

| Symptom | Cause | Fix |
| --- | --- | --- |
| `MISSING_INPUT` | No Hub URL provided | Pass the site root (e.g. `https://hub-lewisville.opendata.arcgis.com`) and retry. |
| `MISSING_TOOLS` | `ogr2ogr`/`tippecanoe`/`duckdb`/`jq` (or duckdb `spatial`) absent | Print the per-OS install line and stop; re-run after installing. |
| `NOT_A_PORTAL` | Target dir has no `datasets.json` / geo components | Run `portaljs-new-portal` first, then re-run. |
| `HARVEST_FAILED` | `/data.json` unreachable or not DCAT-US | Confirm the site is an ArcGIS Hub and the feed loads in a browser. |
| `EXPORT_FAILED` | One FeatureService layer errored or hit a hard transfer cap | Logged and skipped; try keyset paging or a customer FGDB dump for that layer. |
| `LFS_PUSH_FAILED` | Missing/expired Arc token or unset `lfs.url` | Re-mint the JWT (see `portaljs-deploy`); confirm `git config lfs.url`. |

## Examples

### Example 1 — Migrate a City Hub (Lewisville)

```
/arcgis-to-portaljs https://hub-lewisville.opendata.arcgis.com slug=lewisville
```

### Example 2 — Dry-run inventory + plan only (no writes)

```
/arcgis-to-portaljs https://streamwaterdata.co.uk --dry-run
```

### Example 3 — Migrate a subset, one namespace per publisher

```
/arcgis-to-portaljs https://streamwaterdata.co.uk --only sewer-catchments,water-boundaries --namespace-mode owner
```

## Resources

- Full workflow: [`.claude/commands/arcgis-to-portaljs.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/arcgis-to-portaljs.md)
- REST export, classification, parity, and phasing details: [`references/reference.md`](references/reference.md)
- Ongoing sync, parity dashboard, and cutover (Phase 3): [`references/sync-and-cutover.md`](references/sync-and-cutover.md)
- Related skills: `portaljs-migrate`, `portaljs-add-geo`, `portaljs-add-dataset`, `portaljs-deploy`
- ArcGIS REST query API: <https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer/> · tippecanoe: <https://github.com/felt/tippecanoe> · DuckDB spatial: <https://duckdb.org/docs/extensions/spatial>
````

## File: README.md
upstream: https://github.com/datopian/portaljs/blob/main/README.md

````text
<p align="center">
  <img src="assets/portaljs-logo-spin.svg" alt="PortalJS" width="96" height="96" />
  <h1 align="center">PortalJS</h1>
  <p align="center">
    <b>The AI-native framework for building data portals.</b>
    <br />
    Describe the portal you want — your agent helps you choose an architecture, scaffolds it, and loads your data.
    <br />
    <br />
    <a href="https://www.portaljs.com/docs">Docs</a>
    ·
    <a href="https://github.com/datopian/portaljs/discussions">Discussions</a>
    ·
    <a href="https://github.com/datopian/portaljs/issues/new">Report a bug</a>
    <br />
    <br />
    <a href="https://www.npmjs.com/package/create-portaljs"><img src="https://img.shields.io/npm/v/create-portaljs?logo=npm&logoColor=white&label=create-portaljs&color=cb3837" alt="npm version" /></a>
    <a href="https://github.com/datopian/portaljs/stargazers"><img src="https://img.shields.io/github/stars/datopian/portaljs?logo=github&logoColor=white&label=Stars&color=blue" alt="GitHub stars" /></a>
    <a href="https://discord.gg/krmj5HM6He"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white" alt="Join our Discord" /></a>
    <a href="license"><img src="https://img.shields.io/badge/License-MIT-blue" alt="MIT License" /></a>
  </p>
</p>

---

## Quickstart

**Create a portal** — one command, nothing to install beyond Node 22+:

```bash
npm create portaljs@latest my-portal
cd my-portal
npm run dev      # → http://localhost:3000
```

You get the three surfaces — Home, a Catalog (`/search`), and a dataset Showcase
(`/@<namespace>/<slug>`) — over sample data. Plain, editable Next.js, no lock-in. Add your
own CSV/JSON to `datasets.json` and it renders automatically.

**Build it with your AI assistant** — PortalJS ships [Claude Code](https://claude.com/claude-code)
skills that do the assembly. Install them once (into `~/.claude/commands`):

```bash
curl -fsSL https://raw.githubusercontent.com/datopian/portaljs/main/scripts/install-portaljs-skills.sh | bash
```

Then, in a Claude Code session from any directory:

```text
/portaljs-architect    not sure what stack you need? start here
/portaljs-new-portal   "Auckland Council open data portal"
/portaljs-add-dataset  ./data/air-quality.csv
```

`/portaljs-new-portal` scaffolds the three surfaces; `/portaljs-add-dataset` (or `/portaljs-add-resource`) loads data;
`/portaljs-connect-ckan` points it at a CKAN backend; `/portaljs-deploy` ships it. ([All skills + install →](.claude/INSTALL.md))

**Prefer the bare template** — plain Next.js, no AI, no lock-in:

```bash
npx tiged datopian/portaljs/examples/portaljs-catalog my-portal
cd my-portal && npm install && npm run dev      # → http://localhost:3000
```

You get Home, a Catalog (`/search`), and a dataset Showcase (`/@<namespace>/<slug>`) over
sample data. Add your own CSV/JSON to `datasets.json` and it renders automatically.

⭐ If it's useful, a star helps others find it.

## Why PortalJS

Building a data portal has always meant more than a website. You have to decide where
the data lives, how it's versioned, how people search it, how it's served, and how it's
governed — and then wire a frontend on top. Teams either over-build on a heavy data
warehouse they don't need, or under-build on a pile of scripts that doesn't scale.

PortalJS is an **open-source, agentic skills framework that helps data teams build,
develop, and ship data portals — and the data infrastructure underneath them.** It isn't
only a frontend. The skills do two jobs:

- **Advise** — given _what you're building_, _what your data is_, and _what it's for_,
  they recommend an architecture: storage, compute, catalog, access, hosting, metadata.
- **Build** — they scaffold that stack as plain, editable Next.js code with no lock-in.

It is **opinionated but open**: the recommended modern path is git + object storage
(Cloudflare R2) + Parquet, queried with [DuckDB](https://duckdb.org/) — an open lakehouse
instead of a classic warehouse. For living, incremental tables you can layer on
[DuckLake](https://ducklake.select/), and a traditional datastore (CKAN, a warehouse)
stays a first-class option when you need it. You always own plain code.

Built and maintained in the open by [Datopian](https://www.datopian.com/) and the
PortalJS community.

## Architecture at a glance

```text
        🧑  you describe what you want to build
        │
        ▼
╭─ 🤖  AGENTIC SKILLS ──────────────────────────────────  decide + build
│   /portaljs-architect · /portaljs-new-portal · /portaljs-add-dataset · /portaljs-add-chart · /portaljs-add-map …
╰─  generates plain, editable Next.js code — no lock-in
        │
        ▼
╭─ 🖥️  SURFACES ────────────────────────────────────────  what users see
│   🏠 Home /      🔎 Catalog /search      📊 Showcase /@ns/slug
╰─  read data through one DataProvider contract
        │
        ▼
╭─ 🔌  PROVIDERS ───────────────────────────────────────  pluggable backends
│   📁 static·git     🐘 CKAN     🔭 OpenMetadata     🗂️ git-LFS + R2
╰─  swap the source without touching a page
        │
        ▼
📦  STORAGE + COMPUTE  —  choose your point on the spectrum:

      flat files  ─▶  Git-LFS + R2  ─▶  Parquet on R2 + 🦆 DuckDB  ─▶  warehouse / CKAN
      simplest                       ⭐ open lakehouse (default)        heaviest
                                     (+ DuckLake for living tables)

☁️  Substrate  —  Cloudflare R2 (storage) · Workers (runtime) · D1 (catalog) · Pages (static)
     object storage stays S3-compatible — R2 is the default, never a lock-in
```

**Three surfaces.** Every data portal is built from three: a **Home** page that explains
it and offers search, a **Catalog** (`/search`) to discover datasets, and a **Showcase**
(`/@<namespace>/<slug>`) to explore one dataset — metadata, preview, download/API, and
charts/maps. ([Core concepts →](https://www.portaljs.com/docs/core-concepts))

**One seam.** The surfaces read data only through a `DataProvider`, so the source — static
files today, a CKAN or lakehouse backend tomorrow — can change without touching a page.

See [`ROADMAP.md`](ROADMAP.md) for the full model and the
[architecture decision framework](https://www.portaljs.com/docs/architecture/decision-framework)
for how `/portaljs-architect` turns your needs into a stack.

## Build a portal with your AI assistant

PortalJS ships [Claude Code](https://claude.com/claude-code) skills that turn a brief into
a working portal.

### Setup

Install the skills once into your personal scope so they're available from **any**
directory:

```bash
curl -fsSL https://raw.githubusercontent.com/datopian/portaljs/main/scripts/install-portaljs-skills.sh | bash
```

Restart Claude Code (or open a new session) and type `/` to see them. See
[`.claude/INSTALL.md`](.claude/INSTALL.md) for other install options (versioned plugin, or
running straight from a clone of this repo).

### Use

If you're not sure how to set up your portal, start with the advisor, then build:

```text
/portaljs-architect    we have ~200 public CSVs, updated quarterly, and must publish DCAT-AP
/portaljs-new-portal   "Auckland Council open data portal"
/portaljs-add-dataset  ./data/air-quality.csv
/portaljs-add-dataset  https://example.com/parks.geojson
```

The skills are **interactive** — if your brief is thin, they interview you in short rounds
rather than erroring. `/portaljs-architect` recommends a stack and hands off; `/portaljs-new-portal`
scaffolds the three surfaces; `/portaljs-add-dataset` appends to the `datasets.json` manifest and
the showcase renders automatically at `/@<namespace>/<slug>`. Run `npm run dev` and you
have a portal.

**Prefer to build by hand?** The skills are a convenience, not a requirement — scaffold the
template directly with the CLI:

```bash
npm create portaljs@latest my-portal
```

(Or grab the bare template with no prompts: `npx tiged datopian/portaljs/examples/portaljs-catalog my-portal`.)

### Available skills

<!-- BEGIN:skills-table -->

| Skill | What 
````
