v0.1.0

arcgis-to-portaljs

datopian@datopian⭐ 2.4k stars· last commit 19d ago· 15 open issues

Migrate a whole ArcGIS Hub site into a PortalJS Arc portal end-to-end. Harvests the Hub /data.json (DCAT-US) inventory, exports every FeatureService layer through the ArcGIS REST query API with resultOffset paging, converts each to the serverless dual tier (PMTiles render + GeoParquet query) with tabular items to Parquet, pushes everything to Cloudflare R2 via Git LFS, appends dual-tier datasets.json entries, and writes a source-vs-derived parity report. Use to move a City or sector ArcGIS Hub open-data portal onto PortalJS with no server-side compute.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

7.2/10
Verified
Oct 12, 2026

// RATINGS

⭐GitHub Stars
⭐⭐⭐⭐⭐ 2.4k on GitHubGitHub ↗

Very popular

🟢ProSkills ScoreAI Verified
7.2/10
📍

Not yet listed on ClawHub or SkillsMP

// README


Quickstart

Create a portal — one command, nothing to install beyond Node 22+:

npm create portaljs@latest my-portal
cd my-portal
npm run dev      # → http://localhost:3000

You get the three surfaces — Home, a Catalog (/search), and a dataset Showcase (/@<namespace>/<slug>) — over sample data. Plain, editable Next.js, no lock-in. Add your own CSV/JSON to datasets.json and it renders automatically.

Build it with your AI assistant — PortalJS ships Claude Code skills that do the assembly. Install them once (into ~/.claude/commands):

curl -fsSL https://raw.githubusercontent.com/datopian/portaljs/main/scripts/install-portaljs-skills.sh | bash

Then, in a Claude Code session from any directory:

/portaljs-architect    not sure what stack you need? start here
/portaljs-new-portal   "Auckland Council open data portal"
/portaljs-add-dataset  ./data/air-quality.csv

/portaljs-new-portal scaffolds the three surfaces; /portaljs-add-dataset (or /portaljs-add-resource) loads data; /portaljs-connect-ckan points it at a CKAN backend; /portaljs-deploy ships it. (All skills + install →)

Prefer the bare template — plain Next.js, no AI, no lock-in:

npx tiged datopian/portaljs/examples/portaljs-catalog my-portal
cd my-portal && npm install && npm run dev      # → http://localhost:3000

You get Home, a Catalog (/search), and a dataset Showcase (/@<namespace>/<slug>) over sample data. Add your own CSV/JSON to datasets.json and it renders automatically.

⭐ If it's useful, a star helps others find it.

Why PortalJS

Building a data portal has always meant more than a website. You have to decide where the data lives, how it's versioned, how people search it, how it's served, and how it's governed — and then wire a frontend on top. Teams either over-build on a heavy data warehouse they don't need, or under-build on a pile of scripts that doesn't scale.

PortalJS is an open-source, agentic skills framework that helps data teams build, develop, and ship data portals — and the data infrastructure underneath them. It isn't only a frontend. The skills do two jobs:

  • Advise — given what you're building, what your data is, and what it's for, they recommend an architecture: storage, compute, catalog, access, hosting, metadata.
  • Build — they scaffold that stack as plain, editable Next.js code with no lock-in.

It is opinionated but open: the recommended modern path is git + object storage (Cloudflare R2) + Parquet, queried with DuckDB — an open lakehouse instead of a classic warehouse. For living, incremental tables you can layer on DuckLake, and a traditional datastore (CKAN, a warehouse) stays a first-class option when you need it. You always own plain code.

Built and maintained in the open by Datopian and the PortalJS community.

Architecture at a glance

        🧑  you describe what you want to build
        │
        ▼
╭─ 🤖  AGENTIC SKILLS ──────────────────────────────────  decide + build
│   /portaljs-architect · /portaljs-new-portal · /portaljs-add-dataset · /portaljs-add-chart · /portaljs-add-map …
╰─  generates plain, editable Next.js code — no lock-in
        │
        ▼
╭─ 🖥️  SURFACES ────────────────────────────────────────  what users see
│   🏠 Home /      🔎 Catalog /search      📊 Showcase /@ns/slug
╰─  read data through one DataProvider contract
        │
        ▼
╭─ 🔌  PROVIDERS ───────────────────────────────────────  pluggable backends
│   📁 static·git     🐘 CKAN     🔭 OpenMetadata     🗂️ git-LFS + R2
╰─  swap the source without touching a page
        │
        ▼
📦  STORAGE + COMPUTE  —  choose your point on the spectrum:

      flat files  ─▶  Git-LFS + R2  ─▶  Parquet on R2 + 🦆 DuckDB  ─▶  warehouse / CKAN
      simplest                       ⭐ open lakehouse (default)        heaviest
                                     (+ DuckLake for living tables)

☁️  Substrate  —  Cloudflare R2 (storage) · Workers (runtime) · D1 (catalog) · Pages (static)
     object storage stays S3-compatible — R2 is the default, never a lock-in

Three surfaces. Every data portal is built from three: a Home page that explains it and offers search, a Catalog (/search) to discover datasets, and a Showcase (/@<namespace>/<slug>) to explore one dataset — metadata, preview, download/API, and charts/maps. (Core concepts →)

One seam. The surfaces read data only through a DataProvider, so the source — static files today, a CKAN or lakehouse backend tomorrow — can change without touching a page.

See ROADMAP.md for the full model and the architecture decision framework for how /portaljs-architect turns your needs into a stack.

Build a portal with your AI assistant

PortalJS ships Claude Code skills that turn a brief into a working portal.

Setup

Install the skills once into your personal scope so they're available from any directory:

curl -fsSL https://raw.githubusercontent.com/datopian/portaljs/main/scripts/install-portaljs-skills.sh | bash

Restart Claude Code (or open a new session) and type / to see them. See .claude/INSTALL.md for other install options (versioned plugin, or running straight from a clone of this repo).

Use

If you're not sure how to set up your portal, start with the advisor, then build:

/portaljs-architect    we have ~200 public CSVs, updated quarterly, and must publish DCAT-AP
/portaljs-new-portal   "Auckland Council open data portal"
/portaljs-add-dataset  ./data/air-quality.csv
/portaljs-add-dataset  https://example.com/parks.geojson

The skills are interactive — if your brief is thin, they interview you in short rounds rather than erroring. /portaljs-architect recommends a stack and hands off; /portaljs-new-portal scaffolds the three surfaces; /portaljs-add-dataset appends to the datasets.json manifest and the showcase renders automatically at /@<namespace>/<slug>. Run npm run dev and you have a portal.

Prefer to build by hand? The skills are a convenience, not a requirement — scaffold the template directly with the CLI:

npm create portaljs@latest my-portal

(Or grab the bare template with no prompts: npx tiged datopian/portaljs/examples/portaljs-catalog my-portal.)

Available skills

| Skill | What

// HOW IT'S BUILT

// FILES

2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • skills/arcgis-to-portaljs/SKILL.md7.8 KBupstream ↗
    ---
    name: arcgis-to-portaljs
    description: Migrate a whole ArcGIS Hub site into a PortalJS Arc portal end-to-end. Harvests the Hub /data.json (DCAT-US) inventory, exports every FeatureService layer through the ArcGIS REST query API with resultOffset paging, converts each to the serverless dual tier (PMTiles render + GeoParquet query) with tabular items to Parquet, pushes everything to Cloudflare R2 via Git LFS, appends dual-tier datasets.json entries, and writes a source-vs-derived parity report. Use to move a City or sector ArcGIS Hub open-data portal onto PortalJS with no server-side compute.
    allowed-tools: Read, Write, Edit, Bash(ogr2ogr:*), Bash(ogrinfo:*), Bash(tippecanoe:*), Bash(duckdb:*), Bash(git:*), Bash(curl:*), Bash(jq:*), Bash(npx:*), Bash(node:*), Bash(mkdir:*), Bash(cp:*), Bash(wc:*), Bash(command:*), WebFetch
    version: 0.1.0
    author: Datopian <[email protected]>
    license: MIT
    compatibility: Claude Code with PortalJS portals (Next.js 14, React 18/19, Node 18+). Requires native GDAL, tippecanoe, duckdb, and jq on the operator's machine (macOS or Linux/WSL). Runs from any project via the plugin, a personal install, or a portaljs clone.
    tags:
      - portaljs
      - data-portal
      - migration
      - arcgis
      - geospatial
      - pmtiles
    ---
    
    # ArcGIS Hub → PortalJS
    
    ## Overview
    
    Migrate an entire **ArcGIS Hub** open-data site into a **PortalJS Arc** portal in one pass.
    Every Hub site is machine-readable — a DCAT-US catalog at `/data.json`, with every dataset
    backed by an ArcGIS REST FeatureService — so migration is a **harvest → export → convert →
    publish → verify** pipeline that runs almost fully automated on the operator's machine, no
    server-side compute. The tooling is the reusable `arcgis-to-portaljs` migrator: input is one
    Hub URL, output is a ready-to-deploy PortalJS catalog plus a parity report.
    
    The skill is an orchestrator: it reuses the DCAT-US harvest from `portaljs-migrate`, the
    `ogr2ogr`/`tippecanoe`/`duckdb` dual-tier conversion from `portaljs-add-geo`, and the bulk
    Git-LFS → R2 push from `portaljs-migrate`. Its novel parts are the FeatureService REST export
    loop (paged features, not just a link) and the source-vs-derived parity report.
    
    ## Prerequisites
    
    - A scaffolded PortalJS portal whose template ships `components/MapPreview.tsx` and
      `components/GeoQuery.tsx` (PR #1647 or later). Run `portaljs-new-portal` first if none.
    - Native CLIs: **GDAL** (`ogr2ogr`, `ogrinfo`), **tippecanoe**, **duckdb** (with `spatial`),
      and **jq**. macOS: `brew install gdal tippecanoe duckdb jq`; Debian/Ubuntu:
      `apt-get install gdal-bin duckdb jq` plus tippecanoe (apt or build from source); Windows via
      WSL. The skill hard-stops with the install hint if any is missing.
    - Arc credentials for the Git-LFS → R2 push (the token `portaljs-deploy` resolves), or an OSS
      self-hosted Giftless.
    
    ## Instructions
    
    The canonical, full step-by-step workflow is
    [`.claude/commands/arcgis-to-portaljs.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/arcgis-to-portaljs.md)
    — the single source of truth. Read and follow it when executing. Summary:
    
    1. Gather input — Hub URL, portal directory, project slug, optional flags (`--limit`,
       `--only`, `--dry-run`, `--namespace-mode`). Interview if missing; never dead-end.
    2. Check native tools (`ogr2ogr`, `tippecanoe`, `duckdb` + `spatial`, `jq`). Any missing →
       print the per-OS install and stop.
    3. Validate the portal directory and confirm the geo showcase components exist.
    4. Harvest the Hub `/data.json` (reuse the `portaljs-migrate` DCAT-US map) and classify each
       item: vector (FeatureService), table, or non-data (web map / 3D / imagery → skipped).
       Under `--namespace-mode owner`, resolve namespaces through a publisher-normalization
       table with title-prefix fallback for broken `{{source}}` publishers (multi-publisher
       Hubs ship dirty publisher labels). Dedup near-duplicate hosted-`view` layers — but only
       after a mandatory live record-count check on BOTH twins: equal ⇒ dedup (keep the source
       layer, log the pair); different ⇒ keep both as distinct datasets. Consolidate per-year
       dataset series into one year-partitioned Parquet with legacy per-year view entries. Enrich
       from the AGOL item: sanitized metadata (license/description/dates), cleaned display title
       (`cleanTitle` — raw title still drives the slug), `category` (item categories → meaningful
       theme → keyword mapping), and a `thumbnail` snapshot into `public/thumbnails/`.
    5. Export each vector layer through the ArcGIS REST `query` API with `resultOffset` paging
       (`f=geojson`, `outSR=4326`); fall back to keyset paging on transfer limits; accept a
       customer File Geodatabase dump for very large layers.
    6. Convert each layer to the dual tier via the `portaljs-add-geo` recipe (PMTiles +
       GeoParquet); tabular items to Parquet. Preserve the native-CRS original.
    7. Publish — bulk Git-LFS track + one push to R2 through Giftless, then append dual-tier
       `datasets.json` entries (upsert on `(namespace, slug)`).
    8. Write `arcgis-parity-report.md` — record count, extent, attribute schema, and geometry
       validity, source vs derived, per dataset, plus the migrated/skipped/failed accounting.
    9. Report the inventory, migrated datasets, R2 push, and parity summary.
    
    ## Output
    
    - **Created:** `data/<namespace>/<slug>.pmtiles`, `.parquet`, and the original per vector
      dataset (all LFS-tracked → R2); Parquet + original per table; `arcgis-parity-report.md`.
    - **Modified:** `datasets.json` (one dual-tier entry per vector dataset, one resource entry
      per table); `.gitattributes` (LFS tracking).
    - **Verified:** the parity report compares each derived artifact to the live FeatureService.
    - **Result:** `/@<namespace>/<slug>` renders `<MapPreview>` + `<GeoQuery>` for each vector
      dataset with no page edits; the catalog lists everything migrated.
    
    ## Error Handling
    
    | Symptom | Cause | Fix |
    | --- | --- | --- |
    | `MISSING_INPUT` | No Hub URL provided | Pass the site root (e.g. `https://hub-lewisville.opendata.arcgis.com`) and retry. |
    | `MISSING_TOOLS` | `ogr2ogr`/`tippecanoe`/`duckdb`/`jq` (or duckdb `spatial`) absent | Print the per-OS install line and stop; re-run after installing. |
    | `NOT_A_PORTAL` | Target dir has no `datasets.json` / geo components | Run `portaljs-new-portal` first, then re-run. |
    | `HARVEST_FAILED` | `/data.json` unreachable or not DCAT-US | Confirm the site is an ArcGIS Hub and the feed loads in a browser. |
    | `EXPORT_FAILED` | One FeatureService layer errored or hit a hard transfer cap | Logged and skipped; try keyset paging or a customer FGDB dump for that layer. |
    | `LFS_PUSH_FAILED` | Missing/expired Arc token or unset `lfs.url` | Re-mint the JWT (see `portaljs-deploy`); confirm `git config lfs.url`. |
    
    ## Examples
    
    ### Example 1 — Migrate a City Hub (Lewisville)
    
    ```
    /arcgis-to-portaljs https://hub-lewisville.opendata.arcgis.com slug=lewisville
    ```
    
    ### Example 2 — Dry-run inventory + plan only (no writes)
    
    ```
    /arcgis-to-portaljs https://streamwaterdata.co.uk --dry-run
    ```
    
    ### Example 3 — Migrate a subset, one namespace per publisher
    
    ```
    /arcgis-to-portaljs https://streamwaterdata.co.uk --only sewer-catchments,water-boundaries --namespace-mode owner
    ```
    
    ## Resources
    
    - Full workflow: [`.claude/commands/arcgis-to-portaljs.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/arcgis-to-portaljs.md)
    - REST export, classification, parity, and phasing details: [`references/reference.md`](references/reference.md)
    - Ongoing sync, parity dashboard, and cutover (Phase 3): [`references/sync-and-cutover.md`](references/sync-and-cutover.md)
    - Related skills: `portaljs-migrate`, `portaljs-add-geo`, `portaljs-add-dataset`, `portaljs-deploy`
    - ArcGIS REST query API: <https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer/> · tippecanoe: <https://github.com/felt/tippecanoe> · DuckDB spatial: <https://duckdb.org/docs/extensions/spatial>
    
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/datopian-portaljs-skills-arcgis-to-portaljs.md

// REPO STATS

2.4k stars
15 open issues
Last commit: 19d ago