⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

version unknown

capture-alpaca-create

estuary@estuary⭐ 7 stars· last commit 1mo ago· 0 open issues

Create an Alpaca capture using flowctl to stream stock trade data into Estuary collections. Use when setting up an Alpaca Market Data source for historical and real-time stock trades. Use when user says "capture Alpaca", "stream stock trades", "Alpaca market data", or "connect Alpaca to Estuary".

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐ 7 on GitHubGitHub ↗

New / niche

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

Estuary Agent Skills

Set up Estuary data pipelines through your AI assistant. No docs to stitch together, no flowctl commands to remember.

Ask your AI to "capture my Postgres into Snowflake" or "why is this materialization failing?" — these skills give it everything it needs to run the right commands, build the right specs, and explain the gotchas.

Works with Claude Code, GitHub Copilot, Cursor, OpenAI Codex, Gemini CLI, and 30+ other tools via the open SKILL.md standard.

Which skills do I need?

You can do all of this without these skills. Install a group when you want your AI assistant to know the workflow and the gotchas. Install only what you need.

If you want help withInstall
Setting up a sourceestuary-captures, only the skills for your sources
Setting up a destinationestuary-materializations, only the skills for your destinations
Writing a derivationestuary-derivations, derivation-basics plus the types you use
Changing how fields materializeestuary-schema
Running and troubleshooting pipelinesestuary-operations

What's included

Captures (sources)

Install these if you want help creating captures, only the ones matching your sources.

SkillDescription
capture-postgres-createPostgreSQL CDC (vanilla, RDS, Aurora, Cloud SQL, Supabase, Neon)
capture-mysql-createMySQL CDC via binlog replication (RDS, Aurora, Cloud SQL, Azure)
capture-mongodb-createMongoDB CDC (Atlas, DocumentDB, self-hosted)
capture-sqlserver-createSQL Server CDC (RDS, Azure SQL, Cloud SQL)
capture-alpaca-createAlpaca stock trade data (historical backfill + real-time websocket stream)
capture-hubspot-createHubSpot real-time CRM capture (contacts, companies, deals, tickets)
capture-http-ingest-createHTTP webhook capture (GitHub, Shopify, Stripe, or any JSON source)
capture-generic-createAny of 148+ source connectors via dynamic schema discovery

Materializations (destinations)

Install these if you want help creating materializations, only the ones matching your destinations.

SkillDescription
materialize-postgres-createPostgreSQL destination
materialize-snowflake-createSnowflake destination (JWT auth)
materialize-bigquery-createBigQuery destination (GCS staging)
materialize-redshift-createAmazon Redshift destination (S3 staging)
materialize-databricks-createDatabricks destination (Unity Catalog)
materialize-clickhouse-createClickHouse destination (native protocol)
materialize-hubspot-createHubSpot CRM write-back (contacts, companies, deals, tickets)
materialize-generic-createAny destination connector via dynamic schema discovery

Derivations (transformations)

Install these only if you write derivations. Start with derivation-basics, then add the types you use.

SkillDescription
derivation-basicsFoundation: what derivations are, language choice, project layout, and workflow — read first
derivation-filter-transformStateless filtering, field selection, and per-document field transformation
derivation-aggregate-metricsDaily totals, running counts, min/max, and lifetime metrics via reduction annotations
derivation-join-collectionsJoin two or more collections on a shared key into an enriched collection
derivation-flatten-arrayFlatten a nested array into one output document per element
derivation-stateful-logicCustom SQLite state for balances, inventory, approval workflows, and deduplication
derivation-windowingSliding time-window state (e.g. last-24h events) via readDelay expiration
derivation-pythonWrite derivations in Python (async transforms, Pydantic types, uv deps) for ML/embeddings/async APIs

Operations

Install these if you want help running and troubleshooting pipelines.

SkillDescription
estuary-flowctl-setupInstall, authenticate, and update the flowctl CLI
estuary-task-healthEnd-to-end health check for a task: status, data flow, errors, and history
estuary-catalog-statusCheck control-plane status of a task with flowctl catalog status
estuary-task-statsInspect data volume, document counts, and hourly throughput for a task
estuary-catalog-historyReview publication history and recent spec changes
estuary-logsSearch and analyze task logs with jq filtering
estuary-connector-restartPause and restart connectors via shard management
estuary-ssh-tunnelsDiagnose and fix SSH tunnel connection issues

Schema

Install these only if you modify how fields materialize.

SkillDescription
schema-projectionsRename fields or map nested JSON paths to flat column names
schema-field-selectionInclude or exclude fields in materializations using depth modes and overrides
schema-custom-types-ddlOverride default column types via castToString or custom DDL
schema-field-redactionBlock or hash sensitive fields (PII, credentials)
schema-null-defaultsSet default values for missing fields and handle nullable primary keys
schema-define-fieldsAdd or define fields in a collection's readSchema (pruned, refined, or required)

Prerequisites

  • An Estuary account
  • flowctl CLI — the estuary-flowctl-setup skill walks you through installation and authentication

Installation

Skills CLI

Install all 37 skills:

npx skills add estuary/agent-skills

Or list them and pick:

npx skills add estuary/agent-skills --list
npx skills add estuary/agent-skills --skill capture-postgres-create --skill materialize-snowflake-create

Claude Code

Add the Estuary marketplace:

/plugin marketplace add estuary/agent-skills

Then install only the groups you need:

/plugin install estuary-captures@estuary
/plugin install estuary-materializations@estuary
/plugin install estuary-derivations@estuary
/plugin install estuary-operations@estuary
/plugin install estuary-schema@estuary

Or run /plugin to browse from the Discover tab. Installed skills auto-update when the marketplace refreshes.

Manual installation

Clone this repo and copy the skill folders into your AI tool's skill directory:

git clone https://github.com/estuary/agent-skills.git
cp -r agent-skills/skills/* your-project/.claude/skills/

Common paths by tool:

ToolPath
Claude Code.claude/skills/
Cursor.cursor/skills/
GitHub Copilot / VS Code.github/skills/
OpenCode.opencode/skills/
Codex.codex/skills/

Usage

Once installed, ask your AI assistant in plain English:

"Capture my Postgres database into Estuary"

"Materialize my collections into Snowflake"

"Capture from MySQL and materialize into Redshift"

"Why is my materialization failing?"

Resources

Feedback and support

License

Apache 2.0 — see LICENSE for details.

// HOW IT'S BUILT

// FILES

2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • skills/capture-alpaca-create/SKILL.md14 KB565531c7b322upstream ↗
    ---
    name: capture-alpaca-create
    description: Create an Alpaca capture using flowctl to stream stock trade data into Estuary collections. Use when setting up an Alpaca Market Data source for historical and real-time stock trades. Use when user says "capture Alpaca", "stream stock trades", "Alpaca market data", or "connect Alpaca to Estuary".
    ---
    
    # Create Alpaca Capture
    
    Create an Alpaca capture using flowctl to stream stock trade data from the Alpaca Market Data API into Estuary collections — both a historical backfill and a low-latency real-time stream, in parallel.
    
    **Applies to**: source-alpaca (Alpaca Market Data connector)
    
    ## Step 0: Load Connector Documentation
    
    Before proceeding, fetch the official connector docs for prerequisites, config reference, and limitations.
    
    **Load the docs page:**
    https://docs.estuary.dev/reference/Connectors/capture-connectors/alpaca/
    
    Use WebFetch to load this page. It covers:
    
    - The dual-API model (Trades REST API for backfill + Data API websocket for real-time)
    - Supported symbols (8000+ stocks/ETFs, max 20 per capture)
    - Plan requirements (Free/`iex` vs. Unlimited/`sip`)
    - Full config property reference
    - The historical/real-time overlap and how to reconcile duplicates
    
    **Search Kapa for tribal knowledge** (if the Estuary MCP is configured):
    
    ```
    Search kapa ai knowledge sources for "capture alpaca common issues"
    ```
    
    If Kapa MCP is not configured, the user can set it up: https://docs.estuary.dev/features/mcp-integration/
    
    This skill provides the **flowctl workflow** and **troubleshooting** that docs don't cover.
    
    ## How this connector works
    
    Alpaca captures trade data using **two APIs at once**:
    
    - **Trades REST API** — performs a historical backfill from your `start_date` forward until it catches up to the present.
    - **Data API websocket** — opens a real-time stream from the present moment and runs indefinitely until you stop the capture.
    
    All symbols in a capture land in a **single collection** via one binding (`trades`). Because the backfill catches up to where the live stream began, the two streams **overlap and produce some duplicate trade documents** — this is expected and reconcilable (see Limitations and Troubleshooting).
    
    ## Step 1: Gather Requirements
    
    Before writing any YAML, ask the user:
    
    1. **Alpaca account + API credentials?** — They need an Alpaca account and its **API Key ID** and **Secret Key** (generate in the Alpaca dashboard under API Keys). Unlike OAuth connectors, this is plain key auth, so the whole flow can be completed from the CLI.
    2. **Which plan — Free or Unlimited?** This drives the `feed` and `is_free_plan` settings:
       - **Free plan** → `feed: iex`, `advanced.is_free_plan: true`. Data is a smaller sample and **delayed 15 minutes**.
       - **Unlimited plan** → `feed: sip` for complete, real-time SIP data.
    3. **Which symbols?** — A comma-separated list, **maximum 20 per capture**. To track more than 20, set up multiple captures and split symbols between them.
    4. **Start date for the backfill?** — Must be **no earlier than `2016-01-01T00:00:00Z`** (Alpaca's earliest available data). Note this has **no effect if changed after the capture has started**.
    5. **Stop date?** (optional) — Set `advanced.stop_date` to halt the historical backfill at a fixed date (e.g. for a bounded one-time load).
    6. **Backfill-only or real-time-only?** (optional, advanced) — By default the connector runs both. `advanced.disable_real_time: true` gives a backfill-only load; `advanced.disable_backfill: true` gives a stream-only capture.
    7. **Non-default data plane?** — Most users use the default. Ask if they need a non-default data plane.
    
    ## Step 2: Find the Correct Connector Version
    
    Always use the latest numbered version tag. Query the connector registry to find it:
    
    ```bash
    flowctl raw get --table connector_tags \
      --query 'documentation_url=eq.https://go.estuary.dev/source-alpaca' \
      --query 'select=image_tag,documentation_url' \
      --output yaml
    ```
    
    Use the returned `image_tag` — never hardcode a version. (The docs sample shows `:v1`, but confirm against the registry.)
    
    ## Step 3: Help User Complete Prerequisites
    
    1. **Create / locate Alpaca API keys** — In the Alpaca dashboard, generate an **API Key ID** and **Secret Key**. The secret key is shown only once at creation; regenerate if lost.
    2. **Confirm the plan ↔ feed pairing** (see Step 1). Using `feed: sip` without an Unlimited subscription will fail with a subscription/permission error.
    3. **Verify symbols are supported** — Alpaca supports 8000+ stocks and ETFs. To check a symbol, use the Alpaca Broker API. Keep the list to **≤ 20**.
    
    No browser redirect is required — once the keys are in hand, everything else runs in flowctl.
    
    ## Step 4: Create the Capture Spec File
    
    Build `flow.yaml` using the config reference from the docs:
    
    ```yaml
    captures:
      <TENANT>/<PATH>/source-alpaca:
        endpoint:
          connector:
            image: ghcr.io/estuary/source-alpaca:<VERSION>
            config:
              api_key_id: "<ALPACA_API_KEY_ID>"
              api_secret_key: "<ALPACA_API_SECRET_KEY>"
              feed: iex                       # iex (free) or sip (Unlimited plan)
              start_date: "2022-11-01T00:00:00Z"   # must be >= 2016-01-01T00:00:00Z
              symbols: "AAPL,MSFT,AMZN,TSLA,GOOGL"  # max 20, comma-separated
              advanced:
                is_free_plan: true            # set true on a Free plan (delays data 15 min)
                # disable_backfill: false     # set true for a stream-only capture
                # disable_real_time: false    # set true for a backfill-only load
                # stop_date: "2023-01-01T00:00:00Z"
                # max_backfill_interval: "24h"  # smaller helps when tracking many symbols
                # min_backfill_interval: "1m"
        bindings:
          - resource:
              name: trades
            target: <TENANT>/<PATH>/trades
    ```
    
    **Important**:
    - `feed` and `is_free_plan` must match the account's plan. Free plan → `feed: iex` **and** `is_free_plan: true`.
    - `start_date` must be `>= 2016-01-01T00:00:00Z`, and is effectively immutable once the capture starts.
    - `symbols` is a single comma-separated **string**, not a YAML list, and must contain **≤ 20** symbols.
    - `max_backfill_interval` / `min_backfill_interval` are **Go duration strings** (e.g. `24h`, `1m`), not numbers.
    - All symbols capture into the **one** `trades` collection.
    
    ### Protect secrets before committing
    
    `api_secret_key` is a secret — don't commit it in plain text. Encrypt it with Estuary's sops-based mechanism:
    
    ```bash
    sops --encrypt \
      --input-type yaml --output-type yaml \
      --encrypted-suffix "_key" \
      --gcp-kms projects/<PROJECT>/locations/global/keyRings/<RING>/cryptoKeys/<KEY> \
      flow.yaml > flow.encrypted.yaml
    mv flow.encrypted.yaml flow.yaml
    ```
    
    > Note: `--encrypted-suffix "_key"` matches both `api_key_id` and `api_secret_key`. The key ID isn't strictly secret, but encrypting both is harmless. flowctl decrypts sops-encrypted specs at publish time. See https://docs.estuary.dev/concepts/connectors/#protecting-secrets for AWS KMS / Azure Key Vault / age options.
    
    ## Step 5: Discover and Publish
    
    ```bash
    # Validate the spec and confirm the trades binding
    flowctl discover --source flow.yaml
    
    # Review the spec
    cat flow.yaml
    
    # Publish the capture
    flowctl catalog publish --source flow.yaml --auto-approve
    ```
    
    The connector exposes a single `trades` resource, so discovery is mostly a validation step here — you can also author the binding by hand as shown in Step 4.
    
    ## Step 6: Verify
    
    ```bash
    # Check status
    flowctl catalog status <TENANT>/<PATH>/source-alpaca
    
    # View logs
    flowctl logs --task <TENANT>/<PATH>/source-alpaca --since 5m | jq -c '{ts, message}'
    
    # Read captured trade data
    flowctl collections read --collection <TENANT>/<PATH>/trades | head -10
    ```
    
    **Status progression:**
    1. `PENDING` — Normal for ~30 seconds during shard assignment
    2. `BACKFILLING` — Historical backfill via the Trades REST API (runs in parallel with the live stream)
    3. `OK` — Backfill caught up and/or real-time websocket stream running
    
    **What to expect in the data:**
    - On a **Free plan**, real-time trades arrive ~15 minutes delayed and are an `iex`-only sample.
    - You'll see **some duplicate trades** where the backfill overlaps the live stream — expected (see Limitations).
    - During U.S. market closed hours, the real-time stream is quiet; the backfill still runs.
    
    ## Limitations
    
    ### Maximum 20 symbols per capture
    Capturing more than 20 symbols in a single capture can cause API errors. To track more, **split symbols across multiple captures** (each writing to its own collection).
    
    ### Duplicate trade documents from overlapping streams
    The historical backfill and the real-time stream overlap, producing duplicate trade documents with identical Alpaca properties but different Estuary metadata. Resolve them downstream:
    
    - **Standard (non-delta) updates materialization** — Estuary deduplicates during materialization automatically. Most materialization connectors default to standard updates.
    - **Delta-updates-only materialization** — ensure the destination supports the equivalent of `lastWriteWins` reductions.
    
    ## Troubleshooting
    
    ### `401 Unauthorized` / `403 Forbidden` / "access key verification failed"
    
    **Cause**: Wrong `api_key_id` / `api_secret_key`, a regenerated/revoked key, or keys from the wrong Alpaca environment.
    
    **Fix**:
    1. Re-copy the **API Key ID** and **Secret Key** from the Alpaca dashboard (the secret is shown only once — regenerate if you don't have it).
    2. Confirm there's no stray whitespace/newline in the values.
    3. Republish after updating the spec.
    
    ### "subscription does not permit querying recent SIP data" / feed permission errors
    
    **Cause**: `feed: sip` was set without an Alpaca **Unlimited** subscription. The full SIP feed requires a paid plan.
    
    **Fix**: Either upgrade the Alpaca plan, or switch to the free feed: set `feed: iex` and `advanced.is_free_plan: true`. Free-plan data is an `iex` sample delayed 15 minutes.
    
    ### Real-time data is ~15 minutes behind
    
    **Cause**: Free plan. `is_free_plan: true` intentionally delays data by 15 minutes, and `iex` is a partial sample of the market.
    
    **Fix**: This is expected on the Free plan. For complete, low-latency data, upgrade to Unlimited and use `feed: sip` (and remove/disable `is_free_plan`).
    
    ### "websocket connection limit exceeded" / stream repeatedly disconnects
    
    **Cause**: Alpaca permits only **one** concurrent market-data websocket connection per account. Running a second Alpaca capture, or using the same keys for another websocket client (a trading bot, a notebook, etc.), contends for that single connection.
    
    **Fix**:
    1. Ensure only one consumer uses these API keys' websocket at a time.
    2. If you need multiple Alpaca captures (e.g. to exceed 20 symbols), use a **separate Alpaca account / API keys per capture**, since each account allows one stream.
    3. As a workaround for a single account, set `advanced.disable_real_time: true` on all but one capture (the others become backfill-only).
    
    ### "start_date must be no earlier than 2016-01-01" / config schema validation on start_date
    
    **Cause**: `start_date` is before `2016-01-01T00:00:00Z` (Alpaca's earliest data), or not an RFC 3339 timestamp.
    
    **Fix**: Set `start_date` to `2016-01-01T00:00:00Z` or later, in full timestamp form (e.g. `2022-11-01T00:00:00Z`).
    
    ### Changing `start_date` did nothing
    
    **Cause**: `start_date` only takes effect at capture creation — it has **no effect if changed after the capture has started**.
    
    **Fix**: To re-backfill from an earlier date, bump the binding's `backfill` counter (re-reads from the configured `start_date`) or delete and recreate the capture with the new `start_date`.
    
    ```yaml
    bindings:
      - resource: { name: trades }
        target: <TENANT>/<PATH>/trades
        backfill: 1   # increment to force a fresh backfill
    ```
    
    ### API errors when capturing many symbols
    
    **Cause**: More than 20 symbols in one capture, or large backfill windows straining the REST API.
    
    **Fix**:
    1. Keep `symbols` to **≤ 20**; split across multiple captures otherwise.
    2. Lower `advanced.max_backfill_interval` (e.g. `"6h"` or `"1h"`) so each backfill request covers a smaller window — useful when tracking many symbols.
    
    ### Duplicate trades in the destination
    
    **Cause**: Expected — the backfill overlaps the real-time stream (see Limitations).
    
    **Fix**: Materialize with **standard updates** (Estuary deduplicates automatically), or for delta-updates-only destinations ensure `lastWriteWins`-equivalent reduction support.
    
    ### Backfill never finishes / "Config failed schema validation"
    
    **Cause**: Common config mistakes:
    - `symbols` written as a YAML list instead of a comma-separated string
    - `max_backfill_interval` / `min_backfill_interval` given as a number instead of a Go duration string (`"24h"`, `"1m"`)
    - `is_free_plan` not set while using `feed: iex`, or `feed` missing entirely
    
    **Fix**: Match the spec in Step 4. `symbols` is one string; durations are quoted Go-duration strings; `feed` is required.
    
    ### Capture stuck in PENDING
    
    Wait 30-60 seconds — this is normal during shard assignment. If still stuck:
    
    ```bash
    flowctl logs --task <TENANT>/<PATH>/source-alpaca --since 5m | jq 'select(.level == "error")'
    ```
    
    ### No real-time data appearing
    
    **Cause**: U.S. equity markets are closed (no live trades to stream), or `advanced.disable_real_time: true` is set, or the Free plan's 15-minute delay hasn't elapsed yet.
    
    **Fix**:
    1. Confirm market hours — the live stream is quiet when the market is closed; the backfill still runs.
    2. Check `disable_real_time` is not set.
    3. On a Free plan, allow for the 15-minute delay.
    
    ## Related Skills
    
    - `materialize-clickhouse-create` / `materialize-snowflake-create` / `materialize-bigquery-create` — Send captured trades to a warehouse (use standard updates to dedupe)
    - `estuary-connector-restart` — Pause/restart the capture
    - `estuary-logs` — Deep log analysis
    - `estuary-catalog-status` — Status checking
    - `estuary-task-stats` — Confirm trade throughput
    
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/estuary-agent-skills-skills-capture-alpaca-create.md

// REPO STATS

7 stars
0 open issues
Last commit: 1mo ago