# data-analysis

Adopt this skill: https://proskills.md/skills/other/sinaptik-ai-starpod-crates-starpod-skills-data-analysis

Page: https://proskills.md/skills/other/sinaptik-ai-starpod-crates-starpod-skills-data-analysis
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: crates/starpod/skills/data-analysis/SKILL.md
size: 6219
upstream: https://github.com/sinaptik-ai/starpod/blob/main/crates/starpod/skills/data-analysis/SKILL.md

````text
---
name: data-analysis
description: "Use this skill when the user asks to analyze data from CSV, JSON, Excel, or database exports — including exploring datasets, computing statistics, creating visualizations, finding patterns, cleaning data, or building dashboards. Trigger whenever the user provides a data file and wants insights, charts, or transformations."
version: 0.1.0
compatibility: "Requires Python 3. Install deps: pip install pandas matplotlib seaborn"
---

# Data Analysis

## Workflow

1. **Load & inspect** — read data, check shape, types, nulls
2. **Clean** — handle missing values, fix types, remove duplicates
3. **Explore** — summary stats, distributions, correlations
4. **Analyze** — answer the specific question
5. **Visualize** — create clear, labeled charts
6. **Report** — summarize findings in plain language

## Quick Start: Data Profiling

```bash
python scripts/profile.py data.csv                    # print profile to stdout
python scripts/profile.py data.xlsx --output report.md  # save to file
python scripts/profile.py data.xlsx --sheet "Sales"     # specific sheet
```

The profiler auto-detects file format and generates: row/column counts, types, null percentages, numeric statistics, and top categorical values.

## Loading Data

```python
import pandas as pd

# Auto-detect format
df = pd.read_csv("data.csv")
df = pd.read_excel("data.xlsx")
df = pd.read_json("data.json")
df = pd.read_csv("data.tsv", sep="\t")

# Handle encoding issues
df = pd.read_csv("data.csv", encoding="latin-1")

# Large files — read in chunks
for chunk in pd.read_csv("large.csv", chunksize=10000):
    process(chunk)
```

## Inspection

```python
df.shape                    # (rows, cols)
df.dtypes                   # column types
df.head(10)                 # first 10 rows
df.describe()               # numeric statistics
df.describe(include='all')  # include categorical
df.isnull().sum()           # missing values per column
df.nunique()                # unique values per column
df.duplicated().sum()       # duplicate rows
```

## Cleaning

```python
# Drop duplicates
df = df.drop_duplicates()

# Handle missing values
df['col'].fillna(df['col'].median(), inplace=True)  # fill with median
df = df.dropna(subset=['critical_col'])               # drop rows missing critical data

# Fix types
df['date'] = pd.to_datetime(df['date'])
df['amount'] = pd.to_numeric(df['amount'], errors='coerce')
df['category'] = df['category'].astype('category')

# Clean strings
df['name'] = df['name'].str.strip().str.lower()

# Rename columns
df.columns = df.columns.str.strip().str.lower().str.replace(' ', '_')
```

## Analysis Patterns

### Group and aggregate
```python
df.groupby('category')['revenue'].agg(['sum', 'mean', 'count'])
df.groupby(['year', 'region']).agg(
    total_sales=('sales', 'sum'),
    avg_price=('price', 'mean'),
    n_orders=('order_id', 'count')
).reset_index()
```

### Time series
```python
df['date'] = pd.to_datetime(df['date'])
df.set_index('date', inplace=True)
monthly = df.resample('M')['value'].sum()
rolling = df['value'].rolling(window=7).mean()
```

### Pivot tables
```python
pivot = df.pivot_table(
    values='revenue',
    index='region',
    columns='quarter',
    aggfunc='sum',
    margins=True
)
```

### Correlations
```python
corr = df[['price', 'quantity', 'revenue', 'rating']].corr()
```

## Visualization

### Setup
```python
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_theme(style="whitegrid")
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['figure.dpi'] = 150
```

### Bar chart
```python
fig, ax = plt.subplots()
data = df.groupby('category')['revenue'].sum().sort_values(ascending=False)
data.plot(kind='bar', ax=ax, color='#1E2761')
ax.set_title('Revenue by Category', fontsize=14, fontweight='bold')
ax.set_xlabel('')
ax.set_ylabel('Revenue ($)')
ax.bar_label(ax.containers[0], fmt='${:,.0f}')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.savefig('revenue_by_category.png')
```

### Line chart (time series)
```python
fig, ax = plt.subplots()
ax.plot(monthly.index, monthly.values, color='#1E2761', linewidth=2)
ax.fill_between(monthly.index, monthly.values, alpha=0.1, color='#1E2761')
ax.set_title('Monthly Revenue Trend', fontsize=14, fontweight='bold')
ax.set_ylabel('Revenue ($)')
plt.tight_layout()
plt.savefig('trend.png')
```

### Heatmap (correlations)
```python
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, annot=True, fmt='.2f', cmap='RdBu_r', center=0,
            square=True, ax=ax)
ax.set_title('Correlation Matrix', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.savefig('correlations.png')
```

### Histogram / distribution
```python
fig, ax = plt.subplots()
ax.hist(df['value'], bins=30, color='#1E2761', edgecolor='white', alpha=0.8)
ax.axvline(df['value'].mean(), color='#F96167', linestyle='--', label=f"Mean: {df['value'].mean():.1f}")
ax.legend()
ax.set_title('Value Distribution', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.savefig('distribution.png')
```

### Scatter plot
```python
fig, ax = plt.subplots()
ax.scatter(df['x'], df['y'], alpha=0.5, s=20, color='#1E2761')
ax.set_title('X vs Y', fontsize=14, fontweight='bold')
ax.set_xlabel('X')
ax.set_ylabel('Y')
plt.tight_layout()
plt.savefig('scatter.png')
```

## Visualization Rules

- **Always label axes and title** — no unlabeled charts
- **Use consistent colors** — pick a palette and stick with it
- **Annotate key values** — label the most important data points
- **Save at 150+ DPI** — `plt.savefig('chart.png', dpi=150, bbox_inches='tight')`
- **Choose the right chart**: bar for comparison, line for trends, scatter for relationships, histogram for distributions, heatmap for correlations
- **Sort bar charts** — almost always descending by value
- **Limit categories** — show top 10, group the rest as "Other"

## Exporting Results

```python
# To Excel with formatting
with pd.ExcelWriter('report.xlsx', engine='openpyxl') as writer:
    summary.to_excel(writer, sheet_name='Summary')
    details.to_excel(writer, sheet_name='Details')

# To CSV
df.to_csv('output.csv', index=False)

# To markdown table (for reports)
print(df.to_markdown(index=False))
```
````

## File: README.md
upstream: https://github.com/sinaptik-ai/starpod/blob/main/README.md

````text
<p align="center">
  <img src="docs/public/logo.svg" alt="Starpod" width="80" height="80">
</p>

<h1 align="center">Starpod</h1>

<p align="center">
  <strong>Personal AI agents. Built in Rust.</strong>
</p>

<p align="center">
  <a href="https://starpod.sh">Website</a> · <a href="https://docs.starpod.sh">Docs</a> · <a href="https://console.starpod.sh">Console</a> · <a href="https://discord.com/invite/KYKj9F2FRH">Discord</a>
</p>

<p align="center">
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-C0C0C0.svg?style=flat-square" alt="MIT License"></a>
  <a href="https://www.rust-lang.org/"><img src="https://img.shields.io/badge/rust-1.87+-C0C0C0.svg?style=flat-square&logo=rust&logoColor=white" alt="Rust 1.87+"></a>
  <img src="https://img.shields.io/badge/tests-1362-C0C0C0.svg?style=flat-square" alt="1362 tests">
  <img src="https://img.shields.io/badge/crates-16-C0C0C0.svg?style=flat-square" alt="16 crates">
  <a href="https://discord.com/invite/KYKj9F2FRH"><img src="https://img.shields.io/discord/1102146545580785686?label=Discord&logo=discord&logoColor=white&color=5865F2&style=flat-square" alt="Discord"></a>
</p>

---

Starpod is an open-source AI agent runtime built in Rust. Bootstrap an agent in any directory with `starpod init`, then run it locally or deploy to the cloud. Each agent gets its own memory, vault, filesystem, and sessions — all self-contained in a `.starpod/` directory.

## Install

```bash
curl -fsSL https://starpod.sh/install | sh
```

Or via Homebrew:

```bash
brew install sinaptik-ai/tap/starpod
```

<details>
<summary>Other methods</summary>

```bash
# From crates.io
cargo install starpod

# From source
git clone https://github.com/sinaptik-ai/starpod.git
cd starpod
cargo install --path crates/starpod --locked
```

</details>

## Quick start

```bash
# Initialize an agent
starpod init --name "Jarvis" --model anthropic/claude-haiku-4-5

# Seed your API key into the vault
starpod init --env ANTHROPIC_API_KEY=sk-ant-...

# Start the dev server (opens browser)
starpod dev
```

```
  ╭──────────────────────────────────────────╮
  │      Jarvis  ·  AI Assistant             │
  ╰──────────────────────────────────────────╯

  Server 127.0.0.1:3000
  API Key sp-abc123...
```

Or use the terminal:

```bash
starpod chat "What files are in this directory?"
starpod repl
```

## Highlights

- **Simple setup** — `starpod init` bootstraps everything. No workspace files, no blueprints, no separate instance management.
- **Multi-channel** — Web UI, Telegram, CLI, HTTP API, WebSocket streaming. Same agent, every surface.
- **Persistent memory** — markdown files + SQLite FTS5. Per-user memory spaces. The agent remembers across sessions.
- **Self-extending skills** — the agent creates, edits, and deletes its own skill files at runtime. `/skill` and it executes.
- **Encrypted vault** — AES-256-GCM credential storage. API keys never touch disk in plaintext. No `.env` files — vault only.
- **Cron scheduling** — interval, cron expressions, one-shot. Runs through the full agent loop, records history.
- **Channel-aware sessions** — explicit sessions for web/API, time-gap sessions for Telegram. Per-user scoping.
- **Streaming** — real-time text deltas and tool-use events over WebSocket.
- **Built in Rust** — 16 crates, 1,362 tests, zero warnings.

## Architecture

```
crates/
├── agent-sdk/            Claude API client + agent loop
├── starpod-hooks/        Lifecycle hooks, events, permissions
├── starpod-core/         Config, shared types, errors
├── starpod-db/           Unified SQLite (core.db)
├── starpod-memory/       FTS5 full-text search + per-user memory
├── starpod-vault/        AES-256-GCM encrypted credentials
├── starpod-session/      Channel-aware session lifecycle
├── starpod-skills/       Markdown-based self-extension
├── starpod-cron/         Scheduling (interval, cron, one-shot)
├── starpod-agent/        Orchestrator wiring everything together
├── starpod-gateway/      Axum HTTP/WS server + embedded web UI
├── starpod-telegram/     Telegram bot (teloxide)
├── starpod-instances/    Remote instance management
└── starpod/              CLI binary
```

## Agent layout

Each agent is self-contained in a `.starpod/` directory:

```
.starpod/
├── config/
│   ├── agent.toml          Agent configuration
│   ├── SOUL.md             Personality
│   ├── HEARTBEAT.md        Periodic self-reflection
│   ├── BOOT.md             Boot instructions
│   ├── BOOTSTRAP.md        First-run instructions
│   └── frontend.toml       Web UI config
├── skills/                 Skill files
├── db/
│   ├── core.db             Sessions + cron + auth
│   ├── memory.db           FTS5 + vectors
│   └── vault.db            Encrypted credentials
└── users/<id>/
    ├── USER.md             User profile
    ├── MEMORY.md           Memory index
    └── memory/             Daily logs
```

## Configuration

All configuration lives in a single `agent.toml`:

```toml
agent_name = "Nova"
models = ["anthropic/claude-haiku-4-5"]
max_turns = 30
server_addr = "127.0.0.1:3000"

[channels.telegram]
allowed_users = [123456789]
```

**Secrets** live in the vault (`vault.db`), seeded via `starpod init --env KEY=VAL` or the web UI Settings page. No `.env` files.

**Personality** in `SOUL.md` — not in config.

## Agent tools

| Category | Tools |
|----------|-------|
| Built-in | `Read`, `Write`, `Edit`, `Bash`, `Glob`, `Grep` |
| Memory | `MemorySearch`, `MemoryWrite`, `MemoryAppendDaily` |
| Environment | `EnvGet` |
| File sandbox | `FileRead`, `FileWrite`, `FileList`, `FileDelete` |
| Skills | `SkillActivate`, `SkillCreate`, `SkillUpdate`, `SkillDelete`, `SkillList` |
| Cron | `CronAdd`, `CronList`, `CronRemove`, `CronRuns`, `CronRun`, `CronUpdate`, `HeartbeatWake` |

## API

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/chat` | Send a message |
| `GET` | `/api/sessions` | List sessions |
| `GET` | `/api/sessions/:id` | Get session |
| `GET` | `/api/sessions/:id/messages` | Session messages |
| `GET` | `/api/memory/search?q=...` | Full-text search |
| `POST` | `/api/memory/reindex` | Rebuild FTS index |
| `GET` | `/api/health` | Health check |

WebSocket at `ws://localhost:3000/ws`. Auth via `X-API-Key` header or `?token=` param.

```json
← {"type": "message", "text": "Hello"}
→ {"type": "stream_start", "session_id": "..."}
→ {"type": "text_delta", "text": "Hi "}
→ {"type": "text_delta", "text": "there!"}
→ {"type": "tool_use", "name": "Read", "input": {...}}
→ {"type": "tool_result", "content": "..."}
→ {"type": "stream_end", "cost_usd": 0.004}
```

## Telegram

1. Create a bot with [@BotFather](https://t.me/BotFather)
2. Store the token in the vault (via `starpod init --env TELEGRAM_BOT_TOKEN=...` or Settings UI)
3. Add `allowed_users` to `agent.toml`
4. `starpod dev` — look for `Telegram connected`

## CLI

```
starpod init [--name N] [--model M] [--env K=V]   Initialize agent
starpod dev [--port P]                             Dev server (opens browser)
starpod serve                                      Production server
starpod deploy                                     Deploy to remote (coming soon)
starpod chat "message"                             One-shot message
starpod repl                                       Interactive REPL
starpod auth login|logout|status                   Platform authentication
```

## Development

```bash
cargo test --workspace
```

## Community

- [Discord](https://discord.com/invite/KYKj9F2FRH) — questions, feedback, show & tell
- [GitHub Issues](https://github.com/sinaptik-ai/starpod/issues) — bug reports and feature requests
- [Docs](https://docs.starpod.sh) — full documentation

## License

MIT
````
