OpenAI Voice Skill

OpenAI Voice Skill

Real-time voice conversations for OpenClaw agents using OpenAI's Realtime API.

Sub-200ms latency via native SIP — no STT/TTS chain. Built by Nia for OpenClaw agents.

---

🚀 Get Started in 5 Minutes

Prerequisites

• Python 3.10+
• Node.js 18+ (for the channel plugin)
• OpenAI API key with Realtime API access
• Twilio account with a phone number (sign up free)

1. Clone & install

git clone https://github.com/nia-agent-cyber/openai-voice-skill.git
cd openai-voice-skill
pip install -r scripts/requirements.txt

2. Configure

cp .env.example .env

Fill in your keys:

OPENAI_API_KEY=sk-...
OPENAI_PROJECT_ID=proj_...        # platform.openai.com/settings → Project
TWILIO_ACCOUNT_SID=AC...
TWILIO_AUTH_TOKEN=...
TWILIO_PHONE_NUMBER=+1...

3. Start the server

python scripts/webhook-server.py

4. Expose it (for Twilio webhooks)

# Using cloudflared:
cloudflared tunnel --url http://localhost:8080

# Or ngrok:
ngrok http 8080

5. Make your first call

curl -X POST http://localhost:8080/call \
  -H "Content-Type: application/json" \
  -d '{"to": "+1234567890", "message": "Hello from my AI agent!"}'

That's it — your agent is on the phone. 📞

Next: Check out the examples/ folder for ready-to-use recipes like a missed-call → appointment handler.

For full setup, see Twilio SIP trunking and OpenAI webhooks below.

---

What This Does

Voice as a first-class channel for your OpenClaw agent:

• Call your agent - Dial the Twilio number, talk to your agent
• Agent calls you - Outbound calls initiated by the agent
• Session continuity - Same phone number = same conversation, across voice and text channels
• Full agent access - Voice sessions can invoke OpenClaw's tools via ask_openclaw

✅ What's Working

Why Native SIP?

Most voice solutions chain services with cumulative latency:

Phone → Twilio → Server → Deepgram STT → LLM → ElevenLabs TTS → Server → Phone
                   ~300ms      ~500ms        ~500ms              ~300ms

This skill uses OpenAI's Realtime API with native SIP:

Phone → Twilio SIP → OpenAI Realtime API → Phone
                         ~200ms total

Single hop. Native speech-to-speech. Conversations feel natural.

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         OpenClaw Agent                               │
│                                                                      │
│  ┌──────────────────┐    ┌──────────────────┐    ┌───────────────┐  │
│  │ Session Store    │◄───│ Session Bridge   │◄───│ Call Events   │  │
│  │ (voice:+1234...) │    │ (port 8082)      │    │               │  │
│  └──────────────────┘    └──────────────────┘    └───────┬───────┘  │
│           │                                              │          │
│           ▼                                              │          │
│  ┌──────────────────────────────────────────────────────┴───────┐  │
│  │                    webhook-server.py (port 8080)              │  │
│  │  - Receives Twilio webhooks                                   │  │
│  │  - Connects to OpenAI Realtime API                           │  │
│  │  - Handles ask_openclaw function calls                       │  │
│  │  - Stores transcripts                                         │  │
│  └──────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘
                              │
            ┌─────────────────┼─────────────────┐
            ▼                 ▼                 ▼
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│  Twilio SIP     │  │ OpenAI Realtime │  │ OpenClaw CLI    │
│  (phone calls)  │  │ (voice AI)      │  │ (tool execution)│
└─────────────────┘  └─────────────────┘  └─────────────────┘

Key Components

Session Sync Flow

1. Call starts → Bridge creates session key (voice:+15551234567)
2. During call → Transcript events sent to bridge
3. Call ends → Full transcript synced to OpenClaw session JSONL
4. Cross-channel → Same phone = same session in Telegram/other channels

Setup

Prerequisites

• Python 3.10+
• Node.js 18+ (for channel plugin)
• OpenClaw installed and configured
• Twilio account with phone number
• OpenAI API access (with Realtime API enabled)

1. Clone & Install

git clone https://github.com/nia-agent-cyber/openai-voice-skill.git
cd openai-voice-skill/scripts
pip install -r requirements.txt

2. Configure Environment

cp ../.env.example ../.env

Edit .env:

# Required
OPENAI_API_KEY=sk-...
OPENAI_PROJECT_ID=proj_...

# For outbound calls
TWILIO_ACCOUNT_SID=AC...
TWILIO_AUTH_TOKEN=...
TWILIO_PHONE_NUMBER=+14402915517

# Public URL (for Twilio webhooks)
PUBLIC_URL=https://api.niavoice.org

# Optional
PORT=8080
OPENCLAW_TIMEOUT=30

3. Configure Twilio

SIP Trunk (for OpenAI Realtime):

1. Go to Elastic SIP Trunking → Create trunk
2. Termination URI: sip:[email protected];transport=tls
3. Assign your phone number to the trunk

Webhook (for outbound calls):

1. Phone Numbers → Your number → Voice Configuration
2. Webhook URL: https://your-domain/voice/twiml

4. Configure OpenAI

1. Go to platform.openai.com/settings
2. Project → Webhooks
3. Add your server URL + /webhook
4. Subscribe to realtime.call.incoming

5. Run the Server

# Start the voice server
python webhook-server.py

# In production, use the cloudflare tunnel
cloudflared tunnel --url http://localhost:8080

6. Install Channel Plugin (Optional)

For full OpenClaw integration:

cd channel-plugin
npm install
npm run build
cp -r dist/* ~/.openclaw/extensions/voice-channel/

Add to OpenClaw config:

channels:
  voice:
    accounts:
      default:
        enabled: true
        webhookUrl: "https://api.niavoice.org"

Restart OpenClaw:

openclaw gateway restart

Usage

Inbound Calls

Just call your Twilio number! The OpenAI Realtime API handles the conversation with your configured agent personality.

Outbound Calls

HTTP API:

curl -X POST https://api.niavoice.org/call \
  -H "Content-Type: application/json" \
  -d '{
    "to": "+1234567890",
    "message": "Hello! This is your AI assistant calling."
  }'

Response:

{
  "status": "initiated",
  "call_id": "CAxxxxxxxxxxxxxxxxxxxxx",
  "me