# robot-perception

Adopt this skill: https://proskills.md/skills/other/arpitg1304-robotics-agent-skills-skills-robot-perception

Page: https://proskills.md/skills/other/arpitg1304-robotics-agent-skills-skills-robot-perception
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: skills/robot-perception/SKILL.md
size: 60793
sha: 31e9657c8a37b74aeec59c368d039a6674f47860
upstream: https://github.com/arpitg1304/robotics-agent-skills/blob/main/skills/robot-perception/SKILL.md
truncated: true

````text
---
name: robot-perception
description: >
  Comprehensive best practices for robot perception systems covering cameras, LiDARs, depth sensors,
  IMUs, and multi-sensor setups. Use this skill when working with RGB image processing, depth maps,
  point clouds, sensor calibration (intrinsic, extrinsic, hand-eye), object detection, semantic
  segmentation, 3D reconstruction, visual servoing, or perception pipeline optimization. Trigger
  whenever the user mentions OpenCV, Open3D, PCL, RealSense, ZED, OAK-D, camera calibration,
  AprilTags, ArUco markers, stereo vision, RGBD, point cloud filtering, ICP registration, coordinate
  transforms, camera intrinsics, distortion correction, image undistortion, sensor streaming,
  frame synchronization, or any computer vision task in a robotics context. Also covers multi-camera
  rigs, time synchronization across sensors, perception latency budgets, and production deployment
  of perception pipelines.
---

# Robot Perception Skill

## When to Use This Skill
- Setting up and configuring camera, LiDAR, or depth sensors
- Building RGB, depth, or point cloud processing pipelines
- Calibrating cameras (intrinsic, extrinsic, hand-eye)
- Implementing object detection, segmentation, or tracking for robots
- Fusing data from multiple sensor modalities
- Streaming sensor data with proper threading and buffering
- Synchronizing multi-sensor rigs
- Deploying perception models on robot hardware (GPU, edge)
- Debugging perception failures (latency, dropped frames, misalignment)

## Sensor Landscape

### Sensor Types and Characteristics

```
Sensor Type        Output              Range       Rate     Best For
─────────────────────────────────────────────────────────────────────────
RGB Camera         (H,W,3) uint8       ∞           30-120Hz Object detection, tracking, visual servoing
Stereo Camera      (H,W,3)+(H,W,3)    0.3-20m     30-90Hz  Dense depth from passive stereo
Structured Light   (H,W) float + RGB   0.2-10m     30Hz     Indoor manipulation, short range
ToF Depth          (H,W) float + RGB   0.1-10m     30Hz     Indoor, medium range
LiDAR (spinning)   (N,3) or (N,4)     0.5-200m    10-20Hz  Outdoor navigation, mapping
LiDAR (solid-st.)  (N,3)              0.5-200m    10-30Hz  Automotive, outdoor
IMU                (6,) or (9,)        N/A         200-1kHz Orientation, motion estimation
Force/Torque       (6,) float          N/A         1kHz+    Contact detection, force control
Tactile            (H,W) or (N,3)      Contact     30-100Hz Grasp quality, texture
Event Camera       Events (x,y,t,p)    ∞           μs       High-speed tracking, HDR scenes
```

### Common Sensor Hardware

```
Device             Type               SDK/Driver           ROS2 Package
──────────────────────────────────────────────────────────────────────────
Intel RealSense    Structured Light   pyrealsense2         realsense2_camera
Stereolabs ZED     Stereo + IMU       pyzed                zed_wrapper
Luxonis OAK-D      Stereo + Neural    depthai              depthai_ros
FLIR/Basler        Industrial RGB     PySpin/pypylon       spinnaker_camera_driver
Velodyne           Spinning LiDAR     velodyne_driver      velodyne
Ouster             Spinning LiDAR     ouster-sdk           ros2_ouster
Livox              Solid-state LiDAR  livox_sdk            livox_ros2_driver
USB Webcam         RGB                OpenCV VideoCapture  usb_cam / v4l2_camera
```

## Camera Models and Calibration

### Pinhole Camera Model

```
                    3D World Point (X, Y, Z)
                           |
                    [R | t] — Extrinsic (world → camera)
                           |
                    Camera Point (Xc, Yc, Zc)
                           |
                    K — Intrinsic (camera → pixel)
                           |
                    Pixel (u, v)

K = [ fx   0   cx ]      fx, fy = focal lengths (pixels)
    [  0  fy   cy ]      cx, cy = principal point
    [  0   0    1 ]

Projection:  [u, v, 1]^T = K @ [R | t] @ [X, Y, Z, 1]^T
```

### Intrinsic Calibration

```python
import cv2
import numpy as np
from pathlib import Path

class IntrinsicCalibrator:
    """Camera intrinsic calibration using checkerboard pattern"""

    def __init__(self, board_size=(9, 6), square_size_m=0.025):
        self.board_size = board_size
        self.square_size = square_size_m

        # Prepare object points (3D coordinates of checkerboard corners)
        self.objp = np.zeros((board_size[0] * board_size[1], 3), np.float32)
        self.objp[:, :2] = np.mgrid[
            0:board_size[0], 0:board_size[1]
        ].T.reshape(-1, 2) * square_size_m

    def collect_calibration_images(self, camera, num_images=30,
                                    min_coverage=0.6):
        """Collect calibration images with good spatial coverage.

        IMPORTANT: Move the board to cover all regions of the image,
        including corners and edges. Tilt the board at various angles.
        Bad coverage = bad calibration, especially at image edges.
        """
        obj_points = []
        img_points = []
        coverage_map = np.zeros((4, 4), dtype=int)  # Track board positions

        while len(obj_points) < num_images:
            frame = camera.capture()
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            found, corners = cv2.findChessboardCorners(
                gray, self.board_size,
                cv2.CALIB_CB_ADAPTIVE_THRESH |
                cv2.CALIB_CB_NORMALIZE_IMAGE |
                cv2.CALIB_CB_FAST_CHECK
            )

            if found:
                # Sub-pixel refinement — critical for accuracy
                criteria = (cv2.TERM_CRITERIA_EPS +
                           cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
                corners = cv2.cornerSubPix(
                    gray, corners, (11, 11), (-1, -1), criteria)

                # Track coverage
                center = corners.mean(axis=0).flatten()
                grid_x = int(center[0] / gray.shape[1] * 4)
                grid_y = int(center[1] / gray.shape[0] * 4)
                grid_x = min(grid_x, 3)
                grid_y = min(grid_y, 3)
                coverage_map[grid_y, grid_x] += 1

                obj_points.append(self.objp)
                img_points.append(corners)

        coverage = (coverage_map > 0).sum() / coverage_map.size
        if coverage < min_coverage:
            print(f"WARNING: Only {coverage:.0%} coverage. "
                  f"Move board to uncovered regions.")

        return obj_points, img_points, gray.shape[::-1]

    def calibrate(self, obj_points, img_points, image_size):
        """Run calibration and return camera matrix + distortion coeffs"""
        ret, K, dist, rvecs, tvecs = cv2.calibrateCamera(
            obj_points, img_points, image_size, None, None)

        if ret > 1.0:
            print(f"WARNING: High reprojection error ({ret:.3f} px). "
                  f"Check image quality and board detection.")

        # Compute per-image reprojection errors
        errors = []
        for i in range(len(obj_points)):
            projected, _ = cv2.projectPoints(
                obj_points[i], rvecs[i], tvecs[i], K, dist)
            err = cv2.norm(img_points[i], projected, cv2.NORM_L2)
            err /= len(projected)
            errors.append(err)

        print(f"Calibration complete:")
        print(f"  RMS reprojection error: {ret:.4f} px")
        print(f"  Per-image errors: mean={np.mean(errors):.4f}, "
              f"max={np.max(errors):.4f}")
        print(f"  Focal length: fx={K[0,0]:.1f}, fy={K[1,1]:.1f}")
        print(f"  Principal point: cx={K[0,2]:.1f}, cy={K[1,2]:.1f}")

        return CalibrationResult(
            camera_matrix=K, dist_coeffs=dist,
            rms_error=ret, image_size=image_size)

    def save(self, result, path):
        """Save calibration to YAML (OpenCV-compatible format)"""
        fs = cv2.FileStorage(str(path), cv2.FILE_STORAGE_WRITE)
        fs.write("camera_matrix", result.camera_
````

## File: README.md
upstream: https://github.com/arpitg1304/robotics-agent-skills/blob/main/README.md

````text
# 🤖 Robotics Agent Skills

Production-grade robotics knowledge for AI coding agents.

Drop these `SKILL.md` files into Claude Code, Autohand Code, Cursor, Copilot-style
agents, or custom agent frameworks to make them generate better ROS1/ROS2 software:
safer nodes, correct QoS, lifecycle patterns, tests, launch files, Docker, bringup,
perception, and security.

[![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-blue.svg)](LICENSE)
![Skills](https://img.shields.io/badge/skills-10-green)
![ROS](https://img.shields.io/badge/ROS1%20%2B%20ROS2-supported-orange)
![Agent Format](https://img.shields.io/badge/format-SKILL.md-purple)

## Why This Exists

General-purpose coding agents know ROS syntax, but they often miss the robotics details
that matter in production: QoS compatibility, deterministic startup, bounded callbacks,
sensor health checks, safe shutdown, launch ergonomics, and testability.

This repo is the missing robotics engineering layer for AI coding agents.

## Evidence: Same Prompt, With and Without Skills

The `evals/` directory compares the same ROS2 package prompt run once with no skills and
once with the robotics skills loaded as context.

| Result | Without Skills | With Skills |
|--------|----------------|-------------|
| Node architecture | Plain `rclpy.Node`; setup in constructor | Lifecycle node with configure/activate/deactivate/cleanup/shutdown |
| Sensor QoS | Default depth 10 for everything | Sensor QoS for camera/joints, reliable QoS for status |
| Sensor handling | Latest-message assignment | Thread-safe bounded buffers, drop counts, watchdogs |
| Timestamp sync | None | Sync offset checks and dropped-frame accounting |
| Tests | 0 lines | 601 lines |
| Structure | Single node module | Separate writer module and cleaner responsibilities |
| Config surface | 4 parameters | 13 parameters |

This is a single-task, single-run comparison, self-assessed, and neither package was
built against a real ROS2 distribution — treat it as an illustration of the design
decisions the skills push toward, not as a benchmark. Caveats and the known build
defects in both artifacts are documented in the report.

Full report: [evals/EVAL_REPORT.md](evals/EVAL_REPORT.md)

## Quick Start

Install selected skills into a project:

```bash
git clone https://github.com/arpitg1304/robotics-agent-skills.git
cd robotics-agent-skills
./install.sh --target /path/to/your/robot/.claude/skills --skills ros2 robotics-testing robot-bringup
```

For Autohand Code project skills:

```bash
./install.sh --target /path/to/your/robot/.autohand/skills --skills ros2 robotics-testing robot-bringup
```

Or copy manually:

```bash
cp -R skills/ros2 /path/to/your/robot/.claude/skills/
cp -R skills/robotics-testing /path/to/your/robot/.claude/skills/
```

## Recommended Bundles

| Bundle | Skills |
|--------|--------|
| ROS2 application | `ros2`, `robotics-software-principles`, `robotics-testing`, `robot-bringup` |
| Robot perception | `ros2`, `robot-perception`, `robotics-testing`, `docker-ros2-development` |
| Production robot | `ros2`, `robot-bringup`, `robotics-security`, `robotics-testing` |
| Web dashboard | `ros2`, `ros2-web-integration`, `robotics-security`, `robot-bringup` |
| ROS1 maintenance | `ros1`, `robotics-software-principles`, `robotics-testing` |

## Skills

| Skill | Description | Key Topics |
|-------|-------------|------------|
| [robotics-software-principles](skills/robotics-software-principles/SKILL.md) | Design principles | SOLID for robotics, fail-safe defaults, rate separation, composability, graceful degradation |
| [ros1](skills/ros1/SKILL.md) | ROS1 development | catkin, rospy, roscpp, nodelets, tf, actionlib, launch XML, migration |
| [ros2](skills/ros2/SKILL.md) | ROS2 development | rclpy, rclcpp, DDS, QoS, lifecycle nodes, components, Python launch |
| [robotics-design-patterns](skills/robotics-design-patterns/SKILL.md) | Architecture patterns | Behavior trees, FSMs, HAL, safety systems, sensor fusion, sim-to-real |
| [robot-perception](skills/robot-perception/SKILL.md) | Perception systems | Cameras, LiDAR, depth, calibration, point clouds, detection, tracking, sensor fusion |
| [robotics-testing](skills/robotics-testing/SKILL.md) | Testing strategies | pytest + ROS, launch_testing, mock hardware, golden files, CI/CD |
| [docker-ros2-development](skills/docker-ros2-development/SKILL.md) | Docker + ROS2 | Multi-stage Dockerfiles, compose, DDS across containers, GPU passthrough, devcontainers |
| [ros2-web-integration](skills/ros2-web-integration/SKILL.md) | Web integration | rosbridge, FastAPI/Flask bridges, WebSocket streaming, REST APIs, MJPEG/WebRTC, security |
| [robot-bringup](skills/robot-bringup/SKILL.md) | System bringup | systemd services, launch composition, udev rules, watchdogs, log rotation, graceful shutdown |
| [robotics-security](skills/robotics-security/SKILL.md) | Security and hardening | SROS2, DDS encryption, network segmentation, secrets management, e-stop isolation, secure boot |

## How Agents Use These Skills

### Claude Code

Copy or symlink the skills you need into your project's `.claude/skills/` directory.
Claude Code auto-discovers `SKILL.md` files and triggers them based on the YAML
`description` field.

### Autohand Code

Copy or symlink the skills you need into your project's `.autohand/skills/`
directory. User-level skills live under `~/.autohand/skills/`; project-level
skills live under `<project>/.autohand/skills/`.

### Claude Projects

Place the skill directories in your project's `/mnt/skills/user/` directory.

### Custom Agent Frameworks

Load the relevant `SKILL.md` as system-prompt context when a task matches the skill:

```python
from pathlib import Path

SKILLS = {
    "ros2": "skills/ros2/SKILL.md",
    "testing": "skills/robotics-testing/SKILL.md",
    "perception": "skills/robot-perception/SKILL.md",
    "bringup": "skills/robot-bringup/SKILL.md",
    "security": "skills/robotics-security/SKILL.md",
}

def load_skill(task_description: str) -> str:
    text = task_description.lower()
    for trigger, path in SKILLS.items():
        if trigger in text:
            return Path(path).read_text()
    return ""
```

## What Good Skills Contain

- Specific trigger descriptions so agents know when to load them.
- Working examples, not just prose.
- Robotics failure modes: QoS mismatches, stale transforms, unsafe shutdown, blocking callbacks.
- Production defaults: bounded queues, lifecycle management, observability, tests.
- Anti-patterns that show what to avoid.

## Coverage Map

```text
Robot System Architecture
├── Design Principles ---- robotics-software-principles/ (SOLID, safety, composability)
├── Middleware ----------- ros1/, ros2/
├── Behaviors ----------- robotics-design-patterns/ (BT, FSM)
├── Perception ---------- robot-perception/ (cameras, LiDAR, depth, calibration, fusion)
├── Planning ------------ robotics-design-patterns/ (motion planning)
├── Control ------------- robotics-design-patterns/ (control loops)
├── Safety -------------- robotics-design-patterns/ (watchdogs, limits)
├── Testing ------------- robotics-testing/ (unit, integration, sim)
├── Containerization ---- docker-ros2-development/ (Dockerfiles, compose, DDS, GPU)
├── Web Interfaces ------ ros2-web-integration/ (REST, WebSocket, streaming, dashboards)
├── System Bringup ------ robot-bringup/ (systemd, udev, watchdogs, boot sequence)
├── Security ------------ robotics-security/ (SROS2, hardening, e-stop isolation)
└── Deployment ---------- ros2/ (production checklist, CI/CD)
```

## Roadmap

High-impact skills to add next:

- `robot-navigation/` - Nav2, SLAM, localization, costmaps, planners, recovery behaviors.
- `robot-manipulation/` - MoveIt2, grasp planning, planning scenes, TF, pick-and-place.
- `ros2-control/` - hardware interfaces, controllers, update loops, real-time pitfalls.
- `robot-simulation/` - Gazebo, Isaac Sim, MuJoCo, sim clock, resets, domain randomization.
- `robot-description/` - URDF, Xacro
````
