SXD — Distributed Robotics Data-Processing Platform

Overview

SXD is the data backbone for a robotics program: every robot run produces recordings that must be ingested, processed through multiple CPU- and GPU-bound stages, and delivered to different customers reliably. I built the backend and platform services that make that pipeline safe to run, safe to re-run, and easy to operate.

Idempotency as a design principle

The hard problem in a multi-stage pipeline is partial failure: a GPU stage dies halfway, a delivery gets interrupted, someone needs to reprocess last month's episodes with a new model version. I designed artifact processing to be keyed on deterministic identifiers — episode ID, stage, version, and an input fingerprint — so any stage can be re-executed at any time and converge to the same result without duplicating work or corrupting downstream outputs.

What I built

• Ingestion and orchestration services in FastAPI on Postgres, with Alembic-managed schema migrations.
• Multi-stage CPU/GPU workflow coordination with recoverable, resumable execution.
• Deterministic artifact identity (episode ID + stage + version + input fingerprint) enabling safe reruns and clean recovery from partial failures.
• Multi-customer delivery to S3-compatible object storage (Cloudflare R2).

Related tooling

Around the platform I also built VR/robotics capture tooling — a Quest 3 → Foxglove MCAP converter for visualizing capture sessions, and Quest head-pose streaming experiments — published on my GitHub.