AI Story Companion Ecosystem — Hybrid Edge-Cloud Architecture across 5 modular SKUs with magnetic pogo-pin connector interface. v4 updates: Physical Input Architecture (GPIO interrupt format, all controls BACK face + USB-C), Projection Mapping Software Pipeline with 6-step Calibration Engine, and Education Content System integration.
The AI Story Companion platform uses a hybrid edge-cloud architecture. The AI Audio Cube runs lightweight local inference, audio processing, physical input handling, and camera-based projection mapping on-device (edge), while complex AI tasks — story generation, illustration, music, projection scenes, education content — are offloaded to cloud services. All controls (4 buttons, privacy slide, USB-C) are on the BACK face (parent-facing) of the Cube.
All physical controls are mounted on the BACK face of the AI Audio Cube. A single button_handler.py daemon manages all GPIO interrupts in an async event loop. Events are published to an internal message bus consumed by the Audio Pipeline, Camera Service, and Privacy Monitor.
| Control | GPIO (BCM) | Trigger | Handler | Event Published |
|---|---|---|---|---|
| Power Button (BACK face) | GPIO 17 | Falling edge, pull-up; debounce 50 ms | button_handler.py | button_event{id:"power", type:"press"|"long_press"} → power manager |
| Volume Up (BACK face) | GPIO 27 | Falling edge, pull-up; debounce 50 ms; auto-repeat 150 ms on hold | button_handler.py | button_event{id:"vol_up", type:"press"|"auto_repeat"} → audio pipeline |
| Volume Down (BACK face) | GPIO 22 | Falling edge, pull-up; double-press = mute | button_handler.py | button_event{id:"vol_dn", type:"press"|"auto_repeat"|"double"} → audio pipeline |
| Camera Capture (BACK face) | GPIO 5 | Falling edge, pull-up; disabled if privacy slide CLOSED | button_handler.py → camera_svc.py | button_event{id:"cam", type:"press"} → camera_capture task |
| 🔒 Privacy Slide (BACK face) | GPIO 6 | Both edges, pull-up; CLOSED = GPIO LOW | privacy_monitor.py | slide_event{state:"open"|"closed"} → suspends camera pipeline <100 ms; logged to audit |
| ⚡ USB-C Charge State (BACK face) | I²C bus 1, addr 0x6B | PMIC polled 1 Hz (TI BQ25895); INT pin GPIO 4 | pmic_monitor.py | charge_event{status:"charging"|"full"|"disconnected", pct:N} → LED ring + cloud heartbeat |
pmic_monitor.py service tracks charge state and adjusts LED ring breathing animation to reflect it.All 5 SKUs share a common magnetic pogo-pin connector interface. The connector carries a defined set of signals that enable power delivery, high-speed video, module identification, and synchronisation.
| Signal | Protocol | Spec | Purpose |
|---|---|---|---|
| Power | DC Rail | 12V / 3A (36W max) | Powers projection modules via Cube/Dock pass-through |
| Video | HDMI 2.0 | 4K@30fps / 1080p@60fps | Carries projection content from Cube Projection Compositor |
| Module Bus | I2C 400kHz | 7-bit addressing | Module Manager ↔ module: config, status, dim command (brightness control) |
| Frame Sync | GPIO | 5V logic, <1ms jitter | Synchronises frames across 4-Sides Module projectors |
| Module ID | 1-Wire | DS2401 64-bit ROM | Unique module identification; auto-detected in <3s on attach |
| Ground | — | Shared GND plane | Power + signal return |
The Calibration Engine is an on-device Python service that runs a 6-step projection mapping pipeline for each attached projection module. It executes automatically on boot and on any module hot-plug event. Projection begins in fallback (uncorrected) mode and transitions to calibrated mode once the pipeline completes (<15 s per module).
| Step | Process | Algorithm / API | Input | Output | Time Budget |
|---|---|---|---|---|---|
| 1 | Module Detection | I2C 1-Wire ID ROM read (DS2401); Module Manager triggers calibration event | Hot-plug or boot event | Module type enum (Dome / 1-Proj / 4-Sides) | <500 ms |
| 2 | Camera Initialisation | libcamera CameraManager.get() + configure(); CSI-2 stream start | Module type → camera model (OV5647) | Active camera handle; raw frame pipeline open | <1 s |
| 3 | Structured-Light Scan | Project known binary pattern (or checkerboard) via HDMI → capture with module camera; OpenCV findChessboardCorners() | Camera frames + projected pattern | 2D feature point correspondences | <5 s |
| 4 | Distortion Computation | Dome: cv2.calibrateCamera() fisheye model (Brown–Conrady); 1-Proj/4-Sides: cv2.findHomography() perspective model | Feature point correspondences | Camera matrix K; distortion coefficients D; homography matrix H | <3 s |
| 5 | LUT / Warp Map Generation | Dome: cv2.initUndistortRectifyMap(); 1-Proj/4-Sides: cv2.warpPerspective() precomputed map; stored to RAM + flash | Calibration parameters (K, D, H) | Per-pixel warp LUT (float32 map) | <2 s |
| 6 | Compositor Update | Pass warp LUT to OpenGL ES shader via uniform texture; Projection Compositor switches from fallback to calibrated render mode | Warp LUT | Calibrated projection active — geometry-corrected output | <500 ms |
The Projection Compositor (on-device, Custom OpenGL ES) selects a render configuration based on the module set reported by Module Manager. All warp maps from the Calibration Engine are applied as GPU texture lookups.
| Module Attached | Render Config | Warp Type | Resolution | Notes |
|---|---|---|---|---|
| Dome Projector | Fisheye equirectangular warp | Undistort LUT (Brown–Conrady) | 1080p@60fps | ≥95% dome ceiling coverage; obstacle mask applied |
| 1 Projector Module | Perspective warp (wall plane) | Homography H | 720p@60fps | Auto-dims on person detection <200ms; re-brightens on clear |
| 4 Sides Module | 4× independent surfaces, frame sync <1ms | Per-face homography ×4 | 4× 720p@60fps | HDMI 1→4 distribution IC; edge blending at seams; active fan cooling |
| No module / Cube only | Null renderer (audio-only mode) | — | — | LED ring animations only; Calibration Engine idle |
| # | Stage | Component | Location | Technology |
|---|---|---|---|---|
| 1 | Voice Input | Wake word + VAD | Edge (device) | TensorFlow Lite / Porcupine |
| 2 | Speech-to-Text | ASR Engine | Edge + Cloud fallback | Whisper.cpp (local small) / Google STT ($0.024/min) |
| 3 | Safety Check (input) | Content Filter | Cloud | LLM guard / custom classifier + education tier validator |
| 4 | Story Orchestrator + LLM | Narrative Manager & Story Generator | Cloud | LangChain + LangGraph + Claude 3 Sonnet / GPT-4o + module context + education tier context |
| 5 | Memory Retrieval | Vector Store | Cloud | Pinecone / pgvector + education progress store |
| 6 | Music Engine | Adaptive Audio | Hybrid | MusicGen / curated S3 library; tier-appropriate mood mapping |
| 7 | Illustration Engine | Scene Images | Cloud | DALL-E 3 / Stable Diffusion XL; education scene assets from Course CDN |
| 8 | Text-to-Speech | Voice Synthesis | Cloud + Edge cache | ElevenLabs Pro v2 ($0.12/1K chars) / local Piper TTS; rate-limited by age tier |
| 9 | Projection Renderer | Module-aware compositor + Calibration Engine warp | Edge (device) | Custom OpenGL ES + Calibration Engine LUT per module; person safety via 1-Proj camera |
| 10 | Output | Audio + Projection + LED + Physical Controls | Edge (device) | PulseAudio + HDMI over pogo-pin + LED ring + GPIO BACK face button feedback |
The AI Audio Cube runs a custom Linux-based OS (Buildroot or Yocto). v4 adds the Calibration Engine for projection mapping and splits the Camera layer into a Camera Service (libcamera, capture) and CV Pipeline (OpenCV + TFLite, projection mapping + person detection).
| Layer | Component | Notes |
|---|---|---|
| OS | Linux (Yocto/Buildroot) | Lightweight, minimal attack surface |
| Runtime | Python 3.11 + asyncio | Main application runtime |
| Audio | PulseAudio / ALSA | Microphone array + speaker management |
| Wake Word | Porcupine SDK (on-device) | Always-on low power mode |
| Local ASR | Whisper.cpp (tiny/base model) | Offline fallback STT |
| Local TTS | Piper / Coqui TTS | Cache common phrases offline |
| Physical Input Handler | button_handler.py + privacy_monitor.py + pmic_monitor.py | GPIO interrupt daemon: 4 buttons + privacy slide (all BACK face); USB-C charge state via I²C; publishes typed events to internal bus |
| Module Manager | Custom I2C daemon + 1-Wire reader | Auto-detect module via 1-Wire ID in <3s; manage hot-swap; expose module context to cloud; triggers Calibration Engine on detect |
| Calibration Engine | calibration_engine.py (Python + OpenCV) | 6-step projection mapping pipeline per module: detect → camera init → structured-light scan → distortion compute → LUT generate → compositor update; runs <15s non-blocking |
| Camera Service | camera_svc.py (libcamera + V4L2) | Manages all 7 cameras: Sony IMX477 (Cube, FRONT face) + 6× OV5647 (Dome, 1-Proj, 4-Sides); routes frames to Calibration Engine or CV Pipeline; disabled by privacy slide GPIO signal |
| CV Pipeline | cv_pipeline.py (TFLite MobileNet + OpenCV) | Person detection on 1-Proj (<200ms auto-dim via I2C dim cmd); fisheye correction (Dome, passes LUT to Calibration Engine); 4-surface keystone + edge blend (4-Sides) |
| Projection Compositor | Custom OpenGL ES renderer | Selects render config per module set; applies Calibration Engine warp LUT as GPU texture; drives HDMI over pogo-pin |
| Connectivity | NetworkManager + BlueZ | Wi-Fi 6 + BLE 5.3 management |
| OTA | Mender / SWUpdate | Secure firmware update for Cube + all attached modules in single session |
| Security | TPM 2.0 + LUKS encryption | Secure boot + storage encryption |
The cloud backend is a set of microservices on Kubernetes. v4 adds the Education Content Service for Adventure Course pack management and progress tracking.
| Service | Responsibility | Tech Stack |
|---|---|---|
| API Gateway | Auth, rate limiting, routing | Kong / AWS API Gateway |
| Auth Service | JWT, OAuth2, device registration, parental consent | Python/FastAPI + Keycloak |
| Story Orchestrator | Narrative state machine, LLM calls, module context ingestion, education tier context injection | Python/FastAPI + LangChain + LangGraph |
| Memory Service | Child profile, story history, education progress, vector search | PostgreSQL + pgvector / Pinecone |
| Content Safety | AI output moderation, age filter (input + output), education accuracy classifier | Python + custom LLM classifier |
| Projection Scene Service | Module-aware scene selection; narrative context → projection content; returns config per module set in <500ms | Python/FastAPI + Redis cache |
| Education Content Service | Adventure Course pack catalogue, curriculum metadata, progress tracking API, tier management, course pack CDN delivery | Python/FastAPI + S3 + CloudFront CDN |
| Music Service | Dynamic music selection and generation; tier-appropriate mood mapping | Python + MusicGen / S3 library |
| Illustration Service | Scene image generation; education scene assets served from Course CDN | Python + DALL-E 3 / SD API |
| TTS Service | Voice synthesis, caching; age-tier rate control (Seedlings ≤120 WPM) | Python + ElevenLabs Pro v2 / Azure TTS |
| OTA Service | Firmware update orchestration for all 5 SKUs | Go + Mender API |
| Parent App API | Profile mgmt, module status, education tier control, sleep reports, subscription mgmt | Python/FastAPI |
| Analytics Service | Usage tracking, module attach analytics, education progress analytics, sleep reports | Python + ClickHouse |
| Notification Service | Push notifications, email summaries, new course alerts | Python + FCM / SendGrid |
The platform handles sensitive data about children, requiring careful data architecture with privacy by design. Module connection events and education progress are logged for analytics but never associated with raw child behavioural data. All CV frames are on-device only.
| Data Type | Storage | Retention | Compliance |
|---|---|---|---|
| Child profiles, age tiers & preferences | PostgreSQL (encrypted) | Account lifetime | GDPR/COPPA |
| Story history & memory | pgvector + PostgreSQL | 12 months rolling | GDPR |
| Education progress (per subject / tier) | PostgreSQL + Redis cache | Account lifetime; exportable | GDPR/COPPA |
| Adventure Course packs (content assets) | S3 + CloudFront CDN + local eMMC (device) | Indefinite (versioned) | Internal + license terms |
| Voice recordings | Processed in-flight, NOT stored | Real-time only | COPPA compliant |
| Module connection events | ClickHouse (anonymised, device ID only) | 24 months | GDPR Art. 89 |
| Projection scene selections | Redis (short-lived cache) + ClickHouse aggregate | Real-time / 30 days aggregate | GDPR |
| Sleep motion data | Time-series DB (InfluxDB) | 30 days + aggregated | GDPR |
| Camera captures (Cube IMX477) | Processed in-flight → character injection; NOT stored raw | Thumbnail S3 30 days with consent | GDPR/COPPA |
| CV frames (module cameras) | On-device NPU processing only; NOT transmitted | Real-time only | COPPA compliant |
| Privacy slide state events | ClickHouse (anonymised, device ID + timestamp) | 90 days | GDPR Art. 89 |
| Button event logs (debug) | CloudWatch Logs | 7 days (debug tier) | Internal |
| Firmware binaries (all 5 SKUs) | S3 + CDN (versioned per SKU) | All versions retained | Internal |
| Auth tokens | Redis (short TTL) | 15 min access / 7d refresh | OWASP |
Given the child-facing nature of the product, security is a first-class concern at every layer. The module connector interface is isolated by design — I2C commands are validated and sandboxed; no module can issue arbitrary firmware commands.
Secure boot (TPM 2.0), encrypted storage (LUKS), signed firmware updates (per-SKU), no debug ports in production, certificate pinning for cloud comms. Module I2C commands validated by Module Manager whitelist.
mTLS for device-to-cloud, JWT with short expiry, rate limiting per device/account, OWASP Top 10 hardening. Module context passed as signed device attestation token.
Multi-layer: system prompt guardrails → LLM content classifier → education accuracy validator → age-appropriate filter (per tier) → parent override. Projection scene content and Adventure Course content screened independently.
No voice recording storage (COPPA), GDPR data subject rights API, parental consent flows, right-to-erasure implemented. Module analytics anonymised. Education progress data exportable and deletable on request.
All traffic TLS 1.3+, VPC isolation for services, WAF on API Gateway, DDoS protection via Cloudflare.
1-Wire ID read-only; I2C commands restricted to approved command set; no module can escalate to OS level; module firmware signed and verified before execution.
Mechanical privacy slide (BACK face) physically blocks Cube lens — GPIO 6 LOW simultaneously suspends camera pipeline <100ms, no software override possible. CV frames from module cameras processed on-device NPU only — never transmitted. Raw camera data never stored. Privacy slide state changes logged (device ID only, no child association). Camera LED reflects slide state in real time.
Adventure Course packs are signed and verified before installation; curriculum content screened by dedicated education safety classifier (factual accuracy + age appropriateness + curriculum alignment); parental controls enforced server-side — no client-side bypass possible.