An open-source assistive wearable framework that processes visual and auditory streams in real-time, running local model inference on edge hardware to deliver semantic context through spatial audio synthesis — no cloud dependency, full on-device privacy.
$ soundsight init --edge-device rpi5 --model hermes-2-quantized
✓ Camera pipeline initialized [CSI-2 / 30fps]
✓ Audio ingestion active [I²S MEMS / 16kHz]
✓ Vision encoder loaded [SigLIP-Q4 / 41ms inf]
✓ Semantic fusion engine ready [CrossModal v2]
✓ TTS overlay stream active [Piper-ONNX / <80ms]
▸ Cognitive overlay active. Latency: 127ms end-to-end.System Architecture
A three-stage pipeline processes raw sensor data through edge inference and semantic fusion, producing context-aware audio descriptions in under 130ms — entirely on-device.
Dual-stream sensor fusion from CSI-2 camera modules and I²S MEMS microphone arrays. Hardware-accelerated preprocessing via DMA buffers with zero-copy frame passing to the inference engine.
Quantized vision-language models execute on NPU/GPU silicon via ONNX Runtime or llama.cpp with Vulkan compute. Cross-modal embeddings are fused through a lightweight attention bridge before semantic decoding.
Contextual descriptions are synthesized into natural speech via Piper-ONNX and spatialized using binaural HRTF rendering. Audio cues are prioritized by semantic relevance and delivered through bone-conduction transducers for ambient awareness.
Technology
Purpose-selected for edge performance, low-latency inference, and full offline operation.
Research
Developed in collaboration with the School of Computer Science at Technological University Dublin, SoundSight AI bridges applied machine learning research with real-world assistive technology.
Academic Advisor
School of Computer Science, Technological University Dublin. Research focus on embedded systems, assistive computing, and applied machine learning for accessibility. Supervising the theoretical framework and evaluation methodology for the SoundSight cognitive overlay pipeline.
Technological University Dublin
School of Computer Science — Established 2019
Progress
Core inference pipeline operational. Vision-language model integration stable on Raspberry Pi 5 with quantized ONNX exports. Audio synthesis pipeline in active optimization.
Comparative latency profiling across RPi 5, CM5, Jetson Orin Nano, and Khadas VIM4. Power consumption and thermal envelope characterization underway.
Wearable enclosure design finalization, IRB submission for user studies, and open-source SDK publication with reproducible benchmark suite.
Q1 2026 — Completed
Core architecture design, initial vision pipeline, FastAPI control plane
Q2 2026 — Completed
ONNX model quantization, CrossModal v2 attention bridge, TTS integration
Q3 2026 — In Progress
Hardware benchmarking, power optimization, wearable enclosure prototyping
Q4 2026 — Planned
Open-source SDK release, IRB user studies, academic paper submission