Topics

#ai-voice-agent #edge-ai-agents #edge-robotics-voice-assistant #kokora-tts #llama-fine-tune #local-llm-integration #local-voice-agent #nvidia-cuda #offline-ai #ollama-voice-agent #parakeet-model #python-3 #quantizations #rag #sentence-transformers-finetune #setfit-model #setfit-training #sherpa-onnx #speech-synthesis #voice-agent

Language breakdown

No language data available.

▸ Files click to expand

README

AXIOM - Advanced Voice Agent with Conversational Intelligence

AXIOM Mascot

Overview

AXIOM is a voice agent built for robotics / edge environments. It combines modern ML techniques with an efficient inference pipeline to deliver:

Instant Voice Interaction: Real-time speech processing with WebSocket communication
Intelligent Intent Classification: SetFit-based intent recognition using secure .safetensors (no pickle-based model head)
Context-Aware Responses: Semantic RAG + 2,116+ template responses
3D Interactive UI: WebGL-based carousel for visual equipment interaction
Multi-turn Conversation: FIFO history management for contextual understanding
Clean TTS Output: Phonetic + minimal safe correctors (e.g., 5m → 5 meters)

Quick Start

Prerequisites

Python: 3.10+
RAM: 8GB minimum (16GB recommended)
VRAM: 2-3.6GB for GPU acceleration (optional—CPU mode works too)
Disk: 1GB for models (Kokoro, Sherpa, SetFit)

Step 1: Clone & Setup

# Clone repository
git clone https://github.com/pheonix-delta/axiom-voice-agent.git
cd axiom-voice-agent
Create virtual environment (recommended name: axiomvenv)
python3 -m venv axiomvenv
source axiomvenv/bin/activate  # Linux/Mac
or
axiomvenv\Scripts\activate  # Windows
Install dependencies (avoid --break-system-packages; use the venv)
pip install -r requirements.txt

Step 2: Download Models (First Run Only)

Models are symlinked from your system. Verify they're accessible:

# Check symlinks
ls -la models/
Output should show:
kokoro-en-v019 -> ../../kokoro-en-v019
sherpa-onnx-... -> ../../sherpa-onnx-...
If symlinks are broken, set environment variables:
export KOKOROPATH=/path/to/kokoro-en-v019
export SHERPA_PATH=/path/to/sherpa-onnx-...

📖 See MODELPATH_RESOLUTION.md for complete setup options:

Environment variables (recommended)
Creating symlinks
Configuration files (.env)
Troubleshooting broken paths

Step 3: Start the Server

cd backend
python mainagentweb.py
Output:
INFO:     Application startup complete
INFO:     Uvicorn running on http://0.0.0.0:8000

Step 4: Open Browser

Navigate to:

http://localhost:8000

🎙️ Click the microphone icon and start speaking!

⚠️ Important: Use localhost or 127.0.0.1 (not IP addresses) for browser microphone permissions.

Live Demos

🖥️ Web Interface Screenshots

AXIOM Web Interface - Main View
Interactive carousel with equipment cards and voice agent

AXIOM Web Interface - Equipment Details
Detailed equipment specifications and 3D models

AXIOM Web Interface - Voice Interaction
Real-time voice interaction with visual feedback

Real Benchmarks (Measured)

Benchmark scripts and analysis live in benchmarks/.

Live terminal results for resource consumption

Performance Analysis
Component-level latency breakdown and system throughput metrics

Response Time Distribution
End-to-end response time analysis across intent categories

Charts

Latency Benchmarks

Detailed Performance Table

Terminal Demo

Terminal Demo Log - Cleaned excerpts showing key interactions
Asciinema Recording - Full terminal session recording

Architecture

┌─────────────────────┐
│  Browser (Web UI)   │
│  - Voice Capture    │
│  - 3D Visualization │
└──────────┬──────────┘
           │ WebSocket
           ↓
┌──────────────────────────────────────────┐
│         FastAPI Backend Server           │
├──────────────────────────────────────────┤
│ ┌─ STT Pipeline ─────────────────────┐  │
│ │ • Sherpa-ONNX Parakeet             │  │
│ │ • Silero VAD (Voice Detection)     │  │
│ │ • Phonetic + Minimal Safe Corrector│  │
│ └────────────────────────────────────┘  │
│ ┌─ Intent Classification ────────────┐  │
│ │ • SetFit Model (Local inference)   │  │
│ │ • Intent classes                   │  │
│ └────────────────────────────────────┘  │
│ ┌─ Response Pipeline ────────────────┐  │
│ │ • Template-based bypass            │  │
│ │ • Semantic RAG handler             │  │
│ │ • Ollama LLM fallback              │  │
│ └────────────────────────────────────┘  │
│ ┌─ TTS Engine ───────────────────────┐  │
│ │ • Kokoro TTS (Sherpa-ONNX)         │  │
│ │ • Sequential queue (no echo)       │  │
│ │ • TTS-safe text normalization      │  │
│ └────────────────────────────────────┘  │
└──────────────────────────────────────────┘
        ↓ (Data Persistence)
   SQLite Database
   (Conversation History)

System Architecture

High-Level Flow

┌──────────────────────────────────────────────────────────────────────────────┐
│ Browser (frontend/)                                                          │
│  - voice-carousel-integrated.html                                            │
│  - audio-capture-processor.js                                                │
└──────────────────────────────────┬───────────────────────────────────────────┘
                                   │ WebSocket (binary audio + JSON messages)
                                   ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ FastAPI Backend (backend/mainagentweb.py)                                  │
│                                                                              │
│  audio bytes                                                                 │
│     ▼                                                                        │
│  [VAD] backend/vad_handler.py                                                │
│     ▼                                                                        │
│  [STT] backend/stt_handler.py  → transcription                               │
│     ▼                                                                        │
│  [Intent] backend/intent_classifier.py → intent + confidence                 │
│     ▼                                                                        │
│  [Context] backend/conversation_manager.py (SQLite + FIFO)                   │
│     ▼                                                                        │
│  [Response]                                                                  │
│     - Fast path: backend/template_responses.py (2,116 templates)             │
│     - Smart path: backend/semanticraghandler.py (RAG) + backend/axiom_brain.py (LLM)
│     ▼                                                                        │
│  [TTS] backend/sequentialttshandler.py + text normalizers                  │
│     ▼                                                                        │
│  audio out → WebSocket → browser playback                                    │
└──────────────────────────────────────────────────────────────────────────────┘

Component Responsibilities

| Component | Purpose | Tech Stack | |---|---|---| | VAD Handler | Detect speech segments | Silero VAD | | STT Handler | Convert audio → text | Sherpa-ONNX | | Intent Classifier | Detect user intent | SetFit (sentence-transformers) | | Conversation Manager | Maintain context | SQLite + FIFO | | Template Responses | Fast replies | 2,116 JSON templates | | RAG Handler | Search knowledge bases | Sentence-Transformers embeddings | | LLM Interface | Complex queries | Ollama + local model | | TTS Handler | Generate speech | Kokoro-EN (Sherpa-ONNX) | | 3D Mapper | Map keywords → GLB models | Keyword extraction | | WebSocket Server | Real-time communication | FastAPI + uvicorn |

Response Quality (Unique Feature)

Phonetic Corrector: TTS-friendly conversion of units and domain terms

- Example: 5m → 5 meters, jetson nano → Jetson Nano

Minimal Safe Corrector: Removes markdown/noise without changing meaning

- Example: bold, italic, ` code ` → plain text

Template Bypass: Short, verified replies when confidence is high

- Saves GPU/LLM resources and improves latency

Project Structure

axiom-voice-agent/
├── backend/                      # FastAPI server + core voice pipeline
│   ├── mainagentweb.py         # App entrypoint (WebSocket server)
│   ├── vad_handler.py            # Voice activity detection
│   ├── stt_handler.py            # Speech-to-text
│   ├── intent_classifier.py      # Intent routing
│   ├── semanticraghandler.py   # RAG + fallback path
│   ├── template_responses.py     # 2,116 template responses
│   ├── sequentialttshandler.py # TTS queue + generation
│   └── ...
├── frontend/                     # Web UI
│   ├── voice-carousel-integrated.html
│   └── audio-capture-processor.js
├── data/                         # Knowledge bases + templates
│   ├── template_database.json
│   ├── ragknowledgebase.json
│   ├── projectideasrag.json
│   ├── inventory.json
│   └── carousel_mapping.json
├── assets/                       # Images, 3D models, benchmarks
│   ├── screenshots/
│   ├── branding/
│   ├── benchmarks/
│   └── 3d v2/
├── benchmarks/                   # Benchmark scripts + reports
├── demos/                        # Demo logs + recordings
├── docs/                         # Architecture + installation docs
├── models/                       # Symlink-based model directory
├── scripts/                      # Utilities (charts, safetensors tools, etc.)
├── requirements.txt
├── start.sh
└── README.md