PhonemeX Pro – Offline TTS, ASR, Lip-Sync, Avatar & Face Rig System

PhonemeX Pro Edition is a modular, fully offline Text-to-Speech, Speech Recognition, and character voice system built for Unity mobile, desktop, and web builds. It provides real-time, on-device speech synthesis, local speech-to-text transcription, and full access to phoneme timing data at runtime, enabling speech generation, lip-sync, facial animation, and voice-driven character systems. What’s New in Version 1.2.0 PhonemeX Pro has evolved into a complete bidirectional voice suite. Version 1.2.0 introduces a brand-new Speech Recognition (ASR) module powered by local Whisper models, and a zero-latency Audio-Reactive Lip-Sync system for fast, FFT-based mouth movements. The entire codebase has also been deeply modularized, separating Expressions from Lip-Sync to allow for perfectly layered facial animation without conflicts. Existing projects remain fully compatible. Core Systems Included - On-Device Text-to-Speech System A cross-platform, fully offline TTS engine that runs entirely inside Unity. Generates real-time audio along with detailed phoneme timing data for runtime logic and animation. - On-Device Speech Recognition System (New!) A robust ASR pipeline integrating multilingual Whisper models for native speech-to-text inference. Runs completely locally via Unity Sentis (Desktop/Mobile) and WebAssembly (WebGL). Includes a built-in Download Manager UI for easy model fetching. - Real-Time Lip-Sync System A dedicated lip-sync pipeline that schedules phoneme data, blends viseme weights, and drives facial motion in real time. Designed for accurate speech-to-animation synchronization. - Audio-Reactive Lip-Sync (New!) A high-speed, zero-latency alternative to phoneme lip-sync. Uses real-time FFT band analysis, Automatic Gain Control (AGC), and direct jaw bone rotation for instant audio-driven mouth movement directly from an audio source. - Modular Avatar System PhonemeX Pro includes a flexible avatar integration layer compatible with the AvatarX character set. The avatar system applies lip-sync and expression data to fully rigged characters using blendshapes or custom drivers, enabling speech-driven facial animation with minimal setup. The system is designed to work with a wide range of humanoid characters and animation setups, allowing developers to integrate PhonemeX into existing projects without being locked to a specific character format. - Face Rig & Expression System An optional facial rigging and expression framework for authoring, blending, and layering facial poses, emotions, and micro-expressions on top of lip-sync animation. Modular Design - Each system can be used independently depending on your project needs: - Use only the TTS system for speech playback - Combine TTS and lip-sync for talking characters - Integrate the avatar and face rig systems for full facial animation control Modules are loosely coupled and designed to be extended or replaced as your project grows. Key Features - Fully Offline, On-Device TTS – No internet connection or server required by default - Real-Time Phoneme Access – Phoneme timing data available at runtime - Cross-Platform Support – Android, iOS, WebGL, Windows, macOS, and Linux - Lightweight & Fast – Optimized for mobile and browser-based builds - Multi-Voice & Multi-Language ASR Support – Easily switch between models - Multi-Voice Support – Easily switch between supported voice models - Modular Runtime Architecture – Clean separation between voice, lip-sync, and animation systems - Optional online phoneme services can be integrated if desired, but are not required for core functionality. - Local AI Inference – Uses Unity’s official on-device AI inference packages - Demo Scenes Included – Examples showcasing TTS, lip-sync, avatar integration, and face rig workflows Supported Platforms - Android (ARM / ARM64) - iOS / iPadOS - WebGL - Windows (x86 / x64) - macOS - Linux Dependencies / Requirements - Unity 6 LTS or newer - On-device AI inference package: Unity AI