LLM Based Holographic Assistant

Shaping Perspectives Shifting Realities

About The Project

A virtual waifu / assistant that you can speak to through your mic and it'll speak back to you! Has many features such as:

• You can speak to her with a mic
• It can speak back to you
• It can recognize gestures and perform appropriate actions based on it
• Has dynamic voice controls
• Has short-term memory and long-term memory
• Can open apps
• Smarter than you
• Multilingual
• Can learn psychology
• Can recognize different people
• Can sing,dance do lot of other stuff.
• Can control your smart home like Alexa
• Dynamic animations

More features I'm planning to add soon in the Roadmap. Also, here's a summary of how it works for those of you who want to know:

First the speech which is obtained from the mic is first passed into the OpenAI Whisper model for detecting the language of the speech and then its passed on to the speech recognizer, the response is then sent to the client via a websocket, the transcribed text is then sent to gemini LLM, and the response from Jenna is printed to the unity console and appended to conversation_log.json, and finally, the response is spoken by our finedtuned Parler TTS with the expressions for the face of the character and voice and the animation will be dynamically generated for the character by our own custom animation model which we are currently working on. For the time being we are using preset animations.

Created Based on our publication https://ieeexplore.ieee.org/document/10576146

(back to top)

Built With

• Python
• Langchain 3.0
• Whisper
• SpeechRecognition
• PocketSphinx
• Parler TTS
• HyperDB
• Sentence Transformers
• Tuya Cloud IoT
• Unity 2024
• Unity C#
• Rust

(back to top)

Getting Started

Prerequisites

• Install Python 3.10.11 and set it as an environment variable in PATH For both Pythonscripting and Default python shell
• Install GIT
• Install CUDA 11.7 if you have an Nvidia GPU
• Install Visual Studio Community 2022 and select Desktop Development with C++ in the install options
• Install Unity
• Install FFMPEG
• Unity packages such as UniVRM,ULipSync,PythonScripting, Readspeaker TTS(Temporary)

How to run / Steps to reproduce

• Create a new project in unity with URP Settings

• Replace the assets folder with the one in this branch

• Open the Unity Scene from the scenes folder

• Insert the API keys into the .env files

• Also add the API Key in the LANGCHAIN_CONTROLLER as well

• Run the STT service using: python STT.py

• Launch the app by clicking on the play button

• Wait for the application to download the required models on the first start

• Say 'Start' to activate or 'Stop' to deactivate works like an OS

NOTE - IN THE FIRST START IT WILL CAUSE SOME LAG IE IT GETS STUCK AT 'CHECKING FOR UPDATES' THAT'S TOTALLY NORMAL AS IT'S LOADING THE INTENT CLASSIFIER MODEL

Compiled EXE version coming soon !!!

Roadmap

• [x] Long-term memory
• [x] Time and date awareness
• [ ] mixed reality integration
• [x] Gatebox-style hologram
• [x] Dynamic Voice like GPT4o
• [ ] Alexa-like smart home control
• [x] Multilingual
• [ ] Mobile version
• [x] Easier setup
• [ ] Compiling into one exe
• [x] Localized
• [ ] Home Control
• [ ] Outfit Change
• [ ] Add custom skills
• [ ] Music Player
• [ ] Different modes
• [ ] Face Recognition
• [ ] Dynamic Animations
• [X] OS like working
• [X] Wakeword detection
• [ ] Mood recognition
• [ ] Weather Forecast

(back to top)

License

Distributed under the GNU Affero General Public License v3.0 License. See LICENSE.txt for more information.

(back to top)

Contact and Socials

E-mail: support@neuralharbour.com
Project Link: https://github.com/NeuralHarbour/LLM-Based-3D-Avatar-Assistant/v2.0

(back to top)

Acknowledgments

• Choose an Open Source License
• AI Waifu Vtuber

(back to top)