bep40/gemma-avatar-copy

Gemma Avatar

Realtime voice chat with a 3D talking-head avatar. Same AI stack as the smolagents/hf-realtime-voice Space (blog post), but the orb visualization is replaced by a TalkingHead 3D avatar with real-time audio-driven lip-sync.

The pipeline

you speak → silero-VAD → parakeet-tdt-1.1b (STT) → gemma-4-31B-it on Cerebras → Qwen3-TTS → avatar speaks

Transport is the OpenAI Realtime GA protocol over WebSocket against Hugging Face's speech-to-speech backend: mic PCM16 @ 16 kHz goes up as input_audio_buffer.append, TTS PCM16 @ 16 kHz comes back as response.output_audio.delta, transcripts stream alongside.

How the avatar works

• Rendering / body language — TalkingHead (three.js). Blinking, breathing, idle sway, moods, hand gestures, and emoji expressions are its built-in animation system.
• Lip-sync — the backend sends raw PCM only (no word timings, no visemes), so the mouth is driven from the audio itself with HeadAudio: an AudioWorklet that classifies MFCC frames into Oculus visemes (~50 ms latency, fully in-browser). The s2s playback worklet is routed into TalkingHead's audio graph (audioAnalyzerNode → audioSpeechGainNode → reverb → speakers) and HeadAudio taps the speech gain node.
• The model plays the avatar — three function tools are declared to the backend: set_mood, make_hand_gesture, make_facial_expression. Gemma calls them mid-conversation (smiles when greeting, shrugs when unsure, thumbs-up when agreeing).
• Choreography — client statuses drive presence: the avatar makes eye contact when you start talking, gestures with its hands on new utterances, and barge-in clears the playback buffer so the mouth settles instantly.

Run it

bun install

# Pick a backend (one of):
LOAD_BALANCER_URL=https://…            bun run dev   # a speech-to-speech load balancer
SESSION_PROXY_URL=https://…/api        bun run dev   # piggyback another deployment's /api (dev)
bun run dev                                          # direct mode: paste a ws:// URL in Settings

Open http://localhost:3000 and tap Start talking.

?fakemic=1 starts a session with a silent synthetic mic (useful for testing the full loop without a microphone — trigger a reply from the console with getClient().requestResponse()).

Layout

index.ts                    Bun server: HTML import + /api/session proxy + static assets
index.html                  App shell (avatar hero, caption, subtitles, settings)
src/app.js                  Session wiring, tool executor, UI state
src/avatar.js               AvatarStage: TalkingHead + HeadAudio + choreography
src/s2s/s2s-ws-client.js    Realtime WS client (vendored from the Space; orb removed,
                            injectable output node + worklet base URL + shared-ctx close)
src/s2s/codec.js            PCM/base64 + transcript helpers (vendored, unchanged)
src/vendor/headaudio.min.mjs  HeadAudio node class (bundled)
public/worklets/            mic-capture + audio-playback AudioWorklets (vendored, unchanged)
public/vendor/              HeadAudio worklet processor + viseme model (runtime-loaded)
public/avatars/brunette.glb Default avatar (Ready Player Me; CC BY-NC 4.0 — non-commercial)

Notes

• The avatar GLB must have a Mixamo-compatible rig plus ARKit and Oculus-viseme blend shapes. Ready Player Me avatars work with ?morphTargets=ARKit,Oculus%20Visemes on the GLB URL.
• TalkingHead owns the AudioContext; the s2s client is handed head.audioCtx and never closes it.
• Everything animation-related runs on requestAnimationFrame — a backgrounded tab freezes the avatar (audio keeps playing).

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bep40/gemma-avatar-copy"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.