AI generated speech on this lab
What local speech AI can do in 2026, and how far the gliffy lab can go on hardware you already own. Snapshot from moneta 2026-09-11: dual Tesla T4, ACE-Step live on T4 #1, ComfyUI on T4 #0, no dedicated TTS service yet.
Speech is the easiest high-quality AI modality this lab can run. Narration, character VO, zero-shot cloning, and Whisper QC all fit a single 15 GB T4 with headroom. You already have music beds on music.gliffy.tv. What’s missing is a TTS service — not more GPUs. A full always-on voice agent (ASR + LLM + TTS resident together) wants careful host scheduling or a 24 GB CUDA upgrade; batch production does not.
Cloud demos collapse everything into one product. Locally these are different models with different VRAM and latency profiles.
| Lane | Job | Example models | VRAM ballpark |
|---|---|---|---|
| TTS (stock voices) | Text → speech from fixed speakers | Kokoro, Piper, Qwen3-TTS CustomVoice | <1–8 GB |
| Zero-shot cloning | ~3–10 s reference → matching voice | Chatterbox, F5-TTS, XTTS v2, VoxCPM, Qwen3-TTS Base | 4–10 GB |
| Expressive / LLM-TTS | Emotion tags, style, streaming | Orpheus 3B, IndexTTS2, CosyVoice3 | 8–16 GB |
| STT / ASR | Speech → text (captions, QC) | faster-whisper large-v3 | ~3–5 GB FP16 · ~3 GB INT8 |
| Voice conversion | Timbre swap keeping timing | RVC, So-VITS-SVC | 2–6 GB |
| Voice agent | Mic → ASR → LLM → TTS → speaker | Whisper + Ollama + Kokoro/Chatterbox | 12–24 GB combined |
| Singing / score | Music with optional vocals | ACE-Step 1.5 (already live) | ~6–12 GB DiT-only on T4 |
ACE-Step on moneta generates music (and sung vocals in a music context). It does not do dialogue TTS, zero-shot speech cloning, or realtime voice conversion. For spoken lines you still need a speech model. See ACE-Step music on moneta.
| Model | Best for | VRAM | License / notes | T4 fit? |
|---|---|---|---|---|
| Kokoro-82M | Fast narration, many built-in voices | ~2–3 GB (CPU OK) | Apache 2.0 | Yes — trivial |
| Piper | Edge / CPU / batch scripts | <1 GB | GPL-ish forks | Yes — CPU on moneta fine |
| Chatterbox | Best quality + commercial cloning (~5 s ref) | ~4–6 GB | MIT | Yes — preferred install |
| F5-TTS | Research-grade cloning | ~4–8 GB | Code MIT · weights often NC | Yes |
| XTTS v2 | 17-language cloning | ~4–6 GB | CPML non-commercial | Yes |
| VoxCPM 1.5 / 2 | Strong cloning · VoxCPM2 multilingual + voice design | ~6–8 GB | Open weights | Yes for inference |
| Qwen3-TTS 0.6B / 1.7B | Custom voices, voice design, long-form | ~4–8 GB | Open weights · CUDA-first | Caution — Turing/FP16 quirks reported |
| Orpheus 3B | Emotion tags, streaming | ~8–12 GB | Apache 2.0 | Yes alone; tight with ACE-Step |
| faster-whisper large-v3 | Captions + misread QC | ~3–5 GB | MIT | Yes |
| VibeVoice-class 7B | Heavy expressive speech LLM | ~12–20 GB | Varies | Tight / swap to glifftop 32 GB |
Sources: Local AI Master TTS roundup (2026), Demodokos GPU-for-audio guide (2026-06), faster-whisper docs, Orpheus / VoxCPM / Qwen3-TTS model cards. VRAM is inference, not training.
| Host | GPU | Speech role | Live now (2026-09-11) |
|---|---|---|---|
| moneta 10.0.0.10 | 2× Tesla T4 · 15 GB each · CUDA 13 / driver 580 | Best CUDA home for TTS + Whisper; parallel with Comfy/ACE-Step | ComfyUI :8189 · ACE-Step :7865 (~5.9 GB on T4 #1) · ffmpeg · 125 GB RAM · ~157 GB free disk |
| glifftop 10.3.119.177 | RX 7900 XTX · 32 GB ROCm | Script LLM; TTS when Ollama unloaded | Ollama often holds ~16–22 GB — unload before heavy audio |
| bigscreen 10.4.199.235 | RX 9070 XT · 16 GB ROCm | Draft TTS/music when Comfy idle | ROCm path works for many stacks; more friction than CUDA |
Same rule as ACE-Step: each process sees one 15 GB card unless you write multi-GPU code. Speech models are single-GPU. Plan residency: e.g. Chatterbox on T4 #0 when Comfy is idle, or keep ACE-Step on #1 and park TTS on #0 between image jobs.
| Workload | moneta (CUDA T4) | bigscreen (16 GB ROCm) | glifftop (32 GB ROCm) |
|---|---|---|---|
| Kokoro / Piper narration | Easy · even CPU | Easy | Easy |
| Chatterbox / F5 / XTTS clone | Best path (CUDA) | Yes when Comfy free | Yes after LLM unload |
| Qwen3-TTS / VoxCPM inference | VRAM OK · watch Turing FP16 issues on Qwen | Possible · ROCm lag | Best AMD home if CUDA fails |
| Orpheus 3B | Fits alone on a free T4 | Tight with other work | Comfortable after unload |
| Whisper large-v3 QC | Easy (faster-whisper INT8/FP16) | Easy | Easy |
| ACE-Step score (music) | Live DiT-only on T4 #1 | Alternate host | Alternate after unload |
| TTS + ACE-Step resident | Yes: TTS on #0, music on #1 | Kokoro+ACE only if light | Yes with room |
| Full voice agent (ASR+LLM+TTS) | ASR+TTS on moneta · LLM on glifftop over LAN | Possible with small LLM | All-in-one if Ollama stays small |
| LoRA / SFT voice fine-tune | Small LoRA only · slow on T4 | Possible · painful | Better · still prefer 24 GB CUDA |
Throughput on a T4 is typically multiples of realtime for Kokoro and solidly above realtime for Chatterbox-class models. Chunk by paragraph; keep a pronunciation dictionary for names.
One reference WAV per character. Chatterbox or F5-TTS on moneta between Comfy jobs. Score beds stay on ACE-Step (T4 #1). Mix in Resolve / ffmpeg.
Keep ASR + TTS on moneta CUDA; leave the heavy chat model on glifftop. Latency is network + LLM, not TTS. For always-on low latency, prefer Kokoro over Orpheus until you dedicate a card.
| # | Service | Why | GPU |
|---|---|---|---|
| 1 | Chatterbox-TTS-Server (or Gradio) | Best quality/commercial cloning; MIT; ~4–6 GB | T4 #0 when Comfy idle, or alternate port |
| 2 | Kokoro (sidecar) | Fast stock narration; tiny footprint; agent-friendly | CPU or leftover VRAM |
| 3 | faster-whisper large-v3 | Captions + catch TTS misreads before publish | ~3–5 GB; can share #0 |
| 4 | Optional: VoxCPM or F5-TTS | Second opinion clones / multilingual | Swap with Chatterbox |
| 5 | Keep ACE-Step as-is | Music beds already solved | T4 #1 DiT-only |
T4 #0: ComfyUI quality lane or Chatterbox + Whisper between image jobs. T4 #1: ACE-Step music (current). glifftop: scripts / dialogue polish via Ollama. bigscreen: overflow TTS if moneta is busy rendering.
For dialogue-heavy work the bottleneck is listening and fixing, not generation speed. A practical solo pipeline on this lab:
| Artifact | Feasible on current gear? | Notes |
|---|---|---|
| 30–60 min audiobook chapter / evening | Yes | Kokoro or Chatterbox batch; Whisper QC; human pass for names |
| Episode dialogue stems (15 min show) | Yes | ~100–200 lines; clone bank; ACE-Step for score |
| Game NPC bark packs | Yes | Ideal local use case; version the reference WAVs |
| Captions / podcast transcripts | Yes | faster-whisper on either T4 |
| Always-on low-latency voice agent | Partial | Works as split stack; dedicated 24 GB CUDA smoother |
| Train a custom celebrity-grade voice LoRA | Inference yes · train hard | Prefer 24 GB CUDA or rented trainer; keep ethics/legal in mind |
Lab probe 2026-09-11: nvidia-smi T4#0 585 MiB / T4#1 5915 MiB · ace-step.service active · comfyui.service active · no Chatterbox/Kokoro/Piper binaries found on moneta.