Inside a Production Voice Architecture for Igbo, Yoruba, and Hausa

Building a real-time voice chatbot for African languages is not about swapping language codes in a cloud API. Hausa, Yoruba, and Igbo are severely underrepresented in every major commercial model — Whisper hallucinates on silence, NLLB mistranslates common Yoruba phrases, and off-the-shelf TTS has never heard Igbo spoken. This article walks through how we built and deployed a production pipeline serving all four languages (English, Hausa, Yoruba, Igbo), every component fine-tuned and running behind an OpenAI-compatible API.

The Architecture

**Five independent microservices, each owning one concern: **

Every service exposes an OpenAI-compatible HTTP endpoint. The chatbot calls them with the same openai Python client it would use for GPT-4, just with a different base_url. Swapping or scaling any component requires no application code changes.

Speech-to-Text: Fine-Tuned Whisper + Silero VAD

The base whisper-small-multilingual checkpoint produces 30–40% WER on Hausa without fine-tuning. After fine-tuning on approximately 50 hours of native speaker audio per language, WER drops below 12%. The fine-tuned model is loaded as a HuggingFace pipeline with 30-second chunking, so it handles audio of any length.

The more important problem is silence. In production, 15–25% of incoming audio is background noise, accidental activations, or dead air. Without filtering, Whisper hallucinates plausible text on all of it. The fix is running Silero VAD before the model sees anything: if no speech is detected, the request returns an empty transcription immediately. No inference, no hallucination. This single change eliminated the most common class of user-visible errors.

For language identity, Whisper's decoder is hard-forced to the correct language token for English, Hausa, and Yoruba using forced_decoder_ids. Without this, Whisper will silently switch to a language it recognizes better mid-output. Igbo is intentionally excluded from this forcing because it is not in Whisper's vocabulary — attempting to force it raises an error. The fine-tuned model handles Igbo freely without language tokens.

Transcription is streamed as Server-Sent Events, emitting each 30-second chunk as it completes. The client starts receiving text while the model is still processing the tail of a long audio clip.

Translation: NLLB with CTranslate2 Quantization

**We use Meta's NLLB-200 rather than an LLM for translation. **

LLMs translate well for high-resource languages but are unreliable and expensive for Hausa ↔ English or Igbo ↔ English at production throughput. NLLB was purpose-built for exactly these language pairs.

The challenge is latency. The unquantized PyTorch NLLB model runs at ~6 tokens/second on CPU — unusable. CTranslate2 is a C++ inference engine optimized for encoder-decoder models. Converting NLLB to CTranslate2 INT8 format is a one-time offline step:

ct2-transformers-converter \ --model facebook/nllb-200-distilled-600M \ --output_dir nllb-ct2 \ --quantization int8

The converted model runs at ~150ms per sentence on two CPU cores, making it viable without GPU. The CTranslate2 INT8 model is roughly 4x faster than its PyTorch equivalent at the same quality level. Separate model instances are deployed per language — one for Hausa, one for Igbo, one for Yoruba — each as an independent Docker container.

One non-obvious production requirement: translation direction must be explicit. Auto-detection of whether a short query is Hausa or English is unreliable, so the API contract enforces direction: "hausa_to_english" | "english_to_hausa" as a required field with no default. The caller always knows which direction it wants.

LLM Agent: vLLM with Semantic Caching

The core LLM is a fine-tuned model served via vLLM. The two vLLM features that matter most for this use case:

PagedAttention manages KV cache as virtual memory pages rather than pre-allocating contiguous blocks. This prevents OOM crashes under concurrent load — without it, a spike of 10 simultaneous users can exhaust GPU memory if their context lengths vary.

Continuous batching allows requests arriving mid-generation to join the current batch rather than waiting in a queue. Under moderate load, this roughly doubles effective throughput compared to static batching.

On top of vLLM, a Redis semantic cache sits in front of the agent. Before any inference, the incoming query is compared against cached query embeddings at a 0.95 cosine similarity threshold. In production, a large fraction of incoming queries are semantically identical rephrasings of the same underlying question, and they resolve to the same cached answer. This cuts LLM calls by 30–40% in production.

The agent microservice exposes two endpoints from the same LangGraph agent:

• /api/agent/chat — returns a complete JSON response, for integrations that need the full text before proceeding.

• /api/agent/stream — Server-Sent Events, one token at a time, for real-time frontend rendering.

After streaming completes, conversation logging and RAGAS quality evaluation are queued as background tasks without blocking the response.

One practical note: the X-Accel-Buffering: no response header is required when the service runs behind Nginx. Without it, Nginx buffers the entire SSE stream and releases it as a single batch — the client sees no streaming at all.

Text-to-Speech: Custom ChatterboxMultilingual

No commercial TTS service handles tonal Yoruba or Hausa with acceptable naturalness. The custom pipeline is built on ChatterboxTTS — a voice cloning architecture with a T3 acoustic transformer and S3Gen neural vocoder — fine-tuned on native speaker recordings for all four languages. Fourteen named voices are available, each cloned from a real speaker.

The Precision Split

The T3 transformer runs at bfloat16 on CUDA — this halves memory and increases throughput with negligible quality loss, because transformer attention is numerically stable at reduced precision.

The S3Gen vocoder must stay at float32. Vocoders use deep convolutional stacks with accumulated residuals. At bfloat16, numerical error in these accumulations produces audible artifacts ranging from high-frequency buzzing to complete output corruption. Test each submodel independently before assuming bfloat16 is safe across a composite architecture.

Pre-Warming Voice Conditionals

Voice identity in Chatterbox is determined by a speaker embedding (conditional) extracted from a reference audio clip. This extraction is expensive — running it per request would add 200–400ms of latency. All named voice conditionals are computed once at startup and cached in memory. At inference time, switching speakers is a single dictionary lookup.

Streaming WAV for Low Perceived Latency

Long responses are split at sentence boundaries and synthesized chunk-by-chunk. Audio starts streaming to the client as soon as the first chunk is ready. The HTTP response begins with a WAV header that advertises open-ended RIFF and data sizes (0xFFFFFFFF) — a standard trick for streaming PCM audio without knowing the total length upfront. Compatible with every major audio player and browser element.

The difference in perceived latency is significant: a 500-word response synthesized as a single block takes ~8 seconds before audio plays. Streamed chunk-by-chunk, the user hears the first sentence in ~1.2 seconds.

The NLLB services are the most cost-efficient part of the stack. INT8 quantization makes it possible to run three language-pair translators on CPU containers that cost a fraction of a GPU instance. Reserve GPU budget for Whisper and TTS, where latency is user-visible and quality directly depends on model capacity.

The TTS service serializes synthesis through an asyncio lock because the Chatterbox model mutates shared state (model.conds) during generation and is not concurrency-safe. For higher throughput, deploy multiple TTS replicas behind a load balancer rather than trying to parallelize within a single process.

Takeaways

None of these five services is exotic on its own — fine-tuned Whisper, CTranslate2, vLLM, and a voice-cloning TTS model are all individually well-documented. What makes this stack production-grade is the accumulation of small, hard-won decisions: gating STT with VAD, forcing translation direction

explicitly, splitting precision at the submodel boundary rather than the pipeline boundary, and separating evaluation from storage. Each of these was a real incident before it was a design rule. For teams building voice interfaces for underrepresented languages, the lesson generalizes past this specific tech stack: budget time for the boring infrastructure problems, not just the model quality problems — they're usually the ones that actually break in production.