Sauti ASR: Building Speech Recognition for Real Swahili Conversations
Swahili is one of the most widely spoken languages in Africa, yet speech recognition systems for it still break in exactly the environments where they matter most: real conversations, noisy phones, long recordings, regional accents, and code-switched speech.
That is exactly why we built Sauti ASR.
Sauti ASR is MsingiAI’s Swahili automatic speech recognition system: trained, evaluated, deployed on Modal, and now released publicly on Hugging Face in two forms:
Release | Purpose | Hugging Face |
|---|---|---|
| Official public release | |
| Research preview for the Omnilingual lane |
This is not a story about a perfect model.
It is a story about building a real Swahili ASR system that can be benchmarked, deployed, inspected, criticized, improved, and used on actual audio instead of carefully controlled demos.
Why Swahili ASR Matters
Speech technology becomes most useful exactly where typing becomes hardest.
That includes:
healthcare intake and triage
education
accessibility
customer support
search and archive of spoken content
voice interfaces for mobile-first populations
But building ASR for Swahili is harder than simply fine-tuning an English speech model and hoping it generalizes.
A real Swahili ASR system has to survive:
Kenyan and regional accent variation
Swahili-English code-switching
borrowed words and abbreviations
noisy and phone-quality audio
long uninterrupted recordings
domain-specific speech such as maternal health consultations
That is the bar we cared about.
A Different Philosophy for African ASR
We do not believe African speech systems should optimize only for leaderboard metrics on clean speech datasets.
Our focus is robustness under real conversational conditions:
code-switching
noisy recordings
long-form speech
domain vocabulary
clinical and conversational audio
regional speaking variation
Benchmarks matter.
But speech systems only become useful when they survive reality.
What Sauti ASR Is
Sauti ASR is not a single monolithic model.
It is a model family with two active tracks:
Track | Status | Base Model | Role |
|---|---|---|---|
Track A | Official release |
| Public production baseline |
Track B | Research preview |
| Experimental conversational ASR lane |
That split turned out to be the correct engineering decision.
Track A gives us the strongest overall benchmark profile today, the cleanest deployment path, and the best default experience for users.
Track B is more experimental, but it exposed something important during evaluation:
The best benchmark model is not always the most promising model for difficult real-world conversational audio.
The Official Release: Track A
Track A is built on top of microsoft/paza-whisper-large-v3-turbo and fine-tuned for Swahili transcription.
Its current held-out benchmark on 500 Kenyan Swahili test samples is:
Metric | Value |
|---|---|
WER | 13.72% |
CER | 3.88% |
Insertions | 98 |
Deletions | 168 |
Substitutions | 1160 |
Reference words | 10,395 |
Hypothesis words | 10,325 |
That benchmark matters because it gives Sauti ASR a measurable public baseline instead of relying on hand-picked demos.
Why Track A Is the Official Release
We made Track A the primary public release for three reasons:
Reason | Why it matters |
|---|---|
Strongest overall benchmark | Gives the clearest public quality anchor |
Stable deployment behavior | Better operational reliability |
Better first-user experience | More predictable transcription quality |
If someone asks, “What is Sauti ASR today?” Track A is the answer.
The Research Preview: Track B
Track B is our Omnilingual conversational ASR lane built around omniASR_LLM_300M_v2.
Its best current tuned checkpoint achieved:
Item | Value |
|---|---|
Checkpoint step | 250 |
Validation split |
|
Samples | 1000 |
Best WER | 15.1292 |
Later checkpoints actually became worse:
Step | WER |
|---|---|
250 | 15.1292 |
500 | 16.0951 |
750 | 15.7478 |
That taught us an important optimization lesson:
training stability mattered more than simply training longer.
The tuned Track B run still improved significantly over earlier baselines:
Comparison | Earlier Best | Tuned Best | Improvement |
|---|---|---|---|
Validation WER | 16.3555 | 15.1292 | -1.2263 |
Track B does not yet beat Track A on the headline benchmark.
But benchmarks alone did not tell the full story.
What Real Audio Taught Us
Speech systems become honest when they meet real recordings.
We tested Sauti ASR on both short clean clips and long-form conversational audio.
Short Swahili Audio
On a short 6.3-second Swahili clip from SautiTTS, Track A produced:
habari huu ni mfano wa sauti iliyotengenezwa na modeli yetu ya kiswahili
Clean, stable, and exactly what we expected from the public baseline.
Long-Form ANC Consultation Audio
We also tested the system on a 443-second antenatal care consultation recording.
Track A completed long-form transcription using chunked inference:
Item | Value |
|---|---|
Mode | chunked |
Segments | 18 |
Decode time | 12.243s |
Realtime factor | 0.028 |
But long conversational speech exposed weaknesses that benchmark datasets alone do not capture.
Code-switching, speaker drift, names, abbreviations, and clinical vocabulary all became significantly harder.
Interestingly, Track B — despite weaker benchmark numbers — sometimes showed more conversational awareness on difficult clinic-style audio.
That became one of the most important findings from this project:
The best benchmark model is not always the most promising model for the hardest real-world use case.
Where Current ASR Still Breaks
One of the biggest remaining problems is Swahili-English code-switching.
For example:
Audio Intent | Typical Failure |
|---|---|
“Nitumie kwa M-Pesa kesho morning” | English phrase corruption |
Clinical mixed-language speech | Vocabulary substitution |
Names and abbreviations | Phonetic drift |
Long conversational turns | Context fragmentation |
This is one reason we believe conversational African ASR requires more than standard benchmark optimization.
The problem is not only language recognition.
It is conversational robustness.
How We Made Long Audio Work
Long-form transcription is where many promising ASR systems quietly fail.
Sauti ASR currently handles long recordings by chunking audio into segments, transcribing them independently, and stitching the results back together.
Current Long-Form Pipeline
Approach | Benefit | Limitation |
|---|---|---|
Fixed chunking | Reliable and simple operationally | Context breaks across chunks |
Segment stitching | Enables long-form transcription today | Weak conversational continuity |
Fast Modal deployment | Good latency and serving performance | Not yet conversation-aware |
Track A currently uses chunked transcription inside the Modal service path.
Track B currently uses a simpler 25-second chunk-and-stitch research pipeline.
This works today, but it also makes clear where future improvements are needed.
The next frontier is not just better weights.
It is better long-context inference design.
Under the Hood
Sauti ASR is not only a model release.
It is also a systems engineering project.
The repository includes:
Component | Purpose |
|---|---|
| Persistent Modal FastAPI service |
| Train/eval/transcribe orchestration |
| Track A inference and decoding |
| Track B helpers and inference |
| Track A training |
| Track B training |
| Track A Hugging Face publishing |
| Track B preview publishing |
| Direct Modal → Hugging Face release flow |
We used Modal heavily for:
GPU training
checkpoint storage
evaluation
serving
release publishing
dataset preparation
That mattered because it let us build a reproducible workflow where large checkpoints do not need to be manually moved across local machines during training and deployment cycles.
A Better Way To Ship Models
One of the most practical improvements in this project had nothing to do with WER.
We built a direct release pipeline from Modal volumes to Hugging Face.
That means checkpoints can be evaluated, staged, and published directly from the infrastructure layer without downloading massive artifacts locally first.
That release path was used for both:
Operational tooling matters more than many teams realize.
A reproducible release pipeline is part of the product.
What We Learned
Every serious ASR project teaches you that error analysis matters more than vanity metrics.
Here are the biggest lessons from Sauti ASR so far:
Lesson | What we observed |
|---|---|
Long audio changes everything | Strong short-form models can become fragile on extended speech |
Clinical conversational audio is its own domain | ANC recordings exposed code-switching and vocabulary weaknesses |
Training stability matters | Encoder freeze strategy materially affected Track B quality |
Best checkpoint is not always the final checkpoint | Step 250 outperformed later runs |
Deployment infrastructure matters | Shipping models repeatedly requires operational discipline |
Where Sauti ASR Is Strong Today
Sauti ASR already provides several important capabilities:
Strength | Why it matters |
|---|---|
Public benchmark baseline | Clear measurable quality anchor |
Public Hugging Face releases | Researchers can inspect and reproduce |
Real deployment path | Not only a notebook demo |
Long-audio support | Can process real recordings |
Active research lane | Allows architectural experimentation |
That is a strong foundation for a Swahili ASR system.
Where We Still Need To Improve
There is still significant work ahead.
Current weaknesses include:
Gap | Why it matters |
|---|---|
Swahili-English code-switching | Extremely common in real speech |
Clinical vocabulary | Important for healthcare applications |
Names and abbreviations | Major source of substitutions |
Long-context continuity | Important for conversational coherence |
Better Omnilingual packaging | Needed for broader research adoption |
There is no value in pretending these are solved.
What Comes Next
Track A Roadmap
improve general Swahili quality
reduce substitutions on names and numeric expressions
strengthen long-form robustness
simplify and harden serving infrastructure
Track B Roadmap
extend encoder freeze duration
align LR decay with unfreeze scheduling
move beyond naive chunking
expand conversational and clinical speech coverage
test longer-context Omnilingual variants
Product Roadmap
improve upload ergonomics for large audio
add segment-level reporting
support downstream workflows such as search and structured extraction
Why We Are Excited
The difference now is that Sauti ASR exists as an inspectable system rather than a concept.
The models, checkpoints, deployment path, benchmarks, and tradeoffs are public.
That matters.
Because useful African language technology will not be built through polished demos alone.
It will be built by shipping systems, exposing limitations honestly, improving against real-world audio, and making the work reproducible.
Try Sauti ASR
Official Release
🤗 https://huggingface.co/msingiai/sauti-asr
Research Preview
🤗 https://huggingface.co/msingiai/sauti-asr-track-b-preview
Repository
https://github.com/Msingi-AI/sauti-asr
Final Thought
The goal was never to build a model that only performs well inside controlled demos.
The goal was to build Sauti ASR: a Swahili speech recognition system that can survive real conversations, real recordings, and real deployment constraints.
That is a much harder problem.
And that is exactly why we are building it.