Local Voice Assistants: Smarter, Faster, More Efficient

Last year, I wrote about accelerating speech-to-text for Home Assistant using Wyoming, Whisper, and TensorRT. The goal was simple:

Make local voice assistants fast enough that people would actually want to use them.

At the time, getting a fully local voice pipeline working well required compromises. You traded speed for privacy. Capability for efficiency. Accuracy for latency. Getting everything to cooperate often felt more like a science experiment than a polished product.

A year later, that equation is starting to change. The biggest surprise wasn’t a revolutionary new model or dramatically faster hardware; it was dozens of small improvements across every component of the voice stack. Today, my local assistant is faster, smarter, more memory-efficient, and capable of running on more hardware than ever before.

More importantly, it crossed an important threshold. It no longer feels like a project. It feels like an appliance.

Smarter Local LLMs: From Llama 3.2 to Gemma 4

When I originally started experimenting with local assistants, Llama 3.2 provided a solid balance between quality and performance. It worked, but there were still moments where responses felt repetitive, rigid, or just slightly less natural than their cloud counterparts.

Today, Gemma 4 has become my preferred choice. The difference is immediately noticeable. Responses feel more grounded. Context handling has improved. Every day interactions simply feel more conversational.

Even more impressive has been the arrival of Gemma 4’s Quantization-Aware Training (QAT) variants. The improvements weren’t just about better answers; they also changed the economics of running an entire voice stack on a single GPU.

Doing More With the Same GPU

One of the most interesting aspects of these improvements wasn’t speed. It was efficiency.

Recent improvements to my Wyoming Whisper TRT project introduced larger key-value (KV) caches to improve decoding performance and responsiveness. Those changes increased the GPU memory footprint of the speech recognition pipeline.

Historically, that would have forced a difficult decision.

Do you dedicate more VRAM to speech recognition?
Or preserve memory for the language model?

For many Home Assistant users running 8–12 GB GPUs, Jetsons, or repurposed gaming hardware, those trade-offs matter.

Fortunately, Gemma 4 QAT requires less VRAM than the models I had previously been running. The reduced memory footprint of the language model largely offset the increased memory requirements of the ASR improvements. The result wasn’t simply moving resources around. The entire stack improved simultaneously.

Better language understanding
Faster responses
Improved speech recognition performance
Lower perceived latency
Roughly the same overall GPU utilization

For local voice assistants, that’s a significant milestone. Software improvements are finally allowing us to do more with the same hardware.

Simplifying the LLM Layer

Another surprisingly impactful change had nothing to do with the model itself; it was how the model was being served. When I originally built this stack, Home Assistant Assist communicated with Ollama, which in turn served Gemma running on a shared RTX 4070 Ti. Ollama has played a huge role in making local LLMs accessible. For many users, it’s still an excellent choice.

But as I chased lower latency and more predictable behavior, I started digging into why my voice assistant occasionally felt sluggish. Sometimes it wasn’t just slow. It would pause for tens of seconds.

The Investigation

Initially, GPU contention seemed like the obvious culprit. The 4070 Ti wasn’t dedicated to Home Assistant. It was also handling:

Jellyfin
Tdarr
Whisper STT
GLaDOS TTS
Obico

Yet even after background workloads quieted down, latency remained.

Using Ollama, I observed:

Cold requests after VM or Ollama restarts were taking 30–75 seconds
Warm Home Assistant requests averaging 1.7–2.2 seconds
Direct Ollama API requests were taking 1.6–3.1 seconds
Home Assistant prompts averaging roughly 6,000–7,000 tokens

Reducing exposed entities improved response times modestly, confirming prompt size played a role. But it wasn’t the whole story.

After configuring:

OLLAMA_KEEP_ALIVE=-1
OLLAMA_CONTEXT_LENGTH=20224

model unloading between requests stopped.

Ollama’s underlying llama-server logs revealed respectable throughput:

Prompt evaluation around 5,700 tokens per second
Generation around 100 tokens per second

However, there still appeared to be approximately 500–700 milliseconds of overhead beyond the actual inference work. That doesn’t sound like much, until your goal is a voice assistant that responds in under a second or two.

Moving to Native llama.cpp

To eliminate additional abstraction layers, I deployed native llama.cpp alongside Ollama and eventually migrated Home Assistant to hass_local_openai_llm.

docker run -d \
  --name llama-cpp \
  --restart unless-stopped \
  --gpus all \
  -p 18080:8080 \
  -v /home/administrator/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/gemma-4-E4B-it-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --ctx-size 20224 \
  --n-gpu-layers 999 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

The results were eye-opening. Small benchmark requests dropped from:

1.6–3.1 seconds

to:

135–176 milliseconds

Typical Home Assistant requests improved dramatically as well. The first request after startup still incurred a penalty of roughly:

~1.4 seconds

while llama.cpp built the initial checkpoint for Home Assistant’s large context.

After that, prompt reuse took over. Subsequent requests were routinely completed in:

235 ms
306 ms
370 ms
496 ms

Longest Common Prefix (LCP) reuse worked exceptionally well:

sim_best ≈ 0.99
graphs reused > 700

Prompt evaluation for Home Assistant’s large contexts dropped from approximately:

824–933 ms

down to:

33–43 ms

once cached.

One Final Surprise: Gemma’s Thinking

Even after migrating to llama.cpp, occasional latency spikes remained. The culprit turned out to be Gemma itself. By default, Gemma’s reasoning budget was effectively unlimited:

reasoning-budget activated
budget=2147483647 tokens

Occasionally, the model would generate hundreds of internal reasoning tokens before responding. Examples included:

668 reasoning tokens → 8.0 seconds
190 reasoning tokens → 2.1 seconds

Disabling reasoning via:

enable_thinking=false

eliminated these spikes entirely. Responses became far more predictable. The biggest surprise wasn’t that llama.cpp was faster. It was discovering how much latency existed outside the model itself. The model wasn’t the bottleneck. Infrastructure was.

Wyoming Whisper TensorRT: Faster, Smarter, and More Honest

The largest set of improvements over the past year landed in Wyoming Whisper TensorRT. Ironically, the work didn’t begin with a grand redesign.

It started with a simple goal due to a feature request:

Add proper INT8 support.

That investigation quickly evolved into a complete, evidence-driven pass over the encode and decode pipeline.

INT8: Sometimes the Best Optimization Is Knowing It Doesn’t Help

TensorRT’s implicit INT8 mode sounded promising. In reality, TensorRT 10’s encoder-only implicit INT8 path is both optional and deprecated. Testing on Whisper Base using 300 LibriSpeech test-clean utterances on an RTX 3050 revealed that TensorRT selected exactly:

0 out of 105 layers for INT8 execution.

The result?

Identical WER
Identical VRAM usage
Longer build times

Instead of leaving it as a misleading option, the documentation now reflects reality. Sometimes the best optimization is understanding when something doesn’t actually improve performance.

Decode Performance Improved Dramatically

Most of the performance gains came from optimizations that apply regardless of decoder architecture.

These included:

Final-position projection optimizations
GPU-side mel spectrogram generation
Reduced host synchronization
Better cache management
Eliminating unnecessary CPU-GPU transitions

Measured on Whisper Base using TensorRT 10:

Approximately 0.124 seconds p95 decode latency
Roughly 128× realtime performance

Decoder Modes: Choose Speed or Memory

Different hardware benefits from different trade-offs. Rather than forcing a single approach, Whisper TensorRT now supports two decoder modes.

KV Decoder (Default)

Three TensorRT engines
Cross-attention KV precomputed once
Self-attention cache grows per token
Approximately 14% faster

Simple Decoder

Single TensorRT engine
Full recomputation
Roughly 200 MiB lower VRAM usage

Both modes exceed 100× real-time performance. For larger GPUs, KV mode provides the best responsiveness. For memory-constrained systems such as Jetsons, Simple mode provides an excellent alternative.

Better Reliability Through Testing

Some of the most important improvements weren’t performance-related at all. Testing uncovered subtle issues that only appeared outside controlled benchmarks.

Fixes included:

Downmixing stereo audio to mono
Automatic resampling to 16 kHz
Preventing malformed inputs from triggering multi-gigabyte mel allocations
TensorRT tracing compatibility fixes
Moving logs to stderr to avoid conflicts with Wyoming’s stdout transport

The project also gained:

End-to-end Wyoming tests
Audio normalization tests
GPU validation benchmarks
Layer precision reporting tools
Evaluation dataset preparation utilities
Strict linting and type validation

Fast systems are great. Reliable systems are what people actually use.

Wyoming GLaDOS: Making TTS Feel Instant

Speech synthesis saw just as much improvement. Some of the gains came from reducing perceived latency. Others came from challenging assumptions and validating whether existing optimizations were actually doing anything at all.

The Tacotron TensorRT Placebo

For quite some time, Wyoming GLaDOS exposed a Tacotron “TRT” mode. The problem?

Turns out it wasn’t actually accelerating Tacotron. Oops. I guess that happens when you are trying to both grasp these concepts and get coding assistance from LLMs.

Tacotron inference used a custom generate_jit() method rather than forward(), meaning torch.compile never intercepted the execution path used during real inference.
The compiled cache was saved before any compiled execution had occurred.
Subsequent startups loaded what appeared to be TensorRT caches but were actually just copies of the original TorchScript model.

In other words, Tacotron TensorRT support was effectively a placebo.

Fixing it required a different approach.

Rather than relying on torch.compile, the new implementation uses the same TorchScript TensorRT frontend that already worked successfully for the vocoder.
A scripted wrapper exposes generation through forward(), allowing TensorRT to accelerate the portions of Tacotron it can optimize while leaving the autoregressive decoder loop in TorchScript.
If TensorRT isn’t the right choice for a workload, the system gracefully falls back through progressively safer execution paths:
- TensorRT-accelerated generation
- Optimized TorchScript inference
- Standard TorchScript inference

Streaming Audio and Sentence Pipelining

Previously, text-to-speech behaved like a batch process. The entire utterance had to finish synthesizing before the first audio sample reached the client. Now, PCM audio streams back as soon as the vocoder produces it.

Long responses begin playing almost immediately.
Sentence pipelining pushes this even further.
While one sentence is being played, the next sentence is already being synthesized.
The GPU and network remain busy simultaneously.

The result is a voice assistant that feels dramatically more responsive.

Lower Overhead Everywhere

Additional improvements included:

Eliminating per-request torch.cuda.empty_cache() calls
Removing unnecessary synchronizations
Avoiding repeated RNN flattening
Consistent warm-up and inference dtypes
Preventing TensorRT re-specialization
Lazy loading base models when TRT engines already exist
Releasing unused models before warm-up
Thread-safe inference locks for TensorRT contexts
Proper waveform clamping before int16 conversion
Streaming vocoding with support for utterances larger than previous TensorRT limits

None of these changes sounds exciting individually.

Together, they make the assistant feel polished.

Expanding Beyond x86

Perhaps one of the most exciting developments has been expanding support beyond traditional desktop hardware. Thanks to Torch-TensorRT introducing ARM64 packages, Wyoming GLaDOS now has experimental ARM64 TensorRT support. At the same time, thanks to community contributions, Wyoming Whisper TensorRT gained support for JetPack 6.2.1 on NVIDIA Jetson devices!

That means increasingly capable local voice assistants running on:

Jetson Orin Nano
Jetson Orin NX
Jetson AGX Orin
Other ARM64 TensorRT platforms

The barrier to entry keeps falling. The hardware gets smaller. The software gets faster. The experience keeps improving.

The Open Source Advantage

One of the most rewarding parts of this journey has been seeing community feedback and contributions continually push these projects forward. What began as an effort to squeeze more performance out of Whisper evolved into something much larger.

The goal today is simple:

Make private, local voice assistants feel effortless.

The gap between cloud assistants and self-hosted alternatives continues to shrink.

Better models.
Lower latency.
Improved efficiency.
Broader hardware support.
More robust testing.

And perhaps most importantly, none of it requires sending your conversations to someone else’s servers. We’re not finished. There are still optimizations to explore, new models to evaluate, and additional platforms to support. But for the first time, local voice assistants no longer feel like a compromise. They feel ready.

Recent Improvements

If you’d like to dive deeper into the work behind these changes:

Final Thoughts

A year ago, the challenge was proving that a fully local voice assistant could be fast enough to use.

Today, the challenge is becoming much smaller: Convincing people they no longer need the cloud.

When responses arrive in under a second, speech recognition runs more than 100× faster than real-time, audio begins playing before synthesis has finished, and the entire stack runs comfortably on consumer hardware, the old trade-offs start to disappear.

The future of voice assistants isn’t just faster.

It’s private.

And increasingly, it’s local.