One of the things I’ve noticed when talking to people about local voice assistants is that we tend to focus on the wrong metrics.
We compare model benchmarks. We debate which speech-to-text engine is more accurate. We argue over whether a particular language model is smarter than another. We obsess over tokens per second, quantizations, and GPU utilization.
Meanwhile, the people we’re supposedly building these systems for don’t care about any of that.
They care whether they ask a question and get an answer quickly enough that it feels natural.
That’s really the benchmark.
Not whether a model scores higher on some leaderboard or whether it can solve increasingly obscure reasoning problems. The question I’ve found myself coming back to over and over again is much simpler:
Can a local voice assistant feel as responsive as Alexa?
For a long time, I wasn’t sure the answer was yes.
The Difference Between a Demo and a Product
When I first started experimenting with local voice assistants, I was mostly interested in whether it could be done.
Could Home Assistant wake up when spoken to? Could it transcribe speech locally? Could a language model understand what I was asking and generate a useful response? Could it synthesize that response back into speech without relying on cloud services?
The answer to all of those questions turned out to be yes.
The harder question was whether anyone would actually want to use it.
There’s a big difference between something that impresses you during a demonstration and something that quietly becomes part of your daily routine.
If you ask a question and have enough time to wonder whether the assistant heard you, check your phone, or repeat yourself before getting a response, the experience starts to break down. It doesn’t matter how private it is or how impressive the underlying technology might be. It stops feeling conversational and starts feeling like waiting for a computer.
That realization fundamentally changed how I approached the problem.
I stopped asking whether local voice assistants were possible and started asking whether they were practical.
Every Millisecond Counts
One of the challenges with local voice is that there isn’t a single bottleneck.
You’re dealing with a chain of events, and each step contributes a little bit of latency.
The wake word has to be detected. Audio needs to be captured and processed. Speech has to be transcribed. The user’s intent has to be interpreted. The language model has to generate a response. Text-to-speech has to synthesize that response back into audio. Home Assistant integrations need to do whatever work they’ve been asked to perform.
None of those components seemed especially slow when I evaluated them individually.
Together, however, they created an experience that was noticeably slower than what people had come to expect from commercial voice assistants.
That was frustrating because it meant there wasn’t an obvious fix.
There wasn’t a single optimization that suddenly transformed the experience. Improving responsiveness became an exercise in identifying and eliminating dozens of small inefficiencies that compounded into something users could feel.
Rethinking What “Better” Means
At one point, I found myself making the same mistake I see throughout the broader AI community.
I assumed bigger automatically meant better.
Larger models. More context. More capabilities.
On paper, those decisions made sense. In practice, they often made the experience worse.
A voice assistant isn’t a research assistant.
Most of the time, people aren’t asking it to analyze contracts or generate detailed reports. They’re setting timers, asking simple questions, controlling lights, checking the weather, or requesting information they expect to receive immediately.
The difference between a one-second response and a four-second response matters far more than whether the model can solve a particularly difficult reasoning benchmark.
Once I accepted that, my priorities changed.
I stopped chasing the largest models I could fit into memory and started optimizing for responsiveness.
The Optimization Rabbit Hole
Of course, deciding that latency matters doesn’t simplify anything. It just changes what you’re optimizing for.
I experimented with different language models to see how much capability I could preserve while reducing response times. I compared quantizations and adjusted context sizes. I migrated away from Ollama after realizing that native llama.cpp offered a more direct path with less overhead for my particular use case. I spent time tuning GPU offloading, batch sizes, and thread counts. I disabled features that sounded useful but introduced delays that didn’t justify the trade-off.
None of those changes were dramatic in isolation.
Taken together, however, they fundamentally changed the experience.
That’s probably one of the biggest lessons I’ve learned from this project. User experience is often the result of accumulated decisions rather than breakthrough discoveries. Removing two hundred milliseconds here and another three hundred milliseconds there doesn’t sound particularly exciting until you realize you’ve cut response times in half.
The Family Benchmark
As much as I enjoy benchmarks and performance graphs, I’ve come to believe that the most meaningful test has nothing to do with technical metrics.
It’s whether other people voluntarily choose to use it.
Nobody else in my house cares about inference speeds.
Nobody is interested in discussions about CUDA versions, quantizations, or whether a newer model scored better on an evaluation suite.
They care whether the lights turn on when they ask. They care whether timers work. They care whether responses arrive quickly enough that using voice feels easier than pulling out a phone.
That’s a much harder standard to meet because it doesn’t reward technical achievement for its own sake.
It rewards convenience.
A system can be incredibly sophisticated and still fail if the experience introduces friction into everyday life.
Are We Finally Getting There?
I don’t think local voice assistants have completely caught up to the polish of Alexa or Google Assistant.
Those companies have spent years refining their products and collecting enormous amounts of usage data. Expecting hobbyist projects and open-source communities to instantly match that level of maturity was never realistic.
What has surprised me is how much the gap has narrowed.
Speech recognition has improved dramatically. Open models have become faster and more capable. Consumer hardware that would have seemed excessive a few years ago is now relatively accessible. Projects like Home Assistant have made integrating these components significantly easier than they once were.
For the first time, I can imagine recommending a local voice assistant to someone who isn’t specifically looking for a hobby project.
Not because it’s perfect.
Because it’s becoming practical.
Why I Keep Chasing It
If this were only about saving money on subscriptions, I probably would have given up a long time ago.
For me, the appeal of local voice has always been about control. I like knowing that my automations continue functioning even if a cloud service experiences an outage. I appreciate having greater visibility into where my data goes. I enjoy the flexibility that comes with building something tailored to my own environment instead of adapting my life to fit someone else’s platform.
At the same time, I’ve learned that principles alone aren’t enough.
Privacy doesn’t excuse a frustrating experience.
Ownership doesn’t compensate for poor usability.
The people using these systems shouldn’t have to care about the philosophical reasons they exist. They should simply work well enough that the technology fades into the background.
That’s ultimately why I’ve spent so much time trying to shave milliseconds off response times.
The goal was never to beat Alexa on a benchmark chart.
The goal was to build something that my family and other hobbyists’ families would use without thinking twice about it.
After a lot of experimentation, more troubleshooting than I’d like to admit, and a growing appreciation for just how difficult voice interfaces really are, I finally think we’re approaching that point.
Local voice assistants don’t have to be perfect to be compelling.
They just have to feel natural enough that people stop noticing the technology and start focusing on the conversation.
