Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe.

However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.

>The result, up front

It's very welcome, to start with this.

However, just from this heading, I knew with 90% certainty that the post was AI generated.

Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.

I will plug Willow for mac recording. IMO it's basically to me a "better than perfect transcription" as it cleans things up and is almost instant. I liked Superwhisper but switched to Willow as it was a big difference.

Its so good that I'm not sure that it's possible to get any better. Speech to text seems like basically a solved problem, if not now then definitely in 5 years. I don't know if any of these speech to text businesses will work in the long run, but for consumers they are great. My guess is the 2030 version of Apple's SpeechAnalyzer will be so good that nobody will need to use 3rd party software.

I took a swing at bringing this into Handy.computer if anybody's interested: https://github.com/cjpais/Handy/discussions/1031 . Looks like there has been past demand for someone to implement it, but no proposed PRs. This article was inspiring.

Vs Voxtral would be a better comparison. No other model, open or closed, has been able to hit such a low AER (Acronym Error Rate ;)) for my meeting transcripts. Seems to understand/infer all the technobabble I use at work. Never have to edit anything. Whisper was catastrophically bad.

> Apple's new SpeechAnalyzer is the most accurate on-device speech engine we tested. It beat every Whisper model we ship, including Whisper Small, on both the clean and the noisy half of LibriSpeech, while running roughly three times faster than Small.

Impressive. Apple said they improved the models in 27 didn’t they? It would be interesting to see the numbers the beta turns in.

Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?

Is this the new dictation engine that I'm not allowed to run on my 1-YEAR-OLD IPHONE 17 because it's not Pro?

I use Spokenly, offline-only mode with the Nvidia model. All local, totally free. Highly recommend

Any chance you can benchmark against whisper large and large v3 turbo? These run comfortably on older Macbooks and are still far more accurate in real life dictation compared to even the parakeet models( despite ASR leaderboards) with an RTF < 1.

I second that! Can you run your benchmarks against the iOS 27 beta?

Finally. I‘d be delighted though if they actually implemented language autodetection (like everywhere else) though. There’s little more frustrating in my day to day than having dictated half a page to find that it‘s complete gibberish because Apple forces you to select the right language first…

I run SuperWhisper on both my Mac (where is uses Whisper) and my iPhone (where it uses SpeechAnalyzer and have found that SA does indeed run faster and anecdotally more accurately. Super exciting!

For my current purposes, I need a speech-to-text model/API to also emit word-level timestamps - for now, that makes ElevenLabs's Scribe v2 the best multiplatform, multi-language choice though it does look like this SpeechAnalyzer API provides them (although only for English).

I'm always confused by these phrases:

> The new API cuts word error rate by 3.5 to 4x on the same audio: from 9.02% to 2.12% on clean speech

Shouldn't they have said "cuts error rate by 78%" or something?

Whisper large v3 turbo works fine locally on recent iphones, weird to keep it out of the comparison

Lots of comments are "you should compare against X and Y" - even better, just get the results on a standard benchmark, so you can compare against all,

https://huggingface.co/spaces/hf-audio/open_asr_leaderboard

Excited to get next years diarization models - ArgMax is pretty incredible but cost prohibitive (when compared to 0 price from iOS)

Was intrigued by the Inscribe product so I downloaded it to test it out. I uploaded 1 voice file about 4 minutes long and was promptly given a message that I've reached the free plan limits.

Kind of a bait and switch. How can we test the product with such short time limits and what, exactly are you offering if all the processing is done on device by Apple?

Obligatory mention of Handy which is open-source and cross-platform:

https://github.com/cjpais/Handy

It's great for me when configured to use Parakeet v3.

Every single asr model I tested so far did not support timestamps properly though. Some use external aligner to create timestamp, but the accuracy is still much inferior than whipser in case the audio is noisy.

Cloud is so cheap and quick. I use local too but my api bill is like 3 quid a month. You would have to be very cheap or have compliance needs to tolerate the error gap

I wish Apple would update there built in TTS model (on Mac). In this day and age, it's quality is terrible.

If you are running ios 27 beta is this the model when you hit voice to text on the native keyboard?

I'm looking for good and cheap transcription + speaker diarization on my Mac for a small personal project. Recs?

i wish they had benchmarked it against parakeet-unified-en-0.6b and cohere-transcribe-03-2026. i am using parakeet with https://handy.computer daily and it's amazing.

I make an iOS app that uses this API heavily for transcribing diverse audio of varying bitrate and recording quality. The audio often contains music, multiple speakers, sound effects. SpeechAnalyzer almost always gets it.

It can struggle with proper nouns but will return something phonetically similar.

My main gripe is that it requires a separate model download per language. I understand the why they did this (to save disk space). But it makes multi-lingual audio hard to transcribe unless you know ahead of time the languages in the audio.

As an app developer the biggest win from using Apple's model is I don't have to bundle it in my app so my app looks much smaller. If a user has many transcription apps each one could have their own model. If Apple's model is used only one copy is needed.

Would this end up replacing the default iOS keyboard dictation functionality in iOS 27?

Yeah i do find apple's speech to text very good lately and no need to use openai or anything that seem to market their services better

I stopped reading after seeing they compared only with Whisper Small, Base, Tiny

This is useless test and benchmark when you have these day Whisper-V3-Large and Whisper V3-Turbo that you can faster than realtime on 5 years old macbook on apple sillicon (ANE). They didn't even compared to parakeet v2 or parakeet v3. And only english language...

If you want to use this on the CLI: https://github.com/finnvoor/yap.

Supports SRT/TXT/VTT or JSON-with-optional-word-level-timestamps output and progress meter.

Also it can transcribe live system audio.

OK, but how does it compare against Parakeet TDT2 (english), TDT3 (many languages), and Parakeet TDT3 Streaming? And what about whisper large?

This is great marketing, I had no idea what inscribe was, but a blog like this going viral did something no ad could do for me.

Anyone know the best choice these days specifically for speaker diarization?

this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.

Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15.

Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking all the extra layers of privacy and security might be slowing them down?

Overall, Apple who is suing Open AI should just buy them and let me have the best conversational AI out there baked into my old ass iPhone. Because as so far the new Siri on my old phone (tho again GPT works great talking to it and for years) doesnt come close. It's the same old "Could you try that again," Siri. BOO!!!

Author here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro.

Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has been guesswork. Short version: the new API cuts WER 3.5-4x vs the old one (2.12% vs 9.02% on test-clean), and it also beat Whisper Small on both splits at about 3x the speed. The old API came in last on clean speech, behind even Whisper Tiny.

On "why should I trust a vendor benchmark": the Whisper column reproduces OpenAI's published LibriSpeech WERs within +0.11 to +0.42 on all six measurements (same corpus, same normalizer, same scorer for every engine), and the raw per-utterance transcripts are downloadable from the article if anyone wants to rescore with their own normalizer.

Limitations worth stating up front: English only, read speech rather than meeting audio, one machine. Precise per-engine timing isn't in the article yet because the accuracy runs shared the machine with a dev workload; WER is load-independent, timing isn't.

Two things that might interest people migrating: SFSpeechRecognizer sends audio to Apple's servers unless you set requiresOnDeviceRecognition, and with SpeechAnalyzer, finishing your input stream is not enough to end a session. If you never call finalizeAndFinishThroughEndOfInput(), the results sequence never terminates and your await hangs forever. I found that one because it was shipping in my own app.

Happy to answer questions about the harness or the normalizer.

Still nothing beats OpenAI's VTT. Anthropic's sucks and Apple's isn't even usable.

Edit: Getting downvoted by Apple fanboys for telling the truth is a badge of honor.

[flagged]

[flagged]

If this isn't open source/weights and can't run locally, I don't see how this is a replacement for Whisper or other open models, e.g. within Home Assistant.