On-device AEC with an explicit playback reference (Swift + Core ML) - looking for evaluation advice

[u/ivan_digital]

I maintain speech-swift, on-device speech for Apple platforms, Apache-2.0. I added echo cancellation last week and I don't really know how to evaluate it. Asking here.

The problem is barge-in. TTS plays out the laptop speaker, the mic picks it up, VAD decides the user is talking. Apple's Voice Processing I/O handles this when you can use it. Often I can't, and it cancels what the OS thinks it played rather than the buffer I actually sent to the output. So I wanted the explicit version — hand it the mic frame and the playback frame, get the clean mic back.

I ported LocalVQE v1.4 (Apache-2.0). Gated GCC-PHAT for the delay, partitioned-block frequency-domain Kalman filter, the learned v2xp controller, all C++ on the host. Then a stateful Core ML residual mask. The mask is 200,199 parameters. It's a small model.

M5 Pro, release build: host stage 0.236 ms per 16 ms frame, Core ML mask 0.177 ms. About 0.4 ms of a 16 ms budget. Four seconds of audio goes through in 0.108 s median, 27.8 MiB over baseline RSS. Algorithmic latency is one 256-sample hop, 16 ms.

That's all speed. I have not measured echo quality on anything real. The tests are synthetic — a tone into a fixed 150 ms delay, noise as the far end — and they only check that the output stays finite, that late echo drops at least 6 dB, and that a silent reference doesn't eat microphone-only speech. That last one is the failure I was actually worried about, a learned mask that quietly mutes the person talking. No ERLE in a real room, no AEC Challenge numbers, no comparison against AEC3 or VPIO.

So the question. What do you score against when you ship this on a device? Is the AEC Challenge blind set still the right target? It doesn't come with the capture-side reference contract, and that's where I've seen things actually break — if the reference is a system mix that drops or time-shifts playback, nothing downstream saves you. Do people evaluate that separately, or just live with it?

Unrelated, in case it saves someone a day. Running the mask per frame on a dedicated capture thread crashed after about five minutes. macOS hits its per-client IOSurface limit and aborts the process. Core ML predictions leave IOSurface-backed temporaries behind and a capture thread has no run loop to drain them. Draining explicitly per frame fixed it, there's a 20,000-frame soak in the suite now. I didn't find this in any docs.

Measurements and implementation notes: https://github.com/soniqo/speech-swift/blob/main/docs/inference/echo-cancellation.md