Your noise suppression is burning 15 CPU cores to do 1 ms of work

Many apps run their AI models on the CPU with ONNX Runtime, Microsoft's open-source inference engine. By default, its worker threads keep spinning between calls instead of going to sleep. It is documented and takes one line to fix, yet project after project finds it the hard way. On a laptop running live noise suppression, it kept fifteen cores busy for a model that needs a fraction of one.

The number

We ran two small, always-on audio models the way an app runs them: one call per audio frame, on the clock, for 30 seconds. GTCRN, a small noise-suppression model, cleans up speech in 16 ms frames. Silero VAD, a voice activity detector, decides whether someone is talking, in 32 ms chunks. Each needs about a millisecond of computing per frame. For comparison, we also ran both on OpenVINO, Intel's own inference engine, on the CPU and in its automatic mode, which picks the chip for you.

With ONNX Runtime's defaults, GTCRN spent 242 ms of CPU time per 16 ms frame. CPU time adds up every core, so that is fifteen cores held busy, continuously, for a job one core does in a sliver of its time. With one thread it spent 1.5 ms. The voice detector told the same story: 164 ms against 0.6 ms.

OpenVINO's automatic mode also missed 60 frame deadlines in one round, while it moved the voice detector to the GPU.

Power followed. With GTCRN, the laptop's processor drew 4.6 to 16 W more with the defaults than with one thread, depending on the run, and about 3 W more with the voice detector. That is the difference between a quiet laptop and a warm one with its fan on, for a feature meant to run all day in the background.

Why it happens

ONNX Runtime splits each call across a pool of worker threads, by default one per physical core: sixteen on this chip. Between pieces of work, the workers don't sleep. They spin, checking in a tight loop for the next task, because waking a sleeping thread costs time. ONNX Runtime's threading guide is open about the trade: spinning makes each call return sooner, at the cost of CPU time and power, and it is on by default.

On a server that calls a model back to back, that is a good default: the next task is microseconds away. In a live app the next task is a frame away. GTCRN needs about 1.5 ms of each 16 ms frame. For the other 14.5 ms, fifteen workers spin, 62 times a second.

Most benchmarks call the model back to back, so the default looks right in benchmarks and wrong in live apps like these.

Everyone finds it the hard way

This is no secret. The setting that turns spinning off has existed for years, and in 2022 ONNX Runtime added an option to stop spinning after each call, for apps that call a model only now and then. It is off by default. So the fix keeps being rediscovered, one project at a time, in bug reports and fixes like these:

• openpilot, comma.ai's open-source driver assistance: “100% CPU on onnxruntime threads caused by spin-waits”
• BirdNET-Go, live bird-song identification: “disable ONNX Runtime thread-pool spinning”
• opentrack, head tracking for games and simulators: “disable onnxruntime thread pool spinning”
• Wyoming faster-whisper, speech recognition for Home Assistant's voice assistant: “keep all --cpu-threads busy for seconds after a transcript”
• Cleanplex, a scanner for Plex video libraries: “stop onnxruntime spinning”
• Numen, a notes app with AI features: “sessions busy-wait between runs”

Most of these teams traced an odd CPU reading to the thread pool and patched their own code; at least one report is still open. The default stays, because for the workload it was designed and benchmarked for, nothing is broken.

The fix

For a model this small, one thread is enough and gives the lowest CPU time. For a larger model that needs the parallelism, keep the threads and stop the spinning:

import onnxruntime as ort

opts = ort.SessionOptions()

# Small streaming models: one thread.
opts.intra_op_num_threads = 1

# Larger models, instead: keep the threads, but let them sleep between calls.
# opts.add_session_config_entry("session.intra_op.allow_spinning", "0")

session = ort.InferenceSession("gtcrn.onnx", opts)

Which of the two is right depends on the model, the chip and the frame rate. The way to choose is the way the app runs: call the model at its frame rate and measure CPU time, not latency.

Measure at the frame rate

That lesson is bigger than one setting. Any tuner that times a model back to back tends to favor setups that are quick per call and expensive to keep idle, and a live app pays for that idle time on every frame.

So Kestrel, our tool that finds the best way to run a model on the hardware it runs on, has a real-time mode. You give it the frame period; it times every candidate at that cadence and picks the one with the least CPU per frame whose slowest calls still fit the frame. On both models it picks one thread by itself. An engineer who knows ONNX Runtime's thread pool gets the same answer. The point is that you shouldn't need to.

Running a model in a live app? Start a free pilot and we'll run Kestrel on it at your frame rate.

How we measured

An Intel Core Ultra 7 155H laptop (6 performance, 8 efficient and 2 low-power cores) on Linux, on AC power with the display off and a browser open in the background. ONNX Runtime 1.30, and OpenVINO 2025.4 with its latency hint. GTCRN (16 ms frames, gtcrn_simple.onnx from its repository) and Silero VAD v6 as shipped in faster-whisper 1.2.1 (32 ms chunks) carried their recurrent state from frame to frame.

Each setting ran in a fresh process for 30 seconds at the real frame rate, after 200 warm-up calls, in two rounds, the second in reverse order. CPU time is the process's total across all threads, divided by frames. Power is the processor package's RAPL counter.

The inputs were synthetic audio features, so the computing is realistic but audio quality was not scored. Package power drifts by a few watts between runs, so we give only the gap between the defaults and one thread, as a range. We measured one laptop on Linux.