Kestrel benchmark results, November 2026

Kestrel finds the fastest way to run a model on the hardware it runs on, and only picks plans that keep the original model's quality. We measured it at both ends of the range: large language models served from a GPU, and small models running on a laptop.

Fastest LLM speed-up over a default setup

6.1×

Largest cut in GPU energy per token

86%

Faster than the best an engineer can build from public tools

25%

Faster on 36 models it had never seen

1.99×

LLM serving

We served seven open LLMs on one RTX 5090: a 4B distill of Qwen3.8, Llama-3.1-8B, Qwen3-8B, ToolACE-8B, the most downloaded chat fine-tune of Llama-3.1-8B, OpenEuroLLM's 9B, and Qwen3.8-27B in English and in Thai. Kestrel's setup ran 1.5–6.1× faster than the plain model on a default setup.

A default setup is an easy bar, so we set a harder one: the fastest setup an inference engineer can build today from public engines, public quantized checkpoints and public draft models, scored by the same rules. That is the best anyone can download. Kestrel beat it on five of the six models we compared, by up to 24.5% on Qwen3-8B, and matched it on the sixth: on the 27B in English the best public setup was already the fastest, and Kestrel picked it. On ToolACE-8B the comparison is the setup an AI agent found, below.

Each search ran unattended and took under an hour, from download to pick.

Energy

Kestrel used 47–86% less GPU energy per token than the default setup. Most of the saving comes from speed. The GPU draws about the same power, 460–510 W, however fast it runs, so finishing 1.5 to 6 times sooner cuts the energy behind each token nearly in proportion. The 9B, with the smallest speed-up, saved the least.

Against an AI agent

Why not have an AI agent do this? We tested it. A frontier OpenAI agent* got one GPU-hour to make ToolACE-8B as fast as it could, through a tool we built that served a setup on the GPU and timed it.

On the prompts it tuned against, the agent's setup came out ahead of Kestrel's. Then both were served afresh and timed on 24 prompts neither had seen, and the lead was gone: part of it had been fitted to the prompts the agent tuned on. On the new prompts, Kestrel's setup was faster, scored higher on GSM8K and used less energy per token. Kestrel got there in under an hour; the agent took 1 hour 43 minutes.

In a second run the agent started from Kestrel's setup. It spent 24 more trials on it and handed it back unchanged. Put on top of Kestrel, the agent added nothing.

Under load

A server rarely answers one request at a time. On Qwen3-8B, Kestrel's pick stayed ahead of the best public setup at every load we tried: by 24% at one stream, 26% at four and 14% at sixteen, where it served 2,426 tokens per second.

On a rented RTX A6000

We also gave Kestrel Qwen3.8-27B on one rented RTX A6000, under a protocol written down before any measurement, and no drafter. It picked its own int4 copy of the model with the model's built-in MTP drafting. On prompts nothing had seen, that ran 1.15–1.37× faster than the best vLLM setup, FP8 with MTP, and 1.68–2.59× faster than plain FP8, from 1 to 16 requests at once.

Accuracy held: on all 1,319 GSM8K questions, plain FP8 scored 95.22% and Kestrel's pick 95.15%, one question apart.

Models it had never seen

At the other end of the range are small models running on the chips inside a product. We drew models at random from Hugging Face under rules we fixed in advance, none of them used to build Kestrel, froze Kestrel's build and ran it once on an Intel Core Ultra 7 155H laptop.

Against the fastest of each engine's own automatic settings, Kestrel was 1.99× faster on average across 36 models (95% interval 1.59–2.50×). It won on 31, tied on two and lost on three, and was more than three times as fast on 14. Picking a plan took a median of 5 minutes per model.

On a phone

Kestrel's search ran an 8B LLM on a Snapdragon 7s Gen 3 phone at 1.2–2.5× the speed of llama.cpp's default threads, on four quantized files of a community Qwen3-8B fine-tune. It found that four threads on the four big cores run fastest, without being told the layout. Its own LSTM kernel ran a voice-activity detector's LSTM 1.15–1.78× faster than ONNX Runtime on four machines, from that phone to a laptop.

Against GPU tuners

On the RTX 5090 we compared Kestrel with the tools engineers use to tune models for NVIDIA GPUs, on 13 models from our own test set.

• Microsoft Olive. Kestrel was faster on all 10 models where Olive's tuned model kept its accuracy, by 1.20–3.66× and 1.9× on average. On the other three, Olive's result failed the accuracy check.
• TensorRT. On 11 models Kestrel found the same setting as the best of three hand-picked TensorRT settings, within measurement noise. On a voice-activity detector built around a recurrent layer, it ran 2.0× faster than TensorRT alone.
• TensorRT-LLM. On OpenEuroLLM's 9B, Kestrel's pick ran 1.56× TensorRT-LLM's bf16 and 1.18× its FP8.

Speed within an energy budget

Kestrel can pick the fastest plan, or the fastest plan within an energy budget. On the laptop, its fastest plan for a voice-activity detector ran 2.7× faster than OpenVINO's CPU plan at half the energy per call. Capped at 340 millijoules a call, it moved the model to the NPU instead.

Faster at half the energy, with no cap

2.7×

Faster on 39% of the energy, capped at 340 mJ a call

2.2×

Accuracy

Kestrel only picks a plan that passes the same accuracy check as its baseline: within 1 point of the plain model's task accuracy for an LLM, and within 1% output error for every other model, on inputs not used to choose the plan. On six of the seven LLMs, accuracy went up.

How we measured

LLMs. One NVIDIA RTX 5090 with 32 GB, using public serving engines including SGLang and vLLM. One stream unless stated, 512 new tokens, timed on 24 prompts not used to choose the setup (for the 4B and the 9B, the 24 prompts their searches used). Accuracy is GSM8K, all 1,319 questions, or MGSM Thai, with greedy decoding. GPU energy is the board's power reading over each timed run. The default setup is the plain model served by SGLang in bf16. The best public setup is the fastest we could assemble from public engines, public FP8 checkpoints and public draft models, scored by the same rules.

The RTX A6000. One rented RTX A6000 with 48 GB, running vLLM only, so the base there is the best vLLM setup. Final timing on 24 prompts the search had not seen, up to 512 tokens, three rounds in shuffled order, median.

The agent. It worked without network access or Kestrel's code, through one tool that served a setup on the GPU and timed it. Both final setups were timed afresh on 24 prompts neither side had seen, in two rounds of alternating order.

Unseen models. We drew 37 models and froze Kestrel's build before it saw them; 36 could be scored, as one model's own reference output was not finite. Automatic settings means the fastest of ONNX Runtime's defaults, OpenVINO's AUTO device and Olive's tuning, for each model. Each plan ran in a fresh process after a warm-up, over five timed rounds in shuffled order. A plan passes only within 1% output error on inputs not used to choose it, and any Kestrel failure scores as ONNX Runtime's CPU default. Intervals are 95% bootstrap intervals over models.

GPU tuners. TensorRT 10.16 with CUDA graphs, Olive's GPU tuning as it ships, and TensorRT-LLM 1.4.0rc1, on the same RTX 5090.

The phone. llama.cpp 0.7.0 under Termux, two rounds per setting in alternating order.

Try it on your model

Kestrel is available to companies today, starting with a free pilot. Choose a model you run, and we'll run Kestrel on it and show you the results. Start a free pilot.

* GPT 6.1 Sol on Max effort