Kestrel

Makes AI models superfast, on less energy.

Photo: Thibaud Aronson, CC BY-SA 4.0, via iNaturalist, mirrored, sky extended

The fastest way to run your model, found for you.

The same model can run on many engines, at many precisions, on different chips. Kestrel tries them on your exact hardware and keeps the fastest plan that holds the original quality.

  • Engine

    The runtime that serves the model, from vLLM and SGLang to ONNX Runtime, OpenVINO and TensorRT.

  • Precision

    How many bits each number gets, wherever quality holds.

  • Device

    The GPU, the CPU, the integrated GPU or the NPU, chosen per model.

  • Decoding

    How an LLM produces its tokens, tuned for the model and its traffic.

Faster than the best an engineer can build.

Against a default setup, Kestrel served seven open LLMs up to 6× faster on up to 86% less energy. Against the best an inference engineer can build from public engines, quantized checkpoints and draft models, it was ahead on five and level on the sixth, by up to 25%.

Kestrel served seven LLMs 1.5–6.1× faster than a default setupTokens per second, one stream, one RTX 5090. Up to 25% faster than the best public setup. Higher is better.
View data
Default setupBest public setupKestrel
Qwen3.8-4B distill167660686
Llama-3.1-8B101532562
Qwen3-8B100279347
ToolACE-8B101590617
OpenEuroLLM 9B91135137
Qwen3.8-27B, English76239239
Qwen3.8-27B, Thai76233268
Kestrel cut GPU energy per token by 47–86%GPU energy per million tokens, in kWh, one stream, one RTX 5090. Lower is better.
View data
Default setupKestrel
Qwen3.8-4B distill0.900.16
Llama-3.1-8B1.430.21
Qwen3-8B1.460.36
ToolACE-8B1.470.20
OpenEuroLLM 9B1.480.79
Qwen3.8-27B, English1.590.59
Qwen3.8-27B, Thai1.580.55

One NVIDIA RTX 5090. Default setup: the plain model served by SGLang. Best public setup: the fastest we could build for each model from public engines, quantized checkpoints and draft models, scored by the same rules; on ToolACE-8B, the AI agent's setup. Every Kestrel setup kept accuracy within 1 point of the plain model.

Why not just ask an AI agent?

We tried. A frontier OpenAI agent* tuned ToolACE-8B, and on the prompts it tuned against, its setup came out ahead. On prompts it hadn't seen, that lead was gone: Kestrel's setup was faster, more accurate and used less energy. Handed Kestrel's setup, the agent couldn't improve it.

Kestrel against a frontier OpenAI agent*ToolACE-8B on one RTX 5090, timed on prompts neither had seen.
KestrelAgent*
Tokens per second617590
GSM8K accuracy82.2%80.7%
GPU energy per token0.72 J0.74 J
Time to a resultUnder 1 h1 h 43 min

Fast on models it has never seen.

We drew 36 models at random from Hugging Face, none of them used to build Kestrel, and ran it once on a laptop. It beat the engines' own automatic settings on 31.

See all 36 models
Faster than automatic settings, on 36 unseen models
1.99×
Faster than Microsoft Olive's GPU tuning
1.9×
Faster than TensorRT on a recurrent model
2×
Faster on 39% of the energy, under an energy cap
2.2×

Same quality, or it doesn't ship.

Kestrel checks every plan against the original model before it can pick it: within 1 point of task accuracy for an LLM, within 1% output error for anything else. On six of the seven LLMs we tested, accuracy went up.

Try it on your model. Free.

Choose a model you run today. We run Kestrel on it and show you what changes: speed, energy and accuracy.

Start a free pilot

* GPT 6.1 Sol on Max effort