Kestrel
Makes AI models superfast, on less energy.
Photo: Thibaud Aronson, CC BY-SA 4.0, via iNaturalist, mirrored, sky extended
The fastest way to run your model, found for you.
The same model can run on many engines, at many precisions, on different chips. Kestrel tries them on your exact hardware and keeps the fastest plan that holds the original quality.
Engine
The runtime that serves the model, from vLLM and SGLang to ONNX Runtime, OpenVINO and TensorRT.
Precision
How many bits each number gets, wherever quality holds.
Device
The GPU, the CPU, the integrated GPU or the NPU, chosen per model.
Decoding
How an LLM produces its tokens, tuned for the model and its traffic.
Faster than the best an engineer can build.
Against a default setup, Kestrel served seven open LLMs up to 6× faster on up to 86% less energy. Against the best an inference engineer can build from public engines, quantized checkpoints and draft models, it was ahead on five and level on the sixth, by up to 25%.
View data
| Default setup | Best public setup | Kestrel | |
|---|---|---|---|
| Qwen3.8-4B distill | 167 | 660 | 686 |
| Llama-3.1-8B | 101 | 532 | 562 |
| Qwen3-8B | 100 | 279 | 347 |
| ToolACE-8B | 101 | 590 | 617 |
| OpenEuroLLM 9B | 91 | 135 | 137 |
| Qwen3.8-27B, English | 76 | 239 | 239 |
| Qwen3.8-27B, Thai | 76 | 233 | 268 |
View data
| Default setup | Kestrel | |
|---|---|---|
| Qwen3.8-4B distill | 0.90 | 0.16 |
| Llama-3.1-8B | 1.43 | 0.21 |
| Qwen3-8B | 1.46 | 0.36 |
| ToolACE-8B | 1.47 | 0.20 |
| OpenEuroLLM 9B | 1.48 | 0.79 |
| Qwen3.8-27B, English | 1.59 | 0.59 |
| Qwen3.8-27B, Thai | 1.58 | 0.55 |
One NVIDIA RTX 5090. Default setup: the plain model served by SGLang. Best public setup: the fastest we could build for each model from public engines, quantized checkpoints and draft models, scored by the same rules; on ToolACE-8B, the AI agent's setup. Every Kestrel setup kept accuracy within 1 point of the plain model.
Why not just ask an AI agent?
We tried. A frontier OpenAI agent* tuned ToolACE-8B, and on the prompts it tuned against, its setup came out ahead. On prompts it hadn't seen, that lead was gone: Kestrel's setup was faster, more accurate and used less energy. Handed Kestrel's setup, the agent couldn't improve it.
| Kestrel | Agent* | |
|---|---|---|
| Tokens per second | 617 | 590 |
| GSM8K accuracy | 82.2% | 80.7% |
| GPU energy per token | 0.72 J | 0.74 J |
| Time to a result | Under 1 h | 1 h 43 min |
Fast on models it has never seen.
We drew 36 models at random from Hugging Face, none of them used to build Kestrel, and ran it once on a laptop. It beat the engines' own automatic settings on 31.
See all 36 models- Faster than automatic settings, on 36 unseen models
- 1.99×
- Faster than Microsoft Olive's GPU tuning
- 1.9×
- Faster than TensorRT on a recurrent model
- 2×
- Faster on 39% of the energy, under an energy cap
- 2.2×
Same quality, or it doesn't ship.
Kestrel checks every plan against the original model before it can pick it: within 1 point of task accuracy for an LLM, within 1% output error for anything else. On six of the seven LLMs we tested, accuracy went up.
Try it on your model. Free.
Choose a model you run today. We run Kestrel on it and show you what changes: speed, energy and accuracy.
Start a free pilot* GPT 6.1 Sol on Max effort
