Two engines, the same models on the same GPU

Performance and compatibility results come from reports in the basal-rs repository, with data and test conditions. Both engines are always measured on the same GPU, with basal-rs using its default installation settings.

The controls above the chart change the model and measurement. The number marked × shows how much better the winning engine's result is for each GPU. When upstream wins, its name precedes the number.

The chart shows the disk space required to install everything needed to serve Basal models on an NVIDIA GPU. Model files and the GPU driver are excluded. In every mode, upstream runs as basal-serve in Python with torch and transformers, including when Ollama or llama.cpp performs the computation.

Realistic traffic: short questions arrive alongside documents of up to 16k tokensA piece of text, usually part of a word. The model reads text as a sequence of tokens.. p50 is the typical response time; p95 is the time within which 95% of requests finish. High p95 with 32 clients comes from long documents waiting in the queue; short questions have a separate queue and do not wait behind them.

For short documents up to about 2k tokens, upstream is faster on H100. From about 4k tokens, basal-rs is 1.75–2.4 times faster; with five questions about the same document, it is 10–15 times faster because the document is computed once.

On Macs, basal-rs computes with MetalApple's interface for GPU computation on Macs.. The model author's server uses MLXApple's machine-learning library for Macs. in bf16A 16-bit number format with a wider range and lower precision than f16.. The switch shows two comparisons: f16 isolates the difference between engines, while bf16 shows the difference an upstream user would experience. The laptop was plugged in during measurement. Times increase on battery or after it heats up.

900 questions from nine public datasets, 100 from each. We compare decisions with full-precision FP32 calculations. The only basal-rs f16 differences occur on questions where even FP32 is nearly tied, such as 0.503 versus 0.497.

Basal models choose from at most 10 options. When there are more, basal-rs divides them into groups, selects the best from each and resolves them in a final round. The model author's server does not do this, so we compare accuracy with the TypeSafe API on datasets with known correct answers.

We measure basal-rs with basal-1.5-max on an RTX 6000 Ada, whose factory power limit is 300 W. Lowering it to 250 W reduces server throughput by about 20%, while energy per decision stays the same.

All you need is basal-rs installed (the basal command) and a copy of its repository. The repository contains full-precision upstream FP32 results for three models. Commands that save results refuse to overwrite an existing path.