Qwen3-30B-A3B INT4 on Arc B60: OpenVINO/OVMS vs llama.cpp Vulkan — backend shootout
Qwen3-30B-A3B-Instruct-2507 (INT4, OpenVINO IR) served via OpenVINO Model Server 2026.2.1 on Jitori's Intel Arc Pro B60 (Battlemage BMG G21, 24GB). Generates at 67.95 tok/s via the OpenAI streaming API - roughly 1.76x faster than the same B60 running Qwen3-Coder-30B-A3B Q4_K_M via llama.cpp Vulkan (bm-001, 38.6 tok/s). This is the backend comparison the dataset was missing: same GPU, same 30B-A3B model class, two Intel-GPU backends. Finding: OpenVINO is meaningfully faster than Vulkan on Battlemage for this workload. This also corrects a fleet assumption - OpenVINO's GPU plugin (Level Zero) works cleanly on this B60, contradicting the older 'SYCL is broken on Battlemage, Vulkan-only' note.
Configuration
- Model
- Qwen3-30B-A3B-Instruct-2507
- Artifact
- OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov (openvino_model.bin, 16.31GB)
- Checksum
- sha256:not-captured (OVMS git-lfs pull; HF repo digest available on request)
- Quantization
- q4-k-m
- Context length
- 2,048 tokens
- Backend
- vulkan
- Runtime
- OpenVINO Model Server 2026.2.1 (OVMS build 1122f03bf, OpenVINO backend 2026.2.1-21919, GenAI backend 2026.2.1.0); Docker container openvino/model_server:2026.2.1-gpu
- Settings
- OVMS served via Docker: --target_device GPU --pipeline_type LM --task text_generation, default KV cache (dynamic), default batching. Measured via OpenAI /v3/chat/completions streaming endpoint with a 229-token prompt, max_tokens=256, temperature=0.0, 3 reps. Render device /dev/dri/renderD128 (Arc B60 at PCI 0a:00.0) passed through via --device /dev/dri.
- Workload pack
- ovms-streaming-pp-tg-v1
- Author
- Edgar
- Reviewer
- Edgar
Results
OpenAI-compatible streaming chat completions via OVMS /v3/chat/completions. NOT llama-bench - OVMS has no llama-bench equivalent, so this uses a Python harness that times TTFT (time-to-first-chunk) and generation rate from the streaming API, then computes prompt-tokens/TTFT as a pp-rate proxy. 3 reps, prompt of ~229 tokens (intended ~512 but the instruction tokenized shorter), max_tokens=256. This is an APPROXIMATION of the llama-bench pp512/tg256 protocol, not a strict match - see limitations. The decision value is the backend comparison with bm-001 (same GPU, same model class, Vulkan backend), not an absolute throughput claim.
68.0tok/s
GEN · Jitori PC
630tok/s
PROMPT
363ms
TTFT
16.3GB
VRAM USED
| MACHINE | PROMPT tok/s | GEN tok/s | TTFT ms | VRAM GB | RAM GB | POWER W |
|---|---|---|---|---|---|---|
Jitori PC Jitori Intel Arc Pro B60 (Battlemage BMG G21, 24GB VRAM) via OpenVINO Model Server 2026.2.1. OVMS GPU plugin (Level Zero) initialized cleanly: log line 'Available devices for Open VINO: CPU, GPU' and 'state changed to: AVAILABLE'. Generation: 67.95 tok/s avg across 3 reps (67.31 / 68.20 / 68.34 - tight variance). Prompt processing: 229 prompt tokens / 0.363s avg TTFT = 630 tok/s; BUT this includes a cold-start outlier (rep 1 TTFT=0.688s vs reps 2-3 at 0.20s), so the WARM pp rate is closer to 1100+ tok/s (229/0.201). ttft_ms recorded as the avg including warmup. vram_used_gb recorded as the 16.31GB model footprint; OVMS does not expose per-process VRAM via its metrics endpoint. | 630.0 | 68.0 | 363 | 16.3 | 0.0 | 0 |
Limitations
- ◆MEASUREMENT PROTOCOL MISMATCH: this is NOT llama-bench. OVMS exposes an OpenAI API, not llama-bench's pp/tg primitives, so generation rate is measured by counting streaming content chunks and prompt rate is approximated as prompt_tokens/TTFT. This is comparable-in-spirit to bm-001's llama-bench numbers but not strictly apples-to-apples - streaming chunk counting can differ from llama-bench's exact token timing by a few percent. Treat the 1.76x OpenVINO-vs-Vulkan ratio as a directional finding, not a precise figure.
- ◆PROMPT LENGTH MISMATCH: the prompt tokenized to 229 tokens, not the intended 512 (the instruction was shorter than estimated). bm-001 used llama-bench's exact pp512. Prompt-processing rate scales with prompt length on most backends, so a true pp512 comparison may differ. A follow-up with a precisely-512-token prompt would tighten this.
- ◆WARMUP: rep 1 TTFT was 0.688s (cold), reps 2-3 were 0.20s (warm). The reported 363ms ttft_ms is the avg including warmup; the warm steady-state TTFT is ~200ms. llama-bench's -r 3 reps include warmup similarly, so this is roughly comparable, but the cold-start effect is larger on OVMS's first request than llama-bench's first rep.
- ◆MODEL VARIANT DIFFERENCE: this is Qwen3-30B-A3B-Instruct-2507 (the 2507 update, OpenVINO IR INT4 via NNCF). bm-001 is Qwen3-Coder-30B-A3B-Instruct (GGUF Q4_K_M). They are DIFFERENT model variants - Coder vs Instruct-2507 - with the same architecture (30B-A3B MoE) but different weights. The 1.76x speedup therefore reflects backend AND model-variant AND quantization-method differences, not backend alone. OpenVINO NNCF INT4 and llama.cpp Q4_K_M are different INT4 schemes with different accuracy/speed tradeoffs. A same-model, same-quant backend comparison would require converting one model to the other's format.
- ◆QUANTIZATION DIFFERENCE: OpenVINO NNCF INT4 (this record) vs llama.cpp Q4_K_M (bm-001) are different INT4 methods. NNCF INT4 typically uses mixed-precision weight compression with group-wise quantization; Q4_K_M uses k-quant super-blocks. They are not directly comparable on quality - and the speed difference partly reflects the different kernel implementations, not just the backend.
- ◆VRAM NOT CLEANLY CAPTURED: OVMS exposes no per-process VRAM metric via its REST API; the 16.31GB figure is the model's on-disk footprint (openvino_model.bin). Runtime VRAM including KV cache is higher but unmeasured here. The B60 has 24GB so the model fits with ~7GB headroom for KV/runtime.
- ◆ttft_ms reported (363) is the avg including warmup; see the WARMUP limitation.
- ◆v1 of this benchmark. A follow-up with (a) a precisely-512-token prompt, (b) warm-only TTFT, and (c) ideally a same-model GGUF-vs-IR comparison would tighten the backend claim.
- ◆BACKEND LABEL: schema requires backend enum (vulkan|rocm|cuda|...); OpenVINO is not in the enum, so 'vulkan' is used as the closest Intel-GPU backend label. The ACTUAL backend is OpenVINO GPU plugin via Level Zero - a different software stack from Mesa Vulkan. This is a schema limitation, not a data error.
Corrections
2026-07-18
Initial publication. Establishes that OpenVINO Model Server works on the Arc B60 (contradicting the older fleet note that 'SYCL is broken on Battlemage, Vulkan-only' - that note referred to an ad-hoc SYCL build, not Intel's curated OVMS container) and that it delivers ~1.76x generation speedup over llama.cpp Vulkan on the same GPU for the 30B-A3B model class. Caveats: model variant, quant method, and measurement protocol all differ from bm-001, so the 1.76x is directional, not precise. A same-model GGUF-vs-IR comparison is the documented follow-up.