CUDA vs ROCm vs Vulkan vs OpenVINO: measured

Every backend explainer follows the same shape. It describes CUDA (NVIDIA's), ROCm/HIP (AMD's), Vulkan (the cross-vendor escape hatch), and SYCL/OpenVINO (Intel's), then tells you to "pick whatever works on your card." That is correct as far as it goes. It also tells you nothing, because the gaps between backends are large enough that "whatever works" leaves real speed on the table, and one of those backends does not actually work on the card people keep recommending it for.

This is the measured version — four backends on the same hardware, across nine of our benchmark records, with the receipts in the library. The headline finding is uncomfortable for the "just use Vulkan everywhere" school: vendor lock-in is real, the backend gaps are 1.7–2.5×, and one backend (SYCL on Battlemage) is broken in a way you can verify against an upstream bug ticket.

The four backends that matter for local AI

One paragraph each, then we measure them.

• CUDA — NVIDIA's proprietary compute platform. The reference implementation that every other backend is measured against. Runs on GeForce, RTX-series, and data-center NVIDIA cards.
• ROCm / HIP — AMD's answer to CUDA. Mature on data-center Instinct cards and the RDNA3/4 consumer lineup (gfx1100, gfx1151, gfx1201). Less mature on older or non-targeted AMD cards.
• Vulkan — the cross-vendor 3D-and-compute API. The escape hatch: works on NVIDIA, AMD, and Intel, including cards that CUDA and ROCm do not target (Intel Arc, older AMD, mixed-vendor fleets). Pays a speed tax.
• OpenVINO / Level Zero — Intel's inference toolkit and the low-level Intel GPU interface. Ships as the OpenVINO Model Server (OVMS) for serving. The supported Intel path.

Two scope-outs before we start. Metal (Apple's) is a real backend with real users; we have no Apple Silicon in the fleet and cannot claim measured numbers for it. DirectML (Microsoft's Windows-layer backend) is the same story. Both are mentioned in our cross-vendor GPU comparison but neither is measured here.

The vendor lock-in rule

The fastest backend on a given card is almost always the one the vendor wrote. CUDA on NVIDIA. ROCm/HIP on AMD. OpenVINO on Intel Arc. Vulkan is the only cross-vendor escape hatch, and it pays a speed tax for the privilege of portability.

The practical implication: when a runtime like llama.cpp asks you to pick a backend at build time (-G CUDA, -G ROCM, -G VULKAN), the lazy default of "Vulkan because it works everywhere" is the wrong answer for speed. Match the backend to the vendor, fall back to Vulkan only when nothing else targets your card.

Measured: Vulkan vs HIP on the same RDNA4 card

bm-004 ran the same Qwen3.6-27B-A3B-Instruct Q4_K_M-mtp on the same single Radeon AI PRO R9700 (gfx1201 RDNA4), across three backend configurations in back-to-back runs:

Backend (same card, same model)	Generation tok/s	VRAM
llama.cpp Vulkan + MTP speculative decoding	66.0	21.9 GB
llama.cpp Vulkan, no MTP	31.9	18.8 GB
llama.cpp HIP (ROCm-style)	27.1	19.1 GB
Ollama (HIP)	26.7	22.3 GB

Two findings to pull out. First, the MTP speculative-decoding path more than doubles throughput (66 vs 27 tok/s) — see the speculative-decoding piece for the full breakdown. Second, ignoring MTP, Vulkan slightly outperforms HIP on this RDNA4 card (31.9 vs 27.1 tok/s, ~1.18×). That is not supposed to happen — HIP is the vendor backend — and it's a real signal that ROCm maturity on RDNA4 is still catching up. Treat it as one data point on one card, not a universal claim.

Measured: OpenVINO vs Vulkan on the same Arc B60

The cleanest Intel-side result in the library: same Arc B60 Pro, two backends, two records. bm-012 runs OpenVINO Model Server on a Qwen3-30B-A3B INT4. bm-001 runs llama.cpp Vulkan on a Qwen3-Coder-30B-A3B Q4_K_M.

Backend (same Arc B60)	Generation tok/s	VRAM
OpenVINO / OVMS (NNCF INT4, Qwen3-30B-A3B-2507)	67.95	16.31 GB
llama.cpp Vulkan (Q4_K_M GGUF, Qwen3-Coder-30B-A3B)	38.6	18.1 GB

That is a 1.76× generation gap on the same card, in favor of the vendor backend. The full deep-dive is in OpenVINO beats Vulkan on Arc B60.

The honest caveats matter here: the two runs used different model variants (Instruct-2507 vs Coder), different quant methods (NNCF INT4 vs Q4_K_M GGUF), and different measurement protocols (streaming API vs llama-bench). The 1.76× is directional, not precise. But it lines up with the broader pattern — vendor backend beats cross-vendor backend by a meaningful margin — and it is the largest single speedup we have measured from a backend swap alone.

The documented SYCL failure on Battlemage

Negative results are evidence too. When we ran the four-machine baseline in bm-013, the Arc B60 path through stock Ollama produced 8.41 tok/s — Vulkan fallback, not the card's real capability. A SYCL rebuild to try to fix that produced 5.97 tok/s with garbage output, matching the symptoms of upstream bug llama.cpp#22413.

This is the honest version of "does SYCL work on Battlemage": not as of the build we tested, for this configuration, in a way that another user can reproduce by filing the same bug. It may work in the future; it may work for other configurations; we cannot claim it works for ours. Use OpenVINO Model Server on Arc — that is the supported Intel path, and the data above shows it works.

The ecosystem context for this is in Battlemage one year in. The short version: Arc's local-AI story is real as of OVMS, and the old "SYCL is broken on Battlemage" folklore was right about SYCL specifically but wrong about Arc generally.

Measured: CUDA's lead on Blackwell

bm-009 runs Qwen3-Coder-30B-A3B Q4_K_M on a dual-RTX-5070 tensor-split rig through CUDA 13.0 on Blackwell (sm_120): 172 tok/s generation, 3403 tok/s prompt-processing. Same model, same Q4_K_M tier, ~4.5× the Arc B60's Vulkan number.

The more controlled cross-vendor comparison is bm-013, which ran the same Qwen3.6-27B Q4_K_M weights on four machines:

Machine	Backend	Prompt tok/s	Generation tok/s
victor (2× RTX 5070)	CUDA 13.0	344.85	24.28
ray (2× R9700)	ROCm 7.0	217.27	26.74
evox2 (RX 7900 XT)	ROCm 6.17	162.05	25.21
jitori (Arc B60)	Vulkan fallback	111.94	8.41 ⚠

CUDA's lead is concentrated in prompt-processing — the dual-RTX-5070 rig digests prompts ~2× faster than the next-fastest card. For chat workloads where the user is waiting on first-token latency, that is the number that matters. For batch, the three CUDA/ROCm machines cluster within ~10% on generation. The Arc B60 row, again, is the Vulkan-fallback caveat — not the card's real capability.

When Vulkan is the right answer anyway

Despite the speed tax, Vulkan earns its place in three situations:

1. Mixed-vendor fleets. Our four-machine fleet is Intel + AMD + NVIDIA. Standardizing on Vulkan lets the same runtime configuration work across all of them, which matters for the home AI server setup. The price is ~2× slower than the vendor backend on each card.
2. Cards the vendor backend doesn't target. ROCm's supported list does not cover every AMD card ever made; CUDA does not run on anything but NVIDIA; older or unusual Intel hardware may not have OpenVINO support. Vulkan fills the gap.
3. Quick testing and model swaps. Vulkan is the path of least resistance for "does this model even run on my hardware" — fewer build flags, fewer driver assumptions. Use it to validate, then move to the vendor backend for production speed.

The honest rule: Vulkan first to confirm the model runs, then the vendor backend for the speed. Don't ship Vulkan into a production inference path on a card that has a working vendor backend.

What this article does NOT cover

• Metal — Apple's backend, no Apple Silicon in our fleet, no measured numbers. Real and mature for Mac users; we just cannot speak to it.
• DirectML — Microsoft's Windows-layer backend. Same story; no measured data.
• Training backends — every number here is inference. Training has different bottleneck patterns (FP16 bandwidth, gradient accumulation, optimizer state) and a different backend leaderboard.
• vLLM/SGLang continuous batching — those are runtime layers that sit on top of CUDA/ROCm, not backends in the same sense. See Ollama vs llama.cpp on the same hardware for the runtime-layer comparison.
• Driver / SDK install pain — every backend has its own installation gotchas; we cover those in the runtime-specific pieces, not here.

The decision, by vendor

Vendor / card	First-choice backend	Fallback	Measured data
NVIDIA (any CUDA-capable)	CUDA	Vulkan	bm-009, bm-013
AMD RDNA3 (gfx1100, e.g. RX 7900 XT)	ROCm 7.x	Vulkan	bm-007, bm-008, bm-013
AMD RDNA4 (gfx1201, e.g. R9700)	ROCm 7.0 (Vulkan slightly faster in our one test, bm-004)	Vulkan	bm-001, bm-004, bm-005, bm-006
AMD APU unified memory (gfx1151)	ROCm 7.2.2	Vulkan	bm-011
Intel Arc Battlemage (B580, B60)	OpenVINO Model Server	llama.cpp Vulkan	bm-001, bm-003, bm-012
Intel Arc with SYCL	Don't, as of our build — bug llama.cpp#22413	OVMS or Vulkan	bm-013 failures
Apple Silicon	Metal (out of scope here)	—	no data
Windows mixed-vendor	DirectML (out of scope here)	—	no data
Cross-vendor fleet / quick testing	Vulkan (accept the speed tax)	—	every record

The takeaway

Pick the backend the vendor wrote. CUDA on NVIDIA, ROCm on AMD, OpenVINO Model Server on Intel Arc. Vulkan is the cross-vendor escape hatch for mixed fleets, unsupported cards, and quick model validation — not the right answer for production speed on any single card. SYCL on Battlemage is broken in a way you can verify against an upstream ticket; use OVMS instead. The measured gaps are 1.7–2.5×, which is large enough that "just use Vulkan everywhere" is the wrong default, and small enough that Vulkan is still a real option when portability matters more than speed.

The OpenVINO-vs-Vulkan deep dive on Arc is here. The runtime-layer comparison — Ollama vs llama.cpp vs vLLM, which sits on top of these backends — is here. The raw records behind every number above are in the benchmark library.

Last verified: July 2026. Backend maturity moves fast — ROCm on RDNA4, OpenVINO on Battlemage, and SYCL on Arc are all active development areas. Re-validate against the latest runtime builds before relying on these rankings.