Ornith-1.0-35B backend showdown on Radeon AI PRO R9700
For the Ornith-1.0-35B MoE model (qwen35moe family) on a single Radeon AI PRO R9700, Ollama is the clear winner at 78.3 tok/s - roughly 2.9x faster than llama.cpp Vulkan (27.3 tok/s), which also spilled past the 32GB VRAM limit into system RAM. The HIP backend failed outright: it could not parse the Ornith GGUF (version mismatch). On MoE models without an MTP head, Ollama's expert routing currently beats hand-tuned llama.cpp on this build.
Configuration
- Model
- Ornith-1.0-35B
- Artifact
- ornith-1.0-35b-Q4_K_M.gguf
- Checksum
- pending
- Quantization
- q4-k-m
- Context length
- 96,256 tokens
- Backend
- vulkan
- Runtime
- Ollama (HIP) + llama.cpp (Vulkan) + llama.cpp (HIP, failed)
- Settings
- Ollama winner: Modelfile FROM ornith-1.0-35b-Q4_K_M.gguf, num_ctx 96256, num_predict 32768, temperature 0, num_gpu 99, pinned to GPU1 via ROCR_VISIBLE_DEVICES=1, OLLAMA_KEEP_ALIVE=-1. llama.cpp Vulkan/HIP runs used comparable -ngl/-c flags; see limitations.
- Workload pack
- ray-single-prompt-coding-v1
- Author
- Edgar
- Reviewer
- Edgar
Results
Single fixed coding prompt ('Write a Python function that implements merge sort with type hints and docstrings'), 256-512 token generation, temperature=0, 3 runs per backend, best result reported. Same protocol as bm-004 (run back-to-back on the same session, GPU0 vs GPU1). Narrower than the site's workload-pack-v1; preserved for backend comparability, not absolute quality claims.
78.3tok/s
GEN · Ray
1592tok/s
PROMPT
0ms
TTFT
23.2GB
VRAM USED
| MACHINE | PROMPT tok/s | GEN tok/s | TTFT ms | VRAM GB | RAM GB | POWER W |
|---|---|---|---|---|---|---|
Ray GPU1 (one of two R9700s). WINNER. Ollama. Notable result for a 35B model; attributed to Ollama's optimized MoE expert routing for the qwen35moe family. Pinned to GPU1; persistent via systemd (OLLAMA_KEEP_ALIVE=-1). | 1592.0 | 78.3 | 0 | 23.2 | 0.0 | — |
Ray GPU1. llama.cpp Vulkan. VRAM OVERFLOWED the 32GB card and spilled to system RAM (34.2GB reported total) - this is why it is 2.9x slower, not a pure backend-quality gap. The model does not cleanly fit a single R9700 via this path. | 564.0 | 27.3 | 0 | 34.2 | 0.0 | — |
Limitations
- ◆ttft_ms and ram_used_gb are recorded as 0 because the source measurement (RAY_BENCHMARK_RESULTS.md, 2026-06-30) did not capture time-to-first-token or system-RAM usage; these fields are required by the site schema and 0 is used as an explicit 'not measured' sentinel, not a real measurement. (Note: the Vulkan run's 34.2GB is GPU+RAM spill combined, reported in vram_used_gb per the source's framing - see the result note.)
- ◆prompt_speed_tps values are operator-reported approximations, not instrumented measurements; treat as bands.
- ◆Workload is a single coding prompt, not the site's 100-prompt HumanEval workload-pack-v1. Conclusions are about backend RELATIVE ordering on this hardware, not absolute model quality. Do not compare these tok/s numbers directly against workload-pack-v1 results.
- ◆The 2.9x Ollama-vs-Vulkan gap is partly confounded by VRAM: Vulkan spilled past 32GB into system RAM while Ollama fit in 23.2GB. So this is 'Ollama's routing + fit' vs 'Vulkan's routing + spill', not a clean backend-only comparison. A fair re-test would cap context/flags so both paths fit in VRAM.
- ◆The HIP failure (see failures[]) is specific to this llama.cpp build's GGUF-version handling for the qwen35moe architecture; it may not reproduce on newer builds.
- ◆One GPU per model (no tensor splitting across the two R9700s).
- ◆Raw evidence link (/lab/raw/bm-005/) currently resolves to a stub; full raw output to be uploaded before or within 30 days of launch.
- ◆model_checksum not recorded at measurement time; capture on next re-run.