GLM-5.3-Flash
The new daily driver, and the strongest model that fits on the two-card box: on the full benchmark it comes second only to Opus, edging out DeepSeek V4 Flash for the top local spot, though the error bars still overlap so the lead is a narrow one. It is also the heaviest local model, and it only runs at all through a build quantized for this exact hardware.
The current daily driver, and the top local model on the board, a nose ahead of the DeepSeek V4 Flash it replaces and the four-bit Qwen rather than a class above them, so weigh that against its cost: the heaviest footprint on the rig and a custom SGLang SM120 image just to serve it. Worth it when you have the VRAM to spare and want the strongest local answer on the problems that separate models.
The top of the local field, by a nose
On the full benchmark GLM takes the top of the local field at 2.56, behind only Opus and ahead of both DeepSeek V4 Flash and the four-bit Qwen, though you should hold that lead loosely, because the error bars on the three still overlap and a nose ahead is not a step change. The lead is also not spread evenly: it comes almost entirely from one of the hard tasks, where GLM lands a near-complete fix while DeepSeek mostly manages only a partial one, with a smaller edge on a second hard problem, and on the remaining two it sits a touch behind. What the top of this cluster buys you, then, is a model that is a little stronger on the problems that actually separate models, not a broadly different class of result.
Flat decode from the same kernel as DeepSeek
GLM's handful of full-attention layers run on the FlashInfer sparse-MLA path on these SM120 cards, the same kernel that keeps DeepSeek flat, so decode holds around 200 to 255 tok/s the whole way out to 200k rather than tapering. Speculative acceptance moves with how predictable the text is, roughly 0.7 to 0.97 here, and that is what the wobble in the curve is, not a real slowdown.
It only runs because someone quantized it for this exact hardware
There is no GLM-5.3-Flash on these cards off the shelf. It runs on a community W4A16-NVFP4 build, NVFP4 experts with FP8 attention, served through a custom SGLang image built for SM120, because upstream vLLM and SGLang do not load the checkpoint. That image and the sparse-MLA kernel it carries are the whole reason a model this large, 178 GB of weights, is usable on a two-card box at all.
By the numbers
Everything below is read live from the benchmark, the same source as the data page. It reflects the current measurement, not a snapshot pasted in when this page was written.
Serving gotchas
- Needs a custom SGLang SM120 imageUpstream vLLM/SGLang will not load the W4A16-NVFP4 checkpoint. Serve it with the matching SGLang-SM120 image, and confirm the FlashInfer sparse-MLA decode path shows active in the launcher log - that is the kernel doing the work.
- Heaviest footprint on the rigAt 178 GB of weights (187 GB total) it fills almost the whole two-card pair and leaves little KV headroom, so it serves only a few concurrent streams. It is a single-user workhorse, not a fleet server.
- Serving flags the checkpoint requiresShared-experts fusion must stay disabled; reasoning parser glm45, tool-call parser glm47; the linear-attention and mamba backends run on triton.
How it's served
hf download ormandj/GLM-5.3-Flash-W4A16-NVFP4-K32-Experts-FP8-WO # W4A16-NVFP4 experts + FP8 attention; needs ghcr.io/ormandj/sglang-glm53-flash-sm120:v0.4.3 (modelopt_mixed; upstream vLLM/SGLang do NOT load it)Serving compose - ~/vllm/serve-GLM-5.3-Flash-ormandj.sh
#!/usr/bin/env bash
# GLM-5.3-Flash - ormandj W4A16-NVFP4-K32-Experts-FP8-WO, 165.8 GiB. Bring-up wrapper, 2026-09-24.
#
# WHY THIS CHECKPOINT: it is the only GLM-5.3-Flash build that fits 191.2 GiB, because it drops
# attention + shared-expert projections to E4M3 block-FP8 while experts go NVFP4 K=32. Quantizer's own
# target is literally "two SM120 RTX PRO 6000" - our rig. GSM8K 96.9% vs BF16 source.
#
# WHY NOT vLLM: the checkpoint carries `modelopt_mixed` metadata. Upstream Transformers, vLLM and SGLang
# all fail to load it. The pinned SGLang+FlashInfer image below is MANDATORY. Our b12x "jovian" images
# want a mainstream NVFP4 (177-190 GiB) which does NOT fit - hence SGLang, not our usual stack.
#
# Upstream's measured numbers on OUR EXACT HARDWARE (2x RTX PRO 6000 Max-Q 96GB, TP2 over PCIe, 300W):
# decode after MTP - C1 197.5 / C2 311.3 / C3 377.1 / C4 425.1 tok/s ; cold prefill 8k C1 5,627 tok/s
# (NB: our cards are capped at 275W, not 300W - expect slightly under these.)
# *** No decode taper past 64K has ever been published. That is THE number to measure first. ***
# Compare against DS4-0731: 160 @4K -> 253 @64K -> 160 @200K (data/tapers/retaper-ds4-0731-realistic.log)
#
# SPEC-DECODE: native adaptive MTP is ON by default in the launcher (5 steps / top-k 1 / 6 draft tokens,
# 3/5-step ladder). MTP weights are UNQUANTIZED on purpose - quantizing them silently collapses acceptance.
# For a benchmark-matched run use fixed MTP instead: see RUN.md (steps 3 / topk 1 / 4 draft, drop adaptive).
# NOTE: HelixML tested DFlash2 on this stack and REJECTED it (+4.3% at C1, -5.5% at the C4 operating
# point). Don't chase DFlash2 first.
#
# *** GPU - only DS4 stands in the way now ***
# BENCH PORT 8180 (not 8080): cli-proxy-api + sillytavern point at 8080, so a bench server there
# would take real user traffic and contaminate the run. See docs/BENCH-ISOLATION.md.
# bge-m3 was retired 2026-09-24, so MEM_FRACTION=0.99 (the launcher default, which assumes both cards are
# empty) is valid as-is. The ONLY thing to stop first is the DS4 daily driver:
# docker compose -f ~/vllm/compose-DeepSeek-V4-Flash-0731-DSpark-v9.yml down
# Restore it with the same command + `up -d`.
set -euo pipefail
export MODEL_DIR=${MODEL_DIR:-~/models/GLM-5.3-Flash-ormandj-W4A16-NVFP4}
export IMAGE=${IMAGE:-ghcr.io/ormandj/sglang-glm53-flash-sm120:v0.4.3}
export CACHE_DIR=${CACHE_DIR:-~/vllm/glm53-flash-sm120-v84-cache} # v84 cache schema, per release.json
export PORT=${PORT:-8180} # canonical BENCH port, isolated from DS4 on 8080
export TP_SIZE=2
export CONTEXT_LENGTH=${CONTEXT_LENGTH:-524288}
export MAX_TOTAL_TOKENS=${MAX_TOTAL_TOKENS:-524288}
export MAX_RUNNING_REQUESTS=${MAX_RUNNING_REQUESTS:-4}
export MAX_MAMBA_CACHE_SIZE=${MAX_MAMBA_CACHE_SIZE:-28} # must be >= 5 * MAX_RUNNING_REQUESTS
export MEM_FRACTION=${MEM_FRACTION:-0.99} # launcher default; both cards are ours now
export ENABLE_HICACHE=${ENABLE_HICACHE:-0} # =1 + HICACHE_SIZE_GB=32 to match their panel
export CONTAINER_NAME=${CONTAINER_NAME:-glm53-flash-sm120}
exec ~/vllm/sglang-glm53-flash-sm120/examples/serve-glm53-flash.sh