Qwen 3.8 27B

A dense 27B that thinks about twice as hard as anything its size. Quantized to four bits it fits in 23 GB and ties DeepSeek V4 Flash on the coding benchmark - and the four-bit build turned out to be the one to serve.

dense 27Bfour-bit NVFP423 GB weightshybrid attentionMTP spec-decode · 16 August 2026· updated 30 August 2026

Grade (mean)

2.42

±0.12 · 52% gold · n=45

Weights

23 GB

vLLM nightly

VRAM serving

165 GB

2 cards

Decode

71-81

tok/s across context

What to run

Run the four-bit (NVFP4) build at medium reasoning effort. It matches the eight-bit build on quality and beats it on every serving axis - smaller, faster, more concurrent.

It thinks about twice as much, and it pays

The gap over the previous generation is almost entirely on the two hardest tasks, and the new model gets there by generating a lot more. On the hardest one - an atomicity bug in an experience-point ledger - it produces tens of thousands of tokens of reasoning and edits against the old model's shorter, more confident, more wrong attempt. The over-thinking is not waste; it is how it arrives at the answer. The one real cost is that thinking is tokens and tokens are wall-clock.

Reasoning effort is a real dial - set server-side

More effort buys correctness on the hardest task, entirely by lowering the failure rate; the ceiling is the same either way, and medium is the balance worth running. The trap: the agent does not forward a reasoning-effort field to the server at all. The level is set by which chat template the serve mounts - "run this at medium" is a decision you bake into the serve, not a client flag. Once you know it, it is obvious; until you know it, you can lose an afternoon wondering why a flag does nothing.

Quality does not decide the quant - serving does

Four-bit and eight-bit are the same model; benched at medium effort they land on top of each other within noise. So the choice is a serving one, and there it is not close: four-bit decodes a quarter to a third faster, runs in 23 GB against 31, and sustains roughly twice the aggregate throughput under load. There is no axis where eight-bit wins. Run four-bit; keep eight-bit around only to point at and say it bought you nothing.

By the numbers

Everything below is read live from the benchmark, the same source as the data page. It reflects the current measurement, not a snapshot pasted in when this page was written.

Grade distribution (0-3, 95% CI)

Mean tier across all runs with its 95% interval; the bar shows how attempts spread across the four grades.Drawn against DS V4 0731 for scale.

Decode speed vs context (tok/s)

tok/s vs context

How decode holds up as context fills. MTP ON (num=3, native qwen3_5 MTP head). NVFP4 serves natively on SM120 (FlashInfer fp4_gemm, not Marlin). vLLM nightly.

VRAM while serving: weights vs reserved KV cache

Solid is weights, faint is reserved KV cache that scales with how much context you keep live.

Aggregate throughput at 200k context, four-bit vs eight-bit (tok/s)

Deep in the context window the single-stream gap is more than five to one, and four-bit stays ahead the whole way up. Neither quant evicted cache at 32 streams even at 200k tokens each; eight-bit just works from a smaller pool and a slower floor.

Serving gotchas

The reasoning-effort flag is on the server, not the client

The agent never forwards it. Effort is whatever the mounted chat template defaults to - mount the medium template to serve medium.
Do not taper it on filler

It is a hybrid (48 of 64 layers linear-attention, 16 full), so decode is nearly flat by architecture. Repetitive filler + spec-decode made it read both steep and flat, wrongly. Measure on a real corpus.
A wall-clock tail on the hardest task

A handful of runs ran to the three-hour cap with a correct fix already in hand - the agent kept working past useful. A shorter timeout or a notch less effort caps it.

How it's served

Engine: vLLM nightly

Speculative: MTP ON (num=3, native qwen3_5 MTP head). NVFP4 serves natively on SM120 (FlashInfer fp4_gemm, not Marlin). vLLM nightly.

Weights: Qwen/Qwen3.8-27B-NVFP4 (native FP4, FlashInfer) + froggeric v22 medium template

Download: hf download Qwen/Qwen3.8-27B-NVFP4 --local-dir ~/models/Qwen3.8-27B-NVFP4

Serving compose - ~/vllm/compose-Qwen3.8-27B-NVFP4-medium.yml

# Qwen 3.8 27B - NVFP4 (~23G, native Blackwell FP4), MEDIUM-EFFORT variant for the QUALITY bench.
# Quant-matched sibling to compose-Qwen3.6-27B-NVFP4-medium.yml -> the fair generational A/B is
# 3.8-NVFP4 vs 3.6-NVFP4 (both 4-bit). Also gives the within-3.8 FP8-vs-NVFP4 quality+speed A/B.
# Mounts qwen-fixed-v22-medium.jinja (reasoning_effort default = medium; opencode does NOT forward it).
# Same qwen3_5 hybrid + native MTP head (model_mtp.safetensors). bf16 KV (fp8 KV corrupts DeltaNet).
# ⚠ VALIDATE at bring-up: NVFP4 serves NATIVELY on SM120 (FlashInfer, not Marlin fallback / TP garbage).
services:
  vllm:
    image: vllm/vllm-openai:nightly   # nightly proven to load qwen3_5; validate NVFP4 native FP4 path on SM120
    restart: "no"
    ipc: host
    shm_size: "32gb"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      stack: 67108864
    ports:
      - "8080:8000"
    environment:
      - CUDA_VISIBLE_DEVICES=0,1
      - CUDA_DEVICE_ORDER=PCI_BUS_ID
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - HF_HUB_OFFLINE=1
      - TRANSFORMERS_OFFLINE=1
    volumes:
      - ~/models/Qwen3.8-27B-NVFP4:/models:ro
      - ~/vllm/chat-templates/qwen-fixed-v22-medium.jinja:/chat_template.jinja:ro
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]
    command: >
      --model /models
      --served-model-name Qwen3.8-27B latest /latest
      --host 0.0.0.0 --port 8000
      --tensor-parallel-size 2
      --disable-custom-all-reduce
      --gpu-memory-utilization 0.85
      --max-model-len 262144
      --max-num-seqs 32
      --max-num-batched-tokens 8192
      --reasoning-parser qwen3
      --tool-call-parser qwen3_coder
      --enable-auto-tool-choice
      --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
      --chat-template /chat_template.jinja
      --enable-prefix-caching
      --enable-chunked-prefill
      --trust-remote-code