DeepSeek V4 Flash (0731)

The workhorse that held the daily-driver slot until GLM narrowly took it: a sparse-attention MoE that holds its decode speed flat across the whole context window, and, after a July re-post-train, one of the most reliable local models on the benchmark.

sparse-MLA MoEDSpark spec-decodeflat decodestock vLLM build · 4 August 2026· updated 30 August 2026

Grade (mean)

2.43

±0.19 · 60% gold · n=38

Weights

167 GB

vLLM b12x

VRAM serving

187 GB

2 cards

Decode

176-255

tok/s across context

What to run

The familiar fallback now that GLM leads on score, and still the easier one to stand up: it serves on a stock vLLM build with no custom image, holds a flat decode, and sits a notch behind GLM rather than a tier. Serve it with DSpark speculative decode configured correctly, and A/B spec-on against spec-off on real tasks before you trust it, because a subtly wrong build costs quality at the same speed.

The re-post-train moved capability around

The July re-post-train lifts the hard-task average from about 2.0 to about 2.5, but not as a flat bonus. One task jumped from not trying at all to a solid fix, another stepped up a clear notch, a third genuinely regressed right at the edge of the noise, and the fourth was already at the ceiling. A re-post-train moves the model around in capability space rather than adding a constant; here the gains outweigh the one real loss.

Flat decode, and what the speed actually buys

Sparse MLA means most layers do not care how long the context is, so decode stays roughly flat - about 180 to 255 tok/s - the whole way out, where dense models collapse. DSpark speculative decode is what you serve with, but its rate is content-dependent, so the honest speedup over no-spec is roughly 1.4x to 2.2x, not the ~2.6x a repetitive-filler taper flatters it into. And decode is not the clock you wait on: an agentic run is mostly prefill and tool round-trips, so the end-to-end win lands around 1.3x to 1.9x.

By the numbers

Everything below is read live from the benchmark, the same source as the data page. It reflects the current measurement, not a snapshot pasted in when this page was written.

Grade distribution (0-3, 95% CI)

Mean tier across all runs with its 95% interval; the bar shows how attempts spread across the four grades.Drawn against GLM-5.3-Flash for scale.

Decode speed vs context (tok/s)

tok/s vs context

How decode holds up as context fills. DSpark spec-decode ON (b12x v9, num_spec=3 greedy) - ~180-250 t/s, flat across context (sparse MLA); spec speed is content-dependent

VRAM while serving: weights vs reserved KV cache

Solid is weights, faint is reserved KV cache that scales with how much context you keep live.

Serving gotchas

The speculative-decode trap

DSpark is lossless only if the build verifies drafts by exact rejection sampling. A build that verified approximately, and a block size of 7 where the checkpoint wants 5, each quietly cost quality at the same ~290 tok/s. A/B spec-on vs spec-off on real tasks - perplexity will not catch a gap this small; eight graded runs did.
Its taper wobble is spec variance, not a curve

DSpark acceptance depends on how predictable the next tokens are, so the decode line wiggles. Measured on a diverse corpus it is flat ~180-255; a filler taper reads a rising, impossible 183 to 347.

How it's served

Engine: vLLM b12x

Speculative: DSpark spec-decode ON (b12x v9, num_spec=3 greedy) - ~180-250 t/s, flat across context (sparse MLA); spec speed is content-dependent

Weights: deepseek-ai/DeepSeek-V4-Flash-0731

Download: hf download deepseek-ai/DeepSeek-V4-Flash-0731 --local-dir ~/models/DeepSeek-V4-Flash-0731

Serving compose - ~/vllm/compose-DeepSeek-V4-Flash-0731-b12x.yml

# DeepSeek-V4-Flash - vLLM b12x (SM120 sparse MLA) on 2x RTX PRO 6000.
# Image: voipmonitor/vllm b12x build (Martin Vit / local-inference-lab, upstream vLLM contributor).
#   Third-party image, VETTED before use: statically inspected, entrypoint/flags confirmed via
#   `docker inspect`, and egress-verified (forced offline; no phone-home observed at runtime).
# Hardened for defence-in-depth:
#   - weights mounted READ-ONLY, dedicated writable cache only
#   - NO /mnt or home bind mounts (limits blast radius)
#   - published port instead of --network host
#   - HF/telemetry forced OFFLINE (any egress at runtime would be a red flag)
# Needs both GPUs - stop any other model first. MTP=2 (benchmarked decode peak ~190 t/s, +12% vs MTP1; 2026-07-04).

services:
  vllm:
    image: voipmonitor/vllm:chthonic-consecration-f1190eab-b12x0ff2847-pr20-cu132
    restart: unless-stopped
    ipc: host
    shm_size: "32gb"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      stack: 67108864
    ports:
      - "8080:8080"
    environment:
      # --- device / topology (from recipe) ---
      - CUDA_VISIBLE_DEVICES=0,1
      - CUDA_DEVICE_ORDER=PCI_BUS_ID
      - CUTE_DSL_ARCH=sm_120a
      - NCCL_IB_DISABLE=1
      - NCCL_P2P_LEVEL=SYS
      - NCCL_PROTO=LL,LL128,Simple
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      # --- b12x backend switches (required for the fast path) ---
      - USES_B12X=True
      - VLLM_USE_B12X_MOE=1
      - VLLM_USE_B12X_FP8_GEMM=1
      - VLLM_USE_B12X_WO_PROJECTION=1
      - VLLM_USE_B12X_MHC=1
      - VLLM_USE_B12X_SPARSE_INDEXER=1
      - B12X_MHC_MAX_TOKENS=16384
      - B12X_MLA_SM120_UNIFIED=1
      - B12X_DENSE_SPLITK_TURBO=1
      - B12X_W4A16_TC_DECODE=1
      - VLLM_PCIE_ALLREDUCE_BACKEND=b12x
      - VLLM_ENABLE_PCIE_ALLREDUCE=1
      - VLLM_USE_V2_MODEL_RUNNER=1
      - VLLM_USE_AOT_COMPILE=1
      - VLLM_USE_MEGA_AOT_ARTIFACT=1
      - VLLM_USE_BREAKABLE_CUDAGRAPH=0
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - VLLM_MEMORY_PROFILE_INCLUDE_ATTN=1
      - VLLM_PREFIX_CACHE_RETENTION_INTERVAL=4096
      # --- hardening: force offline so any outbound connection is suspicious ---
      - HF_HUB_OFFLINE=1
      - TRANSFORMERS_OFFLINE=1
      - HF_HUB_DISABLE_TELEMETRY=1
      - DO_NOT_TRACK=1
      - VLLM_NO_USAGE_STATS=1
      - VLLM_DO_NOT_TRACK=1
      - HF_HOME=/cache/hf   # writable; offline anyway. JIT caches use image's /cache/jit/* paths.
    volumes:
      - ~/models/DeepSeek-V4-Flash-0731:/models/ds4:ro
      - ~/vllm/ds4-cache:/cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]
    # Keep the image's nvidia_entrypoint.sh (CUDA setup); only replace the command it execs,
    # exactly how Nepherpitu invokes it (image CMD run-kimi26-vllm is just a stub).
    command:
      - /bin/bash
      - -lc
      - >
        unset NCCL_GRAPH_FILE NCCL_GRAPH_DUMP_FILE VLLM_B12X_MLA_EXTEND_MAX_CHUNKS;
        exec /opt/venv/bin/python -m vllm.entrypoints.cli.main serve /models/ds4
        --served-model-name deepseek-v4-flash latest /latest
        --host 0.0.0.0 --port 8080
        --tensor-parallel-size 2
        --kv-cache-dtype fp8
        --block-size 256
        --load-format safetensors
        --moe-backend b12x
        --linear-backend b12x
        --attention-backend B12X_MLA_SPARSE
        --gpu-memory-utilization 0.93
        --max-model-len 512000
        --max-num-seqs 16
        --max-num-batched-tokens 4096
        --max-cudagraph-capture-size 192
        --async-scheduling
        --no-scheduler-reserve-full-isl
        --enable-chunked-prefill
        --enable-prefix-caching
        --enable-flashinfer-autotune
        --tokenizer-mode deepseek_v4
        --tool-call-parser deepseek_v4
        --enable-auto-tool-choice
        --reasoning-parser deepseek_v4
        --default-chat-template-kwargs.thinking=true
        --default-chat-template-kwargs.reasoning_effort=high
        --compilation-config='{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'