MiroThinker-v1.0-30B-FP8

Model Description

This is an FP8 quantized version of miromind-ai/MiroThinker-v1.0-30B, created using llmcompressor (Neural Magic).

Key Benefits:

• ~50% smaller model size (30GB vs 60GB)
• ~2x faster inference on FP8-capable GPUs (Ada Lovelace, Hopper)
• Native vLLM and SGLang support
• Minimal quality loss with FP8 dynamic quantization

Quantization Details

Property	Value
Quantization Method	FP8 Dynamic (W8A8)
Weights Precision	FP8 E4M3 (8-bit)
Activations Precision	FP8 E4M3 (8-bit, dynamic)
Ignored Layers	lm_head (kept in BF16)
Quantization Tool	llmcompressor 0.12.2
Original Model Size	~60GB
Quantized Model Size	~30GB

Quantization Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: FP8_DYNAMIC

Quick Start with Docker

The easiest way to run this model. No setup required - just Docker with NVIDIA runtime.

Docker Compose (Recommended)

# Download docker-compose.yml
wget https://huggingface.co/Doradus/MiroThinker-v1.0-30B-FP8/raw/main/docker/docker-compose.yml

# Run with 2 GPUs (recommended)
docker compose up

# Or single GPU (not recommended - poor performance)
SINGLE_GPU=1 docker compose up

Docker Run

# TP=2 with 2 GPUs (recommended)
docker run --gpus all -p 8000:8000 \
  -v hf_cache:/root/.cache/huggingface \
  --shm-size=16g \
  vllm/vllm-openai:v0.11.2 \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --trust-remote-code

# Single GPU fallback (expect ~1-2 tok/s)
docker run --gpus '"device=0"' -p 8000:8000 \
  -v hf_cache:/root/.cache/huggingface \
  --shm-size=16g \
  vllm/vllm-openai:v0.11.2 \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 1 \
  --max-model-len 2048 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager \
  --trust-remote-code

Test the API

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Doradus/MiroThinker-v1.0-30B-FP8",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

Usage

vLLM (Recommended)

python -m vllm.entrypoints.openai.api_server \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --trust-remote-code

SGLang

python -m sglang.launch_server \
  --model-path Doradus/MiroThinker-v1.0-30B-FP8 \
  --host 0.0.0.0 \
  --port 8000 \
  --tp 2

Transformers (for inspection only)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Doradus/MiroThinker-v1.0-30B-FP8",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("Doradus/MiroThinker-v1.0-30B-FP8")

Recommended Inference Parameters

For optimal performance in agentic tasks (from the original MiroThinker documentation):

temperature = 1.0
top_p = 0.95
repetition_penalty = 1.05
max_context_length = 262144
max_tokens = 16384

Architecture Details

This is a Mixture of Experts (MoE) model based on Qwen3MoE architecture:

Property	Value
Total Parameters	~30B (all experts)
Active Parameters	~3.3B per forward pass
Hidden Size	2048
Attention Heads	32
KV Heads (GQA)	4
Layers	48
Experts	128 total, 8 active per token
MoE Intermediate Size	768 per expert
Max Context	262,144 tokens
Vocabulary	151,936 tokens

Hardware Requirements

VRAM Analysis

Model weights: 30GB (vs 57GB BF16 original)

Context Length	KV Cache (FP16)	Total VRAM	Fits Single GPU?
2K tokens	~0.1 GB	~31 GB	RTX 5090 (tight)
4K tokens	~0.2 GB	~31 GB	RTX 5090 (tight)
8K tokens	~0.4 GB	~31 GB	RTX 5090
16K tokens	~0.8 GB	~32 GB	A100-40GB
32K tokens	~1.6 GB	~33 GB	A100-40GB
64K tokens	~3.2 GB	~35 GB	A100-80GB
131K tokens	~6.4 GB	~38 GB	A100-80GB / H100
262K tokens	~12.8 GB	~45 GB	H100 or TP=2

KV cache calculated for GQA with 4 KV heads, 128 head_dim, 48 layers, FP16 KV

Recommended Configurations

GPU Setup	Max Context	Performance	Notes
1x RTX 4090 (24GB)	OOM	N/A	Model too large
1x RTX 5090 (32GB)	~2K tokens	~1-2 tok/s	Requires --enforce-eager, not recommended
2x RTX 4090 (24GB) TP=2	~16K tokens	~60 tok/s	Recommended consumer config
2x RTX 5090 (32GB) TP=2	~32K tokens	~80 tok/s	Recommended consumer config
1x A100-40GB	~8K tokens	~40 tok/s	Single GPU possible
2x A100-40GB TP=2	~64K tokens	~80 tok/s	Good production config
1x A100-80GB	~131K tokens	~60 tok/s	TP=1 possible
1x H100-80GB	~262K tokens	~120 tok/s	Full context, TP=1

Single 32GB GPU Limitations

The model weights alone require 29.2 GiB, leaving minimal headroom for KV cache on a 32GB GPU. Single RTX 5090 operation is technically possible but not recommended for production:

• Requires --enforce-eager (disables CUDA graphs, significant performance penalty)
• Maximum context: ~2048 tokens
• Throughput: ~1-2 tokens/second (severely memory-bound)
• No headroom for batched requests

If you only have a single 32GB GPU, this configuration will work but with poor performance:

python -m vllm.entrypoints.openai.api_server \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --max-model-len 2048 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager \
  --trust-remote-code

Strongly recommended: Use TP=2 with two 24GB+ GPUs for usable performance.

Note: FP8 inference requires CUDA compute capability 8.9+ (Ada Lovelace) or 9.0+ (Hopper) for optimal performance. On older GPUs, the model will run but may use fallback kernels.

Quality & Performance

FP8 vs BF16 Comparison

Metric	BF16 Original	FP8 Quantized	Delta
Model Size	57 GB	30 GB	-47%
Load Time	~45s	~25s	-44%
Memory BW	~2x	1x (baseline)	FP8 wins

Expected Quality Retention

FP8 dynamic quantization (W8A8) typically preserves >99% of model quality for reasoning tasks. The lm_head is kept in BF16 to maintain output distribution fidelity.

Why FP8 Dynamic?

• No calibration data needed (faster quantization)
• Dynamic activation quantization adapts per-token
• E4M3 format balances range and precision well for LLMs
• Native hardware support on Ada/Hopper (no overhead)

Original Model Benchmarks (from arXiv paper)

MiroThinker is an agentic research model designed for multi-turn tool use, not traditional LLM benchmarks. The original BF16 model was evaluated on agent-specific benchmarks requiring up to 600 tool calls per task:

Benchmark	MiroThinker-30B (BF16)	Description
GAIA	~70%	General AI Assistant (tool use)
BrowseComp	~40%	Web browsing comprehension
BrowseComp-ZH	~50%	Chinese web browsing
HLE-Text	~30%	Humanity's Last Exam

*Scores from paper scaling analysis

Note on FP8 quality: These agentic benchmarks require full agent infrastructure (browser, tools, multi-turn execution) and cannot be directly run on the quantized model in isolation. However, FP8 W8A8 dynamic quantization typically preserves >99% of model quality based on extensive research (Neural Magic, vLLM benchmarks).

Supplementary Benchmarks (lm-evaluation-harness)

For reference, we ran traditional LLM benchmarks, though these don't reflect the model's primary use case:

Benchmark	Metric	Score	Notes
IFEval	Instruction-level (loose)	46.0%	Instruction following
IFEval	Instruction-level (strict)	44.2%	Strict compliance
GSM8K (5-shot)	Exact Match (flexible)	18.0%	Not optimized for this

The GSM8K scores reflect the model's <think> block reasoning behavior (suited for agentic tasks) rather than direct answer generation.

Supplementary benchmarks run 2025-12-03 using lm-evaluation-harness

Measured Throughput (RTX PRO 6000 Blackwell, 96GB)

Tested on vLLM with TP=2, 32K max context:

Test Type	Tokens Generated	Time	Throughput
Short reasoning	100	0.83s	119.9 tok/s
Code generation	256	2.1s	121.7 tok/s
Long explanation	512	4.24s	120.8 tok/s
Average	868	7.17s	121.1 tok/s

VRAM Usage: ~45GB per GPU (TP=2) at 32K context

Tested 2025-12-03 on Doradus infrastructure with vLLM 0.11.x

Reproduction

To reproduce this quantization:

#!/usr/bin/env python3
"""
Quantize MiroThinker-30B to FP8 using llmcompressor (Neural Magic)
Dynamic quantization - no calibration data needed, fast conversion
Output is vLLM-compatible FP8
"""

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

MODEL_PATH = "miromind-ai/MiroThinker-v1.0-30B"
OUTPUT_PATH = "./MiroThinker-v1.0-30B-FP8"

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

oneshot(
    model=MODEL_PATH,
    output_dir=OUTPUT_PATH,
    recipe=recipe,
    num_calibration_samples=0,
    save_compressed=True,
)

Requirements:

pip install llmcompressor torch transformers accelerate

Original Model

This quantization is based on miromind-ai/MiroThinker-v1.0-30B.

MiroThinker v1.0 is an open-source research agent designed to advance tool-augmented reasoning and information-seeking capabilities. Key features:

• 256K context window
• Up to 600 tool calls per task
• Interactive scaling via RL training
• Strong performance on HLE-Text, BrowseComp, GAIA benchmarks

For full details, see the MiroThinker paper and GitHub repository.

License

This model inherits the MIT License from the original MiroThinker model.

Citation

If you use this model, please cite the original MiroThinker paper:

@article{miromind2025mirothinker,
  title={MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling},
  author={MiroMind Team and Bai, Song and Bing, Lidong and Chen, Carson and Chen, Guanzheng and Chen, Yuntao and Chen, Zhe and Chen, Ziyi and Dai, Jifeng and Dong, Xuan and others},
  journal={arXiv preprint arXiv:2511.11793},
  year={2025}
}

Acknowledgements

• MiroMind AI for the original MiroThinker model
• Neural Magic / vLLM for llmcompressor
• DoradusAI for the quantization