README.md

---
library_name: transformers
pipeline_tag: text-generation
license: mit
language:
- en
base_model:
- miromind-ai/MiroThinker-v1.0-30B
tags:
- agent
- open-source
- miromind
- deep-research
- fp8
- quantized
- vllm
- sglang
---

# MiroThinker-v1.0-30B-FP8

<div align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/68525b342230a897a65cc1c0/87mYQ_a-4jpnMkVR4hrgm.png" width="55%" alt="MiroThinker" />
</div>

## Model Description

This is an **FP8 quantized** version of [miromind-ai/MiroThinker-v1.0-30B](https://huggingface.co/miromind-ai/MiroThinker-v1.0-30B), created using [llmcompressor](https://github.com/vllm-project/llm-compressor) (Neural Magic).

**Key Benefits:**
- ~50% smaller model size (30GB vs 60GB)
- ~2x faster inference on FP8-capable GPUs (Ada Lovelace, Hopper)
- Native vLLM and SGLang support
- Minimal quality loss with FP8 dynamic quantization

## Quantization Details

| Property | Value |
|----------|-------|
| Quantization Method | FP8 Dynamic (W8A8) |
| Weights Precision | FP8 E4M3 (8-bit) |
| Activations Precision | FP8 E4M3 (8-bit, dynamic) |
| Ignored Layers | `lm_head` (kept in BF16) |
| Quantization Tool | llmcompressor 0.12.2 |
| Original Model Size | ~60GB |
| Quantized Model Size | ~30GB |

### Quantization Recipe

```yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: FP8_DYNAMIC
```

## Quick Start with Docker

The easiest way to run this model. No setup required - just Docker with NVIDIA runtime.

### Docker Compose (Recommended)

```bash
# Download docker-compose.yml
wget https://huggingface.co/Doradus/MiroThinker-v1.0-30B-FP8/raw/main/docker/docker-compose.yml

# Run with 2 GPUs (recommended)
docker compose up

# Or single GPU (not recommended - poor performance)
SINGLE_GPU=1 docker compose up
```

### Docker Run

```bash
# TP=2 with 2 GPUs (recommended)
docker run --gpus all -p 8000:8000 \
  -v hf_cache:/root/.cache/huggingface \
  --shm-size=16g \
  vllm/vllm-openai:v0.11.2 \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --trust-remote-code

# Single GPU fallback (expect ~1-2 tok/s)
docker run --gpus '"device=0"' -p 8000:8000 \
  -v hf_cache:/root/.cache/huggingface \
  --shm-size=16g \
  vllm/vllm-openai:v0.11.2 \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 1 \
  --max-model-len 2048 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager \
  --trust-remote-code
```

### Test the API

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Doradus/MiroThinker-v1.0-30B-FP8",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'
```

## Usage

### vLLM (Recommended)

```bash
python -m vllm.entrypoints.openai.api_server \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --trust-remote-code
```

### SGLang

```bash
python -m sglang.launch_server \
  --model-path Doradus/MiroThinker-v1.0-30B-FP8 \
  --host 0.0.0.0 \
  --port 8000 \
  --tp 2
```

### Transformers (for inspection only)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Doradus/MiroThinker-v1.0-30B-FP8",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("Doradus/MiroThinker-v1.0-30B-FP8")
```

## Recommended Inference Parameters

For optimal performance in agentic tasks (from the original MiroThinker documentation):

```python
temperature = 1.0
top_p = 0.95
repetition_penalty = 1.05
max_context_length = 262144
max_tokens = 16384
```

## Architecture Details

This is a **Mixture of Experts (MoE)** model based on Qwen3MoE architecture:

| Property | Value |
|----------|-------|
| Total Parameters | ~30B (all experts) |
| Active Parameters | ~3.3B per forward pass |
| Hidden Size | 2048 |
| Attention Heads | 32 |
| KV Heads (GQA) | 4 |
| Layers | 48 |
| Experts | 128 total, 8 active per token |
| MoE Intermediate Size | 768 per expert |
| Max Context | 262,144 tokens |
| Vocabulary | 151,936 tokens |

## Hardware Requirements

### VRAM Analysis

Model weights: **30GB** (vs 57GB BF16 original)

| Context Length | KV Cache (FP16) | Total VRAM | Fits Single GPU? |
|----------------|-----------------|------------|------------------|
| 2K tokens | ~0.1 GB | ~31 GB | RTX 5090 (tight) |
| 4K tokens | ~0.2 GB | ~31 GB | RTX 5090 (tight) |
| 8K tokens | ~0.4 GB | ~31 GB | RTX 5090 |
| 16K tokens | ~0.8 GB | ~32 GB | A100-40GB |
| 32K tokens | ~1.6 GB | ~33 GB | A100-40GB |
| 64K tokens | ~3.2 GB | ~35 GB | A100-80GB |
| 131K tokens | ~6.4 GB | ~38 GB | A100-80GB / H100 |
| 262K tokens | ~12.8 GB | ~45 GB | H100 or TP=2 |

*KV cache calculated for GQA with 4 KV heads, 128 head_dim, 48 layers, FP16 KV*

### Recommended Configurations

| GPU Setup | Max Context | Performance | Notes |
|-----------|-------------|-------------|-------|
| 1x RTX 4090 (24GB) | OOM | N/A | Model too large |
| 1x RTX 5090 (32GB) | ~2K tokens | ~1-2 tok/s | Requires `--enforce-eager`, **not recommended** |
| **2x RTX 4090 (24GB) TP=2** | ~16K tokens | ~60 tok/s | **Recommended consumer config** |
| **2x RTX 5090 (32GB) TP=2** | ~32K tokens | ~80 tok/s | **Recommended consumer config** |
| 1x A100-40GB | ~8K tokens | ~40 tok/s | Single GPU possible |
| 2x A100-40GB TP=2 | ~64K tokens | ~80 tok/s | Good production config |
| 1x A100-80GB | ~131K tokens | ~60 tok/s | TP=1 possible |
| 1x H100-80GB | ~262K tokens | ~120 tok/s | Full context, TP=1 |

### Single 32GB GPU Limitations

The model weights alone require **29.2 GiB**, leaving minimal headroom for KV cache on a 32GB GPU. Single RTX 5090 operation is technically possible but **not recommended for production**:

- Requires `--enforce-eager` (disables CUDA graphs, significant performance penalty)
- Maximum context: ~2048 tokens
- Throughput: ~1-2 tokens/second (severely memory-bound)
- No headroom for batched requests

**If you only have a single 32GB GPU**, this configuration will work but with poor performance:
```bash
python -m vllm.entrypoints.openai.api_server \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --max-model-len 2048 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager \
  --trust-remote-code
```

**Strongly recommended**: Use TP=2 with two 24GB+ GPUs for usable performance.

**Note**: FP8 inference requires CUDA compute capability 8.9+ (Ada Lovelace) or 9.0+ (Hopper) for optimal performance. On older GPUs, the model will run but may use fallback kernels.

## Quality & Performance

### FP8 vs BF16 Comparison

| Metric | BF16 Original | FP8 Quantized | Delta |
|--------|---------------|---------------|-------|
| Model Size | 57 GB | 30 GB | **-47%** |
| Load Time | ~45s | ~25s | **-44%** |
| Memory BW | ~2x | 1x (baseline) | FP8 wins |

### Expected Quality Retention

FP8 dynamic quantization (W8A8) typically preserves >99% of model quality for reasoning tasks. The `lm_head` is kept in BF16 to maintain output distribution fidelity.

**Why FP8 Dynamic?**
- No calibration data needed (faster quantization)
- Dynamic activation quantization adapts per-token
- E4M3 format balances range and precision well for LLMs
- Native hardware support on Ada/Hopper (no overhead)

### Original Model Benchmarks (from [arXiv paper](https://arxiv.org/abs/2511.11793))

MiroThinker is an **agentic research model** designed for multi-turn tool use, not traditional LLM benchmarks. The original BF16 model was evaluated on agent-specific benchmarks requiring up to 600 tool calls per task:

| Benchmark | MiroThinker-30B (BF16) | Description |
|-----------|------------------------|-------------|
| **GAIA** | ~70% | General AI Assistant (tool use) |
| **BrowseComp** | ~40% | Web browsing comprehension |
| **BrowseComp-ZH** | ~50% | Chinese web browsing |
| **HLE-Text** | ~30% | Humanity's Last Exam |

*Scores from paper scaling analysis

**Note on FP8 quality**: These agentic benchmarks require full agent infrastructure (browser, tools, multi-turn execution) and cannot be directly run on the quantized model in isolation. However, FP8 W8A8 dynamic quantization typically preserves **>99% of model quality** based on extensive research ([Neural Magic](https://neuralmagic.com/blog/fp8-quantization/), [vLLM benchmarks](https://docs.vllm.ai/en/latest/quantization/fp8.html)).

### Supplementary Benchmarks (lm-evaluation-harness)

For reference, we ran traditional LLM benchmarks, though these don't reflect the model's primary use case:

| Benchmark | Metric | Score | Notes |
|-----------|--------|-------|-------|
| IFEval | Instruction-level (loose) | 46.0% | Instruction following |
| IFEval | Instruction-level (strict) | 44.2% | Strict compliance |
| GSM8K (5-shot) | Exact Match (flexible) | 18.0% | Not optimized for this |

The GSM8K scores reflect the model's `<think>` block reasoning behavior (suited for agentic tasks) rather than direct answer generation.

*Supplementary benchmarks run 2025-12-03 using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)*

### Measured Throughput (RTX PRO 6000 Blackwell, 96GB)

Tested on vLLM with TP=2, 32K max context:

| Test Type | Tokens Generated | Time | Throughput |
|-----------|------------------|------|------------|
| Short reasoning | 100 | 0.83s | **119.9 tok/s** |
| Code generation | 256 | 2.1s | **121.7 tok/s** |
| Long explanation | 512 | 4.24s | **120.8 tok/s** |
| **Average** | 868 | 7.17s | **121.1 tok/s** |

**VRAM Usage**: ~45GB per GPU (TP=2) at 32K context

*Tested 2025-12-03 on Doradus infrastructure with vLLM 0.11.x*

## Reproduction

To reproduce this quantization:

```python
#!/usr/bin/env python3
"""
Quantize MiroThinker-30B to FP8 using llmcompressor (Neural Magic)
Dynamic quantization - no calibration data needed, fast conversion
Output is vLLM-compatible FP8
"""

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

MODEL_PATH = "miromind-ai/MiroThinker-v1.0-30B"
OUTPUT_PATH = "./MiroThinker-v1.0-30B-FP8"

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

oneshot(
    model=MODEL_PATH,
    output_dir=OUTPUT_PATH,
    recipe=recipe,
    num_calibration_samples=0,
    save_compressed=True,
)
```

**Requirements:**
```
pip install llmcompressor torch transformers accelerate
```

## Original Model

This quantization is based on [miromind-ai/MiroThinker-v1.0-30B](https://huggingface.co/miromind-ai/MiroThinker-v1.0-30B).

MiroThinker v1.0 is an open-source research agent designed to advance tool-augmented reasoning and information-seeking capabilities. Key features:

- 256K context window
- Up to 600 tool calls per task
- Interactive scaling via RL training
- Strong performance on HLE-Text, BrowseComp, GAIA benchmarks

For full details, see the [MiroThinker paper](https://arxiv.org/abs/2511.11793) and [GitHub repository](https://github.com/MiroMindAI/MiroThinker).

## License

This model inherits the **MIT License** from the original MiroThinker model.

## Citation

If you use this model, please cite the original MiroThinker paper:

```bibtex
@article{miromind2025mirothinker,
  title={MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling},
  author={MiroMind Team and Bai, Song and Bing, Lidong and Chen, Carson and Chen, Guanzheng and Chen, Yuntao and Chen, Zhe and Chen, Ziyi and Dai, Jifeng and Dong, Xuan and others},
  journal={arXiv preprint arXiv:2511.11793},
  year={2025}
}
```

## Acknowledgements

- [MiroMind AI](https://miromind.ai/) for the original MiroThinker model
- [Neural Magic / vLLM](https://github.com/vllm-project/llm-compressor) for llmcompressor
- [DoradusAI](https://doradusonline.com) for the quantization