---
base_model:
- ByteDance-Seed/Seed-Coder-8B-Instruct
---
vllm (pretrained=/root/autodl-tmp/Seed-Coder-8B-Instruct,add_bos_token=true,max_model_len=3096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.576|±  |0.0313|
|     |       |strict-match    |     5|exact_match|↑  |0.576|±  |0.0313|

vllm (pretrained=/root/autodl-tmp/Seed-Coder-8B-Instruct,add_bos_token=true,max_model_len=3096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.602|±  |0.0219|
|     |       |strict-match    |     5|exact_match|↑  |0.598|±  |0.0219|

vllm (pretrained=/root/autodl-tmp/Seed-Coder-8B-Instruct,add_bos_token=true,max_model_len=3048,dtype=bfloat16), gen_kwargs: (None), limit: 15.0, num_fewshot: None, batch_size: auto
|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.4386|±  |0.0167|
| - humanities     |      2|none  |      |acc   |↑  |0.4000|±  |0.0343|
| - other          |      2|none  |      |acc   |↑  |0.4872|±  |0.0356|
| - social sciences|      2|none  |      |acc   |↑  |0.4389|±  |0.0364|
| - stem           |      2|none  |      |acc   |↑  |0.4316|±  |0.0288|


vllm (pretrained=/root/autodl-tmp/80-128,add_bos_token=true,max_model_len=3096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  | 0.56|±  |0.0315|
|     |       |strict-match    |     5|exact_match|↑  | 0.56|±  |0.0315|

vllm (pretrained=/root/autodl-tmp/80-128,add_bos_token=true,max_model_len=3096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.590|±  |0.0220|
|     |       |strict-match    |     5|exact_match|↑  |0.584|±  |0.0221|

vllm (pretrained=/root/autodl-tmp/80-128,add_bos_token=true,max_model_len=3048,dtype=bfloat16), gen_kwargs: (None), limit: 15.0, num_fewshot: None, batch_size: auto
|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.4339|±  |0.0166|
| - humanities     |      2|none  |      |acc   |↑  |0.3949|±  |0.0338|
| - other          |      2|none  |      |acc   |↑  |0.4769|±  |0.0355|
| - social sciences|      2|none  |      |acc   |↑  |0.4333|±  |0.0361|
| - stem           |      2|none  |      |acc   |↑  |0.4316|±  |0.0290|


vllm (pretrained=/root/autodl-tmp/80-256,add_bos_token=true,max_model_len=3096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.584|±  |0.0312|
|     |       |strict-match    |     5|exact_match|↑  |0.584|±  |0.0312|

vllm (pretrained=/root/autodl-tmp/80-256,add_bos_token=true,max_model_len=3096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.590|±  | 0.022|
|     |       |strict-match    |     5|exact_match|↑  |0.586|±  | 0.022|

vllm (pretrained=/root/autodl-tmp/80-256,add_bos_token=true,max_model_len=3048,dtype=bfloat16), gen_kwargs: (None), limit: 15.0, num_fewshot: None, batch_size: auto
|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.4246|±  |0.0165|
| - humanities     |      2|none  |      |acc   |↑  |0.3795|±  |0.0336|
| - other          |      2|none  |      |acc   |↑  |0.4872|±  |0.0356|
| - social sciences|      2|none  |      |acc   |↑  |0.4333|±  |0.0360|
| - stem           |      2|none  |      |acc   |↑  |0.4070|±  |0.0282|


vllm (pretrained=/root/autodl-tmp/80-512,add_bos_token=true,max_model_len=3096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.604|±  | 0.031|
|     |       |strict-match    |     5|exact_match|↑  |0.600|±  | 0.031|

vllm (pretrained=/root/autodl-tmp/80-512,add_bos_token=true,max_model_len=3096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.594|±  | 0.022|
|     |       |strict-match    |     5|exact_match|↑  |0.586|±  | 0.022|

vllm (pretrained=/root/autodl-tmp/80-512,add_bos_token=true,max_model_len=3048,dtype=bfloat16), gen_kwargs: (None), limit: 15.0, num_fewshot: None, batch_size: auto
|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.4316|±  |0.0166|
| - humanities     |      2|none  |      |acc   |↑  |0.4000|±  |0.0341|
| - other          |      2|none  |      |acc   |↑  |0.4821|±  |0.0355|
| - social sciences|      2|none  |      |acc   |↑  |0.4278|±  |0.0356|
| - stem           |      2|none  |      |acc   |↑  |0.4211|±  |0.0289|