Commit
·
d93a0ca
0
Parent(s):
initial commit
Browse filesThis view is limited to 50 files because it contains too many changes.
See raw diff
- .gitattributes +36 -0
- NCCL_READY +0 -0
- README.md +68 -0
- STABLE +0 -0
- banner.png +0 -0
- chat_template.jinja +137 -0
- config.json +48 -0
- generation_config.json +10 -0
- model-00001-of-00048.safetensors +3 -0
- model-00002-of-00048.safetensors +3 -0
- model-00003-of-00048.safetensors +3 -0
- model-00004-of-00048.safetensors +3 -0
- model-00005-of-00048.safetensors +3 -0
- model-00006-of-00048.safetensors +3 -0
- model-00007-of-00048.safetensors +3 -0
- model-00008-of-00048.safetensors +3 -0
- model-00009-of-00048.safetensors +3 -0
- model-00010-of-00048.safetensors +3 -0
- model-00011-of-00048.safetensors +3 -0
- model-00012-of-00048.safetensors +3 -0
- model-00013-of-00048.safetensors +3 -0
- model-00014-of-00048.safetensors +3 -0
- model-00015-of-00048.safetensors +3 -0
- model-00016-of-00048.safetensors +3 -0
- model-00017-of-00048.safetensors +3 -0
- model-00018-of-00048.safetensors +3 -0
- model-00019-of-00048.safetensors +3 -0
- model-00020-of-00048.safetensors +3 -0
- model-00021-of-00048.safetensors +3 -0
- model-00022-of-00048.safetensors +3 -0
- model-00023-of-00048.safetensors +3 -0
- model-00024-of-00048.safetensors +3 -0
- model-00025-of-00048.safetensors +3 -0
- model-00026-of-00048.safetensors +3 -0
- model-00027-of-00048.safetensors +3 -0
- model-00028-of-00048.safetensors +3 -0
- model-00029-of-00048.safetensors +3 -0
- model-00030-of-00048.safetensors +3 -0
- model-00031-of-00048.safetensors +3 -0
- model-00032-of-00048.safetensors +3 -0
- model-00033-of-00048.safetensors +3 -0
- model-00034-of-00048.safetensors +3 -0
- model-00035-of-00048.safetensors +3 -0
- model-00036-of-00048.safetensors +3 -0
- model-00037-of-00048.safetensors +3 -0
- model-00038-of-00048.safetensors +3 -0
- model-00039-of-00048.safetensors +3 -0
- model-00040-of-00048.safetensors +3 -0
- model-00041-of-00048.safetensors +3 -0
- model-00042-of-00048.safetensors +3 -0
.gitattributes
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
NCCL_READY
ADDED
|
File without changes
|
README.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: transformers
|
| 3 |
+
tags:
|
| 4 |
+
- prime-rl
|
| 5 |
+
- verifiers
|
| 6 |
+
- prime-intellect
|
| 7 |
+
- reinforcement-learning
|
| 8 |
+
- reasoning
|
| 9 |
+
- agentic
|
| 10 |
+
- mixture-of-experts
|
| 11 |
+
license: mit
|
| 12 |
+
language:
|
| 13 |
+
- en
|
| 14 |
+
base_model:
|
| 15 |
+
- zai-org/GLM-4.5-Air-Base
|
| 16 |
+
pipeline_tag: text-generation
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
# INTELLECT-3.1
|
| 20 |
+
|
| 21 |
+
<div align="center">
|
| 22 |
+
<img src="https://huggingface.co/PrimeIntellect/INTELLECT-3/resolve/main/banner.png" alt="Prime Intellect Logo" />
|
| 23 |
+
</div>
|
| 24 |
+
|
| 25 |
+
<p align="center">
|
| 26 |
+
<strong>INTELLECT-3.1: A 100B+ MoE trained with large-scale RL</strong>
|
| 27 |
+
<br><br>
|
| 28 |
+
Trained with <a href="https://github.com/PrimeIntellect-ai/prime-rl">prime-rl</a> and <a href="https://github.com/PrimeIntellect-ai/verifiers">verifiers</a>
|
| 29 |
+
<br>
|
| 30 |
+
Environments released on <a href="https://app.primeintellect.ai/dashboard/environments">Environments Hub</a>
|
| 31 |
+
<br>
|
| 32 |
+
Read the <a href="https://primeintellect.ai/blog/intellect-3">Blog</a> & <a href="https://storage.googleapis.com/intellect-3-paper/INTELLECT_3_Technical_Report.pdf">Technical Report</a>
|
| 33 |
+
<br>
|
| 34 |
+
<a href="https://x.com/primeintellect">X</a> | <a href="https://discord.gg/RC5GvMbfDf">Discord</a> | <a href="https://app.primeintellect.ai/dashboard/create-cluster">Prime Intellect Platform</a>
|
| 35 |
+
</p>
|
| 36 |
+
|
| 37 |
+
## Introduction
|
| 38 |
+
|
| 39 |
+
**INTELLECT-3.1** is a 106B (A12B) parameter Mixture-of-Experts reasoning model built as a continued training of [INTELLECT-3](https://huggingface.co/PrimeIntellect/INTELLECT-3) with additional reinforcement learning on math, coding, software engineering, and agentic tasks.
|
| 40 |
+
|
| 41 |
+
Training was performed with [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) using environments built with the [verifiers](https://github.com/PrimeIntellect-ai/verifiers) library.
|
| 42 |
+
All training and evaluation environments are available on the [Environments Hub](https://app.primeintellect.ai/dashboard/environments).
|
| 43 |
+
|
| 44 |
+
The model, training frameworks, and environments are open-sourced under fully-permissive licenses (MIT and Apache 2.0).
|
| 45 |
+
|
| 46 |
+
For more details, see the [technical report](https://storage.googleapis.com/intellect-3-paper/INTELLECT_3_Technical_Report.pdf).
|
| 47 |
+
|
| 48 |
+
## Serving with vLLM
|
| 49 |
+
|
| 50 |
+
The model can be served on 2x H200s:
|
| 51 |
+
```bash
|
| 52 |
+
vllm serve PrimeIntellect/INTELLECT-3.1 \
|
| 53 |
+
--tensor-parallel-size 2 \
|
| 54 |
+
--enable-auto-tool-choice \
|
| 55 |
+
--tool-call-parser qwen3_coder \
|
| 56 |
+
--reasoning-parser deepseek_r1
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
## Citation
|
| 60 |
+
|
| 61 |
+
```bibtex
|
| 62 |
+
@misc{intellect3.1,
|
| 63 |
+
title={INTELLECT-3.1: Technical Report},
|
| 64 |
+
author={Prime Intellect Team},
|
| 65 |
+
year={2025},
|
| 66 |
+
url={https://huggingface.co/PrimeIntellect/INTELLECT-3.1}
|
| 67 |
+
}
|
| 68 |
+
```
|
STABLE
ADDED
|
File without changes
|
banner.png
ADDED
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{% macro render_extra_keys(json_dict, handled_keys) %}
|
| 2 |
+
{%- if json_dict is mapping %}
|
| 3 |
+
{%- for json_key in json_dict if json_key not in handled_keys %}
|
| 4 |
+
{%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
|
| 5 |
+
{{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
|
| 6 |
+
{%- else %}
|
| 7 |
+
{{-'\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
|
| 8 |
+
{%- endif %}
|
| 9 |
+
{%- endfor %}
|
| 10 |
+
{%- endif %}
|
| 11 |
+
{% endmacro %}
|
| 12 |
+
|
| 13 |
+
{%- if messages[0]["role"] == "system" %}
|
| 14 |
+
{%- set system_message = messages[0]["content"] %}
|
| 15 |
+
{%- set loop_messages = messages[1:] %}
|
| 16 |
+
{%- else %}
|
| 17 |
+
{%- set loop_messages = messages %}
|
| 18 |
+
{%- endif %}
|
| 19 |
+
|
| 20 |
+
{%- if not tools is defined %}
|
| 21 |
+
{%- set tools = [] %}
|
| 22 |
+
{%- endif %}
|
| 23 |
+
|
| 24 |
+
{%- if system_message is defined %}
|
| 25 |
+
{{- "<|im_start|>system\n" + system_message }}
|
| 26 |
+
{%- else %}
|
| 27 |
+
{%- if tools is iterable and tools | length > 0 %}
|
| 28 |
+
{{- "<|im_start|>system\nYou are INTELLECT-4, a helpful assistant developed by Prime Intellect, that can interact with a computer to solve tasks." }}
|
| 29 |
+
{%- endif %}
|
| 30 |
+
{%- endif %}
|
| 31 |
+
{%- if tools is iterable and tools | length > 0 %}
|
| 32 |
+
{{- "\n\n# Tools\n\nYou have access to the following functions:\n\n" }}
|
| 33 |
+
{{- "<tools>" }}
|
| 34 |
+
{%- for tool in tools %}
|
| 35 |
+
{%- if tool.function is defined %}
|
| 36 |
+
{%- set tool = tool.function %}
|
| 37 |
+
{%- endif %}
|
| 38 |
+
{{- "\n<function>\n<name>" ~ tool.name ~ "</name>" }}
|
| 39 |
+
{%- if tool.description is defined %}
|
| 40 |
+
{{- '\n<description>' ~ (tool.description | trim) ~ '</description>' }}
|
| 41 |
+
{%- endif %}
|
| 42 |
+
{{- '\n<parameters>' }}
|
| 43 |
+
{%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
|
| 44 |
+
{%- for param_name, param_fields in tool.parameters.properties|items %}
|
| 45 |
+
{{- '\n<parameter>' }}
|
| 46 |
+
{{- '\n<name>' ~ param_name ~ '</name>' }}
|
| 47 |
+
{%- if param_fields.type is defined %}
|
| 48 |
+
{{- '\n<type>' ~ (param_fields.type | string) ~ '</type>' }}
|
| 49 |
+
{%- endif %}
|
| 50 |
+
{%- if param_fields.description is defined %}
|
| 51 |
+
{{- '\n<description>' ~ (param_fields.description | trim) ~ '</description>' }}
|
| 52 |
+
{%- endif %}
|
| 53 |
+
{%- set handled_keys = ['name', 'type', 'description'] %}
|
| 54 |
+
{{- render_extra_keys(param_fields, handled_keys) }}
|
| 55 |
+
{{- '\n</parameter>' }}
|
| 56 |
+
{%- endfor %}
|
| 57 |
+
{%- endif %}
|
| 58 |
+
{% set handled_keys = ['type', 'properties'] %}
|
| 59 |
+
{{- render_extra_keys(tool.parameters, handled_keys) }}
|
| 60 |
+
{{- '\n</parameters>' }}
|
| 61 |
+
{%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
|
| 62 |
+
{{- render_extra_keys(tool, handled_keys) }}
|
| 63 |
+
{{- '\n</function>' }}
|
| 64 |
+
{%- endfor %}
|
| 65 |
+
{{- "\n</tools>" }}
|
| 66 |
+
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
| 67 |
+
{%- endif %}
|
| 68 |
+
{%- if system_message is defined %}
|
| 69 |
+
{{- '<|im_end|>\n' }}
|
| 70 |
+
{%- else %}
|
| 71 |
+
{%- if tools is iterable and tools | length > 0 %}
|
| 72 |
+
{{- '<|im_end|>\n' }}
|
| 73 |
+
{%- endif %}
|
| 74 |
+
{%- endif %}
|
| 75 |
+
{%- for message in loop_messages %}
|
| 76 |
+
{%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
|
| 77 |
+
{{- '<|im_start|>' + message.role }}
|
| 78 |
+
{%- if message.content is defined and message.content is string %}
|
| 79 |
+
{%- if message.reasoning_content is defined -%}
|
| 80 |
+
{%- if message.reasoning_content -%}
|
| 81 |
+
{{ '\n<think>' + message.reasoning_content.strip() + '</think>' }}
|
| 82 |
+
{%- else -%}
|
| 83 |
+
{{ '\n<think></think>' }}
|
| 84 |
+
{%- endif -%}
|
| 85 |
+
{%- endif -%}
|
| 86 |
+
{{- '\n' + message.content | trim + '\n' }}
|
| 87 |
+
{%- endif %}
|
| 88 |
+
{%- for tool_call in message.tool_calls %}
|
| 89 |
+
{%- if tool_call.function is defined %}
|
| 90 |
+
{%- set tool_call = tool_call.function %}
|
| 91 |
+
{%- endif %}
|
| 92 |
+
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 93 |
+
{%- if tool_call.arguments is defined %}
|
| 94 |
+
{%- for args_name, args_value in tool_call.arguments|items %}
|
| 95 |
+
{{- '<parameter=' + args_name + '>\n' }}
|
| 96 |
+
{%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
|
| 97 |
+
{{- args_value }}
|
| 98 |
+
{{- '\n</parameter>\n' }}
|
| 99 |
+
{%- endfor %}
|
| 100 |
+
{%- endif %}
|
| 101 |
+
{{- '</function>\n</tool_call>' }}
|
| 102 |
+
{%- endfor %}
|
| 103 |
+
{{- '<|im_end|>\n' }}
|
| 104 |
+
{%- elif message.role == "user" or message.role == "system" or message.role == "assistant" %}
|
| 105 |
+
{{- '<|im_start|>' + message.role }}
|
| 106 |
+
{%- if message.role == "assistant" and message.reasoning_content is defined %}
|
| 107 |
+
{%- if message.reasoning_content -%}
|
| 108 |
+
{{ '\n<think>' + message.reasoning_content.strip() + '</think>' }}
|
| 109 |
+
{%- else -%}
|
| 110 |
+
{{ '\n<think></think>' }}
|
| 111 |
+
{%- endif -%}
|
| 112 |
+
{%- if message.content.strip() -%}
|
| 113 |
+
{{ '\n' + message.content.strip() }}
|
| 114 |
+
{%- endif -%}
|
| 115 |
+
{%- else %}
|
| 116 |
+
{{- '\n' + message.content }}
|
| 117 |
+
{%- endif %}
|
| 118 |
+
{{- '<|im_end|>' + '\n' }}
|
| 119 |
+
{%- elif message.role == "tool" %}
|
| 120 |
+
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
| 121 |
+
{{- '<|im_start|>user\n' }}
|
| 122 |
+
{%- endif %}
|
| 123 |
+
{{- '<tool_response>\n' }}
|
| 124 |
+
{{- message.content }}
|
| 125 |
+
{{- '\n</tool_response>\n' }}
|
| 126 |
+
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
| 127 |
+
{{- '<|im_end|>\n' }}
|
| 128 |
+
{%- elif loop.last %}
|
| 129 |
+
{{- '<|im_end|>\n' }}
|
| 130 |
+
{%- endif %}
|
| 131 |
+
{%- else %}
|
| 132 |
+
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
|
| 133 |
+
{%- endif %}
|
| 134 |
+
{%- endfor %}
|
| 135 |
+
{%- if add_generation_prompt %}
|
| 136 |
+
{{- '<|im_start|>assistant\n<think>' }}
|
| 137 |
+
{%- endif %}
|
config.json
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"Glm4MoeForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attention_bias": true,
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"auto_map": {
|
| 8 |
+
"AutoConfig": "configuration_glm4_moe.Glm4MoeConfig",
|
| 9 |
+
"AutoModel": "modeling_glm4_moe.Glm4MoeModel",
|
| 10 |
+
"AutoModelForCausalLM": "modeling_glm4_moe.Glm4MoeForCausalLM"
|
| 11 |
+
},
|
| 12 |
+
"dtype": "bfloat16",
|
| 13 |
+
"eos_token_id": [
|
| 14 |
+
151334,
|
| 15 |
+
151329
|
| 16 |
+
],
|
| 17 |
+
"first_k_dense_replace": 1,
|
| 18 |
+
"head_dim": 128,
|
| 19 |
+
"hidden_act": "silu",
|
| 20 |
+
"hidden_size": 4096,
|
| 21 |
+
"initializer_range": 0.02,
|
| 22 |
+
"intermediate_size": 10944,
|
| 23 |
+
"max_position_embeddings": 131072,
|
| 24 |
+
"model_type": "glm4_moe",
|
| 25 |
+
"moe_intermediate_size": 1408,
|
| 26 |
+
"n_group": 1,
|
| 27 |
+
"n_routed_experts": 128,
|
| 28 |
+
"n_shared_experts": 1,
|
| 29 |
+
"norm_topk_prob": true,
|
| 30 |
+
"num_attention_heads": 96,
|
| 31 |
+
"num_experts_per_tok": 8,
|
| 32 |
+
"num_hidden_layers": 46,
|
| 33 |
+
"num_key_value_heads": 8,
|
| 34 |
+
"num_nextn_predict_layers": 1,
|
| 35 |
+
"pad_token_id": 151329,
|
| 36 |
+
"partial_rotary_factor": 0.5,
|
| 37 |
+
"rms_norm_eps": 1e-05,
|
| 38 |
+
"rope_scaling": null,
|
| 39 |
+
"rope_theta": 1000000,
|
| 40 |
+
"routed_scaling_factor": 1.0,
|
| 41 |
+
"tie_word_embeddings": false,
|
| 42 |
+
"topk_group": 1,
|
| 43 |
+
"transformers_version": "4.56.1",
|
| 44 |
+
"use_cache": false,
|
| 45 |
+
"use_grouped_mm": true,
|
| 46 |
+
"use_qk_norm": false,
|
| 47 |
+
"vocab_size": 151552
|
| 48 |
+
}
|
generation_config.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_from_model_config": true,
|
| 3 |
+
"eos_token_id": [
|
| 4 |
+
151334,
|
| 5 |
+
151329
|
| 6 |
+
],
|
| 7 |
+
"pad_token_id": 151329,
|
| 8 |
+
"transformers_version": "4.56.1",
|
| 9 |
+
"use_cache": false
|
| 10 |
+
}
|
model-00001-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:967b90d4be2368531443fb9adb8366c41bd8c725293d5d1b771b7abc33e8444b
|
| 3 |
+
size 4900192568
|
model-00002-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:207ffd9658972bec70e358510cd809d1475509b730e21b3d213cfe9201fe5432
|
| 3 |
+
size 4899977872
|
model-00003-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b42cefa57b2af4918421e0a9948eaa361095828a5cc0cd0512d5c37a54f15fdc
|
| 3 |
+
size 4498785360
|
model-00004-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:98ca5291b97e429fd13d1eabf1a275fd10e8c8a0c477e32cfe1d93602ced8cd1
|
| 3 |
+
size 4464885360
|
model-00005-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f46ea18df396aec4917c3609568a6d8c77b26d17c7d86aa46322fb07d765ed19
|
| 3 |
+
size 4464885360
|
model-00006-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ac72dbc54e34505b9a67900411f8d8f735db38e0753d8c6793a544040c6936f6
|
| 3 |
+
size 4464885360
|
model-00007-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:52e89f060ba3dd4bc60fcf2e2fc72fd5dc20c880183ad2beb357b95dd86e2c53
|
| 3 |
+
size 4464885360
|
model-00008-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9609fdf9a158b4d0e42807a3e2a6191eef3890c0799a87def86b2cc1eb652925
|
| 3 |
+
size 4464885360
|
model-00009-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:945d345943b9a9f83cd3620eccee5a1abdd3d140c859668ffdb27f1f2a9deea4
|
| 3 |
+
size 4464885360
|
model-00010-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2cf5fe8a934476572d5a1c0e9f49ec94f62093b139e26f9604911fc948964806
|
| 3 |
+
size 4464885360
|
model-00011-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:dd7050d5120945e22cbd1fad72a236866b58d4e9bfe71a35bc62ff8f7a48a04a
|
| 3 |
+
size 4464885360
|
model-00012-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:beebfb05c494db2beda081c60e92710c3b878ceef6daf8b3f07cba90ea9ea3ea
|
| 3 |
+
size 4464885496
|
model-00013-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3cfd505ddf9a1e6b5d9ce64b73be6d211c46f316006887848b9f0f8de3742def
|
| 3 |
+
size 4464885752
|
model-00014-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c5a8f4c75aa38211be6dfce55d2962b0297956d636bd7a895a35e0d4ee4e6de1
|
| 3 |
+
size 4464885752
|
model-00015-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2341508ad8d980555684ab079eaa3f1ae4876b2b56c736d87f3e4ec91b46e597
|
| 3 |
+
size 4464885752
|
model-00016-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2ea5a9f64286fb5a721674da6be19dd22a99500e24a05c8ff0b12d9be02213ab
|
| 3 |
+
size 4464885752
|
model-00017-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:45bcad8d7734173abc7dadae105a82558fe8709d366d9d3e55779c8ff85562e7
|
| 3 |
+
size 4464885752
|
model-00018-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:dc1cf09608b51d266df825ccf74f93bec92b1fc3db684d9d6eeff8c81dd31341
|
| 3 |
+
size 4464885752
|
model-00019-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bc08c2864c2184941f3480073448db02eda18959f7b8270976f078f489e59d91
|
| 3 |
+
size 4464885752
|
model-00020-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3ffff3e4771e1cfaa129360b7c20ad0ee66f393d1cbf38d58509002ac02e4f7b
|
| 3 |
+
size 4464885752
|
model-00021-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d34db1267502392d821a269970d472110a5cabc3a06f416fb30b0362c4e7ee64
|
| 3 |
+
size 4464885752
|
model-00022-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3cfd9ca0f1e1bc78471572fdeb5062c9d5aecb954b64e4afc606c5226d58730f
|
| 3 |
+
size 4464885752
|
model-00023-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4d9bdf0743557667a293b68300c16b1c96118dbb958f575c975af4f5361bb2e8
|
| 3 |
+
size 4464885752
|
model-00024-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:48e9d683696914f1de83464546eb67c5f48b02de75a650143af696de0d57aee6
|
| 3 |
+
size 4464885752
|
model-00025-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cc960e31117861d4d921ad4f8346d845cbcf7d4ca5ecff02dbd078866969d7e0
|
| 3 |
+
size 4464885752
|
model-00026-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8fe9fd855efe8bc05a853779831dbc3386055812badee4e8f4b54f82da4cbf2d
|
| 3 |
+
size 4464885752
|
model-00027-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cd6ac423a7a2614fa271d58a07dccba7acdbd6dfad3b02a4a20d3be4417ecfac
|
| 3 |
+
size 4464885752
|
model-00028-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c9d1973ec13ba62da83a89950041ddf66e4281ad42f2ef288e06b1e98acf0efa
|
| 3 |
+
size 4464885752
|
model-00029-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1093a9c656afa78bf0cbcfb076224d09935761fa876a7e4f08a4bf32e574340f
|
| 3 |
+
size 4464885752
|
model-00030-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4d3e06e46d7485d17936b06f018a8ffba7c1de863572f4a6fff7c73d1ce6f4b7
|
| 3 |
+
size 4464885752
|
model-00031-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b0c222c8e1c4bbd274f9a2bf76a9f515729f5dcb7d29b45d6081382bcc6ac00a
|
| 3 |
+
size 4464885752
|
model-00032-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7cca3840390f9ea224783b133f20720e790fec52ebe2c319c84505f4bfdc35b5
|
| 3 |
+
size 4464885752
|
model-00033-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:da66e0a649c4727669c53e61f3a9a16363cb1d25ecfe18d881786c703ceb29ff
|
| 3 |
+
size 4464885752
|
model-00034-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:049ef2fffb7dd275cac9cd4408263d5950a4a0830b6c31160f556a511e0ac90b
|
| 3 |
+
size 4464885752
|
model-00035-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5a8e4fd694bb02b5d99595f679b40505bd1ad916349845bde834b592b6693a98
|
| 3 |
+
size 4464885752
|
model-00036-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:41cb430e60029d4e5e4b44d9a9902bd830c0d70a14485a57c0b0e8e20f3b3ad3
|
| 3 |
+
size 4464885752
|
model-00037-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1bc707e48560d0d180fd92605b1684256a255ac8bc01dae71fbcbdd6fea71349
|
| 3 |
+
size 4464885752
|
model-00038-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cfd6a6e0394fe93187d246c3758b2377a882db5df3a686254466b0c179ef925a
|
| 3 |
+
size 4464885752
|
model-00039-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d256836cdc23400f874a012fc669278a11161d1ec05253bb2255473d57d62b89
|
| 3 |
+
size 4464885752
|
model-00040-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:23581accb9dd228fffbdb4359fbba9c707babe47f65e588a8b38e2e565e6b4a6
|
| 3 |
+
size 4464885752
|
model-00041-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d9fcc2b5950f69b2a7446f67c72430fd1d154ebc1e0fbb18669213fb2f63456e
|
| 3 |
+
size 4464885752
|
model-00042-of-00048.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3d7543c9661050a50e102b8ed9076bea819167abbcb1f9c338fc32db3f0f33cf
|
| 3 |
+
size 4464885752
|