LM Studio Support with Q4_K_S please?
#5
by
Cagannn
- opened
README.md
CHANGED
|
@@ -25,8 +25,6 @@ library_name: transformers
|
|
| 25 |
| IQuest-Coder-V1-40B-Instruct | [🤗 Hugging Face](https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Instruct) |
|
| 26 |
| IQuest-Coder-V1-40B-Loop-Instruct | [🤗 Hugging Face](https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Loop-Instruct) |
|
| 27 |
|
| 28 |
-
[Clarification: Regarding the Performance of IQuest-Coder-V1](https://github.com/IQuestLab/IQuest-Coder-V1/issues/14#issuecomment-3705756919)
|
| 29 |
-
|
| 30 |
## Sampling Parameters
|
| 31 |
For the IQuest-Coder-V1-Instruct: We suggest using Temperature=0.6, TopP=0.85, TopK=20.
|
| 32 |
|
|
@@ -124,13 +122,13 @@ model_name = "IQuest/IQuest-Coder-V1-40B-Thinking"
|
|
| 124 |
For production deployment, you can use vLLM to create an OpenAI-compatible API endpoint. Please refer to the [vLLM PR](https://github.com/vllm-project/vllm/pull/31575/files) for implementation details.
|
| 125 |
|
| 126 |
```bash
|
| 127 |
-
vllm serve
|
| 128 |
```
|
| 129 |
|
| 130 |
For Thinking models with reasoning support:
|
| 131 |
|
| 132 |
```bash
|
| 133 |
-
vllm serve
|
| 134 |
```
|
| 135 |
|
| 136 |
|
|
|
|
| 25 |
| IQuest-Coder-V1-40B-Instruct | [🤗 Hugging Face](https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Instruct) |
|
| 26 |
| IQuest-Coder-V1-40B-Loop-Instruct | [🤗 Hugging Face](https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Loop-Instruct) |
|
| 27 |
|
|
|
|
|
|
|
| 28 |
## Sampling Parameters
|
| 29 |
For the IQuest-Coder-V1-Instruct: We suggest using Temperature=0.6, TopP=0.85, TopK=20.
|
| 30 |
|
|
|
|
| 122 |
For production deployment, you can use vLLM to create an OpenAI-compatible API endpoint. Please refer to the [vLLM PR](https://github.com/vllm-project/vllm/pull/31575/files) for implementation details.
|
| 123 |
|
| 124 |
```bash
|
| 125 |
+
vllm serve IQuest/IQuest-Coder-V1-40B-Instruct --tensor-parallel-size 8
|
| 126 |
```
|
| 127 |
|
| 128 |
For Thinking models with reasoning support:
|
| 129 |
|
| 130 |
```bash
|
| 131 |
+
vllm serve IQuest/IQuest-Coder-V1-40B-Thinking --reasoning-parser qwen3 --tensor-parallel-size 8
|
| 132 |
```
|
| 133 |
|
| 134 |
|