|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: Kwaipilot/KAT-Dev-72B-Exp |
|
|
tags: |
|
|
- gptq |
|
|
- quantized |
|
|
- vllm |
|
|
- 4bit |
|
|
- group_size_32 |
|
|
quantization_config: |
|
|
quant_method: gptq |
|
|
bits: 4 |
|
|
group_size: 32 |
|
|
damp_percent: 0.1 |
|
|
--- |
|
|
|
|
|
# KAT-Dev-72B-Exp - GPTQ INT4 (group_size=32) |
|
|
|
|
|
This is a GPTQ quantized version of [Kwaipilot/KAT-Dev-72B-Exp](https://huggingface.co/Kwaipilot/KAT-Dev-72B-Exp). |
|
|
|
|
|
## Quantization Details |
|
|
|
|
|
- **Method**: GPTQ (GPT Quantization) |
|
|
- **Bits**: 4 |
|
|
- **Group Size**: 32 |
|
|
- **Quantization Type**: INT |
|
|
- **Symmetric**: True |
|
|
- **Calibration Samples**: 128 |
|
|
- **Calibration Dataset**: allenai/c4 |
|
|
- **Max Sequence Length**: 512 |
|
|
|
|
|
## Hardware Used for Quantization |
|
|
|
|
|
- 6x NVIDIA GeForce RTX 5090 (32GB each) |
|
|
- CUDA 12.8+ |
|
|
- Sequential layer-by-layer processing (OOM-safe) |
|
|
|
|
|
## Usage |
|
|
|
|
|
### With vLLM |
|
|
|
|
|
```python |
|
|
from vllm import LLM, SamplingParams |
|
|
|
|
|
# Initialize the model |
|
|
llm = LLM(model="Vykyan/KAT-Dev-72B-Exp-GPTQ-INT4-gs32", trust_remote_code=True) |
|
|
|
|
|
# Create sampling parameters |
|
|
sampling_params = SamplingParams( |
|
|
temperature=0.7, |
|
|
top_p=0.9, |
|
|
max_tokens=512, |
|
|
) |
|
|
|
|
|
# Generate text |
|
|
prompts = ["Hello, how are you?"] |
|
|
outputs = llm.generate(prompts, sampling_params) |
|
|
|
|
|
for output in outputs: |
|
|
print(output.outputs[0].text) |
|
|
``` |
|
|
|
|
|
### With Transformers |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
"Vykyan/KAT-Dev-72B-Exp-GPTQ-INT4-gs32", |
|
|
device_map="auto", |
|
|
trust_remote_code=True |
|
|
) |
|
|
tokenizer = AutoTokenizer.from_pretrained( |
|
|
"Vykyan/KAT-Dev-72B-Exp-GPTQ-INT4-gs32", |
|
|
trust_remote_code=True |
|
|
) |
|
|
|
|
|
# Generate |
|
|
inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device) |
|
|
outputs = model.generate(**inputs, max_new_tokens=100) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
## Inference Performance |
|
|
|
|
|
This quantized model offers: |
|
|
- ~4x memory reduction compared to FP16 |
|
|
- Faster inference on compatible hardware |
|
|
- Maintained accuracy through GPTQ quantization |
|
|
|
|
|
### Recommended Hardware |
|
|
|
|
|
- NVIDIA GPUs with compute capability 7.5+ (RTX 20-series or newer) |
|
|
- Minimum 24GB VRAM for single-GPU inference |
|
|
- Multi-GPU setup for larger batch sizes |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Base Model**: Kwaipilot/KAT-Dev-72B-Exp |
|
|
- **Quantization Tool**: llm-compressor |
|
|
- **Compatible Inference Engines**: vLLM, TGI (Text Generation Inference) |
|
|
|
|
|
## Limitations |
|
|
|
|
|
- Quantization may affect model accuracy on certain tasks |
|
|
- Requires vLLM or compatible inference engine for optimal performance |
|
|
|
|
|
## Acknowledgements |
|
|
|
|
|
- Base model: [Kwaipilot/KAT-Dev-72B-Exp](https://huggingface.co/Kwaipilot/KAT-Dev-72B-Exp) |
|
|
- Quantization: [llm-compressor](https://github.com/vllm-project/llm-compressor) |
|
|
- Inference: [vLLM](https://github.com/vllm-project/vllm) |
|
|
|
