pytorch
/

gemma-3-27b-it-AWQ-INT4

text-generation-inference

Model card Files Files and versions

jerryzh168 commited on Oct 11

Commit

9de4d92

·

verified ·

1 Parent(s): 9cdb596

Update README.md

Files changed (1) hide show

README.md +3 -0

README.md CHANGED Viewed

@@ -132,6 +132,9 @@ pip install accelerate
 Use the following code to get the quantized model:
 ```Py
 model_id = "google/gemma-3-27b-it"
 model_to_quantize = "google/gemma-3-27b-it"
 from torchao.quantization import Int4WeightOnlyConfig, quantize_, ModuleFqnToConfig

 Use the following code to get the quantized model:
 ```Py
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig
 model_id = "google/gemma-3-27b-it"
 model_to_quantize = "google/gemma-3-27b-it"
 from torchao.quantization import Int4WeightOnlyConfig, quantize_, ModuleFqnToConfig