Create README.md

README.md

@@ -0,0 +1,56 @@
+---
+license: llama3.1
+base_model: meta-llama/Llama-3.1-8B-Instruct
+base_model_relation: quantized
+tags:
+- Llama 3.1 8b
+- GGUF
+- quantized
+- 6-bit
+---
+
+## Llama.cpp hybrid layer quantization of Llama 3.1 8B Instruct by meta-llama
+
+Original model: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
+
+The hybrid quant employs different quantization levels on a per layer basis to enable
+both high performance and small file size at the same time.  This particular quant
+achieves a ~6G gguf with the same perplexity as a ~6.6G Q6_K GGUF.  The quants
+employed are all K to avoid slow CPU or older GPU processing of IQ quants.  For this
+file the layer quants are as follows:
+```
+   LAYER_TYPES='[
+   [0 ,"Q6_K"  ],[1 ,"Q5_K_M"],[2 ,"Q4_K_M"],[3 ,"Q4_K_M"],[4 ,"Q4_K_M"],[5 ,"Q4_K_M"],[6 ,"Q4_K_M"],[7 ,"Q4_K_M"],
+   [8 ,"Q5_K_M"],[9 ,"Q5_K_S"],[10,"Q5_K_M"],[11,"Q5_K_S"],[12,"Q5_K_M"],[13,"Q5_K_S"],[14,"Q5_K_M"],[15,"Q5_K_S"],
+   [16,"Q5_K_M"],[17,"Q5_K_M"],[18,"Q5_K_M"],[19,"Q5_K_M"],[20,"Q5_K_M"],[21,"Q5_K_M"],[22,"Q5_K_M"],[23,"Q5_K_M"],
+   [24,"Q5_K_M"],[25,"Q5_K_M"],[26,"Q6_K"  ],[27,"Q6_K"  ],[28,"Q6_K"  ],[29,"Q8_0"  ],[30,"Q8_0"  ],[31,"Q8_0"  ]
+   ]'
+   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K"
+```
+Comparison:
+
+Quant |  size  |  PPL |   Comment
+---------|---------|------|-----------
+Q6_K  | 6.6e9 | 7.2 |  Q6_K with default embedding and output
+Q6_K_H |  6.0e9  |  7.2  | Hybrid quant with Q6_K embedding Q6_K output
+
+Usage:
+
+This model may be used together with fixie-ai ultravox-v0_5-llama-3_1-8b to enable it to process audio (.mp3 and .wav files) and text inputs
+and generate text outputs.  The mmproj file is made available here: https://huggingface.co/steampunque/ultravox-v0_5-llama-3_1-8b-Hybrid-GGUF
+More information about running multimedia may be found in the docs in the mtmd readme in the tools directory of the llama.cpp source tree
+https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/README.md.
+
+Benchmarks:
+
+A full set of benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm
+
+## Download the file from below:
+| Link | Type | Size/e9 B | Notes |
+|------|------|-----------|-------|
+| [Llama-3.1-8B-Instruct.Q6_K_H.gguf](https://huggingface.co/steampunque/Llama-3.1-8B-Instruct-Hybrid-GGUF/resolve/main/Llama-3.1-8B-Instruct.Q6_K_H.gguf) | Q6_K_H | 6e9 B | 0.6B smaller than Q6_K |
+| [ultravox-v0_5-llama-3_1-8b.mmproj.gguf](https://huggingface.co/steampunque/ultravox-v0_5-llama-3_1-8b-Hybrid-GGUF/resolve/main/ultravox-v0_5-llama-3_1-8b.mmproj.gguf) | mmproj | 1.38e9 B | multimedia projector |
+ 
+A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:
+
+https://github.com/ggml-org/llama.cpp/discussions/13040