File size: 4,016 Bytes
78223de 1e88adc 78223de 1e88adc |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 |
---
license: apache-2.0
datasets:
- jtatman/python-code-dataset-500k
metrics:
- bleu
- rouge
- perplexity
- chrf
- codebertscore
base_model:
- codellama/CodeLlama-7b-Python-hf
pipeline_tag: text-generation
tags:
- code
- python
- codellama
- lora
- peft
- sft
- programming
---
# CodeLlama-7b-Python-hf-ft
This repository contains a **LoRA fine-tuned adapter** for **[CodeLlama-7b-Python-hf](https://huggingface.co/codellama/CodeLlama-7b-Python-hf)**, trained to improve **Python instruction-following and code generation**.
**Note:**
This is a **PEFT LoRA adapter**, not a fully merged standalone model. You must load it on top of the base model.
---
## Model Details
- **Base model**: [codellama/CodeLlama-7b-Python-hf](https://huggingface.co/codellama/CodeLlama-7b-Python-hf)
- **Fine-tuned for**: Python instruction-following and code generation
- **Fine-tuning method**: SFT + LoRA (PEFT)
- **Framework**: Transformers + PEFT + TRL
---
## Dataset Used
This adapter was fine-tuned on:
1. [jtatman/python-code-dataset-500k](https://huggingface.co/datasets/jtatman/python-code-dataset-500k)
- Large-scale Python instruction → solution pairs
- Parquet format (~500k+ examples)
---
## Training Configuration
### LoRA Configuration
- **r:** 32
- **lora_alpha:** 16
- **Target modules:**
`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
### SFT Configuration
- **Epochs:** 1
- **Learning rate:** 2e-4
- **Scheduler:** cosine
- **Warmup ratio:** 0.03
- **Weight decay:** 0.0
- **Train batch size:** 4
- **Eval batch size:** 4
- **Gradient accumulation steps:** 16
- **Precision:** bf16
- **Attention:** flash_attention_2
- **Packing:** enabled
- **Gradient checkpointing:** enabled
- **Logging:** every 50 steps + per epoch
- **Saving:** per epoch (`save_total_limit=2`)
---
## Evaluation Results
The model was evaluated using both language-modeling metrics and generation-quality metrics.
### 📉 Perplexity / Loss
- **Base model loss:** `1.3214`
- **Base model perplexity:** `3.7486`
- **Fine-tuned (LoRA) val/test loss:** `0.7126`
- **Fine-tuned (LoRA) val/test perplexity:** `2.0394`
### 📊 Generation Quality Metrics (Test)
- **Exact Match:** `0.0033`
- **Normalized Exact Match:** `0.0033`
- **BLEU:** `18.43`
- **chrF:** `34.06`
- **ROUGE-L (F1):** `0.2417`
### 🧠 CodeBERTScore (Mean)
- **Precision:** `0.7187`
- **Recall:** `0.7724`
- **F1:** `0.7421`
- **F3:** `0.7657`
### 🧾 Training Summary (from logs)
- **Train loss:** `~0.6903`
- **Eval loss:** `~0.6877`
---
## Example Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Base + adapter
base_id = "codellama/CodeLlama-7b-Python-hf"
adapter_id = "Tanneru/CodeLlama-7b-Python-hf-ft"
# Load tokenizer (repo includes tokenizer files)
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
base_id,
device_map="auto",
torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
)
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()
prompt = "Write a Python function that checks if a number is prime."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```
## Citation
If you use this model in your research or project, please cite it:
```bibtex
@misc{tanneru2025codellamapythonft,
title = {CodeLlama-7b-Python-hf-ft},
author = {Tanneru},
year = {2025},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Tanneru/CodeLlama-7b-Python-hf-ft}}
}
``` |