---
license: mit
language:
- en
tags:
- text-generation
- sql
- text-to-sql
- llama
- llama-3
- fine-tuned
- database
- nlp
- lora
base_model: meta-llama/Meta-Llama-3-8B
datasets:
- estu-research/sql-training-dataset
metrics:
- accuracy
- exact_match
library_name: transformers
pipeline_tag: text-generation
---

# Llama-3-8B SQL Expert (Fine-Tuned)

Fine-tuned version of Meta's Llama-3-8B model for converting natural language questions to SQL queries using LoRA adapters.

## Model Details

- **Base Model**: [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
- **Fine-tuned by**: ESTU Research Team (Kulalı, Aydın, Alhan, Fidan)
- **Institution**: Eskisehir Technical University
- **Project**: TÜBİTAK 2209-A Research
- **License**: MIT
- **Language**: English
- **Task**: Natural Language to SQL Translation
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation)

## Performance

- **Execution Accuracy**: 78.2%
- **Exact Match**: 68.9%
- **Average Latency**: 450ms
- **Model Size**: 16.2 GB (full) / 196 MB (LoRA adapters only)

## Training Details

### Training Data
- **Dataset**: [estu-research/sql-training-dataset](https://huggingface.co/datasets/estu-research/sql-training-dataset)
- **Examples**: 1,000+ natural language to SQL pairs
- **Domain**: Sales database queries (customers, orders, products, employees)

### Training Configuration
```python
{
  "base_model": "meta-llama/Meta-Llama-3-8B",
  "method": "LoRA",
  "rank": 16,
  "alpha": 32,
  "dropout": 0.05,
  "target_modules": [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj"
  ],
  "epochs": 4,
  "batch_size": 8,
  "learning_rate": 2e-4,
  "training_time": "12.4 hours (A100 GPU)"
}
```

### Training Results
```
Epoch 1: Loss 1.234 | Val Loss 1.289 | Accuracy 69.4%
Epoch 2: Loss 0.543 | Val Loss 0.612 | Accuracy 74.1%
Epoch 3: Loss 0.298 | Val Loss 0.334 | Accuracy 76.8%
Epoch 4: Loss 0.187 | Val Loss 0.221 | Accuracy 78.2%
```

## Usage

### Installation
```bash
pip install transformers torch peft
```

### Quick Start (LoRA)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

# Load LoRA adapters
model = PeftModel.from_pretrained(base_model, "estu-research/llama3-8b-sql-ft")

# Example query
question = """
Schema: CREATE TABLE customers (customerNumber INT, customerName VARCHAR(50), country VARCHAR(50));
Question: List all customers from France
"""

inputs = tokenizer(question, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
sql = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(sql)
# Output: SELECT * FROM customers WHERE country = 'France';
```

### Full Model Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load merged model
model = AutoModelForCausalLM.from_pretrained("estu-research/llama3-8b-sql-ft")
tokenizer = AutoTokenizer.from_pretrained("estu-research/llama3-8b-sql-ft")

question = "Show top 10 products by price"
inputs = tokenizer(question, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.1)
sql = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(sql)
```

### Pipeline Usage
```python
from transformers import pipeline

pipe = pipeline("text-generation", model="estu-research/llama3-8b-sql-ft")

result = pipe(
    "Schema: CREATE TABLE orders (orderDate DATE, amount DECIMAL);\nQuestion: Total sales by month in 2024",
    max_new_tokens=200
)
print(result[0]['generated_text'])
```

## Example Queries

| Natural Language | Generated SQL |
|------------------|---------------|
| Count orders per customer | `SELECT customerName, COUNT(orderNumber) FROM customers JOIN orders USING(customerNumber) GROUP BY customerNumber;` |
| Average order value | `SELECT AVG(quantityOrdered * priceEach) as avg_value FROM orderDetails;` |
| Customers with no orders | `SELECT customerName FROM customers WHERE customerNumber NOT IN (SELECT DISTINCT customerNumber FROM orders);` |

## Comparison with Other Models

| Model | Accuracy | Latency | Cost | Size |
|-------|----------|---------|------|------|
| GPT-4o-mini (FT) | 97.8% | 800ms | $0.30/1K | Cloud |
| GPT-4 | 92.1% | 1200ms | $3.00/1K | Cloud |
| **Llama-3-8B (FT)** | **78.2%** | 450ms | Free | 16GB |
| Gemma-7B (FT) | 76.0% | 500ms | Free | 14GB |
| GPT-3.5 Turbo | 78.9% | 500ms | $0.05/1K | Cloud |

## Advantages

✅ **Open Source**: Fully downloadable and modifiable  
✅ **Cost-Effective**: Free self-hosting  
✅ **Privacy**: On-premise deployment  
✅ **Fast**: 450ms average latency  
✅ **Efficient**: LoRA adapters only 196 MB  

## Limitations

- Trained primarily on sales database schema
- May struggle with highly complex nested queries
- Best performance on English language queries
- Requires GPU for optimal inference (4-8 GB VRAM minimum)

## Intended Use

- **Primary**: Natural language to SQL translation for analytics
- **Secondary**: SQL query assistance and education
- **Self-Hosted**: Privacy-sensitive applications
- **Not For**: Production databases without query validation

## Hardware Requirements

### Inference
- **Minimum**: 8 GB VRAM (with quantization)
- **Recommended**: 16 GB VRAM (full precision)
- **Optimal**: 24 GB VRAM (batch processing)

### LoRA Only
- **Minimum**: 4 GB VRAM
- **Recommended**: 8 GB VRAM

## Citation

```bibtex
@misc{llama3-8b-sql-ft,
  title={Llama-3-8B SQL Expert: Fine-Tuned Model for Text-to-SQL},
  author={Kulalı and Aydın and Alhan and Fidan},
  institution={Eskisehir Technical University},
  year={2024},
  url={https://huggingface.co/estu-research/llama3-8b-sql-ft}
}
```

## Links

- **GitHub**: [Japyh/llm-based-dbms](https://github.com/Japyh/llm-based-dbms)
- **Research Paper**: [docs/research_paper_draft.md](https://github.com/Japyh/llm-based-dbms/blob/main/docs/research_paper_draft.md)
- **Dataset**: [estu-research/sql-training-dataset](https://huggingface.co/datasets/estu-research/sql-training-dataset)
- **Organization**: [estu-research](https://huggingface.co/estu-research)

## Acknowledgments

This work was supported by TÜBİTAK 2209-A Research Grant at Eskisehir Technical University.

Special thanks to Meta AI for releasing Llama-3.

## License

MIT License - See LICENSE file for details