tasks/README.md

# Evaluation Tasks

This directory contains evaluation tasks organized by use case.

## Structure

```
tasks/
├── sql_generation/           # SQL generation tasks
│   └── nyc_taxi_small/      # NYC Taxi dataset
├── code_generation/          # Code generation tasks
│   ├── python_algorithms/   # Python algorithm tasks
│   └── go_algorithms/       # Go algorithm tasks
└── documentation/           # Documentation generation tasks
    ├── technical_docs/      # Technical documentation tasks
    └── api_documentation/   # API documentation tasks
```

## Use Cases

### 1. SQL Generation
- **Purpose**: Evaluate models on natural language to SQL query generation
- **Datasets**: NYC Taxi Small
- **Dialects**: Presto, BigQuery, Snowflake
- **Metrics**: Correctness, execution success, result matching, dialect compliance

### 2. Code Generation
- **Purpose**: Evaluate models on natural language to source code generation
- **Languages**: Python, Go, JavaScript, Java
- **Datasets**: Algorithm implementations, web services, data structures
- **Metrics**: Syntax correctness, compilation success, execution success, code quality

### 3. Documentation Generation
- **Purpose**: Evaluate models on natural language to technical documentation
- **Formats**: Markdown, HTML, JSON, YAML
- **Datasets**: API docs, technical guides, installation instructions
- **Metrics**: Accuracy, completeness, clarity, format compliance

## Task Structure

Each task directory contains:

### Required Files
- `cases.yaml` - Test cases with questions and reference outputs
- `loader.py` - Data loading and test execution utilities
- `schema.sql` - Database schema (for SQL tasks)
- `test_data.json` - Test data for evaluation (for code/doc tasks)

### Optional Files
- `README.md` - Task-specific documentation
- `requirements.txt` - Task-specific dependencies
- `config.yaml` - Task-specific configuration

## Adding New Tasks

1. Create a new directory under the appropriate use case
2. Add the required files (`cases.yaml`, `loader.py`)
3. Define test cases with questions and reference outputs
4. Implement data loading and evaluation logic
5. Update the main configuration files

## Evaluation Metrics

### SQL Generation
- **Correctness**: Exact match with reference SQL
- **Execution Success**: SQL executes without errors
- **Result Matching**: F1 score comparing query results
- **Dialect Compliance**: Proper SQL transpilation
- **Readability**: SQL structure and formatting

### Code Generation
- **Syntax Correctness**: Code compiles without syntax errors
- **Compilation Success**: Code builds successfully
- **Execution Success**: Code runs and produces expected output
- **Code Quality**: Follows language best practices
- **Performance**: Code efficiency and optimization

### Documentation Generation
- **Accuracy**: Content matches reference documentation
- **Completeness**: Covers all required information
- **Clarity**: Easy to understand and follow
- **Format Compliance**: Follows specified documentation format
- **Technical Correctness**: Technically accurate information