README.md

---
base_model:
- microsoft/codebert-base
datasets:
- devngho/the-stack-llm-annotations-v2
language:
- code
library_name: transformers
license: mit
metrics:
- f1
---

# devngho/code_edu_classifier-v3-microsoft_codebert-base

이 모델은 [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base)에 classifier를 추가한 모델입니다. [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier)의 코드 버전을 목표로, 코드의 교육성 점수를 평가합니다.
학습에는 [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup)에서 추출한 샘플을 [Qwen/Qwen2.5-Coder-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct)로 평가한 [devngho/the-stack-llm-annotations-v2](https://huggingface.co/datasets/devngho/the-stack-llm-annotations-v2) 데이터셋이 사용되었습니다.

이 연구는 Google의 TPU Research Cloud [(TRC)](https://sites.research.google/trc/about/)의 Cloud TPU 제공으로 수행되었습니다. ⚡

## 상세

- **제작:** devngho
- **언어:** code
- **라이선스:** mit
- **기반 모델:** [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base)

## 학습 상세

- learning_rate: 3e-4 (cosine)
- warmup_ratio: 0.1
- batch_size: 2048(512*4)
- optimizer: adamw(b1=0.9, b2=0.98, eps=1e-8, weight_decay=0.01)
- duration: 4h 41m
- steps: 6080

## 학습 장비

TPU v4-8

## 성능

```
Validation Report:
              precision    recall  f1-score   support

           0       0.80      0.06      0.10        72
           1       0.62      0.40      0.48       835
           2       0.61      0.62      0.61      2722
           3       0.48      0.72      0.58      1891
           4       0.62      0.02      0.05       623
           5       0.00      0.00      0.00         1

    accuracy                           0.55      6144
   macro avg       0.52      0.30      0.30      6144
weighted avg       0.58      0.55      0.52      6144

Confusion Matrix:
[[   4   36   30    2    0    0]
 [   1  330  464   40    0    0]
 [   0  157 1684  881    0    0]
 [   0    5  516 1361    9    0]
 [   0    0   71  537   15    0]
 [   0    0    0    1    0    0]]
```

3 이상과 미만으로 구분할 때 f1 score는 약 0.72입니다.

# devngho/code_edu_classifier-v3-microsoft_codebert-base

This model is [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) with classfier head. It is designed to evaluate the educational value of codes, similar to the [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier), but focused on code. The training data comes from [devngho/the-stack-llm-annotations-v2](https://huggingface.co/datasets/devngho/the-stack-llm-annotations-v2) dataset, contains samples extracted from [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) and evaluated using [Qwen/Qwen2.5-Coder-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct).

This research was supported with Cloud TPUs from Google's TPU Research Cloud [(TRC)](https://sites.research.google/trc/about/).⚡

- **Developed by:** devngho
- **Language(s):** code
- **License:** mit
- **Base model:** [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base)

## Training detail

- learning_rate: 3e-4 (cosine)
- warmup_ratio: 0.1
- batch_size: 2048(512*4)
- optimizer: adamw(b1=0.9, b2=0.98, eps=1e-8, weight_decay=0.01)
- duration: 4h 41m
- steps: 6080

## Training hardware

TPU v4-8

## Performance

```
Validation Report:
              precision    recall  f1-score   support

           0       0.80      0.06      0.10        72
           1       0.62      0.40      0.48       835
           2       0.61      0.62      0.61      2722
           3       0.48      0.72      0.58      1891
           4       0.62      0.02      0.05       623
           5       0.00      0.00      0.00         1

    accuracy                           0.55      6144
   macro avg       0.52      0.30      0.30      6144
weighted avg       0.58      0.55      0.52      6144

Confusion Matrix:
[[   4   36   30    2    0    0]
 [   1  330  464   40    0    0]
 [   0  157 1684  881    0    0]
 [   0    5  516 1361    9    0]
 [   0    0   71  537   15    0]
 [   0    0    0    1    0    0]]
```

The F1 score is about 0.72 when separating above and below 3.