Hugging Face's logo

ysr
/
hyperparam-rust-sft-lora

PEFT
TensorBoard
Safetensors
trl
sft
Generated from Trainer
Model card Files
xet
Metrics Community
ysr commited on
Commit
c0a97ac
·
verified ·
1 Parent(s): b27d1ae

ysr/deepseek-rust-1.3b-lora-32

Browse files
Files changed (4) hide show
  1. README.md +20 -12
  2. adapter_config.json +4 -4
  3. adapter_model.safetensors +2 -2
  4. training_args.bin +1 -1
README.md CHANGED
@@ -20,7 +20,7 @@ should probably proofread and complete it, then remove this comment. -->
20
 
21
  This model is a fine-tuned version of [deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base) on the generator dataset.
22
  It achieves the following results on the evaluation set:
23
- - Loss: 0.4360
24
 
25
  ## Model description
26
 
@@ -39,7 +39,7 @@ More information needed
39
  ### Training hyperparameters
40
 
41
  The following hyperparameters were used during training:
42
- - learning_rate: 0.0002
43
  - train_batch_size: 1
44
  - eval_batch_size: 1
45
  - seed: 42
@@ -48,27 +48,35 @@ The following hyperparameters were used during training:
48
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
49
  - lr_scheduler_type: cosine
50
  - lr_scheduler_warmup_ratio: 0.05
51
- - lr_scheduler_warmup_steps: 1
52
  - num_epochs: 5
53
 
54
  ### Training results
55
 
56
  | Training Loss | Epoch | Step | Validation Loss |
57
  |:-------------:|:-----:|:----:|:---------------:|
58
- | 0.6641 | 0.59 | 25 | 0.5224 |
59
- | 0.4747 | 1.18 | 50 | 0.4685 |
60
- | 0.4392 | 1.77 | 75 | 0.4519 |
61
- | 0.4378 | 2.37 | 100 | 0.4437 |
62
- | 0.4189 | 2.96 | 125 | 0.4389 |
63
- | 0.4209 | 3.55 | 150 | 0.4367 |
64
- | 0.4201 | 4.14 | 175 | 0.4360 |
65
- | 0.4163 | 4.73 | 200 | 0.4360 |
 
 
 
 
 
 
 
 
66
 
67
 
68
  ### Framework versions
69
 
70
  - PEFT 0.10.0
71
- - Transformers 4.39.2
72
  - Pytorch 2.2.1
73
  - Datasets 2.18.0
74
  - Tokenizers 0.15.2
 
20
 
21
  This model is a fine-tuned version of [deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base) on the generator dataset.
22
  It achieves the following results on the evaluation set:
23
+ - Loss: 0.4301
24
 
25
  ## Model description
26
 
 
39
  ### Training hyperparameters
40
 
41
  The following hyperparameters were used during training:
42
+ - learning_rate: 0.0003
43
  - train_batch_size: 1
44
  - eval_batch_size: 1
45
  - seed: 42
 
48
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
49
  - lr_scheduler_type: cosine
50
  - lr_scheduler_warmup_ratio: 0.05
51
+ - lr_scheduler_warmup_steps: 20
52
  - num_epochs: 5
53
 
54
  ### Training results
55
 
56
  | Training Loss | Epoch | Step | Validation Loss |
57
  |:-------------:|:-----:|:----:|:---------------:|
58
+ | 0.8205 | 0.3 | 25 | 0.5577 |
59
+ | 0.5013 | 0.59 | 50 | 0.4856 |
60
+ | 0.4588 | 0.89 | 75 | 0.4663 |
61
+ | 0.4503 | 1.18 | 100 | 0.4556 |
62
+ | 0.4319 | 1.48 | 125 | 0.4482 |
63
+ | 0.4293 | 1.77 | 150 | 0.4429 |
64
+ | 0.4272 | 2.07 | 175 | 0.4387 |
65
+ | 0.4153 | 2.37 | 200 | 0.4365 |
66
+ | 0.4111 | 2.66 | 225 | 0.4343 |
67
+ | 0.4031 | 2.96 | 250 | 0.4322 |
68
+ | 0.406 | 3.25 | 275 | 0.4317 |
69
+ | 0.3996 | 3.55 | 300 | 0.4309 |
70
+ | 0.3959 | 3.84 | 325 | 0.4303 |
71
+ | 0.3989 | 4.14 | 350 | 0.4302 |
72
+ | 0.4035 | 4.43 | 375 | 0.4303 |
73
+ | 0.3929 | 4.73 | 400 | 0.4301 |
74
 
75
 
76
  ### Framework versions
77
 
78
  - PEFT 0.10.0
79
+ - Transformers 4.39.3
80
  - Pytorch 2.2.1
81
  - Datasets 2.18.0
82
  - Tokenizers 0.15.2
adapter_config.json CHANGED
@@ -16,14 +16,14 @@
16
  "megatron_core": "megatron.core",
17
  "modules_to_save": null,
18
  "peft_type": "LORA",
19
- "r": 64,
20
  "rank_pattern": {},
21
  "revision": null,
22
  "target_modules": [
23
- "v_proj",
24
- "o_proj",
25
  "k_proj",
26
- "q_proj"
 
27
  ],
28
  "task_type": "CAUSAL_LM",
29
  "use_dora": false,
 
16
  "megatron_core": "megatron.core",
17
  "modules_to_save": null,
18
  "peft_type": "LORA",
19
+ "r": 32,
20
  "rank_pattern": {},
21
  "revision": null,
22
  "target_modules": [
23
+ "q_proj",
 
24
  "k_proj",
25
+ "v_proj",
26
+ "o_proj"
27
  ],
28
  "task_type": "CAUSAL_LM",
29
  "use_dora": false,
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2fd6460db179ca1736b9c7447f6f1ac3ef8d5ca5bc0c962a0a70f4f375627c71
3
- size 50357632
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84a723e022024cdfece83514750f0c6efcf4dc33aefc626f199a1553a4a60f8d
3
+ size 25191728
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d3d201d74214f993646ae033925c94201c44cfcc9da3d21d28f5ab25ce99f8ac
3
  size 4920
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2b08e1878e76116da9d989483eca71f784ccf29c23ca5c4fb9e809222fcba214
3
  size 4920