Hugging Face's logo

AhmedCodes64
/
Merged_Model_DPO_v2

PEFT
Safetensors
Model card Files
xet
Community
AhmedCodes64 commited on
Commit
402bb48
·
verified ·
1 Parent(s): ddef5be

Upload 15 files

Browse files
Files changed (16) hide show
  1. .gitattributes +1 -0
  2. README.md +202 -3
  3. adapter_config.json +37 -0
  4. adapter_model.safetensors +3 -0
  5. added_tokens.json +24 -0
  6. merges.txt +0 -0
  7. optimizer.pt +3 -0
  8. rng_state.pth +3 -0
  9. scaler.pt +3 -0
  10. scheduler.pt +3 -0
  11. special_tokens_map.json +31 -0
  12. tokenizer.json +3 -0
  13. tokenizer_config.json +209 -0
  14. trainer_state.json +2494 -0
  15. training_args.bin +3 -0
  16. vocab.json +0 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,202 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Bharatdeep-H/portfolio-query-qwen-2.5-1.5b-v10-cot_merged
3
+ library_name: peft
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
200
+ ### Framework versions
201
+
202
+ - PEFT 0.14.0
adapter_config.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": null,
4
+ "base_model_name_or_path": "Bharatdeep-H/portfolio-query-qwen-2.5-1.5b-v10-cot_merged",
5
+ "bias": "none",
6
+ "eva_config": null,
7
+ "exclude_modules": null,
8
+ "fan_in_fan_out": false,
9
+ "inference_mode": true,
10
+ "init_lora_weights": true,
11
+ "layer_replication": null,
12
+ "layers_pattern": null,
13
+ "layers_to_transform": null,
14
+ "loftq_config": {},
15
+ "lora_alpha": 64,
16
+ "lora_bias": false,
17
+ "lora_dropout": 0,
18
+ "megatron_config": null,
19
+ "megatron_core": "megatron.core",
20
+ "modules_to_save": null,
21
+ "peft_type": "LORA",
22
+ "r": 64,
23
+ "rank_pattern": {},
24
+ "revision": null,
25
+ "target_modules": [
26
+ "o_proj",
27
+ "gate_proj",
28
+ "down_proj",
29
+ "v_proj",
30
+ "up_proj",
31
+ "k_proj",
32
+ "q_proj"
33
+ ],
34
+ "task_type": "CAUSAL_LM",
35
+ "use_dora": false,
36
+ "use_rslora": false
37
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b6a4ffcbc94ed2f35f42ae0c9a21edefc662ec0c9705bcb9e475d8cb44f02e0f
3
+ size 295488936
added_tokens.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "</tool_call>": 151658,
3
+ "<tool_call>": 151657,
4
+ "<|box_end|>": 151649,
5
+ "<|box_start|>": 151648,
6
+ "<|endoftext|>": 151643,
7
+ "<|file_sep|>": 151664,
8
+ "<|fim_middle|>": 151660,
9
+ "<|fim_pad|>": 151662,
10
+ "<|fim_prefix|>": 151659,
11
+ "<|fim_suffix|>": 151661,
12
+ "<|im_end|>": 151645,
13
+ "<|im_start|>": 151644,
14
+ "<|image_pad|>": 151655,
15
+ "<|object_ref_end|>": 151647,
16
+ "<|object_ref_start|>": 151646,
17
+ "<|quad_end|>": 151651,
18
+ "<|quad_start|>": 151650,
19
+ "<|repo_name|>": 151663,
20
+ "<|video_pad|>": 151656,
21
+ "<|vision_end|>": 151653,
22
+ "<|vision_pad|>": 151654,
23
+ "<|vision_start|>": 151652
24
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8b25019aab8258a033622face0d3e112eee5309de9d10c70a7c8965992728330
3
+ size 150486964
rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2b66e3cc7c452b707ddac5caf0aa17618afb9bc1a0333600a22c4afb353f3165
3
+ size 14244
scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2b9d1805b40936af6e0fa36185411489941dca6f187679680dc66eadf6b0eff4
3
+ size 988
scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c151e959b286b48822476eca05cb64802424ffcf0c6d033c7a022d195f26e704
3
+ size 1064
special_tokens_map.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|im_start|>",
4
+ "<|im_end|>",
5
+ "<|object_ref_start|>",
6
+ "<|object_ref_end|>",
7
+ "<|box_start|>",
8
+ "<|box_end|>",
9
+ "<|quad_start|>",
10
+ "<|quad_end|>",
11
+ "<|vision_start|>",
12
+ "<|vision_end|>",
13
+ "<|vision_pad|>",
14
+ "<|image_pad|>",
15
+ "<|video_pad|>"
16
+ ],
17
+ "eos_token": {
18
+ "content": "<|im_end|>",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ },
24
+ "pad_token": {
25
+ "content": "<|vision_pad|>",
26
+ "lstrip": false,
27
+ "normalized": false,
28
+ "rstrip": false,
29
+ "single_word": false
30
+ }
31
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
3
+ size 11421896
tokenizer_config.json ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": false,
3
+ "add_prefix_space": false,
4
+ "added_tokens_decoder": {
5
+ "151643": {
6
+ "content": "<|endoftext|>",
7
+ "lstrip": false,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": true
12
+ },
13
+ "151644": {
14
+ "content": "<|im_start|>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": true
20
+ },
21
+ "151645": {
22
+ "content": "<|im_end|>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": true
28
+ },
29
+ "151646": {
30
+ "content": "<|object_ref_start|>",
31
+ "lstrip": false,
32
+ "normalized": false,
33
+ "rstrip": false,
34
+ "single_word": false,
35
+ "special": true
36
+ },
37
+ "151647": {
38
+ "content": "<|object_ref_end|>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false,
43
+ "special": true
44
+ },
45
+ "151648": {
46
+ "content": "<|box_start|>",
47
+ "lstrip": false,
48
+ "normalized": false,
49
+ "rstrip": false,
50
+ "single_word": false,
51
+ "special": true
52
+ },
53
+ "151649": {
54
+ "content": "<|box_end|>",
55
+ "lstrip": false,
56
+ "normalized": false,
57
+ "rstrip": false,
58
+ "single_word": false,
59
+ "special": true
60
+ },
61
+ "151650": {
62
+ "content": "<|quad_start|>",
63
+ "lstrip": false,
64
+ "normalized": false,
65
+ "rstrip": false,
66
+ "single_word": false,
67
+ "special": true
68
+ },
69
+ "151651": {
70
+ "content": "<|quad_end|>",
71
+ "lstrip": false,
72
+ "normalized": false,
73
+ "rstrip": false,
74
+ "single_word": false,
75
+ "special": true
76
+ },
77
+ "151652": {
78
+ "content": "<|vision_start|>",
79
+ "lstrip": false,
80
+ "normalized": false,
81
+ "rstrip": false,
82
+ "single_word": false,
83
+ "special": true
84
+ },
85
+ "151653": {
86
+ "content": "<|vision_end|>",
87
+ "lstrip": false,
88
+ "normalized": false,
89
+ "rstrip": false,
90
+ "single_word": false,
91
+ "special": true
92
+ },
93
+ "151654": {
94
+ "content": "<|vision_pad|>",
95
+ "lstrip": false,
96
+ "normalized": false,
97
+ "rstrip": false,
98
+ "single_word": false,
99
+ "special": true
100
+ },
101
+ "151655": {
102
+ "content": "<|image_pad|>",
103
+ "lstrip": false,
104
+ "normalized": false,
105
+ "rstrip": false,
106
+ "single_word": false,
107
+ "special": true
108
+ },
109
+ "151656": {
110
+ "content": "<|video_pad|>",
111
+ "lstrip": false,
112
+ "normalized": false,
113
+ "rstrip": false,
114
+ "single_word": false,
115
+ "special": true
116
+ },
117
+ "151657": {
118
+ "content": "<tool_call>",
119
+ "lstrip": false,
120
+ "normalized": false,
121
+ "rstrip": false,
122
+ "single_word": false,
123
+ "special": false
124
+ },
125
+ "151658": {
126
+ "content": "</tool_call>",
127
+ "lstrip": false,
128
+ "normalized": false,
129
+ "rstrip": false,
130
+ "single_word": false,
131
+ "special": false
132
+ },
133
+ "151659": {
134
+ "content": "<|fim_prefix|>",
135
+ "lstrip": false,
136
+ "normalized": false,
137
+ "rstrip": false,
138
+ "single_word": false,
139
+ "special": false
140
+ },
141
+ "151660": {
142
+ "content": "<|fim_middle|>",
143
+ "lstrip": false,
144
+ "normalized": false,
145
+ "rstrip": false,
146
+ "single_word": false,
147
+ "special": false
148
+ },
149
+ "151661": {
150
+ "content": "<|fim_suffix|>",
151
+ "lstrip": false,
152
+ "normalized": false,
153
+ "rstrip": false,
154
+ "single_word": false,
155
+ "special": false
156
+ },
157
+ "151662": {
158
+ "content": "<|fim_pad|>",
159
+ "lstrip": false,
160
+ "normalized": false,
161
+ "rstrip": false,
162
+ "single_word": false,
163
+ "special": false
164
+ },
165
+ "151663": {
166
+ "content": "<|repo_name|>",
167
+ "lstrip": false,
168
+ "normalized": false,
169
+ "rstrip": false,
170
+ "single_word": false,
171
+ "special": false
172
+ },
173
+ "151664": {
174
+ "content": "<|file_sep|>",
175
+ "lstrip": false,
176
+ "normalized": false,
177
+ "rstrip": false,
178
+ "single_word": false,
179
+ "special": false
180
+ }
181
+ },
182
+ "additional_special_tokens": [
183
+ "<|im_start|>",
184
+ "<|im_end|>",
185
+ "<|object_ref_start|>",
186
+ "<|object_ref_end|>",
187
+ "<|box_start|>",
188
+ "<|box_end|>",
189
+ "<|quad_start|>",
190
+ "<|quad_end|>",
191
+ "<|vision_start|>",
192
+ "<|vision_end|>",
193
+ "<|vision_pad|>",
194
+ "<|image_pad|>",
195
+ "<|video_pad|>"
196
+ ],
197
+ "bos_token": null,
198
+ "chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0]['role'] == 'system' %}\n {{- messages[0]['content'] }}\n {%- else %}\n {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n {%- endif %}\n {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0]['role'] == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n {%- else %}\n {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {{- '<|im_start|>' + message.role }}\n {%- if message.content %}\n {{- '\\n' + message.content }}\n {%- endif %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n<tool_call>\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- '}\\n</tool_call>' }}\n {%- endfor %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %} {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n<tool_response>\\n' }}\n {{- message.content }}\n {{- '\\n</tool_response>' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
199
+ "clean_up_tokenization_spaces": false,
200
+ "eos_token": "<|im_end|>",
201
+ "errors": "replace",
202
+ "extra_special_tokens": {},
203
+ "model_max_length": 32768,
204
+ "pad_token": "<|vision_pad|>",
205
+ "padding_side": "right",
206
+ "split_special_tokens": false,
207
+ "tokenizer_class": "Qwen2Tokenizer",
208
+ "unk_token": null
209
+ }
trainer_state.json ADDED
@@ -0,0 +1,2494 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 1.981818181818182,
6
+ "eval_steps": 500,
7
+ "global_step": 164,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.012121212121212121,
14
+ "grad_norm": 2.573503255844116,
15
+ "learning_rate": 2.9411764705882356e-07,
16
+ "logits/chosen": 0.5989066958427429,
17
+ "logits/rejected": 0.3433399796485901,
18
+ "logps/chosen": -80.92005920410156,
19
+ "logps/rejected": -97.3714599609375,
20
+ "loss": 0.6931,
21
+ "rewards/accuracies": 0.0,
22
+ "rewards/chosen": 0.0,
23
+ "rewards/margins": 0.0,
24
+ "rewards/rejected": 0.0,
25
+ "step": 1
26
+ },
27
+ {
28
+ "epoch": 0.024242424242424242,
29
+ "grad_norm": 2.060006856918335,
30
+ "learning_rate": 5.882352941176471e-07,
31
+ "logits/chosen": 0.5360614061355591,
32
+ "logits/rejected": 0.42208123207092285,
33
+ "logps/chosen": -78.2723388671875,
34
+ "logps/rejected": -92.45564270019531,
35
+ "loss": 0.6931,
36
+ "rewards/accuracies": 0.0,
37
+ "rewards/chosen": 0.0,
38
+ "rewards/margins": 0.0,
39
+ "rewards/rejected": 0.0,
40
+ "step": 2
41
+ },
42
+ {
43
+ "epoch": 0.03636363636363636,
44
+ "grad_norm": 2.1472530364990234,
45
+ "learning_rate": 8.823529411764707e-07,
46
+ "logits/chosen": 0.46590447425842285,
47
+ "logits/rejected": 0.5246831178665161,
48
+ "logps/chosen": -82.09193420410156,
49
+ "logps/rejected": -88.78755187988281,
50
+ "loss": 0.6928,
51
+ "rewards/accuracies": 0.625,
52
+ "rewards/chosen": 0.0004730225191451609,
53
+ "rewards/margins": 0.0007421494228765368,
54
+ "rewards/rejected": -0.00026912690373137593,
55
+ "step": 3
56
+ },
57
+ {
58
+ "epoch": 0.048484848484848485,
59
+ "grad_norm": 2.800305128097534,
60
+ "learning_rate": 1.1764705882352942e-06,
61
+ "logits/chosen": 0.65618497133255,
62
+ "logits/rejected": 0.49275052547454834,
63
+ "logps/chosen": -82.10594940185547,
64
+ "logps/rejected": -95.1839370727539,
65
+ "loss": 0.6947,
66
+ "rewards/accuracies": 0.25,
67
+ "rewards/chosen": -0.0007750511867925525,
68
+ "rewards/margins": -0.003024006262421608,
69
+ "rewards/rejected": 0.00224895472638309,
70
+ "step": 4
71
+ },
72
+ {
73
+ "epoch": 0.06060606060606061,
74
+ "grad_norm": 2.5324952602386475,
75
+ "learning_rate": 1.4705882352941177e-06,
76
+ "logits/chosen": 0.5559582114219666,
77
+ "logits/rejected": 0.3742131292819977,
78
+ "logps/chosen": -87.20143127441406,
79
+ "logps/rejected": -105.94652557373047,
80
+ "loss": 0.693,
81
+ "rewards/accuracies": 0.5,
82
+ "rewards/chosen": -4.3773557990789413e-05,
83
+ "rewards/margins": 0.00021791458129882812,
84
+ "rewards/rejected": -0.00026168813928961754,
85
+ "step": 5
86
+ },
87
+ {
88
+ "epoch": 0.07272727272727272,
89
+ "grad_norm": 2.026254177093506,
90
+ "learning_rate": 1.7647058823529414e-06,
91
+ "logits/chosen": 0.6321487426757812,
92
+ "logits/rejected": 0.6650270223617554,
93
+ "logps/chosen": -83.1277847290039,
94
+ "logps/rejected": -85.23698425292969,
95
+ "loss": 0.6928,
96
+ "rewards/accuracies": 0.625,
97
+ "rewards/chosen": -0.0016042710049077868,
98
+ "rewards/margins": 0.000760841416195035,
99
+ "rewards/rejected": -0.0023651123046875,
100
+ "step": 6
101
+ },
102
+ {
103
+ "epoch": 0.08484848484848485,
104
+ "grad_norm": 2.4757308959960938,
105
+ "learning_rate": 2.058823529411765e-06,
106
+ "logits/chosen": 0.2787664830684662,
107
+ "logits/rejected": 0.3368262052536011,
108
+ "logps/chosen": -79.63035583496094,
109
+ "logps/rejected": -90.24547576904297,
110
+ "loss": 0.691,
111
+ "rewards/accuracies": 0.75,
112
+ "rewards/chosen": 0.0005692481645382941,
113
+ "rewards/margins": 0.004281234461814165,
114
+ "rewards/rejected": -0.003711986355483532,
115
+ "step": 7
116
+ },
117
+ {
118
+ "epoch": 0.09696969696969697,
119
+ "grad_norm": 2.419964551925659,
120
+ "learning_rate": 2.3529411764705885e-06,
121
+ "logits/chosen": 0.5775762796401978,
122
+ "logits/rejected": 0.47042325139045715,
123
+ "logps/chosen": -80.90266418457031,
124
+ "logps/rejected": -86.38180541992188,
125
+ "loss": 0.6898,
126
+ "rewards/accuracies": 0.875,
127
+ "rewards/chosen": -0.00029315962456166744,
128
+ "rewards/margins": 0.006642055697739124,
129
+ "rewards/rejected": -0.006935215089470148,
130
+ "step": 8
131
+ },
132
+ {
133
+ "epoch": 0.10909090909090909,
134
+ "grad_norm": 2.376508951187134,
135
+ "learning_rate": 2.647058823529412e-06,
136
+ "logits/chosen": 0.2030642330646515,
137
+ "logits/rejected": 0.4143352508544922,
138
+ "logps/chosen": -78.35124969482422,
139
+ "logps/rejected": -89.64823913574219,
140
+ "loss": 0.6877,
141
+ "rewards/accuracies": 0.875,
142
+ "rewards/chosen": -0.010747241787612438,
143
+ "rewards/margins": 0.010888958349823952,
144
+ "rewards/rejected": -0.021636201068758965,
145
+ "step": 9
146
+ },
147
+ {
148
+ "epoch": 0.12121212121212122,
149
+ "grad_norm": 2.344287157058716,
150
+ "learning_rate": 2.9411764705882355e-06,
151
+ "logits/chosen": 0.5994730591773987,
152
+ "logits/rejected": 0.5396036505699158,
153
+ "logps/chosen": -79.0880355834961,
154
+ "logps/rejected": -88.75253295898438,
155
+ "loss": 0.6849,
156
+ "rewards/accuracies": 0.875,
157
+ "rewards/chosen": -0.010178089141845703,
158
+ "rewards/margins": 0.016606714576482773,
159
+ "rewards/rejected": -0.026784801855683327,
160
+ "step": 10
161
+ },
162
+ {
163
+ "epoch": 0.13333333333333333,
164
+ "grad_norm": 2.1255571842193604,
165
+ "learning_rate": 3.2352941176470594e-06,
166
+ "logits/chosen": 0.3612106442451477,
167
+ "logits/rejected": 0.6012680530548096,
168
+ "logps/chosen": -81.41439819335938,
169
+ "logps/rejected": -89.69815063476562,
170
+ "loss": 0.6846,
171
+ "rewards/accuracies": 1.0,
172
+ "rewards/chosen": -0.01244363747537136,
173
+ "rewards/margins": 0.01724729686975479,
174
+ "rewards/rejected": -0.029690932482481003,
175
+ "step": 11
176
+ },
177
+ {
178
+ "epoch": 0.14545454545454545,
179
+ "grad_norm": 1.613459825515747,
180
+ "learning_rate": 3.529411764705883e-06,
181
+ "logits/chosen": 0.45081156492233276,
182
+ "logits/rejected": 0.519508957862854,
183
+ "logps/chosen": -82.76570129394531,
184
+ "logps/rejected": -86.98448944091797,
185
+ "loss": 0.6856,
186
+ "rewards/accuracies": 0.875,
187
+ "rewards/chosen": -0.004742145538330078,
188
+ "rewards/margins": 0.01519632339477539,
189
+ "rewards/rejected": -0.01993846893310547,
190
+ "step": 12
191
+ },
192
+ {
193
+ "epoch": 0.15757575757575756,
194
+ "grad_norm": 2.1008141040802,
195
+ "learning_rate": 3.8235294117647055e-06,
196
+ "logits/chosen": 0.5754950046539307,
197
+ "logits/rejected": 0.32084736227989197,
198
+ "logps/chosen": -77.88233947753906,
199
+ "logps/rejected": -90.71653747558594,
200
+ "loss": 0.6702,
201
+ "rewards/accuracies": 0.875,
202
+ "rewards/chosen": -0.023272134363651276,
203
+ "rewards/margins": 0.04726962745189667,
204
+ "rewards/rejected": -0.07054176181554794,
205
+ "step": 13
206
+ },
207
+ {
208
+ "epoch": 0.1696969696969697,
209
+ "grad_norm": 1.8546743392944336,
210
+ "learning_rate": 4.11764705882353e-06,
211
+ "logits/chosen": 0.26316455006599426,
212
+ "logits/rejected": 0.2455248236656189,
213
+ "logps/chosen": -85.9008560180664,
214
+ "logps/rejected": -90.04121398925781,
215
+ "loss": 0.6757,
216
+ "rewards/accuracies": 1.0,
217
+ "rewards/chosen": -0.023431111127138138,
218
+ "rewards/margins": 0.035515978932380676,
219
+ "rewards/rejected": -0.058947086334228516,
220
+ "step": 14
221
+ },
222
+ {
223
+ "epoch": 0.18181818181818182,
224
+ "grad_norm": 2.233098268508911,
225
+ "learning_rate": 4.411764705882353e-06,
226
+ "logits/chosen": 0.39422497153282166,
227
+ "logits/rejected": 0.2617032527923584,
228
+ "logps/chosen": -83.47624206542969,
229
+ "logps/rejected": -92.22642517089844,
230
+ "loss": 0.6549,
231
+ "rewards/accuracies": 0.75,
232
+ "rewards/chosen": -0.009575080126523972,
233
+ "rewards/margins": 0.07998533546924591,
234
+ "rewards/rejected": -0.08956041187047958,
235
+ "step": 15
236
+ },
237
+ {
238
+ "epoch": 0.19393939393939394,
239
+ "grad_norm": 2.212238073348999,
240
+ "learning_rate": 4.705882352941177e-06,
241
+ "logits/chosen": 0.6788235306739807,
242
+ "logits/rejected": 0.37267640233039856,
243
+ "logps/chosen": -79.50228118896484,
244
+ "logps/rejected": -85.6736068725586,
245
+ "loss": 0.6502,
246
+ "rewards/accuracies": 0.875,
247
+ "rewards/chosen": -0.007221986539661884,
248
+ "rewards/margins": 0.090502068400383,
249
+ "rewards/rejected": -0.0977240651845932,
250
+ "step": 16
251
+ },
252
+ {
253
+ "epoch": 0.20606060606060606,
254
+ "grad_norm": 2.0433385372161865,
255
+ "learning_rate": 5e-06,
256
+ "logits/chosen": 0.4306856095790863,
257
+ "logits/rejected": 0.4344051778316498,
258
+ "logps/chosen": -80.9012680053711,
259
+ "logps/rejected": -86.1850357055664,
260
+ "loss": 0.6826,
261
+ "rewards/accuracies": 0.75,
262
+ "rewards/chosen": -0.10336600244045258,
263
+ "rewards/margins": 0.022557254880666733,
264
+ "rewards/rejected": -0.1259232461452484,
265
+ "step": 17
266
+ },
267
+ {
268
+ "epoch": 0.21818181818181817,
269
+ "grad_norm": 2.405224323272705,
270
+ "learning_rate": 4.965986394557824e-06,
271
+ "logits/chosen": 0.6965266466140747,
272
+ "logits/rejected": 0.4544612169265747,
273
+ "logps/chosen": -79.70795440673828,
274
+ "logps/rejected": -100.73465728759766,
275
+ "loss": 0.6404,
276
+ "rewards/accuracies": 0.875,
277
+ "rewards/chosen": -0.06364002823829651,
278
+ "rewards/margins": 0.11611214280128479,
279
+ "rewards/rejected": -0.1797521561384201,
280
+ "step": 18
281
+ },
282
+ {
283
+ "epoch": 0.23030303030303031,
284
+ "grad_norm": 2.102670192718506,
285
+ "learning_rate": 4.931972789115647e-06,
286
+ "logits/chosen": 0.07950831949710846,
287
+ "logits/rejected": 0.24685823917388916,
288
+ "logps/chosen": -78.33619689941406,
289
+ "logps/rejected": -84.86385345458984,
290
+ "loss": 0.6501,
291
+ "rewards/accuracies": 0.75,
292
+ "rewards/chosen": -0.06548452377319336,
293
+ "rewards/margins": 0.09256687760353088,
294
+ "rewards/rejected": -0.15805140137672424,
295
+ "step": 19
296
+ },
297
+ {
298
+ "epoch": 0.24242424242424243,
299
+ "grad_norm": 2.0822105407714844,
300
+ "learning_rate": 4.897959183673469e-06,
301
+ "logits/chosen": 0.32838794589042664,
302
+ "logits/rejected": 0.3617649972438812,
303
+ "logps/chosen": -82.71783447265625,
304
+ "logps/rejected": -90.81163787841797,
305
+ "loss": 0.6378,
306
+ "rewards/accuracies": 1.0,
307
+ "rewards/chosen": -0.06530027091503143,
308
+ "rewards/margins": 0.11812734603881836,
309
+ "rewards/rejected": -0.183427631855011,
310
+ "step": 20
311
+ },
312
+ {
313
+ "epoch": 0.2545454545454545,
314
+ "grad_norm": 2.3999321460723877,
315
+ "learning_rate": 4.863945578231293e-06,
316
+ "logits/chosen": 0.6818695068359375,
317
+ "logits/rejected": 0.24048802256584167,
318
+ "logps/chosen": -80.49371337890625,
319
+ "logps/rejected": -96.09458923339844,
320
+ "loss": 0.6311,
321
+ "rewards/accuracies": 0.75,
322
+ "rewards/chosen": -0.07628040015697479,
323
+ "rewards/margins": 0.1365867704153061,
324
+ "rewards/rejected": -0.21286717057228088,
325
+ "step": 21
326
+ },
327
+ {
328
+ "epoch": 0.26666666666666666,
329
+ "grad_norm": 2.2768161296844482,
330
+ "learning_rate": 4.829931972789116e-06,
331
+ "logits/chosen": 0.18232905864715576,
332
+ "logits/rejected": 0.39266088604927063,
333
+ "logps/chosen": -80.37593078613281,
334
+ "logps/rejected": -89.72500610351562,
335
+ "loss": 0.6031,
336
+ "rewards/accuracies": 0.875,
337
+ "rewards/chosen": 0.06079893559217453,
338
+ "rewards/margins": 0.20091038942337036,
339
+ "rewards/rejected": -0.14011144638061523,
340
+ "step": 22
341
+ },
342
+ {
343
+ "epoch": 0.2787878787878788,
344
+ "grad_norm": 1.9448617696762085,
345
+ "learning_rate": 4.795918367346939e-06,
346
+ "logits/chosen": 0.6265424489974976,
347
+ "logits/rejected": 0.510066032409668,
348
+ "logps/chosen": -79.49093627929688,
349
+ "logps/rejected": -94.0775146484375,
350
+ "loss": 0.583,
351
+ "rewards/accuracies": 0.875,
352
+ "rewards/chosen": -0.029067613184452057,
353
+ "rewards/margins": 0.2460225224494934,
354
+ "rewards/rejected": -0.27509012818336487,
355
+ "step": 23
356
+ },
357
+ {
358
+ "epoch": 0.2909090909090909,
359
+ "grad_norm": 2.10017728805542,
360
+ "learning_rate": 4.761904761904762e-06,
361
+ "logits/chosen": 0.4965704679489136,
362
+ "logits/rejected": 0.425863116979599,
363
+ "logps/chosen": -82.37673950195312,
364
+ "logps/rejected": -92.39176940917969,
365
+ "loss": 0.5989,
366
+ "rewards/accuracies": 1.0,
367
+ "rewards/chosen": -0.1384635865688324,
368
+ "rewards/margins": 0.202981099486351,
369
+ "rewards/rejected": -0.3414447009563446,
370
+ "step": 24
371
+ },
372
+ {
373
+ "epoch": 0.30303030303030304,
374
+ "grad_norm": 1.8628658056259155,
375
+ "learning_rate": 4.727891156462586e-06,
376
+ "logits/chosen": 0.36799079179763794,
377
+ "logits/rejected": 0.37271881103515625,
378
+ "logps/chosen": -80.91612243652344,
379
+ "logps/rejected": -94.26983642578125,
380
+ "loss": 0.5859,
381
+ "rewards/accuracies": 0.875,
382
+ "rewards/chosen": -0.02675161324441433,
383
+ "rewards/margins": 0.24192678928375244,
384
+ "rewards/rejected": -0.26867836713790894,
385
+ "step": 25
386
+ },
387
+ {
388
+ "epoch": 0.3151515151515151,
389
+ "grad_norm": 1.5512022972106934,
390
+ "learning_rate": 4.693877551020409e-06,
391
+ "logits/chosen": 0.45694056153297424,
392
+ "logits/rejected": 0.36823397874832153,
393
+ "logps/chosen": -80.49776458740234,
394
+ "logps/rejected": -91.63525390625,
395
+ "loss": 0.6118,
396
+ "rewards/accuracies": 0.875,
397
+ "rewards/chosen": -0.13241948187351227,
398
+ "rewards/margins": 0.19294482469558716,
399
+ "rewards/rejected": -0.32536429166793823,
400
+ "step": 26
401
+ },
402
+ {
403
+ "epoch": 0.32727272727272727,
404
+ "grad_norm": 2.0366673469543457,
405
+ "learning_rate": 4.659863945578232e-06,
406
+ "logits/chosen": 0.44295981526374817,
407
+ "logits/rejected": 0.3648220896720886,
408
+ "logps/chosen": -81.94073486328125,
409
+ "logps/rejected": -94.45509338378906,
410
+ "loss": 0.5649,
411
+ "rewards/accuracies": 1.0,
412
+ "rewards/chosen": 0.006222818046808243,
413
+ "rewards/margins": 0.2877269685268402,
414
+ "rewards/rejected": -0.28150415420532227,
415
+ "step": 27
416
+ },
417
+ {
418
+ "epoch": 0.3393939393939394,
419
+ "grad_norm": 2.866046667098999,
420
+ "learning_rate": 4.6258503401360546e-06,
421
+ "logits/chosen": 0.37108322978019714,
422
+ "logits/rejected": 0.5524705052375793,
423
+ "logps/chosen": -81.45274353027344,
424
+ "logps/rejected": -93.38094329833984,
425
+ "loss": 0.6103,
426
+ "rewards/accuracies": 0.75,
427
+ "rewards/chosen": -0.1701315939426422,
428
+ "rewards/margins": 0.20057372748851776,
429
+ "rewards/rejected": -0.3707053065299988,
430
+ "step": 28
431
+ },
432
+ {
433
+ "epoch": 0.3515151515151515,
434
+ "grad_norm": 1.883554220199585,
435
+ "learning_rate": 4.591836734693878e-06,
436
+ "logits/chosen": 0.635645866394043,
437
+ "logits/rejected": 0.4668920636177063,
438
+ "logps/chosen": -81.63595581054688,
439
+ "logps/rejected": -90.26708984375,
440
+ "loss": 0.6138,
441
+ "rewards/accuracies": 0.875,
442
+ "rewards/chosen": -0.28862398862838745,
443
+ "rewards/margins": 0.17766514420509338,
444
+ "rewards/rejected": -0.4662891626358032,
445
+ "step": 29
446
+ },
447
+ {
448
+ "epoch": 0.36363636363636365,
449
+ "grad_norm": 2.250345230102539,
450
+ "learning_rate": 4.557823129251701e-06,
451
+ "logits/chosen": 0.49621516466140747,
452
+ "logits/rejected": 0.5221789479255676,
453
+ "logps/chosen": -81.66897583007812,
454
+ "logps/rejected": -90.41416931152344,
455
+ "loss": 0.5329,
456
+ "rewards/accuracies": 1.0,
457
+ "rewards/chosen": -0.04887208715081215,
458
+ "rewards/margins": 0.39742839336395264,
459
+ "rewards/rejected": -0.4463005065917969,
460
+ "step": 30
461
+ },
462
+ {
463
+ "epoch": 0.37575757575757573,
464
+ "grad_norm": 1.950496792793274,
465
+ "learning_rate": 4.523809523809524e-06,
466
+ "logits/chosen": 0.5327723622322083,
467
+ "logits/rejected": 0.46856898069381714,
468
+ "logps/chosen": -82.16653442382812,
469
+ "logps/rejected": -87.493408203125,
470
+ "loss": 0.6341,
471
+ "rewards/accuracies": 0.75,
472
+ "rewards/chosen": -0.19954414665699005,
473
+ "rewards/margins": 0.14190863072872162,
474
+ "rewards/rejected": -0.34145280718803406,
475
+ "step": 31
476
+ },
477
+ {
478
+ "epoch": 0.3878787878787879,
479
+ "grad_norm": 2.1350300312042236,
480
+ "learning_rate": 4.489795918367348e-06,
481
+ "logits/chosen": 0.5370631814002991,
482
+ "logits/rejected": 0.5145618915557861,
483
+ "logps/chosen": -86.60906982421875,
484
+ "logps/rejected": -93.3792495727539,
485
+ "loss": 0.5004,
486
+ "rewards/accuracies": 1.0,
487
+ "rewards/chosen": -0.13417404890060425,
488
+ "rewards/margins": 0.4657081365585327,
489
+ "rewards/rejected": -0.5998822450637817,
490
+ "step": 32
491
+ },
492
+ {
493
+ "epoch": 0.4,
494
+ "grad_norm": 1.427173376083374,
495
+ "learning_rate": 4.45578231292517e-06,
496
+ "logits/chosen": 0.6590886116027832,
497
+ "logits/rejected": 0.5974253416061401,
498
+ "logps/chosen": -87.329345703125,
499
+ "logps/rejected": -94.36528778076172,
500
+ "loss": 0.6501,
501
+ "rewards/accuracies": 0.75,
502
+ "rewards/chosen": -0.4314802885055542,
503
+ "rewards/margins": 0.09496374428272247,
504
+ "rewards/rejected": -0.5264440774917603,
505
+ "step": 33
506
+ },
507
+ {
508
+ "epoch": 0.4121212121212121,
509
+ "grad_norm": 2.568857431411743,
510
+ "learning_rate": 4.421768707482993e-06,
511
+ "logits/chosen": 0.30280208587646484,
512
+ "logits/rejected": 0.24079981446266174,
513
+ "logps/chosen": -84.2959213256836,
514
+ "logps/rejected": -93.14737701416016,
515
+ "loss": 0.5052,
516
+ "rewards/accuracies": 0.875,
517
+ "rewards/chosen": 0.05571327358484268,
518
+ "rewards/margins": 0.4796813726425171,
519
+ "rewards/rejected": -0.4239681363105774,
520
+ "step": 34
521
+ },
522
+ {
523
+ "epoch": 0.42424242424242425,
524
+ "grad_norm": 2.2260830402374268,
525
+ "learning_rate": 4.3877551020408165e-06,
526
+ "logits/chosen": 0.6506168842315674,
527
+ "logits/rejected": 0.4573900103569031,
528
+ "logps/chosen": -75.35042572021484,
529
+ "logps/rejected": -94.74085998535156,
530
+ "loss": 0.4573,
531
+ "rewards/accuracies": 0.875,
532
+ "rewards/chosen": 0.18374451994895935,
533
+ "rewards/margins": 0.6279397010803223,
534
+ "rewards/rejected": -0.4441951811313629,
535
+ "step": 35
536
+ },
537
+ {
538
+ "epoch": 0.43636363636363634,
539
+ "grad_norm": 2.2206430435180664,
540
+ "learning_rate": 4.35374149659864e-06,
541
+ "logits/chosen": 0.46530038118362427,
542
+ "logits/rejected": 0.15938237309455872,
543
+ "logps/chosen": -84.88845825195312,
544
+ "logps/rejected": -93.98237609863281,
545
+ "loss": 0.5184,
546
+ "rewards/accuracies": 0.875,
547
+ "rewards/chosen": -0.13509225845336914,
548
+ "rewards/margins": 0.4553631842136383,
549
+ "rewards/rejected": -0.5904554128646851,
550
+ "step": 36
551
+ },
552
+ {
553
+ "epoch": 0.4484848484848485,
554
+ "grad_norm": 2.2873029708862305,
555
+ "learning_rate": 4.319727891156463e-06,
556
+ "logits/chosen": 0.5647754669189453,
557
+ "logits/rejected": 0.32446718215942383,
558
+ "logps/chosen": -83.01586151123047,
559
+ "logps/rejected": -105.61540222167969,
560
+ "loss": 0.4647,
561
+ "rewards/accuracies": 1.0,
562
+ "rewards/chosen": -0.2565247416496277,
563
+ "rewards/margins": 0.611173152923584,
564
+ "rewards/rejected": -0.8676979541778564,
565
+ "step": 37
566
+ },
567
+ {
568
+ "epoch": 0.46060606060606063,
569
+ "grad_norm": 2.183913230895996,
570
+ "learning_rate": 4.2857142857142855e-06,
571
+ "logits/chosen": 0.10570168495178223,
572
+ "logits/rejected": -0.027524828910827637,
573
+ "logps/chosen": -80.10659790039062,
574
+ "logps/rejected": -94.95332336425781,
575
+ "loss": 0.4946,
576
+ "rewards/accuracies": 0.875,
577
+ "rewards/chosen": 0.09906873852014542,
578
+ "rewards/margins": 0.5269591212272644,
579
+ "rewards/rejected": -0.427890419960022,
580
+ "step": 38
581
+ },
582
+ {
583
+ "epoch": 0.4727272727272727,
584
+ "grad_norm": 2.2503180503845215,
585
+ "learning_rate": 4.251700680272109e-06,
586
+ "logits/chosen": 0.14078444242477417,
587
+ "logits/rejected": 0.30571305751800537,
588
+ "logps/chosen": -83.16555786132812,
589
+ "logps/rejected": -92.3938980102539,
590
+ "loss": 0.4761,
591
+ "rewards/accuracies": 0.875,
592
+ "rewards/chosen": 0.07579489052295685,
593
+ "rewards/margins": 0.5589980483055115,
594
+ "rewards/rejected": -0.48320311307907104,
595
+ "step": 39
596
+ },
597
+ {
598
+ "epoch": 0.48484848484848486,
599
+ "grad_norm": 2.3518598079681396,
600
+ "learning_rate": 4.217687074829933e-06,
601
+ "logits/chosen": 0.3999345302581787,
602
+ "logits/rejected": 0.020384781062602997,
603
+ "logps/chosen": -86.66480255126953,
604
+ "logps/rejected": -100.3143539428711,
605
+ "loss": 0.5238,
606
+ "rewards/accuracies": 1.0,
607
+ "rewards/chosen": -0.27970850467681885,
608
+ "rewards/margins": 0.416584312915802,
609
+ "rewards/rejected": -0.6962927579879761,
610
+ "step": 40
611
+ },
612
+ {
613
+ "epoch": 0.49696969696969695,
614
+ "grad_norm": 2.3239002227783203,
615
+ "learning_rate": 4.183673469387755e-06,
616
+ "logits/chosen": 0.32163870334625244,
617
+ "logits/rejected": 0.1246703565120697,
618
+ "logps/chosen": -82.92484283447266,
619
+ "logps/rejected": -99.36587524414062,
620
+ "loss": 0.5421,
621
+ "rewards/accuracies": 0.875,
622
+ "rewards/chosen": -0.5212937593460083,
623
+ "rewards/margins": 0.40337201952934265,
624
+ "rewards/rejected": -0.9246657490730286,
625
+ "step": 41
626
+ },
627
+ {
628
+ "epoch": 0.509090909090909,
629
+ "grad_norm": 1.7185702323913574,
630
+ "learning_rate": 4.1496598639455785e-06,
631
+ "logits/chosen": 0.1402607560157776,
632
+ "logits/rejected": 0.189698725938797,
633
+ "logps/chosen": -81.78995513916016,
634
+ "logps/rejected": -86.25436401367188,
635
+ "loss": 0.5575,
636
+ "rewards/accuracies": 0.875,
637
+ "rewards/chosen": -0.14212313294410706,
638
+ "rewards/margins": 0.31536924839019775,
639
+ "rewards/rejected": -0.4574923515319824,
640
+ "step": 42
641
+ },
642
+ {
643
+ "epoch": 0.5212121212121212,
644
+ "grad_norm": 2.684324026107788,
645
+ "learning_rate": 4.115646258503402e-06,
646
+ "logits/chosen": 0.1822155863046646,
647
+ "logits/rejected": 0.09680613875389099,
648
+ "logps/chosen": -84.12541198730469,
649
+ "logps/rejected": -95.77808380126953,
650
+ "loss": 0.5116,
651
+ "rewards/accuracies": 0.875,
652
+ "rewards/chosen": -0.055333998054265976,
653
+ "rewards/margins": 0.5083685517311096,
654
+ "rewards/rejected": -0.5637024641036987,
655
+ "step": 43
656
+ },
657
+ {
658
+ "epoch": 0.5333333333333333,
659
+ "grad_norm": 2.2760534286499023,
660
+ "learning_rate": 4.081632653061225e-06,
661
+ "logits/chosen": 0.2796591818332672,
662
+ "logits/rejected": 0.08514690399169922,
663
+ "logps/chosen": -79.44110107421875,
664
+ "logps/rejected": -105.64237976074219,
665
+ "loss": 0.3924,
666
+ "rewards/accuracies": 0.875,
667
+ "rewards/chosen": -0.08903113752603531,
668
+ "rewards/margins": 0.9599316120147705,
669
+ "rewards/rejected": -1.0489627122879028,
670
+ "step": 44
671
+ },
672
+ {
673
+ "epoch": 0.5454545454545454,
674
+ "grad_norm": 2.321897506713867,
675
+ "learning_rate": 4.047619047619048e-06,
676
+ "logits/chosen": 0.3646478056907654,
677
+ "logits/rejected": 0.08748706430196762,
678
+ "logps/chosen": -81.45243072509766,
679
+ "logps/rejected": -101.58081817626953,
680
+ "loss": 0.4698,
681
+ "rewards/accuracies": 0.75,
682
+ "rewards/chosen": -0.11360569298267365,
683
+ "rewards/margins": 0.6234503984451294,
684
+ "rewards/rejected": -0.7370560765266418,
685
+ "step": 45
686
+ },
687
+ {
688
+ "epoch": 0.5575757575757576,
689
+ "grad_norm": 1.6828770637512207,
690
+ "learning_rate": 4.013605442176871e-06,
691
+ "logits/chosen": -0.027117550373077393,
692
+ "logits/rejected": -0.1517198383808136,
693
+ "logps/chosen": -83.84929656982422,
694
+ "logps/rejected": -92.15609741210938,
695
+ "loss": 0.4976,
696
+ "rewards/accuracies": 0.875,
697
+ "rewards/chosen": -0.05189989507198334,
698
+ "rewards/margins": 0.542182445526123,
699
+ "rewards/rejected": -0.5940823554992676,
700
+ "step": 46
701
+ },
702
+ {
703
+ "epoch": 0.5696969696969697,
704
+ "grad_norm": 2.0993480682373047,
705
+ "learning_rate": 3.979591836734694e-06,
706
+ "logits/chosen": 0.17406637966632843,
707
+ "logits/rejected": -0.06627433747053146,
708
+ "logps/chosen": -79.99982452392578,
709
+ "logps/rejected": -97.92196655273438,
710
+ "loss": 0.3913,
711
+ "rewards/accuracies": 1.0,
712
+ "rewards/chosen": -0.10610990226268768,
713
+ "rewards/margins": 0.9161709547042847,
714
+ "rewards/rejected": -1.0222809314727783,
715
+ "step": 47
716
+ },
717
+ {
718
+ "epoch": 0.5818181818181818,
719
+ "grad_norm": 2.8575243949890137,
720
+ "learning_rate": 3.945578231292517e-06,
721
+ "logits/chosen": 0.19870872795581818,
722
+ "logits/rejected": 0.2058108150959015,
723
+ "logps/chosen": -82.90513610839844,
724
+ "logps/rejected": -104.39181518554688,
725
+ "loss": 0.4777,
726
+ "rewards/accuracies": 0.75,
727
+ "rewards/chosen": -0.52581387758255,
728
+ "rewards/margins": 0.6539483666419983,
729
+ "rewards/rejected": -1.179762363433838,
730
+ "step": 48
731
+ },
732
+ {
733
+ "epoch": 0.593939393939394,
734
+ "grad_norm": 2.676713705062866,
735
+ "learning_rate": 3.9115646258503405e-06,
736
+ "logits/chosen": 0.018811197951436043,
737
+ "logits/rejected": 0.04601475968956947,
738
+ "logps/chosen": -87.0106430053711,
739
+ "logps/rejected": -95.17454528808594,
740
+ "loss": 0.5488,
741
+ "rewards/accuracies": 0.75,
742
+ "rewards/chosen": -0.2954186499118805,
743
+ "rewards/margins": 0.5057048797607422,
744
+ "rewards/rejected": -0.8011234998703003,
745
+ "step": 49
746
+ },
747
+ {
748
+ "epoch": 0.6060606060606061,
749
+ "grad_norm": 2.592785120010376,
750
+ "learning_rate": 3.877551020408164e-06,
751
+ "logits/chosen": -0.20389169454574585,
752
+ "logits/rejected": 0.15457789599895477,
753
+ "logps/chosen": -84.461181640625,
754
+ "logps/rejected": -94.97840881347656,
755
+ "loss": 0.5292,
756
+ "rewards/accuracies": 0.75,
757
+ "rewards/chosen": -0.47971272468566895,
758
+ "rewards/margins": 0.5050724148750305,
759
+ "rewards/rejected": -0.9847851991653442,
760
+ "step": 50
761
+ },
762
+ {
763
+ "epoch": 0.6181818181818182,
764
+ "grad_norm": 1.5183725357055664,
765
+ "learning_rate": 3.843537414965986e-06,
766
+ "logits/chosen": 0.014184877276420593,
767
+ "logits/rejected": -0.06476259976625443,
768
+ "logps/chosen": -84.92556762695312,
769
+ "logps/rejected": -98.85646057128906,
770
+ "loss": 0.4731,
771
+ "rewards/accuracies": 0.625,
772
+ "rewards/chosen": -0.3806217312812805,
773
+ "rewards/margins": 0.7172415256500244,
774
+ "rewards/rejected": -1.0978631973266602,
775
+ "step": 51
776
+ },
777
+ {
778
+ "epoch": 0.6303030303030303,
779
+ "grad_norm": 2.276607036590576,
780
+ "learning_rate": 3.80952380952381e-06,
781
+ "logits/chosen": -0.08753400295972824,
782
+ "logits/rejected": -0.06234829127788544,
783
+ "logps/chosen": -88.94854736328125,
784
+ "logps/rejected": -100.52336120605469,
785
+ "loss": 0.3788,
786
+ "rewards/accuracies": 1.0,
787
+ "rewards/chosen": -0.18584024906158447,
788
+ "rewards/margins": 0.8677559494972229,
789
+ "rewards/rejected": -1.0535962581634521,
790
+ "step": 52
791
+ },
792
+ {
793
+ "epoch": 0.6424242424242425,
794
+ "grad_norm": 1.763636589050293,
795
+ "learning_rate": 3.7755102040816327e-06,
796
+ "logits/chosen": -0.09077857434749603,
797
+ "logits/rejected": -0.15150956809520721,
798
+ "logps/chosen": -87.24674224853516,
799
+ "logps/rejected": -104.22818756103516,
800
+ "loss": 0.377,
801
+ "rewards/accuracies": 0.875,
802
+ "rewards/chosen": -0.0016011148691177368,
803
+ "rewards/margins": 0.9522408843040466,
804
+ "rewards/rejected": -0.953842043876648,
805
+ "step": 53
806
+ },
807
+ {
808
+ "epoch": 0.6545454545454545,
809
+ "grad_norm": 2.4671847820281982,
810
+ "learning_rate": 3.7414965986394563e-06,
811
+ "logits/chosen": -0.016229011118412018,
812
+ "logits/rejected": -0.06034049391746521,
813
+ "logps/chosen": -84.28144073486328,
814
+ "logps/rejected": -105.29624938964844,
815
+ "loss": 0.3391,
816
+ "rewards/accuracies": 1.0,
817
+ "rewards/chosen": -0.5438804626464844,
818
+ "rewards/margins": 1.0644160509109497,
819
+ "rewards/rejected": -1.6082963943481445,
820
+ "step": 54
821
+ },
822
+ {
823
+ "epoch": 0.6666666666666666,
824
+ "grad_norm": 1.968894124031067,
825
+ "learning_rate": 3.7074829931972796e-06,
826
+ "logits/chosen": -0.1544817090034485,
827
+ "logits/rejected": -0.1875695288181305,
828
+ "logps/chosen": -89.4288101196289,
829
+ "logps/rejected": -106.10208892822266,
830
+ "loss": 0.4345,
831
+ "rewards/accuracies": 1.0,
832
+ "rewards/chosen": -0.3140699565410614,
833
+ "rewards/margins": 0.6972665190696716,
834
+ "rewards/rejected": -1.0113364458084106,
835
+ "step": 55
836
+ },
837
+ {
838
+ "epoch": 0.6787878787878788,
839
+ "grad_norm": 2.3655052185058594,
840
+ "learning_rate": 3.6734693877551024e-06,
841
+ "logits/chosen": -0.2433435171842575,
842
+ "logits/rejected": -0.15462861955165863,
843
+ "logps/chosen": -89.85462188720703,
844
+ "logps/rejected": -101.96684265136719,
845
+ "loss": 0.4306,
846
+ "rewards/accuracies": 0.875,
847
+ "rewards/chosen": -0.44760817289352417,
848
+ "rewards/margins": 0.8661949634552002,
849
+ "rewards/rejected": -1.3138031959533691,
850
+ "step": 56
851
+ },
852
+ {
853
+ "epoch": 0.6909090909090909,
854
+ "grad_norm": 2.2802484035491943,
855
+ "learning_rate": 3.6394557823129257e-06,
856
+ "logits/chosen": -0.005502855405211449,
857
+ "logits/rejected": -0.03438833728432655,
858
+ "logps/chosen": -86.78486633300781,
859
+ "logps/rejected": -108.69119262695312,
860
+ "loss": 0.3527,
861
+ "rewards/accuracies": 1.0,
862
+ "rewards/chosen": -0.5136454701423645,
863
+ "rewards/margins": 1.0334694385528564,
864
+ "rewards/rejected": -1.5471149682998657,
865
+ "step": 57
866
+ },
867
+ {
868
+ "epoch": 0.703030303030303,
869
+ "grad_norm": 2.532391309738159,
870
+ "learning_rate": 3.6054421768707485e-06,
871
+ "logits/chosen": -0.05163462832570076,
872
+ "logits/rejected": -0.15335048735141754,
873
+ "logps/chosen": -87.60224914550781,
874
+ "logps/rejected": -105.06623840332031,
875
+ "loss": 0.437,
876
+ "rewards/accuracies": 0.875,
877
+ "rewards/chosen": -0.7067617774009705,
878
+ "rewards/margins": 0.7635065913200378,
879
+ "rewards/rejected": -1.4702682495117188,
880
+ "step": 58
881
+ },
882
+ {
883
+ "epoch": 0.7151515151515152,
884
+ "grad_norm": 3.486032485961914,
885
+ "learning_rate": 3.5714285714285718e-06,
886
+ "logits/chosen": -0.49854233860969543,
887
+ "logits/rejected": -0.42069506645202637,
888
+ "logps/chosen": -84.32408142089844,
889
+ "logps/rejected": -98.28240966796875,
890
+ "loss": 0.6605,
891
+ "rewards/accuracies": 0.625,
892
+ "rewards/chosen": -0.8036391735076904,
893
+ "rewards/margins": 0.3057122826576233,
894
+ "rewards/rejected": -1.109351396560669,
895
+ "step": 59
896
+ },
897
+ {
898
+ "epoch": 0.7272727272727273,
899
+ "grad_norm": 3.032853364944458,
900
+ "learning_rate": 3.537414965986395e-06,
901
+ "logits/chosen": -0.31167668104171753,
902
+ "logits/rejected": -0.21693065762519836,
903
+ "logps/chosen": -82.40815734863281,
904
+ "logps/rejected": -104.02395629882812,
905
+ "loss": 0.4043,
906
+ "rewards/accuracies": 0.875,
907
+ "rewards/chosen": -0.5048532485961914,
908
+ "rewards/margins": 0.9285488724708557,
909
+ "rewards/rejected": -1.433402180671692,
910
+ "step": 60
911
+ },
912
+ {
913
+ "epoch": 0.7393939393939394,
914
+ "grad_norm": 2.9223711490631104,
915
+ "learning_rate": 3.503401360544218e-06,
916
+ "logits/chosen": -0.28187453746795654,
917
+ "logits/rejected": -0.477741003036499,
918
+ "logps/chosen": -90.93452453613281,
919
+ "logps/rejected": -99.23526000976562,
920
+ "loss": 0.5541,
921
+ "rewards/accuracies": 0.75,
922
+ "rewards/chosen": -0.9466086626052856,
923
+ "rewards/margins": 0.38151228427886963,
924
+ "rewards/rejected": -1.3281209468841553,
925
+ "step": 61
926
+ },
927
+ {
928
+ "epoch": 0.7515151515151515,
929
+ "grad_norm": 2.4500951766967773,
930
+ "learning_rate": 3.469387755102041e-06,
931
+ "logits/chosen": -0.8493286371231079,
932
+ "logits/rejected": -0.5042667388916016,
933
+ "logps/chosen": -85.09526824951172,
934
+ "logps/rejected": -102.77671813964844,
935
+ "loss": 0.3626,
936
+ "rewards/accuracies": 0.875,
937
+ "rewards/chosen": -0.7808952331542969,
938
+ "rewards/margins": 1.1093262434005737,
939
+ "rewards/rejected": -1.8902214765548706,
940
+ "step": 62
941
+ },
942
+ {
943
+ "epoch": 0.7636363636363637,
944
+ "grad_norm": 2.2984867095947266,
945
+ "learning_rate": 3.435374149659864e-06,
946
+ "logits/chosen": -0.6557090282440186,
947
+ "logits/rejected": -0.68489009141922,
948
+ "logps/chosen": -86.66493225097656,
949
+ "logps/rejected": -99.13359069824219,
950
+ "loss": 0.4207,
951
+ "rewards/accuracies": 0.875,
952
+ "rewards/chosen": -0.8295649290084839,
953
+ "rewards/margins": 0.7867836952209473,
954
+ "rewards/rejected": -1.6163487434387207,
955
+ "step": 63
956
+ },
957
+ {
958
+ "epoch": 0.7757575757575758,
959
+ "grad_norm": 2.1829628944396973,
960
+ "learning_rate": 3.4013605442176872e-06,
961
+ "logits/chosen": 0.057308848947286606,
962
+ "logits/rejected": -0.05564488470554352,
963
+ "logps/chosen": -96.36569213867188,
964
+ "logps/rejected": -111.1490478515625,
965
+ "loss": 0.4997,
966
+ "rewards/accuracies": 0.875,
967
+ "rewards/chosen": -0.7329980134963989,
968
+ "rewards/margins": 0.6487807631492615,
969
+ "rewards/rejected": -1.3817787170410156,
970
+ "step": 64
971
+ },
972
+ {
973
+ "epoch": 0.7878787878787878,
974
+ "grad_norm": 2.3714816570281982,
975
+ "learning_rate": 3.3673469387755105e-06,
976
+ "logits/chosen": -0.6570330262184143,
977
+ "logits/rejected": -0.6089777946472168,
978
+ "logps/chosen": -87.11174774169922,
979
+ "logps/rejected": -109.03756713867188,
980
+ "loss": 0.325,
981
+ "rewards/accuracies": 1.0,
982
+ "rewards/chosen": -0.430930495262146,
983
+ "rewards/margins": 1.1537929773330688,
984
+ "rewards/rejected": -1.5847234725952148,
985
+ "step": 65
986
+ },
987
+ {
988
+ "epoch": 0.8,
989
+ "grad_norm": 2.326923370361328,
990
+ "learning_rate": 3.3333333333333333e-06,
991
+ "logits/chosen": -0.4283097982406616,
992
+ "logits/rejected": -0.5568526387214661,
993
+ "logps/chosen": -82.62942504882812,
994
+ "logps/rejected": -104.06118774414062,
995
+ "loss": 0.3385,
996
+ "rewards/accuracies": 0.875,
997
+ "rewards/chosen": -0.24116915464401245,
998
+ "rewards/margins": 1.3031015396118164,
999
+ "rewards/rejected": -1.5442705154418945,
1000
+ "step": 66
1001
+ },
1002
+ {
1003
+ "epoch": 0.8121212121212121,
1004
+ "grad_norm": 2.2417383193969727,
1005
+ "learning_rate": 3.2993197278911566e-06,
1006
+ "logits/chosen": -0.2863921821117401,
1007
+ "logits/rejected": -0.34131675958633423,
1008
+ "logps/chosen": -82.39275360107422,
1009
+ "logps/rejected": -103.30265808105469,
1010
+ "loss": 0.3566,
1011
+ "rewards/accuracies": 0.75,
1012
+ "rewards/chosen": -0.5457499623298645,
1013
+ "rewards/margins": 1.191537618637085,
1014
+ "rewards/rejected": -1.7372875213623047,
1015
+ "step": 67
1016
+ },
1017
+ {
1018
+ "epoch": 0.8242424242424242,
1019
+ "grad_norm": 1.7097468376159668,
1020
+ "learning_rate": 3.2653061224489794e-06,
1021
+ "logits/chosen": -0.8362722992897034,
1022
+ "logits/rejected": -1.0511568784713745,
1023
+ "logps/chosen": -83.73776245117188,
1024
+ "logps/rejected": -99.80606842041016,
1025
+ "loss": 0.3407,
1026
+ "rewards/accuracies": 0.75,
1027
+ "rewards/chosen": -0.3446452021598816,
1028
+ "rewards/margins": 1.332738995552063,
1029
+ "rewards/rejected": -1.6773841381072998,
1030
+ "step": 68
1031
+ },
1032
+ {
1033
+ "epoch": 0.8363636363636363,
1034
+ "grad_norm": 2.3758466243743896,
1035
+ "learning_rate": 3.231292517006803e-06,
1036
+ "logits/chosen": -0.5980571508407593,
1037
+ "logits/rejected": -0.612303614616394,
1038
+ "logps/chosen": -90.42843627929688,
1039
+ "logps/rejected": -111.27342987060547,
1040
+ "loss": 0.3426,
1041
+ "rewards/accuracies": 0.875,
1042
+ "rewards/chosen": -0.7758303880691528,
1043
+ "rewards/margins": 1.0578529834747314,
1044
+ "rewards/rejected": -1.8336834907531738,
1045
+ "step": 69
1046
+ },
1047
+ {
1048
+ "epoch": 0.8484848484848485,
1049
+ "grad_norm": 2.9294631481170654,
1050
+ "learning_rate": 3.1972789115646264e-06,
1051
+ "logits/chosen": -0.5315291285514832,
1052
+ "logits/rejected": -0.5597429275512695,
1053
+ "logps/chosen": -90.48263549804688,
1054
+ "logps/rejected": -105.86819458007812,
1055
+ "loss": 0.327,
1056
+ "rewards/accuracies": 0.875,
1057
+ "rewards/chosen": -0.6664019823074341,
1058
+ "rewards/margins": 1.3921841382980347,
1059
+ "rewards/rejected": -2.0585861206054688,
1060
+ "step": 70
1061
+ },
1062
+ {
1063
+ "epoch": 0.8606060606060606,
1064
+ "grad_norm": 2.242730140686035,
1065
+ "learning_rate": 3.1632653061224496e-06,
1066
+ "logits/chosen": -0.6034741997718811,
1067
+ "logits/rejected": -0.6490014791488647,
1068
+ "logps/chosen": -86.84711456298828,
1069
+ "logps/rejected": -98.25587463378906,
1070
+ "loss": 0.3063,
1071
+ "rewards/accuracies": 0.75,
1072
+ "rewards/chosen": -0.6566754579544067,
1073
+ "rewards/margins": 1.320617914199829,
1074
+ "rewards/rejected": -1.9772932529449463,
1075
+ "step": 71
1076
+ },
1077
+ {
1078
+ "epoch": 0.8727272727272727,
1079
+ "grad_norm": 2.2084603309631348,
1080
+ "learning_rate": 3.1292517006802725e-06,
1081
+ "logits/chosen": -0.6398088932037354,
1082
+ "logits/rejected": -0.6443818211555481,
1083
+ "logps/chosen": -87.65351104736328,
1084
+ "logps/rejected": -121.56532287597656,
1085
+ "loss": 0.2704,
1086
+ "rewards/accuracies": 1.0,
1087
+ "rewards/chosen": -0.693427324295044,
1088
+ "rewards/margins": 1.869689702987671,
1089
+ "rewards/rejected": -2.563117027282715,
1090
+ "step": 72
1091
+ },
1092
+ {
1093
+ "epoch": 0.8848484848484849,
1094
+ "grad_norm": 2.9377784729003906,
1095
+ "learning_rate": 3.0952380952380957e-06,
1096
+ "logits/chosen": -0.9345231652259827,
1097
+ "logits/rejected": -0.8481395244598389,
1098
+ "logps/chosen": -90.87952423095703,
1099
+ "logps/rejected": -111.86048889160156,
1100
+ "loss": 0.4141,
1101
+ "rewards/accuracies": 0.875,
1102
+ "rewards/chosen": -0.99037766456604,
1103
+ "rewards/margins": 1.1532552242279053,
1104
+ "rewards/rejected": -2.1436328887939453,
1105
+ "step": 73
1106
+ },
1107
+ {
1108
+ "epoch": 0.896969696969697,
1109
+ "grad_norm": NaN,
1110
+ "learning_rate": 3.0952380952380957e-06,
1111
+ "logits/chosen": -0.7441086769104004,
1112
+ "logits/rejected": -0.5624651312828064,
1113
+ "logps/chosen": -91.13578796386719,
1114
+ "logps/rejected": -105.57829284667969,
1115
+ "loss": 0.5834,
1116
+ "rewards/accuracies": 0.75,
1117
+ "rewards/chosen": -1.249877691268921,
1118
+ "rewards/margins": 0.8648021221160889,
1119
+ "rewards/rejected": -2.1146798133850098,
1120
+ "step": 74
1121
+ },
1122
+ {
1123
+ "epoch": 0.9090909090909091,
1124
+ "grad_norm": 2.4748213291168213,
1125
+ "learning_rate": 3.0612244897959185e-06,
1126
+ "logits/chosen": -0.8197535276412964,
1127
+ "logits/rejected": -0.777995228767395,
1128
+ "logps/chosen": -91.41878509521484,
1129
+ "logps/rejected": -118.0909423828125,
1130
+ "loss": 0.2613,
1131
+ "rewards/accuracies": 1.0,
1132
+ "rewards/chosen": -1.1850990056991577,
1133
+ "rewards/margins": 1.63706636428833,
1134
+ "rewards/rejected": -2.8221654891967773,
1135
+ "step": 75
1136
+ },
1137
+ {
1138
+ "epoch": 0.9212121212121213,
1139
+ "grad_norm": 3.374300003051758,
1140
+ "learning_rate": 3.027210884353742e-06,
1141
+ "logits/chosen": -0.9236270785331726,
1142
+ "logits/rejected": -0.8337404727935791,
1143
+ "logps/chosen": -96.32296752929688,
1144
+ "logps/rejected": -125.73896026611328,
1145
+ "loss": 0.3092,
1146
+ "rewards/accuracies": 0.875,
1147
+ "rewards/chosen": -1.3610727787017822,
1148
+ "rewards/margins": 1.4623136520385742,
1149
+ "rewards/rejected": -2.8233861923217773,
1150
+ "step": 76
1151
+ },
1152
+ {
1153
+ "epoch": 0.9333333333333333,
1154
+ "grad_norm": 2.5666401386260986,
1155
+ "learning_rate": 2.993197278911565e-06,
1156
+ "logits/chosen": -1.0324219465255737,
1157
+ "logits/rejected": -0.9907780885696411,
1158
+ "logps/chosen": -88.73812103271484,
1159
+ "logps/rejected": -126.26348876953125,
1160
+ "loss": 0.2357,
1161
+ "rewards/accuracies": 0.875,
1162
+ "rewards/chosen": -1.0420153141021729,
1163
+ "rewards/margins": 1.8817570209503174,
1164
+ "rewards/rejected": -2.9237725734710693,
1165
+ "step": 77
1166
+ },
1167
+ {
1168
+ "epoch": 0.9454545454545454,
1169
+ "grad_norm": 4.485978126525879,
1170
+ "learning_rate": 2.959183673469388e-06,
1171
+ "logits/chosen": -1.0606157779693604,
1172
+ "logits/rejected": -0.9346469044685364,
1173
+ "logps/chosen": -90.07966613769531,
1174
+ "logps/rejected": -98.59324645996094,
1175
+ "loss": 0.5322,
1176
+ "rewards/accuracies": 0.875,
1177
+ "rewards/chosen": -1.1819792985916138,
1178
+ "rewards/margins": 0.40667369961738586,
1179
+ "rewards/rejected": -1.5886529684066772,
1180
+ "step": 78
1181
+ },
1182
+ {
1183
+ "epoch": 0.9575757575757575,
1184
+ "grad_norm": 2.4757275581359863,
1185
+ "learning_rate": 2.925170068027211e-06,
1186
+ "logits/chosen": -0.8977766633033752,
1187
+ "logits/rejected": -0.924136221408844,
1188
+ "logps/chosen": -91.82334899902344,
1189
+ "logps/rejected": -112.15080261230469,
1190
+ "loss": 0.2769,
1191
+ "rewards/accuracies": 1.0,
1192
+ "rewards/chosen": -0.6803606152534485,
1193
+ "rewards/margins": 1.7069063186645508,
1194
+ "rewards/rejected": -2.3872668743133545,
1195
+ "step": 79
1196
+ },
1197
+ {
1198
+ "epoch": 0.9696969696969697,
1199
+ "grad_norm": 3.1851117610931396,
1200
+ "learning_rate": 2.891156462585034e-06,
1201
+ "logits/chosen": -0.98236083984375,
1202
+ "logits/rejected": -0.9971530437469482,
1203
+ "logps/chosen": -86.82650756835938,
1204
+ "logps/rejected": -117.18161010742188,
1205
+ "loss": 0.2453,
1206
+ "rewards/accuracies": 1.0,
1207
+ "rewards/chosen": -0.8689044713973999,
1208
+ "rewards/margins": 2.010478973388672,
1209
+ "rewards/rejected": -2.8793835639953613,
1210
+ "step": 80
1211
+ },
1212
+ {
1213
+ "epoch": 0.9818181818181818,
1214
+ "grad_norm": 3.0963189601898193,
1215
+ "learning_rate": 2.8571428571428573e-06,
1216
+ "logits/chosen": -1.145169973373413,
1217
+ "logits/rejected": -1.1914927959442139,
1218
+ "logps/chosen": -96.40525817871094,
1219
+ "logps/rejected": -118.5788345336914,
1220
+ "loss": 0.3285,
1221
+ "rewards/accuracies": 0.75,
1222
+ "rewards/chosen": -1.5041391849517822,
1223
+ "rewards/margins": 1.3836085796356201,
1224
+ "rewards/rejected": -2.8877477645874023,
1225
+ "step": 81
1226
+ },
1227
+ {
1228
+ "epoch": 0.9939393939393939,
1229
+ "grad_norm": 2.4315578937530518,
1230
+ "learning_rate": 2.8231292517006805e-06,
1231
+ "logits/chosen": -1.113650918006897,
1232
+ "logits/rejected": -1.160982370376587,
1233
+ "logps/chosen": -95.52598571777344,
1234
+ "logps/rejected": -122.83209228515625,
1235
+ "loss": 0.2272,
1236
+ "rewards/accuracies": 1.0,
1237
+ "rewards/chosen": -1.3614020347595215,
1238
+ "rewards/margins": 1.9207268953323364,
1239
+ "rewards/rejected": -3.2821288108825684,
1240
+ "step": 82
1241
+ },
1242
+ {
1243
+ "epoch": 1.0,
1244
+ "grad_norm": 1.3302698135375977,
1245
+ "learning_rate": 2.7891156462585034e-06,
1246
+ "logits/chosen": -1.3629558086395264,
1247
+ "logits/rejected": -1.263519525527954,
1248
+ "logps/chosen": -89.43887329101562,
1249
+ "logps/rejected": -117.10709381103516,
1250
+ "loss": 0.0766,
1251
+ "rewards/accuracies": 1.0,
1252
+ "rewards/chosen": -0.4000343680381775,
1253
+ "rewards/margins": 2.2837576866149902,
1254
+ "rewards/rejected": -2.6837921142578125,
1255
+ "step": 83
1256
+ },
1257
+ {
1258
+ "epoch": 1.0121212121212122,
1259
+ "grad_norm": 2.363071918487549,
1260
+ "learning_rate": 2.7551020408163266e-06,
1261
+ "logits/chosen": -1.3711086511611938,
1262
+ "logits/rejected": -1.2426812648773193,
1263
+ "logps/chosen": -95.48851013183594,
1264
+ "logps/rejected": -117.56849670410156,
1265
+ "loss": 0.2692,
1266
+ "rewards/accuracies": 0.875,
1267
+ "rewards/chosen": -1.3288049697875977,
1268
+ "rewards/margins": 1.447179913520813,
1269
+ "rewards/rejected": -2.775984764099121,
1270
+ "step": 84
1271
+ },
1272
+ {
1273
+ "epoch": 1.0242424242424242,
1274
+ "grad_norm": 2.7445614337921143,
1275
+ "learning_rate": 2.7210884353741503e-06,
1276
+ "logits/chosen": -0.9873888492584229,
1277
+ "logits/rejected": -1.0433766841888428,
1278
+ "logps/chosen": -93.86320495605469,
1279
+ "logps/rejected": -116.3147201538086,
1280
+ "loss": 0.3151,
1281
+ "rewards/accuracies": 1.0,
1282
+ "rewards/chosen": -1.4767199754714966,
1283
+ "rewards/margins": 1.3222770690917969,
1284
+ "rewards/rejected": -2.798996925354004,
1285
+ "step": 85
1286
+ },
1287
+ {
1288
+ "epoch": 1.0363636363636364,
1289
+ "grad_norm": 4.278541088104248,
1290
+ "learning_rate": 2.687074829931973e-06,
1291
+ "logits/chosen": -1.0000330209732056,
1292
+ "logits/rejected": -1.0293034315109253,
1293
+ "logps/chosen": -96.55633544921875,
1294
+ "logps/rejected": -115.9503173828125,
1295
+ "loss": 0.354,
1296
+ "rewards/accuracies": 0.875,
1297
+ "rewards/chosen": -1.2256011962890625,
1298
+ "rewards/margins": 1.4643216133117676,
1299
+ "rewards/rejected": -2.68992280960083,
1300
+ "step": 86
1301
+ },
1302
+ {
1303
+ "epoch": 1.0484848484848486,
1304
+ "grad_norm": 4.383340835571289,
1305
+ "learning_rate": 2.6530612244897964e-06,
1306
+ "logits/chosen": -1.4603015184402466,
1307
+ "logits/rejected": -1.3601713180541992,
1308
+ "logps/chosen": -93.01313781738281,
1309
+ "logps/rejected": -115.10786437988281,
1310
+ "loss": 0.291,
1311
+ "rewards/accuracies": 0.875,
1312
+ "rewards/chosen": -1.329301118850708,
1313
+ "rewards/margins": 1.7043461799621582,
1314
+ "rewards/rejected": -3.0336475372314453,
1315
+ "step": 87
1316
+ },
1317
+ {
1318
+ "epoch": 1.0606060606060606,
1319
+ "grad_norm": 3.752023696899414,
1320
+ "learning_rate": 2.6190476190476192e-06,
1321
+ "logits/chosen": -1.0595191717147827,
1322
+ "logits/rejected": -1.129692554473877,
1323
+ "logps/chosen": -98.60609436035156,
1324
+ "logps/rejected": -129.41033935546875,
1325
+ "loss": 0.2992,
1326
+ "rewards/accuracies": 0.875,
1327
+ "rewards/chosen": -1.1922227144241333,
1328
+ "rewards/margins": 2.3776955604553223,
1329
+ "rewards/rejected": -3.569918155670166,
1330
+ "step": 88
1331
+ },
1332
+ {
1333
+ "epoch": 1.0727272727272728,
1334
+ "grad_norm": 1.9620897769927979,
1335
+ "learning_rate": 2.5850340136054425e-06,
1336
+ "logits/chosen": -1.139256238937378,
1337
+ "logits/rejected": -1.2131175994873047,
1338
+ "logps/chosen": -86.63563537597656,
1339
+ "logps/rejected": -117.04485321044922,
1340
+ "loss": 0.217,
1341
+ "rewards/accuracies": 1.0,
1342
+ "rewards/chosen": -1.0059623718261719,
1343
+ "rewards/margins": 2.2501814365386963,
1344
+ "rewards/rejected": -3.256143569946289,
1345
+ "step": 89
1346
+ },
1347
+ {
1348
+ "epoch": 1.084848484848485,
1349
+ "grad_norm": 5.18034029006958,
1350
+ "learning_rate": 2.5510204081632657e-06,
1351
+ "logits/chosen": -1.088368535041809,
1352
+ "logits/rejected": -1.310705304145813,
1353
+ "logps/chosen": -88.38562774658203,
1354
+ "logps/rejected": -115.90390014648438,
1355
+ "loss": 0.3847,
1356
+ "rewards/accuracies": 0.875,
1357
+ "rewards/chosen": -0.9935359358787537,
1358
+ "rewards/margins": 1.7633311748504639,
1359
+ "rewards/rejected": -2.7568671703338623,
1360
+ "step": 90
1361
+ },
1362
+ {
1363
+ "epoch": 1.096969696969697,
1364
+ "grad_norm": 3.5002970695495605,
1365
+ "learning_rate": 2.5170068027210886e-06,
1366
+ "logits/chosen": -1.2597205638885498,
1367
+ "logits/rejected": -1.279348611831665,
1368
+ "logps/chosen": -96.96222686767578,
1369
+ "logps/rejected": -126.3550796508789,
1370
+ "loss": 0.2583,
1371
+ "rewards/accuracies": 1.0,
1372
+ "rewards/chosen": -1.492140531539917,
1373
+ "rewards/margins": 1.8623967170715332,
1374
+ "rewards/rejected": -3.35453724861145,
1375
+ "step": 91
1376
+ },
1377
+ {
1378
+ "epoch": 1.1090909090909091,
1379
+ "grad_norm": 4.057419300079346,
1380
+ "learning_rate": 2.482993197278912e-06,
1381
+ "logits/chosen": -1.357700228691101,
1382
+ "logits/rejected": -1.3373640775680542,
1383
+ "logps/chosen": -98.337890625,
1384
+ "logps/rejected": -108.06039428710938,
1385
+ "loss": 0.4725,
1386
+ "rewards/accuracies": 0.625,
1387
+ "rewards/chosen": -1.7533509731292725,
1388
+ "rewards/margins": 0.8783911466598511,
1389
+ "rewards/rejected": -2.631742000579834,
1390
+ "step": 92
1391
+ },
1392
+ {
1393
+ "epoch": 1.121212121212121,
1394
+ "grad_norm": 1.610083818435669,
1395
+ "learning_rate": 2.4489795918367347e-06,
1396
+ "logits/chosen": -1.2951648235321045,
1397
+ "logits/rejected": -1.405087947845459,
1398
+ "logps/chosen": -88.3757553100586,
1399
+ "logps/rejected": -124.96995544433594,
1400
+ "loss": 0.0841,
1401
+ "rewards/accuracies": 1.0,
1402
+ "rewards/chosen": -0.8665078282356262,
1403
+ "rewards/margins": 2.740098476409912,
1404
+ "rewards/rejected": -3.6066062450408936,
1405
+ "step": 93
1406
+ },
1407
+ {
1408
+ "epoch": 1.1333333333333333,
1409
+ "grad_norm": 3.589198112487793,
1410
+ "learning_rate": 2.414965986394558e-06,
1411
+ "logits/chosen": -1.2180225849151611,
1412
+ "logits/rejected": -1.3185333013534546,
1413
+ "logps/chosen": -103.8935317993164,
1414
+ "logps/rejected": -126.54220581054688,
1415
+ "loss": 0.3401,
1416
+ "rewards/accuracies": 1.0,
1417
+ "rewards/chosen": -1.6970596313476562,
1418
+ "rewards/margins": 1.3950029611587524,
1419
+ "rewards/rejected": -3.0920627117156982,
1420
+ "step": 94
1421
+ },
1422
+ {
1423
+ "epoch": 1.1454545454545455,
1424
+ "grad_norm": 1.59670090675354,
1425
+ "learning_rate": 2.380952380952381e-06,
1426
+ "logits/chosen": -1.5853278636932373,
1427
+ "logits/rejected": -1.414257287979126,
1428
+ "logps/chosen": -89.67636108398438,
1429
+ "logps/rejected": -134.74940490722656,
1430
+ "loss": 0.1001,
1431
+ "rewards/accuracies": 1.0,
1432
+ "rewards/chosen": -0.6401633024215698,
1433
+ "rewards/margins": 2.851132392883301,
1434
+ "rewards/rejected": -3.49129581451416,
1435
+ "step": 95
1436
+ },
1437
+ {
1438
+ "epoch": 1.1575757575757575,
1439
+ "grad_norm": 4.376671314239502,
1440
+ "learning_rate": 2.3469387755102044e-06,
1441
+ "logits/chosen": -1.38065767288208,
1442
+ "logits/rejected": -1.435377836227417,
1443
+ "logps/chosen": -96.53055572509766,
1444
+ "logps/rejected": -119.9242935180664,
1445
+ "loss": 0.4199,
1446
+ "rewards/accuracies": 0.75,
1447
+ "rewards/chosen": -1.8974272012710571,
1448
+ "rewards/margins": 1.4000160694122314,
1449
+ "rewards/rejected": -3.297443151473999,
1450
+ "step": 96
1451
+ },
1452
+ {
1453
+ "epoch": 1.1696969696969697,
1454
+ "grad_norm": 2.5332932472229004,
1455
+ "learning_rate": 2.3129251700680273e-06,
1456
+ "logits/chosen": -1.302667260169983,
1457
+ "logits/rejected": -1.3088089227676392,
1458
+ "logps/chosen": -93.29157257080078,
1459
+ "logps/rejected": -123.02129364013672,
1460
+ "loss": 0.234,
1461
+ "rewards/accuracies": 1.0,
1462
+ "rewards/chosen": -0.987712562084198,
1463
+ "rewards/margins": 2.1374382972717285,
1464
+ "rewards/rejected": -3.1251509189605713,
1465
+ "step": 97
1466
+ },
1467
+ {
1468
+ "epoch": 1.1818181818181819,
1469
+ "grad_norm": 2.5916943550109863,
1470
+ "learning_rate": 2.2789115646258505e-06,
1471
+ "logits/chosen": -1.6041796207427979,
1472
+ "logits/rejected": -1.516028642654419,
1473
+ "logps/chosen": -93.6436767578125,
1474
+ "logps/rejected": -125.10226440429688,
1475
+ "loss": 0.1684,
1476
+ "rewards/accuracies": 1.0,
1477
+ "rewards/chosen": -1.393079400062561,
1478
+ "rewards/margins": 2.149301290512085,
1479
+ "rewards/rejected": -3.5423808097839355,
1480
+ "step": 98
1481
+ },
1482
+ {
1483
+ "epoch": 1.1939393939393939,
1484
+ "grad_norm": 4.822098731994629,
1485
+ "learning_rate": 2.244897959183674e-06,
1486
+ "logits/chosen": -1.5234119892120361,
1487
+ "logits/rejected": -1.4276056289672852,
1488
+ "logps/chosen": -98.25959014892578,
1489
+ "logps/rejected": -122.98600769042969,
1490
+ "loss": 0.3363,
1491
+ "rewards/accuracies": 0.875,
1492
+ "rewards/chosen": -1.4268699884414673,
1493
+ "rewards/margins": 1.5049833059310913,
1494
+ "rewards/rejected": -2.9318532943725586,
1495
+ "step": 99
1496
+ },
1497
+ {
1498
+ "epoch": 1.206060606060606,
1499
+ "grad_norm": 3.437187671661377,
1500
+ "learning_rate": 2.2108843537414966e-06,
1501
+ "logits/chosen": -1.6394751071929932,
1502
+ "logits/rejected": -1.606034517288208,
1503
+ "logps/chosen": -97.52091217041016,
1504
+ "logps/rejected": -119.31391906738281,
1505
+ "loss": 0.2759,
1506
+ "rewards/accuracies": 0.875,
1507
+ "rewards/chosen": -1.413924217224121,
1508
+ "rewards/margins": 1.6457183361053467,
1509
+ "rewards/rejected": -3.0596425533294678,
1510
+ "step": 100
1511
+ },
1512
+ {
1513
+ "epoch": 1.2181818181818183,
1514
+ "grad_norm": 3.6992416381835938,
1515
+ "learning_rate": 2.17687074829932e-06,
1516
+ "logits/chosen": -1.4257758855819702,
1517
+ "logits/rejected": -1.4925386905670166,
1518
+ "logps/chosen": -89.6732177734375,
1519
+ "logps/rejected": -111.02645111083984,
1520
+ "loss": 0.3846,
1521
+ "rewards/accuracies": 0.875,
1522
+ "rewards/chosen": -1.1798646450042725,
1523
+ "rewards/margins": 1.4852337837219238,
1524
+ "rewards/rejected": -2.6650984287261963,
1525
+ "step": 101
1526
+ },
1527
+ {
1528
+ "epoch": 1.2303030303030302,
1529
+ "grad_norm": 2.463331937789917,
1530
+ "learning_rate": 2.1428571428571427e-06,
1531
+ "logits/chosen": -1.5652334690093994,
1532
+ "logits/rejected": -1.4604835510253906,
1533
+ "logps/chosen": -97.5364761352539,
1534
+ "logps/rejected": -120.79109954833984,
1535
+ "loss": 0.1744,
1536
+ "rewards/accuracies": 1.0,
1537
+ "rewards/chosen": -1.2595030069351196,
1538
+ "rewards/margins": 1.9318180084228516,
1539
+ "rewards/rejected": -3.1913208961486816,
1540
+ "step": 102
1541
+ },
1542
+ {
1543
+ "epoch": 1.2424242424242424,
1544
+ "grad_norm": 2.5878117084503174,
1545
+ "learning_rate": 2.1088435374149664e-06,
1546
+ "logits/chosen": -1.325555682182312,
1547
+ "logits/rejected": -1.428547978401184,
1548
+ "logps/chosen": -87.47943878173828,
1549
+ "logps/rejected": -122.22314453125,
1550
+ "loss": 0.2459,
1551
+ "rewards/accuracies": 1.0,
1552
+ "rewards/chosen": -0.502652108669281,
1553
+ "rewards/margins": 2.3220510482788086,
1554
+ "rewards/rejected": -2.8247032165527344,
1555
+ "step": 103
1556
+ },
1557
+ {
1558
+ "epoch": 1.2545454545454544,
1559
+ "grad_norm": 2.1508100032806396,
1560
+ "learning_rate": 2.0748299319727892e-06,
1561
+ "logits/chosen": -1.6749191284179688,
1562
+ "logits/rejected": -1.5175896883010864,
1563
+ "logps/chosen": -96.2158432006836,
1564
+ "logps/rejected": -134.41329956054688,
1565
+ "loss": 0.1496,
1566
+ "rewards/accuracies": 1.0,
1567
+ "rewards/chosen": -1.7228033542633057,
1568
+ "rewards/margins": 2.3363349437713623,
1569
+ "rewards/rejected": -4.059138298034668,
1570
+ "step": 104
1571
+ },
1572
+ {
1573
+ "epoch": 1.2666666666666666,
1574
+ "grad_norm": 4.549570560455322,
1575
+ "learning_rate": 2.0408163265306125e-06,
1576
+ "logits/chosen": -1.583759069442749,
1577
+ "logits/rejected": -1.595736026763916,
1578
+ "logps/chosen": -91.94076538085938,
1579
+ "logps/rejected": -116.45210266113281,
1580
+ "loss": 0.3024,
1581
+ "rewards/accuracies": 1.0,
1582
+ "rewards/chosen": -1.4231905937194824,
1583
+ "rewards/margins": 1.6255862712860107,
1584
+ "rewards/rejected": -3.048776865005493,
1585
+ "step": 105
1586
+ },
1587
+ {
1588
+ "epoch": 1.2787878787878788,
1589
+ "grad_norm": 3.135284185409546,
1590
+ "learning_rate": 2.0068027210884353e-06,
1591
+ "logits/chosen": -1.4998533725738525,
1592
+ "logits/rejected": -1.4802157878875732,
1593
+ "logps/chosen": -97.09971618652344,
1594
+ "logps/rejected": -123.02347564697266,
1595
+ "loss": 0.3073,
1596
+ "rewards/accuracies": 1.0,
1597
+ "rewards/chosen": -1.9578745365142822,
1598
+ "rewards/margins": 1.374443769454956,
1599
+ "rewards/rejected": -3.3323183059692383,
1600
+ "step": 106
1601
+ },
1602
+ {
1603
+ "epoch": 1.290909090909091,
1604
+ "grad_norm": 2.2556681632995605,
1605
+ "learning_rate": 1.9727891156462586e-06,
1606
+ "logits/chosen": -1.5938842296600342,
1607
+ "logits/rejected": -1.5801693201065063,
1608
+ "logps/chosen": -96.47604370117188,
1609
+ "logps/rejected": -124.70124053955078,
1610
+ "loss": 0.1763,
1611
+ "rewards/accuracies": 1.0,
1612
+ "rewards/chosen": -1.65388822555542,
1613
+ "rewards/margins": 1.8723074197769165,
1614
+ "rewards/rejected": -3.526195526123047,
1615
+ "step": 107
1616
+ },
1617
+ {
1618
+ "epoch": 1.303030303030303,
1619
+ "grad_norm": 2.146841287612915,
1620
+ "learning_rate": 1.938775510204082e-06,
1621
+ "logits/chosen": -1.673741102218628,
1622
+ "logits/rejected": -1.6758308410644531,
1623
+ "logps/chosen": -95.465087890625,
1624
+ "logps/rejected": -127.52227020263672,
1625
+ "loss": 0.123,
1626
+ "rewards/accuracies": 1.0,
1627
+ "rewards/chosen": -1.3014137744903564,
1628
+ "rewards/margins": 2.439565658569336,
1629
+ "rewards/rejected": -3.7409796714782715,
1630
+ "step": 108
1631
+ },
1632
+ {
1633
+ "epoch": 1.3151515151515152,
1634
+ "grad_norm": 3.1009531021118164,
1635
+ "learning_rate": 1.904761904761905e-06,
1636
+ "logits/chosen": -1.7371501922607422,
1637
+ "logits/rejected": -1.6622573137283325,
1638
+ "logps/chosen": -95.71591186523438,
1639
+ "logps/rejected": -119.20672607421875,
1640
+ "loss": 0.2911,
1641
+ "rewards/accuracies": 1.0,
1642
+ "rewards/chosen": -1.3894450664520264,
1643
+ "rewards/margins": 1.329725742340088,
1644
+ "rewards/rejected": -2.7191710472106934,
1645
+ "step": 109
1646
+ },
1647
+ {
1648
+ "epoch": 1.3272727272727272,
1649
+ "grad_norm": 2.3910417556762695,
1650
+ "learning_rate": 1.8707482993197282e-06,
1651
+ "logits/chosen": -1.6049320697784424,
1652
+ "logits/rejected": -1.6519778966903687,
1653
+ "logps/chosen": -97.65145111083984,
1654
+ "logps/rejected": -135.92715454101562,
1655
+ "loss": 0.2255,
1656
+ "rewards/accuracies": 0.875,
1657
+ "rewards/chosen": -1.2323026657104492,
1658
+ "rewards/margins": 2.811933755874634,
1659
+ "rewards/rejected": -4.044236183166504,
1660
+ "step": 110
1661
+ },
1662
+ {
1663
+ "epoch": 1.3393939393939394,
1664
+ "grad_norm": 3.1409387588500977,
1665
+ "learning_rate": 1.8367346938775512e-06,
1666
+ "logits/chosen": -1.7363131046295166,
1667
+ "logits/rejected": -1.609194278717041,
1668
+ "logps/chosen": -98.52182006835938,
1669
+ "logps/rejected": -124.78396606445312,
1670
+ "loss": 0.322,
1671
+ "rewards/accuracies": 0.75,
1672
+ "rewards/chosen": -1.5491598844528198,
1673
+ "rewards/margins": 1.6308727264404297,
1674
+ "rewards/rejected": -3.180032730102539,
1675
+ "step": 111
1676
+ },
1677
+ {
1678
+ "epoch": 1.3515151515151516,
1679
+ "grad_norm": 3.7841951847076416,
1680
+ "learning_rate": 1.8027210884353743e-06,
1681
+ "logits/chosen": -1.6172230243682861,
1682
+ "logits/rejected": -1.550255298614502,
1683
+ "logps/chosen": -92.85408020019531,
1684
+ "logps/rejected": -112.62004089355469,
1685
+ "loss": 0.2447,
1686
+ "rewards/accuracies": 1.0,
1687
+ "rewards/chosen": -0.998050332069397,
1688
+ "rewards/margins": 1.8484382629394531,
1689
+ "rewards/rejected": -2.8464884757995605,
1690
+ "step": 112
1691
+ },
1692
+ {
1693
+ "epoch": 1.3636363636363638,
1694
+ "grad_norm": 1.7370332479476929,
1695
+ "learning_rate": 1.7687074829931975e-06,
1696
+ "logits/chosen": -1.7049037218093872,
1697
+ "logits/rejected": -1.7044804096221924,
1698
+ "logps/chosen": -86.98583984375,
1699
+ "logps/rejected": -121.7904052734375,
1700
+ "loss": 0.1767,
1701
+ "rewards/accuracies": 0.875,
1702
+ "rewards/chosen": -0.86982661485672,
1703
+ "rewards/margins": 2.5970449447631836,
1704
+ "rewards/rejected": -3.466871500015259,
1705
+ "step": 113
1706
+ },
1707
+ {
1708
+ "epoch": 1.3757575757575757,
1709
+ "grad_norm": 4.671180248260498,
1710
+ "learning_rate": 1.7346938775510206e-06,
1711
+ "logits/chosen": -1.6064000129699707,
1712
+ "logits/rejected": -1.7420430183410645,
1713
+ "logps/chosen": -103.08202362060547,
1714
+ "logps/rejected": -126.36719512939453,
1715
+ "loss": 0.4506,
1716
+ "rewards/accuracies": 0.625,
1717
+ "rewards/chosen": -2.119563579559326,
1718
+ "rewards/margins": 1.3957490921020508,
1719
+ "rewards/rejected": -3.515312671661377,
1720
+ "step": 114
1721
+ },
1722
+ {
1723
+ "epoch": 1.387878787878788,
1724
+ "grad_norm": 2.891258478164673,
1725
+ "learning_rate": 1.7006802721088436e-06,
1726
+ "logits/chosen": -1.7240793704986572,
1727
+ "logits/rejected": -1.7777044773101807,
1728
+ "logps/chosen": -88.76716613769531,
1729
+ "logps/rejected": -127.64596557617188,
1730
+ "loss": 0.1338,
1731
+ "rewards/accuracies": 1.0,
1732
+ "rewards/chosen": -0.8680980801582336,
1733
+ "rewards/margins": 2.9895691871643066,
1734
+ "rewards/rejected": -3.8576672077178955,
1735
+ "step": 115
1736
+ },
1737
+ {
1738
+ "epoch": 1.4,
1739
+ "grad_norm": 3.575765371322632,
1740
+ "learning_rate": 1.6666666666666667e-06,
1741
+ "logits/chosen": -1.6597156524658203,
1742
+ "logits/rejected": -1.6605198383331299,
1743
+ "logps/chosen": -88.46395874023438,
1744
+ "logps/rejected": -116.02509307861328,
1745
+ "loss": 0.2241,
1746
+ "rewards/accuracies": 1.0,
1747
+ "rewards/chosen": -0.8981451988220215,
1748
+ "rewards/margins": 2.229868173599243,
1749
+ "rewards/rejected": -3.1280133724212646,
1750
+ "step": 116
1751
+ },
1752
+ {
1753
+ "epoch": 1.412121212121212,
1754
+ "grad_norm": 1.8933932781219482,
1755
+ "learning_rate": 1.6326530612244897e-06,
1756
+ "logits/chosen": -1.6874178647994995,
1757
+ "logits/rejected": -1.6887404918670654,
1758
+ "logps/chosen": -98.41668701171875,
1759
+ "logps/rejected": -132.19210815429688,
1760
+ "loss": 0.1343,
1761
+ "rewards/accuracies": 1.0,
1762
+ "rewards/chosen": -1.6015653610229492,
1763
+ "rewards/margins": 2.4682466983795166,
1764
+ "rewards/rejected": -4.069811820983887,
1765
+ "step": 117
1766
+ },
1767
+ {
1768
+ "epoch": 1.4242424242424243,
1769
+ "grad_norm": 1.5824294090270996,
1770
+ "learning_rate": 1.5986394557823132e-06,
1771
+ "logits/chosen": -1.7633352279663086,
1772
+ "logits/rejected": -1.6844078302383423,
1773
+ "logps/chosen": -91.18302917480469,
1774
+ "logps/rejected": -133.2034912109375,
1775
+ "loss": 0.1243,
1776
+ "rewards/accuracies": 1.0,
1777
+ "rewards/chosen": -1.005885124206543,
1778
+ "rewards/margins": 3.0340609550476074,
1779
+ "rewards/rejected": -4.03994607925415,
1780
+ "step": 118
1781
+ },
1782
+ {
1783
+ "epoch": 1.4363636363636363,
1784
+ "grad_norm": 2.5562682151794434,
1785
+ "learning_rate": 1.5646258503401362e-06,
1786
+ "logits/chosen": -1.7930512428283691,
1787
+ "logits/rejected": -1.770517110824585,
1788
+ "logps/chosen": -101.37489318847656,
1789
+ "logps/rejected": -118.3425064086914,
1790
+ "loss": 0.3446,
1791
+ "rewards/accuracies": 0.75,
1792
+ "rewards/chosen": -2.229304313659668,
1793
+ "rewards/margins": 1.3117181062698364,
1794
+ "rewards/rejected": -3.541022539138794,
1795
+ "step": 119
1796
+ },
1797
+ {
1798
+ "epoch": 1.4484848484848485,
1799
+ "grad_norm": 3.2759103775024414,
1800
+ "learning_rate": 1.5306122448979593e-06,
1801
+ "logits/chosen": -1.6340594291687012,
1802
+ "logits/rejected": -1.6577837467193604,
1803
+ "logps/chosen": -85.42779541015625,
1804
+ "logps/rejected": -113.1285171508789,
1805
+ "loss": 0.2232,
1806
+ "rewards/accuracies": 1.0,
1807
+ "rewards/chosen": -0.880210280418396,
1808
+ "rewards/margins": 2.2023112773895264,
1809
+ "rewards/rejected": -3.082521677017212,
1810
+ "step": 120
1811
+ },
1812
+ {
1813
+ "epoch": 1.4606060606060607,
1814
+ "grad_norm": 1.992604374885559,
1815
+ "learning_rate": 1.4965986394557825e-06,
1816
+ "logits/chosen": -1.3893667459487915,
1817
+ "logits/rejected": -1.6112309694290161,
1818
+ "logps/chosen": -89.03352355957031,
1819
+ "logps/rejected": -123.7257308959961,
1820
+ "loss": 0.215,
1821
+ "rewards/accuracies": 0.875,
1822
+ "rewards/chosen": -0.9930751919746399,
1823
+ "rewards/margins": 2.7060134410858154,
1824
+ "rewards/rejected": -3.6990885734558105,
1825
+ "step": 121
1826
+ },
1827
+ {
1828
+ "epoch": 1.4727272727272727,
1829
+ "grad_norm": 1.8438624143600464,
1830
+ "learning_rate": 1.4625850340136056e-06,
1831
+ "logits/chosen": -1.597573161125183,
1832
+ "logits/rejected": -1.6522390842437744,
1833
+ "logps/chosen": -95.18002319335938,
1834
+ "logps/rejected": -149.4505615234375,
1835
+ "loss": 0.1243,
1836
+ "rewards/accuracies": 1.0,
1837
+ "rewards/chosen": -1.2069129943847656,
1838
+ "rewards/margins": 3.2835397720336914,
1839
+ "rewards/rejected": -4.490452766418457,
1840
+ "step": 122
1841
+ },
1842
+ {
1843
+ "epoch": 1.4848484848484849,
1844
+ "grad_norm": 2.364157199859619,
1845
+ "learning_rate": 1.4285714285714286e-06,
1846
+ "logits/chosen": -1.5696862936019897,
1847
+ "logits/rejected": -1.589264154434204,
1848
+ "logps/chosen": -98.0048828125,
1849
+ "logps/rejected": -129.3928680419922,
1850
+ "loss": 0.1642,
1851
+ "rewards/accuracies": 1.0,
1852
+ "rewards/chosen": -1.5141558647155762,
1853
+ "rewards/margins": 2.3787457942962646,
1854
+ "rewards/rejected": -3.892901659011841,
1855
+ "step": 123
1856
+ },
1857
+ {
1858
+ "epoch": 1.496969696969697,
1859
+ "grad_norm": 1.409197211265564,
1860
+ "learning_rate": 1.3945578231292517e-06,
1861
+ "logits/chosen": -1.530297040939331,
1862
+ "logits/rejected": -1.5341331958770752,
1863
+ "logps/chosen": -97.44312286376953,
1864
+ "logps/rejected": -143.15103149414062,
1865
+ "loss": 0.1037,
1866
+ "rewards/accuracies": 1.0,
1867
+ "rewards/chosen": -1.2367117404937744,
1868
+ "rewards/margins": 3.409477472305298,
1869
+ "rewards/rejected": -4.646188735961914,
1870
+ "step": 124
1871
+ },
1872
+ {
1873
+ "epoch": 1.509090909090909,
1874
+ "grad_norm": 1.676865577697754,
1875
+ "learning_rate": 1.3605442176870751e-06,
1876
+ "logits/chosen": -1.8658342361450195,
1877
+ "logits/rejected": -1.7735984325408936,
1878
+ "logps/chosen": -95.08421325683594,
1879
+ "logps/rejected": -126.56539154052734,
1880
+ "loss": 0.1224,
1881
+ "rewards/accuracies": 1.0,
1882
+ "rewards/chosen": -1.3045662641525269,
1883
+ "rewards/margins": 2.744955539703369,
1884
+ "rewards/rejected": -4.0495219230651855,
1885
+ "step": 125
1886
+ },
1887
+ {
1888
+ "epoch": 1.5212121212121212,
1889
+ "grad_norm": 1.2884773015975952,
1890
+ "learning_rate": 1.3265306122448982e-06,
1891
+ "logits/chosen": -1.664272427558899,
1892
+ "logits/rejected": -1.6544904708862305,
1893
+ "logps/chosen": -91.6380615234375,
1894
+ "logps/rejected": -132.34695434570312,
1895
+ "loss": 0.1712,
1896
+ "rewards/accuracies": 0.875,
1897
+ "rewards/chosen": -1.1254020929336548,
1898
+ "rewards/margins": 2.629539966583252,
1899
+ "rewards/rejected": -3.754941701889038,
1900
+ "step": 126
1901
+ },
1902
+ {
1903
+ "epoch": 1.5333333333333332,
1904
+ "grad_norm": 1.5515812635421753,
1905
+ "learning_rate": 1.2925170068027212e-06,
1906
+ "logits/chosen": -1.6638004779815674,
1907
+ "logits/rejected": -1.7308552265167236,
1908
+ "logps/chosen": -90.78031921386719,
1909
+ "logps/rejected": -127.66264343261719,
1910
+ "loss": 0.1479,
1911
+ "rewards/accuracies": 1.0,
1912
+ "rewards/chosen": -1.2343765497207642,
1913
+ "rewards/margins": 2.645479202270508,
1914
+ "rewards/rejected": -3.8798556327819824,
1915
+ "step": 127
1916
+ },
1917
+ {
1918
+ "epoch": 1.5454545454545454,
1919
+ "grad_norm": 3.9256937503814697,
1920
+ "learning_rate": 1.2585034013605443e-06,
1921
+ "logits/chosen": -1.8101837635040283,
1922
+ "logits/rejected": -1.7593592405319214,
1923
+ "logps/chosen": -96.83201599121094,
1924
+ "logps/rejected": -118.23445129394531,
1925
+ "loss": 0.2606,
1926
+ "rewards/accuracies": 1.0,
1927
+ "rewards/chosen": -2.00610613822937,
1928
+ "rewards/margins": 1.5677779912948608,
1929
+ "rewards/rejected": -3.5738840103149414,
1930
+ "step": 128
1931
+ },
1932
+ {
1933
+ "epoch": 1.5575757575757576,
1934
+ "grad_norm": 3.4023656845092773,
1935
+ "learning_rate": 1.2244897959183673e-06,
1936
+ "logits/chosen": -1.6174266338348389,
1937
+ "logits/rejected": -1.5729625225067139,
1938
+ "logps/chosen": -101.54139709472656,
1939
+ "logps/rejected": -124.65895080566406,
1940
+ "loss": 0.2789,
1941
+ "rewards/accuracies": 1.0,
1942
+ "rewards/chosen": -2.163264036178589,
1943
+ "rewards/margins": 1.4744569063186646,
1944
+ "rewards/rejected": -3.637721061706543,
1945
+ "step": 129
1946
+ },
1947
+ {
1948
+ "epoch": 1.5696969696969698,
1949
+ "grad_norm": 8.019857406616211,
1950
+ "learning_rate": 1.1904761904761906e-06,
1951
+ "logits/chosen": -1.9090921878814697,
1952
+ "logits/rejected": -1.7606130838394165,
1953
+ "logps/chosen": -98.2342300415039,
1954
+ "logps/rejected": -117.97206115722656,
1955
+ "loss": 0.434,
1956
+ "rewards/accuracies": 0.875,
1957
+ "rewards/chosen": -2.0541114807128906,
1958
+ "rewards/margins": 1.012056589126587,
1959
+ "rewards/rejected": -3.0661683082580566,
1960
+ "step": 130
1961
+ },
1962
+ {
1963
+ "epoch": 1.5818181818181818,
1964
+ "grad_norm": 2.7587668895721436,
1965
+ "learning_rate": 1.1564625850340136e-06,
1966
+ "logits/chosen": -1.561621904373169,
1967
+ "logits/rejected": -1.6865174770355225,
1968
+ "logps/chosen": -96.38162994384766,
1969
+ "logps/rejected": -122.79660034179688,
1970
+ "loss": 0.2854,
1971
+ "rewards/accuracies": 0.875,
1972
+ "rewards/chosen": -1.4686121940612793,
1973
+ "rewards/margins": 2.149606466293335,
1974
+ "rewards/rejected": -3.618218421936035,
1975
+ "step": 131
1976
+ },
1977
+ {
1978
+ "epoch": 1.593939393939394,
1979
+ "grad_norm": 2.6992170810699463,
1980
+ "learning_rate": 1.122448979591837e-06,
1981
+ "logits/chosen": -1.7638099193572998,
1982
+ "logits/rejected": -1.6800596714019775,
1983
+ "logps/chosen": -100.1748275756836,
1984
+ "logps/rejected": -128.0324249267578,
1985
+ "loss": 0.1708,
1986
+ "rewards/accuracies": 1.0,
1987
+ "rewards/chosen": -2.1165099143981934,
1988
+ "rewards/margins": 2.189871311187744,
1989
+ "rewards/rejected": -4.3063812255859375,
1990
+ "step": 132
1991
+ },
1992
+ {
1993
+ "epoch": 1.606060606060606,
1994
+ "grad_norm": 4.169434070587158,
1995
+ "learning_rate": 1.08843537414966e-06,
1996
+ "logits/chosen": -1.5525397062301636,
1997
+ "logits/rejected": -1.6112422943115234,
1998
+ "logps/chosen": -100.83575439453125,
1999
+ "logps/rejected": -128.28770446777344,
2000
+ "loss": 0.2272,
2001
+ "rewards/accuracies": 1.0,
2002
+ "rewards/chosen": -1.8389804363250732,
2003
+ "rewards/margins": 2.285094738006592,
2004
+ "rewards/rejected": -4.124075412750244,
2005
+ "step": 133
2006
+ },
2007
+ {
2008
+ "epoch": 1.6181818181818182,
2009
+ "grad_norm": 4.582830905914307,
2010
+ "learning_rate": 1.0544217687074832e-06,
2011
+ "logits/chosen": -1.811753511428833,
2012
+ "logits/rejected": -1.7734105587005615,
2013
+ "logps/chosen": -99.59288024902344,
2014
+ "logps/rejected": -122.28144836425781,
2015
+ "loss": 0.2968,
2016
+ "rewards/accuracies": 1.0,
2017
+ "rewards/chosen": -2.2128849029541016,
2018
+ "rewards/margins": 1.3973000049591064,
2019
+ "rewards/rejected": -3.610184907913208,
2020
+ "step": 134
2021
+ },
2022
+ {
2023
+ "epoch": 1.6303030303030304,
2024
+ "grad_norm": 3.154338836669922,
2025
+ "learning_rate": 1.0204081632653063e-06,
2026
+ "logits/chosen": -1.5578339099884033,
2027
+ "logits/rejected": -1.6350841522216797,
2028
+ "logps/chosen": -108.85205078125,
2029
+ "logps/rejected": -149.6986083984375,
2030
+ "loss": 0.1819,
2031
+ "rewards/accuracies": 1.0,
2032
+ "rewards/chosen": -2.148804187774658,
2033
+ "rewards/margins": 2.981112480163574,
2034
+ "rewards/rejected": -5.129916191101074,
2035
+ "step": 135
2036
+ },
2037
+ {
2038
+ "epoch": 1.6424242424242426,
2039
+ "grad_norm": 3.0729594230651855,
2040
+ "learning_rate": 9.863945578231293e-07,
2041
+ "logits/chosen": -1.7794138193130493,
2042
+ "logits/rejected": -1.7279512882232666,
2043
+ "logps/chosen": -99.13277435302734,
2044
+ "logps/rejected": -116.2926025390625,
2045
+ "loss": 0.219,
2046
+ "rewards/accuracies": 1.0,
2047
+ "rewards/chosen": -1.7788490056991577,
2048
+ "rewards/margins": 1.6601810455322266,
2049
+ "rewards/rejected": -3.439030170440674,
2050
+ "step": 136
2051
+ },
2052
+ {
2053
+ "epoch": 1.6545454545454545,
2054
+ "grad_norm": 4.573628902435303,
2055
+ "learning_rate": 9.523809523809525e-07,
2056
+ "logits/chosen": -1.7974026203155518,
2057
+ "logits/rejected": -1.820143461227417,
2058
+ "logps/chosen": -94.99871826171875,
2059
+ "logps/rejected": -131.11312866210938,
2060
+ "loss": 0.1895,
2061
+ "rewards/accuracies": 1.0,
2062
+ "rewards/chosen": -1.565184473991394,
2063
+ "rewards/margins": 2.551455020904541,
2064
+ "rewards/rejected": -4.116639614105225,
2065
+ "step": 137
2066
+ },
2067
+ {
2068
+ "epoch": 1.6666666666666665,
2069
+ "grad_norm": 5.590139865875244,
2070
+ "learning_rate": 9.183673469387756e-07,
2071
+ "logits/chosen": -1.8338170051574707,
2072
+ "logits/rejected": -1.789681077003479,
2073
+ "logps/chosen": -101.36553955078125,
2074
+ "logps/rejected": -121.39546203613281,
2075
+ "loss": 0.2882,
2076
+ "rewards/accuracies": 0.875,
2077
+ "rewards/chosen": -1.8100769519805908,
2078
+ "rewards/margins": 1.7394788265228271,
2079
+ "rewards/rejected": -3.549555778503418,
2080
+ "step": 138
2081
+ },
2082
+ {
2083
+ "epoch": 1.6787878787878787,
2084
+ "grad_norm": 3.2985379695892334,
2085
+ "learning_rate": 8.843537414965988e-07,
2086
+ "logits/chosen": -1.7100659608840942,
2087
+ "logits/rejected": -1.621692180633545,
2088
+ "logps/chosen": -100.19327545166016,
2089
+ "logps/rejected": -127.35150146484375,
2090
+ "loss": 0.194,
2091
+ "rewards/accuracies": 1.0,
2092
+ "rewards/chosen": -2.047415256500244,
2093
+ "rewards/margins": 2.123518466949463,
2094
+ "rewards/rejected": -4.170933246612549,
2095
+ "step": 139
2096
+ },
2097
+ {
2098
+ "epoch": 1.690909090909091,
2099
+ "grad_norm": 2.877264976501465,
2100
+ "learning_rate": 8.503401360544218e-07,
2101
+ "logits/chosen": -1.6704832315444946,
2102
+ "logits/rejected": -1.6993167400360107,
2103
+ "logps/chosen": -112.37958526611328,
2104
+ "logps/rejected": -143.90924072265625,
2105
+ "loss": 0.2125,
2106
+ "rewards/accuracies": 0.875,
2107
+ "rewards/chosen": -2.7939703464508057,
2108
+ "rewards/margins": 2.070646286010742,
2109
+ "rewards/rejected": -4.864616870880127,
2110
+ "step": 140
2111
+ },
2112
+ {
2113
+ "epoch": 1.7030303030303031,
2114
+ "grad_norm": 1.3344841003417969,
2115
+ "learning_rate": 8.163265306122449e-07,
2116
+ "logits/chosen": -1.6487529277801514,
2117
+ "logits/rejected": -1.616715669631958,
2118
+ "logps/chosen": -89.38975524902344,
2119
+ "logps/rejected": -136.5968780517578,
2120
+ "loss": 0.0842,
2121
+ "rewards/accuracies": 1.0,
2122
+ "rewards/chosen": -0.927839994430542,
2123
+ "rewards/margins": 3.7124204635620117,
2124
+ "rewards/rejected": -4.640260219573975,
2125
+ "step": 141
2126
+ },
2127
+ {
2128
+ "epoch": 1.7151515151515153,
2129
+ "grad_norm": 2.1570088863372803,
2130
+ "learning_rate": 7.823129251700681e-07,
2131
+ "logits/chosen": -1.778488039970398,
2132
+ "logits/rejected": -1.7679212093353271,
2133
+ "logps/chosen": -93.74765014648438,
2134
+ "logps/rejected": -121.31202697753906,
2135
+ "loss": 0.1657,
2136
+ "rewards/accuracies": 1.0,
2137
+ "rewards/chosen": -1.2899467945098877,
2138
+ "rewards/margins": 2.4230360984802246,
2139
+ "rewards/rejected": -3.712982654571533,
2140
+ "step": 142
2141
+ },
2142
+ {
2143
+ "epoch": 1.7272727272727273,
2144
+ "grad_norm": 2.719651460647583,
2145
+ "learning_rate": 7.482993197278913e-07,
2146
+ "logits/chosen": -1.889997124671936,
2147
+ "logits/rejected": -1.7798057794570923,
2148
+ "logps/chosen": -101.00578308105469,
2149
+ "logps/rejected": -145.29190063476562,
2150
+ "loss": 0.1061,
2151
+ "rewards/accuracies": 1.0,
2152
+ "rewards/chosen": -1.5723130702972412,
2153
+ "rewards/margins": 3.213724136352539,
2154
+ "rewards/rejected": -4.786037445068359,
2155
+ "step": 143
2156
+ },
2157
+ {
2158
+ "epoch": 1.7393939393939393,
2159
+ "grad_norm": 2.279040813446045,
2160
+ "learning_rate": 7.142857142857143e-07,
2161
+ "logits/chosen": -1.5846905708312988,
2162
+ "logits/rejected": -1.7222152948379517,
2163
+ "logps/chosen": -92.99658203125,
2164
+ "logps/rejected": -149.00729370117188,
2165
+ "loss": 0.0996,
2166
+ "rewards/accuracies": 1.0,
2167
+ "rewards/chosen": -1.1021888256072998,
2168
+ "rewards/margins": 3.8253209590911865,
2169
+ "rewards/rejected": -4.927509784698486,
2170
+ "step": 144
2171
+ },
2172
+ {
2173
+ "epoch": 1.7515151515151515,
2174
+ "grad_norm": 3.1770360469818115,
2175
+ "learning_rate": 6.802721088435376e-07,
2176
+ "logits/chosen": -1.4982367753982544,
2177
+ "logits/rejected": -1.6536548137664795,
2178
+ "logps/chosen": -92.8003921508789,
2179
+ "logps/rejected": -129.96063232421875,
2180
+ "loss": 0.2177,
2181
+ "rewards/accuracies": 0.875,
2182
+ "rewards/chosen": -1.4197006225585938,
2183
+ "rewards/margins": 2.827155590057373,
2184
+ "rewards/rejected": -4.246856212615967,
2185
+ "step": 145
2186
+ },
2187
+ {
2188
+ "epoch": 1.7636363636363637,
2189
+ "grad_norm": 2.86728572845459,
2190
+ "learning_rate": 6.462585034013606e-07,
2191
+ "logits/chosen": -1.6697306632995605,
2192
+ "logits/rejected": -1.7117178440093994,
2193
+ "logps/chosen": -105.58975219726562,
2194
+ "logps/rejected": -140.0699005126953,
2195
+ "loss": 0.196,
2196
+ "rewards/accuracies": 1.0,
2197
+ "rewards/chosen": -2.083402156829834,
2198
+ "rewards/margins": 2.635946750640869,
2199
+ "rewards/rejected": -4.719348907470703,
2200
+ "step": 146
2201
+ },
2202
+ {
2203
+ "epoch": 1.7757575757575759,
2204
+ "grad_norm": 3.762331008911133,
2205
+ "learning_rate": 6.122448979591837e-07,
2206
+ "logits/chosen": -1.9138250350952148,
2207
+ "logits/rejected": -1.9125970602035522,
2208
+ "logps/chosen": -94.5777587890625,
2209
+ "logps/rejected": -125.44267272949219,
2210
+ "loss": 0.1805,
2211
+ "rewards/accuracies": 1.0,
2212
+ "rewards/chosen": -1.2487887144088745,
2213
+ "rewards/margins": 2.5635106563568115,
2214
+ "rewards/rejected": -3.8122997283935547,
2215
+ "step": 147
2216
+ },
2217
+ {
2218
+ "epoch": 1.7878787878787878,
2219
+ "grad_norm": 2.9627037048339844,
2220
+ "learning_rate": 5.782312925170068e-07,
2221
+ "logits/chosen": -1.6655495166778564,
2222
+ "logits/rejected": -1.669325351715088,
2223
+ "logps/chosen": -110.57474517822266,
2224
+ "logps/rejected": -125.62745666503906,
2225
+ "loss": 0.3147,
2226
+ "rewards/accuracies": 0.75,
2227
+ "rewards/chosen": -2.6180191040039062,
2228
+ "rewards/margins": 1.3960981369018555,
2229
+ "rewards/rejected": -4.014117240905762,
2230
+ "step": 148
2231
+ },
2232
+ {
2233
+ "epoch": 1.8,
2234
+ "grad_norm": 2.934206247329712,
2235
+ "learning_rate": 5.4421768707483e-07,
2236
+ "logits/chosen": -1.6896376609802246,
2237
+ "logits/rejected": -1.7967206239700317,
2238
+ "logps/chosen": -103.97714233398438,
2239
+ "logps/rejected": -150.354248046875,
2240
+ "loss": 0.1075,
2241
+ "rewards/accuracies": 1.0,
2242
+ "rewards/chosen": -2.2765142917633057,
2243
+ "rewards/margins": 3.3366103172302246,
2244
+ "rewards/rejected": -5.613123893737793,
2245
+ "step": 149
2246
+ },
2247
+ {
2248
+ "epoch": 1.812121212121212,
2249
+ "grad_norm": 2.11407208442688,
2250
+ "learning_rate": 5.102040816326531e-07,
2251
+ "logits/chosen": -1.774599552154541,
2252
+ "logits/rejected": -1.7382519245147705,
2253
+ "logps/chosen": -94.33448791503906,
2254
+ "logps/rejected": -130.95513916015625,
2255
+ "loss": 0.1393,
2256
+ "rewards/accuracies": 1.0,
2257
+ "rewards/chosen": -1.2320460081100464,
2258
+ "rewards/margins": 2.8650190830230713,
2259
+ "rewards/rejected": -4.097064971923828,
2260
+ "step": 150
2261
+ },
2262
+ {
2263
+ "epoch": 1.8242424242424242,
2264
+ "grad_norm": 1.7938166856765747,
2265
+ "learning_rate": 4.7619047619047623e-07,
2266
+ "logits/chosen": -1.6386427879333496,
2267
+ "logits/rejected": -1.734755277633667,
2268
+ "logps/chosen": -99.16139221191406,
2269
+ "logps/rejected": -138.26210021972656,
2270
+ "loss": 0.0658,
2271
+ "rewards/accuracies": 1.0,
2272
+ "rewards/chosen": -1.9007340669631958,
2273
+ "rewards/margins": 2.9712603092193604,
2274
+ "rewards/rejected": -4.871994972229004,
2275
+ "step": 151
2276
+ },
2277
+ {
2278
+ "epoch": 1.8363636363636364,
2279
+ "grad_norm": 2.793008804321289,
2280
+ "learning_rate": 4.421768707482994e-07,
2281
+ "logits/chosen": -1.838673710823059,
2282
+ "logits/rejected": -1.7541086673736572,
2283
+ "logps/chosen": -95.26689910888672,
2284
+ "logps/rejected": -133.28399658203125,
2285
+ "loss": 0.1334,
2286
+ "rewards/accuracies": 1.0,
2287
+ "rewards/chosen": -1.4139292240142822,
2288
+ "rewards/margins": 2.946502208709717,
2289
+ "rewards/rejected": -4.360431671142578,
2290
+ "step": 152
2291
+ },
2292
+ {
2293
+ "epoch": 1.8484848484848486,
2294
+ "grad_norm": 2.561596393585205,
2295
+ "learning_rate": 4.0816326530612243e-07,
2296
+ "logits/chosen": -1.7521519660949707,
2297
+ "logits/rejected": -1.7829943895339966,
2298
+ "logps/chosen": -95.3616714477539,
2299
+ "logps/rejected": -145.51622009277344,
2300
+ "loss": 0.0941,
2301
+ "rewards/accuracies": 1.0,
2302
+ "rewards/chosen": -1.6735217571258545,
2303
+ "rewards/margins": 3.370741605758667,
2304
+ "rewards/rejected": -5.04426383972168,
2305
+ "step": 153
2306
+ },
2307
+ {
2308
+ "epoch": 1.8606060606060606,
2309
+ "grad_norm": 5.654742240905762,
2310
+ "learning_rate": 3.7414965986394563e-07,
2311
+ "logits/chosen": -1.8284450769424438,
2312
+ "logits/rejected": -1.8849406242370605,
2313
+ "logps/chosen": -93.88221740722656,
2314
+ "logps/rejected": -123.63079833984375,
2315
+ "loss": 0.2585,
2316
+ "rewards/accuracies": 0.875,
2317
+ "rewards/chosen": -1.3463689088821411,
2318
+ "rewards/margins": 2.314800977706909,
2319
+ "rewards/rejected": -3.66117000579834,
2320
+ "step": 154
2321
+ },
2322
+ {
2323
+ "epoch": 1.8727272727272726,
2324
+ "grad_norm": 3.4941039085388184,
2325
+ "learning_rate": 3.401360544217688e-07,
2326
+ "logits/chosen": -1.8094770908355713,
2327
+ "logits/rejected": -1.8442440032958984,
2328
+ "logps/chosen": -102.26832580566406,
2329
+ "logps/rejected": -135.0662841796875,
2330
+ "loss": 0.2172,
2331
+ "rewards/accuracies": 0.875,
2332
+ "rewards/chosen": -1.8880977630615234,
2333
+ "rewards/margins": 2.399691343307495,
2334
+ "rewards/rejected": -4.287789344787598,
2335
+ "step": 155
2336
+ },
2337
+ {
2338
+ "epoch": 1.8848484848484848,
2339
+ "grad_norm": 7.033143043518066,
2340
+ "learning_rate": 3.0612244897959183e-07,
2341
+ "logits/chosen": -1.8748410940170288,
2342
+ "logits/rejected": -1.8568613529205322,
2343
+ "logps/chosen": -91.27564239501953,
2344
+ "logps/rejected": -110.9326171875,
2345
+ "loss": 0.5425,
2346
+ "rewards/accuracies": 0.75,
2347
+ "rewards/chosen": -1.5733274221420288,
2348
+ "rewards/margins": 1.3791325092315674,
2349
+ "rewards/rejected": -2.9524598121643066,
2350
+ "step": 156
2351
+ },
2352
+ {
2353
+ "epoch": 1.896969696969697,
2354
+ "grad_norm": 4.322800159454346,
2355
+ "learning_rate": 2.72108843537415e-07,
2356
+ "logits/chosen": -1.6863696575164795,
2357
+ "logits/rejected": -1.7218561172485352,
2358
+ "logps/chosen": -92.07785034179688,
2359
+ "logps/rejected": -128.896240234375,
2360
+ "loss": 0.197,
2361
+ "rewards/accuracies": 0.75,
2362
+ "rewards/chosen": -1.2870597839355469,
2363
+ "rewards/margins": 2.7534267902374268,
2364
+ "rewards/rejected": -4.040486812591553,
2365
+ "step": 157
2366
+ },
2367
+ {
2368
+ "epoch": 1.9090909090909092,
2369
+ "grad_norm": 5.76566219329834,
2370
+ "learning_rate": 2.3809523809523811e-07,
2371
+ "logits/chosen": -1.7858214378356934,
2372
+ "logits/rejected": -1.7693607807159424,
2373
+ "logps/chosen": -99.64199829101562,
2374
+ "logps/rejected": -123.77627563476562,
2375
+ "loss": 0.3982,
2376
+ "rewards/accuracies": 0.75,
2377
+ "rewards/chosen": -2.0218448638916016,
2378
+ "rewards/margins": 1.6399569511413574,
2379
+ "rewards/rejected": -3.66180157661438,
2380
+ "step": 158
2381
+ },
2382
+ {
2383
+ "epoch": 1.9212121212121214,
2384
+ "grad_norm": 5.00474739074707,
2385
+ "learning_rate": 2.0408163265306121e-07,
2386
+ "logits/chosen": -1.7164349555969238,
2387
+ "logits/rejected": -1.7813513278961182,
2388
+ "logps/chosen": -98.73097229003906,
2389
+ "logps/rejected": -117.31221008300781,
2390
+ "loss": 0.3695,
2391
+ "rewards/accuracies": 0.75,
2392
+ "rewards/chosen": -1.9404692649841309,
2393
+ "rewards/margins": 1.20289146900177,
2394
+ "rewards/rejected": -3.1433606147766113,
2395
+ "step": 159
2396
+ },
2397
+ {
2398
+ "epoch": 1.9333333333333333,
2399
+ "grad_norm": 3.1765851974487305,
2400
+ "learning_rate": 1.700680272108844e-07,
2401
+ "logits/chosen": -1.8153291940689087,
2402
+ "logits/rejected": -1.8343583345413208,
2403
+ "logps/chosen": -96.27183532714844,
2404
+ "logps/rejected": -123.16444396972656,
2405
+ "loss": 0.2216,
2406
+ "rewards/accuracies": 0.875,
2407
+ "rewards/chosen": -1.5373660326004028,
2408
+ "rewards/margins": 2.2644829750061035,
2409
+ "rewards/rejected": -3.801848888397217,
2410
+ "step": 160
2411
+ },
2412
+ {
2413
+ "epoch": 1.9454545454545453,
2414
+ "grad_norm": 2.6219356060028076,
2415
+ "learning_rate": 1.360544217687075e-07,
2416
+ "logits/chosen": -1.6747124195098877,
2417
+ "logits/rejected": -1.690348744392395,
2418
+ "logps/chosen": -104.59769439697266,
2419
+ "logps/rejected": -125.69312286376953,
2420
+ "loss": 0.2246,
2421
+ "rewards/accuracies": 1.0,
2422
+ "rewards/chosen": -1.8719542026519775,
2423
+ "rewards/margins": 1.6488584280014038,
2424
+ "rewards/rejected": -3.520812511444092,
2425
+ "step": 161
2426
+ },
2427
+ {
2428
+ "epoch": 1.9575757575757575,
2429
+ "grad_norm": 3.1105170249938965,
2430
+ "learning_rate": 1.0204081632653061e-07,
2431
+ "logits/chosen": -1.9918447732925415,
2432
+ "logits/rejected": -1.8305941820144653,
2433
+ "logps/chosen": -93.89464569091797,
2434
+ "logps/rejected": -129.2601318359375,
2435
+ "loss": 0.1517,
2436
+ "rewards/accuracies": 1.0,
2437
+ "rewards/chosen": -1.2167264223098755,
2438
+ "rewards/margins": 2.7442147731781006,
2439
+ "rewards/rejected": -3.9609413146972656,
2440
+ "step": 162
2441
+ },
2442
+ {
2443
+ "epoch": 1.9696969696969697,
2444
+ "grad_norm": 3.005218505859375,
2445
+ "learning_rate": 6.802721088435375e-08,
2446
+ "logits/chosen": -1.897176742553711,
2447
+ "logits/rejected": -1.8900220394134521,
2448
+ "logps/chosen": -94.99845123291016,
2449
+ "logps/rejected": -142.2918243408203,
2450
+ "loss": 0.2873,
2451
+ "rewards/accuracies": 0.75,
2452
+ "rewards/chosen": -1.604986310005188,
2453
+ "rewards/margins": 3.0710902214050293,
2454
+ "rewards/rejected": -4.676076412200928,
2455
+ "step": 163
2456
+ },
2457
+ {
2458
+ "epoch": 1.981818181818182,
2459
+ "grad_norm": 2.6735050678253174,
2460
+ "learning_rate": 3.4013605442176873e-08,
2461
+ "logits/chosen": -1.938112497329712,
2462
+ "logits/rejected": -1.9410051107406616,
2463
+ "logps/chosen": -95.70248413085938,
2464
+ "logps/rejected": -139.81271362304688,
2465
+ "loss": 0.1507,
2466
+ "rewards/accuracies": 1.0,
2467
+ "rewards/chosen": -1.3574395179748535,
2468
+ "rewards/margins": 3.3296563625335693,
2469
+ "rewards/rejected": -4.687096118927002,
2470
+ "step": 164
2471
+ }
2472
+ ],
2473
+ "logging_steps": 1,
2474
+ "max_steps": 164,
2475
+ "num_input_tokens_seen": 0,
2476
+ "num_train_epochs": 2,
2477
+ "save_steps": 1650,
2478
+ "stateful_callbacks": {
2479
+ "TrainerControl": {
2480
+ "args": {
2481
+ "should_epoch_stop": false,
2482
+ "should_evaluate": false,
2483
+ "should_log": false,
2484
+ "should_save": true,
2485
+ "should_training_stop": true
2486
+ },
2487
+ "attributes": {}
2488
+ }
2489
+ },
2490
+ "total_flos": 0.0,
2491
+ "train_batch_size": 2,
2492
+ "trial_name": null,
2493
+ "trial_params": null
2494
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:db5b7ada2a7891adec96cc5803dca9ed20b9d9121612730a5dee0a3041f63f4f
3
+ size 6264
vocab.json ADDED
The diff for this file is too large to render. See raw diff