Yuan-Li-FNLP/R3-RAG-CS-Llama

llama

This model is a fine-tuned version of /remote-home1/yli/Model/Generator/Llama3_1_hf/8B/base on the 2wikimultihopqa_train, the hotpotqa_train and the musique_train datasets. It achieves the following results on the evaluation set:

• Loss: 0.1818

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.2053	0.1997	500	0.1904
0.2035	0.3994	1000	0.1856
0.1931	0.5990	1500	0.1751
0.187	0.7987	2000	0.1665
0.1916	0.9984	2500	0.1609
0.1085	1.1981	3000	0.1631
0.1153	1.3978	3500	0.1600
0.1205	1.5974	4000	0.1545
0.102	1.7971	4500	0.1496
0.0737	1.9968	5000	0.1455
0.0261	2.1965	5500	0.1799
0.0349	2.3962	6000	0.1782
0.0269	2.5958	6500	0.1810
0.0296	2.7955	7000	0.1819
0.0258	2.9952	7500	0.1818

Framework versions

• Transformers 4.46.1
• Pytorch 2.4.0
• Datasets 3.1.0
• Tokenizers 0.20.3