phi3_finetunev2

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.1320

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 4
• seed: 3407
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 5
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss
0.2859	2.6667	40	0.5632
0.2014	5.3333	80	0.6352
0.1367	8.0	120	0.6947
0.0955	10.6667	160	0.7761
0.0756	13.3333	200	0.8871
0.0666	16.0	240	0.8893
0.0609	18.6667	280	0.9542
0.0584	21.3333	320	0.9722
0.0565	24.0	360	1.0021
0.0548	26.6667	400	1.0579
0.0543	29.3333	440	1.0642
0.0542	32.0	480	1.0696
0.0532	34.6667	520	1.0861
0.0526	37.3333	560	1.0901
0.0522	40.0	600	1.1002
0.0517	42.6667	640	1.1150
0.0512	45.3333	680	1.1166
0.0509	48.0	720	1.1320

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.3.0+cu121
• Datasets 2.16.0
• Tokenizers 0.19.1