lora-roberta-large-finetuned-reduced_captures

This model is a fine-tuned version of roberta-large on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2458
• Accuracy: 0.9345

Tests metrics:

• loss: 0.23798689246177673
• accuracy: 0.9321285694578563

Model description

Captures prediction based on LoRA-adapted RoBERTa-large. This includes labels 0-12 but excluding 9 (criminality).

Intended uses & limitations

Impero Safeguarding & Wellbeing

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 10
• eval_batch_size: 10
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 40
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.3924	0.9996	616	0.3862	0.8906
0.4385	1.9992	1232	0.3267	0.9048
0.388	2.9988	1848	0.2849	0.9175
0.3256	4.0	2465	0.2728	0.9207
0.2718	4.9996	3081	0.2939	0.9170
0.2877	5.9992	3697	0.2522	0.9267
0.233	6.9988	4313	0.2624	0.9260
0.1832	8.0	4930	0.2512	0.9317
0.2399	8.9996	5546	0.2458	0.9345
0.1506	9.9959	6160	0.2390	0.9336

Framework versions

• PEFT 0.12.0
• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.0