emessy/Flash_Fiction-FineLlama-3.1-8B

Uploaded model

• Developed by: emessy
• License: apache-2.0
• Finetuned from model : unsloth/Meta-Llama-3.1-8B-bnb-4bit

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.

Configure LoRA

lora_config = LoraConfig( r=16, lora_alpha=16, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM" )

Training arguments

training_args = TrainingArguments( output_dir="./results", num_train_epochs=5, per_device_train_batch_size=4, gradient_accumulation_steps=4, learning_rate=2e-4, fp16=True, # Use half-precision logging_steps=10, save_steps=50, eval_steps=50, )