README.md

metadata

license: other
base_model: microsoft/phi-2
tags:
  - generated_from_trainer
  - medical
  - peft
  - 'lora '
model-index:
  - name: Thealth-phi-2-tunned-9_medalpaca_medical_meadow
    results: []
datasets:
  - medalpaca/medical_meadow_mediqa
  - medalpaca/medical_meadow_mmmlu
  - medalpaca/medical_meadow_medical_flashcards
  - medalpaca/medical_meadow_wikidoc_patient_information
  - medalpaca/medical_meadow_wikidoc
  - medalpaca/medical_meadow_pubmed_causal
  - medalpaca/medical_meadow_medqa
  - medalpaca/medical_meadow_health_advice
  - medalpaca/medical_meadow_cord19
pipeline_tag: conversational

Thealth-phi-2-tunned-9_medalpaca_medical_meadow

This model is a fine-tuned version of microsoft/phi-2 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 6.6588

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

Training is done one 9 medalpaca/medical_meadow datasets combined and splited to 90% train and 10% Evaluation

Dataset
medalpaca/medical_meadow_mediqa
medalpaca/medical_meadow_mmmlu
medalpaca/medical_meadow_medical_flashcards
medalpaca/medical_meadow_wikidoc_patient_information
medalpaca/medical_meadow_wikidoc
medalpaca/medical_meadow_pubmed_causal
medalpaca/medical_meadow_medqa
medalpaca/medical_meadow_health_advice
medalpaca/medical_meadow_cord19

Training procedure

Used different tokenizer stanford-crfm/BioMedLM

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss
6.8245	0.0	500	6.7654
6.7944	0.0	1000	6.6588

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("stanford-crfm/BioMedLM", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("TachyHealthResearch/Thealth-phi-2-tunned-9_medalpaca_medical_meadow", trust_remote_code=True, torch_dtype=torch.float32)

inputs = tokenizer(
    """
    question: ****** ? answer:
    """,
    return_tensors="pt",
    return_attention_mask=False)

outputs = model.generate(**inputs, max_length=512)
text = tokenizer.batch_decode(outputs)[0]
print(text)

Framework versions

• Transformers 4.35.2
• Pytorch 2.1.0+cu121
• Datasets 2.15.0
• Tokenizers 0.15.0