metadata
library_name: transformers
How to run it
setup enviroment
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121
pip install fbgemm-gpu==0.8.0rc4
Download the enablement fork, https://huggingface.co/sllhf/transformers_enablement_fork/tree/main
unzip the file
cd transformers
# add changes from this PR https://github.com/huggingface/transformers/pull/32047
git fetch origin pull/32047/head:new-quant-method
git merge new-quant-method
pip install -e .
git clone https://github.com/huggingface/accelerate.git
cd accelerate
pip install -e .
# Next install vLLM PR https://github.com/vllm-project/vllm/pull/6559
git clone https://github.com/vllm-project/vllm.git
cd vllm
git fetch origin pull/6559/head:fbgemm-checkpoints
git checkout fbgemm-checkpoints
pip install -e .
Load back the HF model
from transformers import FbgemmFp8Config, AutoModelForCausalLM, AutoTokenizer
model_name = "sllhf/Meta-Llama-3.1-405B-Instruct-FP8"
quantization_config = FbgemmFp8Config()
quantized_model = AutoModelForCausalLM.from_pretrained(
model_name, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
# make sure to set up your own params, temperature, top_p etc.
output = quantized_model.generate(**input_ids, max_new_tokens=10)
print(tokenizer.decode(output[0], skip_special_tokens=True))
How to run in it with vLLM
from vllm import LLM
model = LLM("sllhf/Meta-Llama-3.1-405B-Instruct-FP8")