ValueError: Tokenizer class YiTokenizer does not exist or is not currently imported.
#1
by
Starlento
- opened
Sorry that maybe you are not the right person to answer this question.
I tried transformers-4.36.0.dev0 and transformers==4.34.0 in Yi's repo, they both show the error in my title.
I just wonder whether it is an issue from your side (I guest it is not)... But I do not find the same issue in Yi's repo and currently not able to test the original ones.
I found the reason...
the right code should be tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True) for loading the tokenizer.
Could you kindly modify your README as it is tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True) which cannot import from tokenization_yi.py.
Sorry to bother again.
@TheBloke
Starlento
changed discussion status to
closed