Add fast tokenizer

#9
by Narsil HF staff - opened
Files changed (3) hide show
  1. NOTICE +0 -24
  2. README.md +10 -11
  3. tokenizer.json +0 -0
NOTICE DELETED
@@ -1,24 +0,0 @@
1
- Copyright 2023-2024 01.AI
2
-
3
- This work, [Your Model Name], is based on the work originally authored by 01.AI.
4
-
5
- Licensed under the Apache License, Version 2.0 (the "License");
6
- you may not use this file except in compliance with the License.
7
- You may obtain a copy of the License at
8
-
9
- http://www.apache.org/licenses/LICENSE-2.0
10
-
11
- Unless required by applicable law or agreed to in writing, software
12
- distributed under the License is distributed on an "AS IS" BASIS,
13
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
- See the License for the specific language governing permissions and
15
- limitations under the License.
16
-
17
- Attribution
18
-
19
- If you create derivative works based on this model, please include the following attribution in your derivative works:
20
-
21
- This work is a derivative of [The Yi-1.5 Series Model You Base On] by 01.AI, used under the Apache 2.0 License.
22
-
23
-
24
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
README.md CHANGED
@@ -16,7 +16,6 @@ license: apache-2.0
16
  <a href="https://github.com/01-ai/Yi-1.5/issues/2">πŸ’¬ WeChat</a>
17
  <br/>
18
  <a href="https://arxiv.org/abs/2403.04652">πŸ“ Paper</a> β€’
19
- <a href="https://01-ai.github.io/">πŸ’ͺ Tech Blog</a> β€’
20
  <a href="https://github.com/01-ai/Yi/tree/main?tab=readme-ov-file#faq">πŸ™Œ FAQ</a> β€’
21
  <a href="https://github.com/01-ai/Yi/tree/main?tab=readme-ov-file#learning-hub">πŸ“— Learning Hub</a>
22
  </p>
@@ -43,11 +42,11 @@ Model | Context Length | Pre-trained Tokens
43
 
44
  | Name | Download |
45
  | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
46
- | Yi-1.5-34B-Chat | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI)|
47
- | Yi-1.5-34B-Chat-16K | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
48
- | Yi-1.5-9B-Chat | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
49
- | Yi-1.5-9B-Chat-16K | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
50
- | Yi-1.5-6B-Chat | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
51
 
52
  </div>
53
 
@@ -57,11 +56,11 @@ Model | Context Length | Pre-trained Tokens
57
 
58
  | Name | Download |
59
  | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
60
- | Yi-1.5-34B | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
61
- | Yi-1.5-34B-32K | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
62
- | Yi-1.5-9B | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
63
- | Yi-1.5-9B-32K | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
64
- | Yi-1.5-6B | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) β€’ [🟣 wisemodel](https://wisemodel.cn/organization/01.AI) |
65
 
66
  </div>
67
 
 
16
  <a href="https://github.com/01-ai/Yi-1.5/issues/2">πŸ’¬ WeChat</a>
17
  <br/>
18
  <a href="https://arxiv.org/abs/2403.04652">πŸ“ Paper</a> β€’
 
19
  <a href="https://github.com/01-ai/Yi/tree/main?tab=readme-ov-file#faq">πŸ™Œ FAQ</a> β€’
20
  <a href="https://github.com/01-ai/Yi/tree/main?tab=readme-ov-file#learning-hub">πŸ“— Learning Hub</a>
21
  </p>
 
42
 
43
  | Name | Download |
44
  | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
45
+ | Yi-1.5-34B-Chat | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
46
+ | Yi-1.5-34B-Chat-16K | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
47
+ | Yi-1.5-9B-Chat | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
48
+ | Yi-1.5-9B-Chat-16K | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
49
+ | Yi-1.5-6B-Chat | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
50
 
51
  </div>
52
 
 
56
 
57
  | Name | Download |
58
  | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
59
+ | Yi-1.5-34B | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
60
+ | Yi-1.5-34B-32K | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
61
+ | Yi-1.5-9B | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
62
+ | Yi-1.5-9B-32K | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
63
+ | Yi-1.5-6B | β€’ [πŸ€— Hugging Face](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) β€’ [πŸ€– ModelScope](https://www.modelscope.cn/organization/01ai) |
64
 
65
  </div>
66
 
tokenizer.json DELETED
The diff for this file is too large to render. See raw diff