LLaMa 3.1 can't do the killers problem
Not sure what's going on here. But in my tests, it's giving worse ouputs than LLaMa 3 8B. Laughable results:
(It should be noted that I'm using the newest ollama version (0.2.8) and newest open-webui (0.3.10)
Might have something to do with this... https://github.com/ggerganov/llama.cpp/issues/8650 The wrong pretokenizer is being used. Smaug-BPE vs LLAMA-BPE. 🤔
@bartowski Are you aware of this issue with converting LLaMa-3.1?
I'm aware but not sure what the appropriate fix is, I imagine that setting it manually to llama 3 may work but is not the best solution since it should be figured out why it's detecting Smaug
The 8B model is terrible for all reasoning and math. It didn't solve any reasoning questions I presented. Perhaps it needs a good fine tuning yet. I see potential, but just now, it is good only for chatting. 👍🙏💥
https://github.com/ggerganov/llama.cpp/issues/8650
@Dihelson Way to not read our comments... 👆 The wrong pretokenizer is being used. Smaug-BPE vs LLAMA-BPE. They are working on fixing the issue in llama.cpp.
https://github.com/ggerganov/llama.cpp/issues/8650
@Dihelson Way to not read our comments... 👆 The wrong pretokenizer is being used. Smaug-BPE vs LLAMA-BPE. They are working on fixing the issue in llama.cpp.
It somehow still gets it wrong in lmsys arena
I believe they use full weight models in lmsys?
https://github.com/ggerganov/llama.cpp/issues/8650
@Dihelson Way to not read our comments... 👆 The wrong pretokenizer is being used. Smaug-BPE vs LLAMA-BPE. They are working on fixing the issue in llama.cpp.
This problem is happening also in Groq llama 3.1 8B in Ollama. Is it the pretokenizen also ??? Same thing happening there. Even the 70B model couldn't solve most of smart questions. The only model which solved the triple age problem was the 405B parameters. The problem is the following:
"We're in 2024. I am 80 years old. My daughter was born in 1966. When was the year I had the triple of her age ? "
( The correct answer is 1977, when I had 33 and the daughter, 11 ).
🙏💥👍
@Dihelson Several files have been updated over the last several hours. I suggest downloading them and reconverting LLaMa-3.1-8B-Instruct and quantizing again. 🤔
"We're in 2024. I am 80 years old. My daughter was born in 1966. When was the year I had the triple of her age ? "
( The correct answer is 1977, when I had 33 and the daughter, 11 ).
🙏💥👍
It's so strange that even phi-3-small solves this but many other models don't. And btw, phi3small also answers question about killers.
Yep, same gemma7b solve this but newer and 'better' ones, no.. Llm magic I guess.
Waiting few days to run LLAMA3. 1
We're in 2024. I am 80 years old. My daughter was born in 1966. When was the year I had the triple of her age ?
When your daughter was 7 years old (i.e., in the year 1973), your age would have been 33 years. 3 times 7 is 21. 1973 - 21 = 1952.
Answer: The year was 1952.