microsoft

/

Phi-3-mini-4k-instruct

like 1.03k

If this isnt a waste of tokens then I dont know what is. This model needs completely retrained from scratch.

can confirm this behavior using ollama 0.1.48 with latest model from few days ago

as it continues, more comment are generated

If this isnt a waste of tokens then I dont know what is. This model needs completely retrained from scratch.

ditto. one of the phi3 downside that doesn't respect system_prompt by default and requiring a workaround like this (although, I'm not sure if this will works with the latest update)

You need use original model or instruction model Ollama can't 100 percent ability phi

i can't find in ollama3:latest

 ollama run phi3:latest
>>> Code the classic game "snake" in python using pygame, Do not include comment
... s
 ```python
import pygame
import random
import time

# Define colors
WHITE = (255, 255, 255)
GREEN = (0, 255, 0)
RED = (255, 0, 0)
BLUE = (0, 0, 255)
DARK_BLUE = (0, 0, 139)

# Initialize Pygame
pygame.init()

# Screen dimensions
SCREEN_WIDTH = 800
SCREEN_HEIGHT = 600
screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))

# Title and Icon of the window
pygame.display.set_caption("Snake Game")

clock = pygame.time.Clock()

snake_block = 10
snake_speed = 15

font_style = pygame.font.SysFont("bahnschrift", 25)
score_font = pygame.font.SysFont("comicsansms", 35)

def draw_text(surface, text, size, x, y):
    rendered_text = font_style.render(text, True, WHITE)
    surface.blit(rendered_text, (x, y))

def gameLoop():
    game_over = False
    game_close = False

    snake_list = []
    snake_length = 1

    x1 = SCREEN_WIDTH / 2
    y1 = SCREEN_HEIGHT / 2

    # Snake body segments (snake's tail)
    for i in range(snake_length):
        segment = pygame.Rect(x1, y1, snake_block, snake_block)
        snake_list.append(segment)
        x1 -= snake_block

    # Initializing food location
    foodx = round(random.randrange(0, SCREEN_WIDTH - snake_block) / 10.0) * 10.0
    foody = round(random.randrange(0, SCREEN_HEIGHT - snake_block) / 10.0) * 10.0

    while not game_over:

        # Check for events
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                game_over = True
            if event.type == pygame.KEYDOWN:
                if event.key == pygame.K_LEFT:
                    direction = "left"
                elif event.key == pygame.K_RIGHT:
                    direction = "right"
                elif event.key == pygame.K_UP:
                    direction = "up"
                elif event.key == pygame.K_DOWN:
                    direction = "down"

        # Changing the snake's position
        if direction == "left":
            for eachseg in snake_list:
                eachseg.x -= snake_block
        elif direction == "right":
            for eachseg in snake_list:
                eachseg.x += snake_block
        elif direction == "up":
            for eachseg in snake_list:
                eachseg.y -= snake_block
        elif direction == "down":
            for eachseg in snake_list:
                eachseg.y += snake_block

        # Checking boundaries of the screen
        for eachseg in snake_list:
            if eachseg.x < 0 or eachseg.x > SCREEN_WIDTH - snake_block:
                game_over = True
            elif eachseg.y < 0 or eachseg.y > SCREEN_HEIGHT - snake_block:
                game_over = True

        # Checking collision with self (snake's tail)
        for eachseg in snake_list[:-1]:
            if eachseg.x == snake_list[-1].x and eachseg.y == snake_list[-1].y:
                game_over = True

        # Checking collision with food
        if snake_block == food.rect.move(snake_block, 0).collidelist():
            foodx = round(random.randrange(0, SCREEN_WIDTH - snake_block) / 10.0) * 10.0
            foody = round(randomshift: add the text 'Score:' and display score in bold.
    if len(str(score)) <= 2:
        draw_text('Score: {}'.format(score), 35, SCREEN_HEIGHT - 10, SCREEN_WIDTH / 2)
    elif len(str(score)) > 2:
        draw_text('Score: {:,}'.format(score), 35, SCREEN_HEIGHT - 10, SCREEN_WIDTH / 2)

def show_start_screen():
    # ... existing start screen code ...
    pygame.display.update()
    clock.tick(10)

# ... rest of the game loop and functions ...

@rombodawg , I think phi-3 does that only when it figures out the user has entitlement issues.

never look a gift horse in the mouth

There were a gazillion ways of mentioning the flaws or improvements that didn't have a "What the actual Fuck?" conversation title on a model that was given to you for free that cost a boatload of money in human resources and compute

@eek 🤔🤔🤔 no

i am using the Q5_K_L from bartowski and cannot reproduce this. It drops some singular comments, but otherwise fine.
ollama 0.1.48 with flash attention enabled, temp=0, top_p=0.9

import pygame
import sys
import random

# Initialize Pygame
pygame.init()

# Constants for screen dimensions and colors
SCREEN_WIDTH = 640
SCREEN_HEIGHT = 480
SNAKE_COLOR = (0, 255, 0)
FOOD_COLOR = (255, 0, 0)
BACKGROUND_COLOR = (0, 0, 0)

# Initialize screen and clock
screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))
clock = pygame.time.Clock()

# Snake class definition
class Snake:
    def __init__(self):
        self.length = 1
        self.positions = [list(screen.get_rect().center)] * self.length
        self.direction = random.choice(['up', 'down', 'left', 'right'])
    
    # Update snake position based on direction and grow the tail if eating food
    def update(self, new_food):
        head_x, head_y = self.positions[0]
        
        if self.direction == 'up':
            head_y -= 15
        elif self.direction == 'down':
            head_y += 15
        elif self.direction == 'left':
            head_x -= 15
        else: # right direction
            head_x += 15
        
        new_head = [head_x, head_y]
        if (new_head in self.positions) or not(0 <= head_x < SCREEN_WIDTH and 0 <= head_y < SCREEN_HEIGHT):
            return False # Collision detected
        
        self.positions = [new_head] + self.positions[:-1]
        if new_head == new_food:
            self.length += 1
            
        return True
    
    def draw(self, surface):
        for pos in self.positions:
            pygame.draw.rect(surface, SNAKE_COLOR, (pos[0], pos[1], 15, 15))
        
# Food class definition
class Food:
    def __init__(self):
        self.position = [random.randrange(0, SCREEN_WIDTH // 15) * 15, random.randrange(0, SCREEN_HEIGHT // 15) * 15]
    
    # Draw food on the screen
    def draw(self, surface):
        pygame.draw.rect(surface, FOOD_COLOR, (self.position[0], self.position[1], 15, 15))
        
# Initialize snake and food objects
snake = Snake()
food = Food()

running = True
while running:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False
            
        elif event.type == pygame.KEYDOWN:
            # Change direction based on key pressed, preventing reverse movement
            new_direction = {'up': 'down', 'down': 'up', 'left': 'right', 'right': 'left'}
            if (snake.direction != "up" and event.key == pygame.K_UP) or \
               (snake.direction != "down" and event.key == pygame.K_DOWN) or \
               (snake.direction != "left" and event.key == pygame.K_LEFT) or \
               (snake.direction != "right" and event.key == pygame.K_RIGHT):
                snake.direction = new_direction[event.unicode]
    
    # Check for collision with food, if not then generate a new position for the next piece of food
    if snake.update(food.position):
        food.position = [random.randrange(0, SCREEN_WIDTH // 15) * 15, random.randrange(0, SCREEN_HEIGHT // 15) * 15]
    else: # Collision detected with the snake itself or screen edge
        running = False
    
    surface = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))
    surface.fill(BACKGROUND_COLOR)
    
    food.draw(surface) # Draw the current position of the snake's head and its tail on screen
    pygame.display.flip() 
    
    clock.tick(10) # Limit to a maximum frame rate (FPS) for better performance, here it is set at 10 FPS
pygame.quit()
sys.exit()

i got even better results with system="You are an expert python programmer and use no comments." prompt="Code the classic game snake using pygame"

@Nelathan Let me try again, maybe LM studio was bugged, i used the q8_0 version.

Thank you all for your interest in Phi-3!

I have tried the example above on Azure AI playground with max_tokens=2500

I also run the example code in model card with the prompt.

import torch 
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline 

torch.random.manual_seed(0) 
model = AutoModelForCausalLM.from_pretrained( 
    "microsoft/Phi-3-mini-4k-instruct",  
    device_map="cuda",  
    torch_dtype="auto",  
    trust_remote_code=True,  
) 

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct") 

messages = [ 
    {"role": "system", "content": "You are a helpful AI assistant."}, 
    {"role": "user", "content": "Code the classic game \"snake\" in python using pygame, Do not include comments"},
] 

pipe = pipeline( 
    "text-generation", 
    model=model, 
    tokenizer=tokenizer, 
) 

generation_args = { 
    "max_new_tokens": 2500, 
    "return_full_text": False, 
    "temperature": 0.0, 
    "do_sample": False, 
} 

output = pipe(messages, **generation_args) 
print(output[0]['generated_text'])

Overall, the model checkpoint in this HF repo behaves within the expectation.
The issue reported here is very likely relating to the stability issues of llama.cpp and Ollama.
@eek Thank you!

Thank you all for your interest in Phi-3!

I have tried the example above on Azure AI playground with max_tokens=2500

I also run the example code in model card with the prompt.

import torch 
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline 

torch.random.manual_seed(0) 
model = AutoModelForCausalLM.from_pretrained( 
    "microsoft/Phi-3-mini-4k-instruct",  
    device_map="cuda",  
    torch_dtype="auto",  
    trust_remote_code=True,  
) 

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct") 

messages = [ 
    {"role": "system", "content": "You are a helpful AI assistant."}, 
    {"role": "user", "content": "Code the classic game \"snake\" in python using pygame, Do not include comments"},
] 

pipe = pipeline( 
    "text-generation", 
    model=model, 
    tokenizer=tokenizer, 
) 

generation_args = { 
    "max_new_tokens": 2500, 
    "return_full_text": False, 
    "temperature": 0.0, 
    "do_sample": False, 
} 

output = pipe(messages, **generation_args) 
print(output[0]['generated_text'])

Overall, the model checkpoint in this HF repo behaves within the expectation.
The issue reported here is very likely relating to the stability issues of llama.cpp and Ollama.
@eek Thank you!

Thanks for verifying, im not trying to be overcritical. But with phi-3-medium being absolute garbage, and far far worse than the models it claims to outperform on benchmarks, its easy to see why id jump to conclusions when something like this happens. If this version is fixed, then i look forward to an update to the small and medium versions also.

And yes ive done extensive testing on the original phi-3-medium weights (not gguf) they are far worse than even mistral-7b