Papers
arxiv:2401.15024

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

Published on Jan 26
ยท Submitted by akhaliq on Jan 29
#1 Paper of the day

Abstract

Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression

Community

Not APril FOols, Rad!

Code is 404

Paper author
โ€ข
edited Jan 29

Code is 404

it's there now!

On the calibration set - it looks like this is running a bunch of samples of Wiki2 and Alpaca truncated to the maximum length of the LLM.

I'm guessing the intuition here is to have a representative set of samples that's either seen during pretraining or are likely to be encountered in day-to-day use.

Have you guys explored the idea of using this for feature activation steering? E.g. prime the model (maybe even using contrastive pairs to get positive and negatives) using calibration prompts that steer behavior to, for e.g., avoid sycophancy (vs emulates typical interactions).

Admittedly, it's really heavy for what can be done, but it'd be interesting to see if the PCA picks up anything interesting (e.g. do they decompose into basic feature directions? can they be interpreted? can they be manipulated to amplify/suppress certain behaviors? do these components become sparser as steering becomes more directed?)

Paper author

Hi @leegao19 ,

Your intuition is correct. I think about it as trying to perturb all the directions in the model that are going to be important. I've been pleasantly surprised by how few samples are needed to get a representative set. Just 2k samples from Alpaca seems to be enough to preserve llm-eval metrics.

We have not experimented with steering! Feel free to check out the code and pull out the Principal components to see what they each do. I'd be interested to hear what happens.

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2401.15024 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2401.15024 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2401.15024 in a Space README.md to link it from this page.

Collections including this paper 32