CodeGPT on XTC

CodeGPT on XTC: Compressing a CodeGPT Model Using Hybrid Layer Reduction and Extreme Quantisation through Knowledge Distillation

de Moor, Aral (TU Delft Electrical Engineering, Mathematics and Computer Science)

Izadi, M. (mentor)
Al-Kaswan, A. (mentor)
van Deursen, A. (mentor)
Anand, A. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2023-06-27

Large language models are powerful because of their state-of-the-art language processing abilities. But, they come at the cost of being extremely resource-intensive, and are steadily growing in size. As a result, compressing such models for resource- constrained devices is an active and promising re- search area. In spite of their current popular- ity, many novel compression techniques lack im- plementation for GPT models. We apply the XTC pipeline, consisting of layer-reduction and quantisation through knowledge distillation, to a CodeGPT generative model. The resulting mod- els are evaluated on the CodeXGLUE line-level code-completion benchmark. Based on this, we demonstrate that (1) XTC can be adapted to GPT- like models, translating many of the findings of the original study; (2) a 6-layer reduction with 1-bit weight and 8-bit activation quantisation is able to reduce model size 15×, in addition to almost dou- bling inference speed, with minimal performance degradation. The resulting compressed models show promise for use in local code generation. By showing that a novel compression technique can be adapted to GPT-like models, we hope to further in- spire research in this field.

Large Language Models
Compression
Knowledge Distillation
Layer Reduction
Quantisation
GPT

http://resolver.tudelft.nl/uuid:f37924fc-ecac-4bd4-b923-7d4c73f74a72

https://github.com/AISE-TUDelft/LLM4CodeCompression Replication Code Repository

Student theses

bachelor thesis

© 2023 Aral de Moor