AI Quantization Laboratory
An offline-capable browser laboratory for learning how number formats and quantization work, from individual floating-point bits to quantized matrix math and the predictions of small neural networks.
About this project
At around this time, I was starting to explore tools that enabled me to run LLMs locally on my own hardware as opposed to running them in the cloud. As part of this endeavor, I was often faced with choosing between models at different quantization levels. There was a balancing act: the more heavily a model was quantized, the smaller it became (which would allow me to run a larger model on more moderate hardware with a smaller memory footprint), but the less accurate it became.
I had utilized quantization before, primarily as a means for compressing large data files. While I had an understanding of the basics, I really desired to learn more about the underlying mathematics that described the type of quantization that existed within LLMs. With that goal, the AI Quantization Laboratory was created as a means to learn this subject. Not only was this project effective at increasing my own understanding of quantization for LLMs, but it also deepened my appreciation of software-composing AI agents as tools for fun, tailored, and interactive learning.
Screenshots
(view full size)
(view full size)
(view full size)