quantization

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Faster Whisper transcription with CTranslate2

中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)

[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant

A vector index built on TurboQuant, written in Rust with Python bindings

Accessible large language models via k-bit quantization for PyTorch.

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

Lossy PNG compressor — pngquant command based on libimagequant library

Fast inference engine for Transformer models

[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

PyTorch native quantization for training and inference

ComfyUI Plugin of Nunchaku

Your Cheat Sheet for AI Engineering Interview – Questions and Answers.

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

Build, personalize and control your own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6

A list of papers, docs, codes about model quantization. This repo is aimed to provide the info for model quantization research, we are continuously improving the project. Welcome to PR the works (papers, repositories) that are missed by the repo.

Mastering Applied AI, One Concept at a Time

Run Mixtral-8x7B models in Colab or consumer desktops

micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape

Run a 1-billion parameter LLM on a $10 board with 256MB RAM

PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.

OpenMMLab Model Compression Toolbox and Benchmark.

PaddleSlim is an open-source library for deep model compression and architecture search.

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

A SOTA quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的量化工具包