quantization

INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

Brevitas: neural network quantization in PyTorch

Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel XPU (CPU-only and CUDA also supported).

Efficient computing methods developed by Huawei Noah's Ark Lab

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

Neural Network Compression Framework for enhanced OpenVINO™ inference

Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

Z80-μLM is a 2-bit quantized language model small enough to run on an 8-bit Z80 processor. Train conversational models in Python, export them as CP/M .COM binaries, and chat with your vintage computer.

A pytorch quantization backend for optimum

Dataflow compiler for QNN inference on FPGAs

[ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

TinyChatEngine: On-Device LLM Inference Library

Official implementation of Half-Quadratic Quantization (HQQ)

[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning; [NeurIPS 2022] MCUNetV3: On-Device Training Under 256KB Memory

Palette quantization library that powers pngquant and other PNG optimizers

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.

List of papers related to neural network quantization in recent AI conferences and journals.

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

[ICML 2024] SqueezeLLM: Dense-and-Sparse Quantization

The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment

Welcome to the official repository of SINQ! A novel, fast and high-quality quantization method designed to make any Large Language Model smaller while preserving accuracy [ICML 2026]

RDNA-native LLM inference engine in Rust.

Offline semantic Text-to-Image and Image-to-Image search on Android powered by quantized state-of-the-art vision-language pretrained CLIP model and ONNX Runtime inference engine

QKeras: a quantization deep learning library for Tensorflow Keras

Tool for onnx->keras or onnx->tflite. Hope this tool can help you.

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.