inference

Manages Unified Access to Generative AI Services built on Envoy Gateway

Run a 1-billion parameter LLM on a $10 board with 256MB RAM

The inference module for AgiBot X1.

Acceleration package for neural networks on multi-core CPUs

Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀

A high-performance inference engine for AI models

Open Source Continuous Inference Benchmark Research Platform — Kimi K3 2.8T, MiniMax M3, DeepSeekv4, GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72 & soon™ TPUv6e/v7/Trainium2/3 | 开源持续推理基准研究平台 — Kimi K2.7-Code、MiniMax M3、DeepSeekv4、GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72，即将推出™ TPUv6e/v7/Trainium2/3

Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.

Nvidia GPU exporter for prometheus using nvidia-smi binary OR using NVML

Deploy a ML inference service on a budget in less than 10 lines of code.

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀

Package for causal inference in graphs and in the pairwise settings. Tools for graph structure recovery and dependencies are included.

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

LLM KV cache compression made easy

Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Includes MLX Core macOS app with chat, agent mode, and tool calling.

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

A throughput-oriented high-performance serving framework for LLMs

MLX Studio - Home of JANG_Q - Image Gen/Edit + Chat/Code All in one - + OpenClaw (Anthropic API)

Bolt is a deep learning library with high performance and heterogeneous flexibility.

📚 Introduction to Modern Statistics - A college-level open-source textbook with a modern approach highlighting multivariable relationships and simulation-based inference.

A uniform interface to run deep learning models from multiple frameworks

A scalable inference server for models optimized with OpenVINO™

Suno AI's Bark model in C/C++ for fast text-to-speech generation

An open-source computer vision framework to build and deploy apps in minutes

PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.