inference

TensorRT C++ API Tutorial

Run TensorFlow models in C++ without installation and without Bazel

Example notebooks for working with SageMaker Studio Lab. Sign up for an account at the link below!

Small, dependency-free, fast Python package to infer binary file types checking the magic numbers signature

AI Productivity Tool - Free and open source, improve user productivity, and protect privacy and data security. Including but not limited to: built-in local exclusive ChatGPT, DeepSeek, Phi, Qwen and other models, one-click batch intelligent processing of pictures, videos, audio, etc.

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.

Machine Learning inference engine for Microcontrollers and Embedded devices

a lightweight LLM model inference framework

Blind WAF identification tool

yolort is a runtime stack for yolov5 on specialized accelerators such as tensorrt, libtorch, onnxruntime, tvm and ncnn.

Privacy Meter: An open-source library to audit data privacy in statistical and machine learning algorithms.

Continual learning infra for self-improving agents

Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, NetworkX, RAPIDS, RDFlib, pySHACL, PyVis, morph-kgc, pslpython, pyarrow, etc.

Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.

Train a state-of-the-art yolov3 object detector from scratch!

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Accurate, large-scale, and extensible simulator for LLM inference Systems

A lightweight, portable pure C99 onnx inference engine for embedded devices with hardware acceleration support.

High-performance serverless GPU task orchestration — the scheduling layer behind wavespeed.ai

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

Deep Learning Streamer (DL Streamer) Pipeline Framework is an open-source streaming media analytics framework, based on GStreamer* multimedia framework, for creating complex media analytics pipelines for the Cloud or at the Edge.

convert mmdetection model to tensorrt, support fp16, int8, batch input, dynamic shape etc.

Inference-native Tokenmaxxing Agent Harness for Loop Engineering

Python library for YOLO small object detection and instance segmentation

Run MoE models bigger than your RAM. Frontier-size MoE on a 12 GB phone, CPU only, lossless, on stock llama.cpp

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

Run your deep learning workloads on Kubernetes more easily and efficiently.

Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.

A project that optimizes OWL-ViT for real-time inference with NVIDIA TensorRT.

Confidential AI deployment with secure enclaves :lock: