evaluation

Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!

Avalanche: an End-to-End Library for Continual Learning based on PyTorch.

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.

(IROS 2020, ECCVW 2020) Official Python Implementation for "3D Multi-Object Tracking: A Baseline and New Evaluation Metrics"

WFGY is heading toward WFGY 5.0 Polaris Protocol, a major open-source release for AI reasoning, RAG, agents, and real-world workflows. Includes Problem Map, Global Debug Card, WFGY 4.0, and the CFV Easter Egg.

心理健康大模型 (LLM x Mental Health), Pre & Post-training & Dataset & Evaluation & Depoly & RAG, with InternLM / Qwen / Baichuan / DeepSeek / Mixtral / LLama / GLM series models

A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.

Building blocks for rapid development of GenAI applications

The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

Multi-class confusion matrix library in Python

Evaluation code for various unsupervised automated metrics for Natural Language Generation.

Short and sweet LISP editing

XAI - An eXplainability toolbox for machine learning

A framework for comprehensive diagnosis and optimization of agents using simulated, realistic synthetic interactions

KernelBench: Can LLMs Write GPU Kernels? - Benchmark + Toolkit with Torch -> CUDA (+ more DSLs)

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

FuzzBench - Fuzzer benchmarking as a service.

High-fidelity performance metrics for generative models in PyTorch

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

NCalc is a fast and lightweight expression evaluator library for .NET, designed for flexibility and high performance. It supports a wide range of mathematical and logical operations.

Evaluate and improve models and agents using environments

Evaluate your LLM's response with Prometheus and GPT4 💯

LangSmith Client SDK Implementations

SemanticKITTI API for visualizing dataset, processing data, and evaluating results.

Generate High-Quality Synthetics, Train, Measure, and Evaluate in a Single Pipeline

ClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.

OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards

Expression evaluation in golang

Web Codegen Scorer is a tool for evaluating the quality of web code generated by LLMs.