#Evaluation
Showing 60 of 166 repositories tagged #evaluation, ranked by stars
๐ชข Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. ๐YC W23
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
Supercharge Your LLM Application Evaluations ๐
Easily fine-tune, evaluate and deploy Gemma 4, Qwen3.5, Qwen3.6, gpt-oss, DeepSeek-R1, or any open source LLM / VLM!
OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
๐ง Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 ๐
Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation
The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
Arbitrary expression evaluation for golang
The platform for LLM evaluations and AI agent testing
SuperCLUE: ไธญๆ้็จๅคงๆจกๅ็ปผๅๆงๅบๅ | A Benchmark for Foundation Models in Chinese
Klipse is a JavaScript plugin for embedding interactive code snippets in tech blogs.
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
An open-source visual programming environment for battle-testing prompts to LLMs.
End-to-end Automatic Speech Recognition for Madarian and English in Tensorflow
A unified evaluation framework for large language models
๐ค Evaluate: A library for easily evaluating machine learning models and datasets.
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
Observal is a sandboxed artifactory and analytics platform for your AI development stack. Setup Observal, define the scope and share your Skills, MCPs and Agents.
๐ฐ Must-read papers and blogs on LLM based Long Context Modeling ๐ฅ
Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!
Avalanche: an End-to-End Library for Continual Learning based on PyTorch.
Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.
:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.
(IROS 2020, ECCVW 2020) Official Python Implementation for "3D Multi-Object Tracking: A Baseline and New Evaluation Metrics"
WFGY is heading toward WFGY 5.0 Polaris Protocol, a major open-source release for AI reasoning, RAG, agents, and real-world workflows. Includes Problem Map, Global Debug Card, WFGY 4.0, and the CFV Easter Egg.
ๅฟ็ๅฅๅบทๅคงๆจกๅ (LLM x Mental Health), Pre & Post-training & Dataset & Evaluation & Depoly & RAG, with InternLM / Qwen / Baichuan / DeepSeek / Mixtral / LLama / GLM series models
Building blocks for rapid development of GenAI applications
The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.
Multi-class confusion matrix library in Python
XAI - An eXplainability toolbox for machine learning
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
FuzzBench - Fuzzer benchmarking as a service.
Evaluate your LLM's response with Prometheus and GPT4 ๐ฏ
Evaluate and improve models and agents using environments
LangSmith Client SDK Implementations
SemanticKITTI API for visualizing dataset, processing data, and evaluating results.
Generate High-Quality Synthetics, Train, Measure, and Evaluate in a Single Pipeline
Expression evaluation in golang
ClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.
Framework for enhancing LLMs for RAG tasks using fine-tuning.
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards
Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. ไธไธช็ฑๅทฅๅ ทใๅบๅ/ๆฐๆฎใๆผ็คบใๆ่กๆฆๅๅคงๆจกๅ็ญ็ปๆ็็ฒพ้ๅ่กจ๏ผไธป่ฆ้ขๅๅบ็กๅคงๆจกๅ่ฏๆต๏ผๆจๅจๆขๆฑ็ๆๅผAI็ๆๆฏ่พน็.
[ICML 2024] TrustLLM: Trustworthiness in Large Language Models
A resource repository for machine unlearning in large language models
A collection of datasets that pair questions with SQL queries.
This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"
Meta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.
ParseBench - A Document Parsing Benchmark for AI Agents
The official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"