#Evaluation-framework
Showing 31 of 31 repositories tagged #evaluation-framework, ranked by stars
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
The LLM Evaluation Framework
A framework for few-shot evaluation of language models.
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.
Repo for AI Agents The Definitive Guide
AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
Data-Driven Evaluation for LLM-Powered Applications
PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection
Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.
The official evaluation suite and dynamic data release for MixEval.
AI Data Management & Evaluation Platform
The robust European language model benchmark.
A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.
A New End-to-end Framework for Evaluating Voice Agents
βοΈ A curated list of tools, methods & platforms for evaluating AI reliability in real applications
A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.
A unified evaluation toolkit and leaderboard for rigorously assessing the scientific intelligence of large language and visionβlanguage models across the full research workflow.
[NeurIPS 2024] Evaluation harness for SWT-Bench, a benchmark for evaluating LLM repository-level test-generation
We believe that every SOTA result is only valid on its own dataset. RAGView provides a unified evaluation platform to benchmark different RAG methods on your specific data.
Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs
Agent Skills Evaluation Framework
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
Scalable Meta-Evaluation of LLMs as Evaluators
Diagnose the performance of your RAGπ©Ί
This repository helps you evaluate your models on the FreshStack benchmark!
A controlled-experiment lab for studying how tool-using LLM agents behave β vary tools, personas, and conversation histories, run repeated trials across models, and measure the effect on safety and tool-call behavior.
WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors
Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration
Evaluation framework for LLM knowledge inputs β prompts, RAG corpora, skills, agent workflows. Fix the model, vary the artifact. Built-in statistical rigor: bootstrap CI, Krippendorff Ξ±, length-debias, saturation curves.