#Evals
Showing 28 of 28 repositories tagged #evals, ranked by stars
Mastra is the modern TypeScript framework for AI-powered applications and agents.
AI Observability & Evaluation
Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
AI observability platform for production LLM and agent systems.
Evaluation and Tracking for LLM Experiments and AI Agents
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
Harbor is a framework for running agent evaluations and creating and using RL environments.
AI system design guide for engineers building production AI systems and evals.
Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.
Evaluate your LLM-powered apps with TypeScript
OpenSource Production ready Customer service with built in Evals and monitoring
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL
A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.
Build AI workflows by prompt or visual canvas. Heym is source-available and self-hosted, with agents, RAG, MCP, HITL, observability, evals, token cost tracking and more.
Test Generation for Prompts
A benchmark for evaluating AI agents on realistic business workflows
An opinionated list of awesome Pydantic-AI frameworks, libraries, software and resources.
Composable pi coding agent with MCP, LSP, agent chains, prompt presets, and local eval telemetry
Lumen — learner-owned AI education platform. Tell the AI what you want to learn: it builds you a private course in ~a minute, tutors you with course-scoped RAG + citations, and lets you share, clone & remix via a moderated catalog. BYOK, custom no-LangChain multi-agent orchestrator, golden evals in CI, MCP server. Live demo + public /eval.
Which coding agent wins on real work?
Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs
This course uses a deliberately vulnerable banking application to demonstrate common security vulnerabilities, their impact, and how to fix them. The application is built with Flask (backend) and React (frontend).
A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).
Analyze and generate unstructured data using LLMs, from quick experiments to billion token jobs.
The canonical standard for building production-grade agentic products — autonomy ladder, composition patterns, the 7-layer harness, eval discipline — plus a Claude Code skill set that operationalizes it.