#Ai-safety
Showing 60 of 85 repositories tagged #ai-safety, ranked by stars
AI Agent Governance Toolkit β Policy enforcement, zero-trust identity, execution sandboxing, and reliability engineering for autonomous AI agents. Covers 10/10 OWASP Agentic Top 10.
A curated list of awesome responsible machine learning resources.
Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback
Catch your AI's mistakes and blind spots before your customers or regulators do. iFixAi runs 45 inspections, 32 graded core plus 13 extended for frontier risks like sabotage, sandbagging, and oversight evasion. It returns a letter grade in under 5 minutes. Industry and model agnostic.
UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection
Safety-first guardrails for AI-driven cloud and Kubernetes operations
Agentlens is a trusted agent trading platform. Here, you can quickly find the Agent that meets your needs, and you can also publish your own Agent to turn it into your digital asset. We encourage everyone to transform their areas of expertise into Agents and turn them into digital assets, allowing others to see your unique strengths.
This is a repository that aims to provide updates on the status of jailbreaking the OpenAI GPT language model.
A resource repository for machine unlearning in large language models
Deliver safe & effective language models
PromptInject is a framework that assembles prompts in a modular fashion to provide a quantitative analysis of the robustness of LLMs to adversarial prompt attacks. π Best Paper Awards @ NeurIPS ML Safety Workshop 2022
The open agent control plane. Govern autonomous AI agents with pre-execution policy enforcement, approval gates, and audit trails. Works with LangChain, CrewAI, MCP, and any framework.
Open Source LLM toolkit to build trustworthy LLM applications. TigerArmor (AI safety), TigerRAG (embedding, RAG), TigerTune (fine-tuning)
Self-hosted, OpenAI-compatible AI gateway for private RAG, natural-language data access, and tool-calling agents.
π¦ Prevents outdated Rust code suggestions from AI assistants. This MCP server fetches current crate docs, uses embeddings/LLMs, and provides accurate context via a tool call.
[NeurIPS '23 Spotlight] Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
LangFair is a Python library for conducting use-case level LLM bias and fairness assessments
π A curated list of papers & technical articles on AI Quality & Safety
[ICLR'24 Spotlight] A language model (LM)-based emulation framework for identifying the risks of LM agents with tool use
BeaverTails is a collection of datasets designed to facilitate research on safety alignment in large language models (LLMs).
How to Make Safe AI? Let's Discuss! π‘|π¬|π|π
Agentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.
[CCS'24] SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models
Open-source EDR for AI agents. Monitor processes, files, network, and behavior of autonomous AI agents.
Agent orchestration & security template featuring MCP tool building, agent2agent workflows, mechanistic interpretability on sleeper agents, and agent integration via CLI wrappers
Alignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface
AI Agent Security Middleware β 8-layer defense, DLP data flow, prompt injection detection, zero dependencies. SDK + MCP server for Claude Code, Cursor, LangChain, Hermes Agent & more.
A Living Library You Can Talk To. Open-source educational platform with 30 historical figures from philosophy, science, art, mysticism, and activism. Stories, dialogues, AI conversation, multi-figure councils. Nonprofit, BYOK, self-hostable, no behavioral tracking.
An orchestration runtime for multi-agent AI systems. Declare agents, tools, and policies as YAML; Orloj schedules, executes, routes, and governs them for production-grade operation.
AI Security Platform: Defense (61 Rust engines + Micro-Model Swarm) + Offense (39K+ payloads)
Runtime enforcement boundary for AI agents: a local sidecar that gates every outbound call against Cedar policies you own. Deterministic, call-level, no model on the hot path
[ICLR'24] RAIN: Your Language Models Can Align Themselves without Finetuning
Practices, protocols, and skills for AI-driven software development. Skills and safety hooks for Claude Code, Codex, OpenCode, Cursor, Antigravity, and any agent supporting the Agent Skills standard.
LLM alignment jailbreak; a set of instructions for auditing their internal reasoning and uncovering biases
Deploy and manage AI agents in seconds.
Governance gateway for AI agents β bounded, auditable, session-aware control with MCP proxy, shell proxy & HTTP API. Works with Cursor, Claude Code, Codex, and any MCP-compatible agent.
γACL 2026 MainγAgentMark: Utility-Preserving Behavioral Watermarking for Agents
[ICLR 2024 Spotlight π₯ ] - [ Best Paper Award SoCal NLP 2023 π] - Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models
Don't make LLMs honest. Make every factual claim auditable. β An LLM Claim Auditing Layer with T1-T7 truth gradients. 98.1% business effectiveness on LiarBench v0.2.
Framework for specifying and proving propertiesβsuch as robustness, fairness, and interpretabilityβof machine learning models using Lean 4.
AgentGuard: Zero-Trust Security Foundation for AI Agents
A project to add scalable state-of-the-art out-of-distribution detection (open set recognition) support by changing two lines of code! Perform efficient inferences (i.e., do not increase inference time) and detection without classification accuracy drop, hyperparameter tuning, or collecting additional data.
Runtime safety for AI coding agents with real-time enforcement, system-event monitoring, and long-horizon provenance. Supports Claude Code and Codex on native macOS.
Guardrail capabilities for Pydantic AI β cost tracking, prompt injection detection, PII filtering, secret redaction, tool permissions, and async guardrails. Built on pydantic-ai's native capabilities API.
AI model health monitor for LLM apps β runtime checks for drift, hallucination risk, latency, and JSON/format quality on any OpenAI, Anthropic, or local client.
Prompt-injection guardrail for LLM applications. Compact model that outperforms larger open-source guards. No regex, no signatures. Demo: anton.securelayer7.net
[DEPRECATED] Moved to microsoft/agent-governance-toolkit
Persistent Claude Code agents with scheduling, sessions, memory, and Telegram.
OWASP Foundation web repository
When the stakes are high, intelligence is only half the equation - reliability is the other β οΈ
Desktop App with Built-In LLM for Removing Personal Identifiable Information in Documents
Utterly unelegant prompts for local LLMs, with scary results.
DoomArena is a Framework for Testing AI Agents Against Evolving Security Threats
AISecOps (AI Security Operations) framework for deterministic verification of AI systems. QWED verifies LLM outputs using math, logic, and symbolic execution β creating an auditable trust boundary for agentic AI systems. Not generation. Verification.
AI-powered Driver Drowsiness Detection System using Computer Vision & Machine Learning for real-time driver alertness monitoring and accident prevention.
Claude Code best practices -- applied to application design. Interactive HLD/LLD visualization, implementation example. LLM-agnostic, DB-governed, GDPR-ready.
ΠΠ½ΠΈΡΠΈΠ°ΡΠΈΠ²Π°, ΠΏΠΎΡΠ²ΡΡΠ΅Π½Π½Π°Ρ Π±Π΅Π·ΠΎΠΏΠ°ΡΠ½ΠΎΡΡΠΈ Π°Π³Π΅Π½ΡΠΎΠ² Π½Π° Π±Π°Π·Π΅ ΠΈΡΠΊΡΡΡΡΠ²Π΅Π½Π½ΠΎΠ³ΠΎ ΠΈΠ½ΡΠ΅Π»Π»Π΅ΠΊΡΠ°
Open-source AI agent red-team engine, SDK, and CLI. Run offline or against the Humanbound Platform.
arifOS β Constitutional MCP kernel for governed AI execution. AAA trinity: AGI (Mind) Β· ASI (Heart) Β· APEX (Judge). Built for the sovereign agentic era.
π¬ Verifiable AI-Augmented Engineering Framework - Stop AI hallucinations with formal traceability (REQβARTβTC). Agent Skills for Claude Code, Cursor, VS Code & Copilot. Enterprise-grade: ISO 9001, ISO 27001, GxP-ready. Red Team verification, multi-cycle lifecycle, behavioral anti-patterns.