#Llm-security
Showing 60 of 85 repositories tagged #llm-security, ranked by stars
Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data. ๐ณDocker-friendly.โกAlways in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.
Open-source AI penetration testing tool to find and fix your appโs vulnerabilities.
the LLM vulnerability scanner
NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
๐ข Open-Source Evaluation & Testing library for LLM Agents
A full-stack AI Red Teaming platform securing AI ecosystems via OpenClaw Security Scan, Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
[CCS'24] A dataset consists of 15,140 ChatGPT prompts from Reddit, Discord, websites, and open-source datasets (including 1,405 jailbreak prompts).
The Security Toolkit for LLM Interactions
Sandbox any AI agent in seconds - zero setup, zero latency.
A secure low code deception runtime framework, leveraging AI for System Virtualization.
A powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers identify and mitigate potential jailbreaks in their LLM APIs.
OWASP Top 10 for Large Language Model Apps (Part of the GenAI Security Project)
Catch your AI's mistakes and blind spots before your customers or regulators do. iFixAi runs 45 inspections, 32 graded core plus 13 extended for frontier risks like sabotage, sandbagging, and oversight evasion. It returns a letter grade in under 5 minutes. Industry and model agnostic.
A security scanner for your LLM agentic workflows
AI-first security scanner. NEW in v2026.7: Claude Code compromise detection โ vet .claude/ hooks, permissions & skills before you clone โ plus an always-on AI attack-signature scanner and native Rust & PHP rules. Also: medusa scan --git to vet any repo, medusa secrets scan for leaked API keys. 40,000+ patterns, zero setup.
An easy-to-use Python framework to generate adversarial jailbreak prompts.
Open-source AI agent firewall for MCP security and agent egress. Scans mediated HTTP, MCP, A2A, and WebSocket traffic for exfiltration, SSRF, and prompt injection, and emits mediator-signed action receipts: verifiable audit evidence from outside the agent.
Local security audit for AI API relays and LLM proxies: detects prompt injection, model substitution, tool-call rewriting, SSE anomalies, error leakage, and Web3 wallet risks.
Give each AI agent its own isolated machine with root, Docker, and systemd. Active defense detects and stops threats automatically.
Papers and resources related to the security and privacy of LLMs ๐ค
This repository provides a benchmark for prompt injection attacks and defenses in LLMs
This is The most comprehensive prompt hacking course available, which record our progress on a prompt engineering and prompt hacking course.
Runtime security for Agents. Blocks dangerous commands, prevent secret leaks, stop prompt injection, gate risky package installs and visibility and control over tool calls.
๐ดโโ ๏ธ Hacking Guides, Demos and Proof-of-Concepts ๐ฅท
Red Teaming python-framework for testing chatbots and GenAI systems.
An open-source knowledge base of defensive countermeasures to protect AI/ML systems. Features interactive views and maps defenses to known threats from frameworks like MITRE ATLAS, MAESTRO, and OWASP.
Whistleblower is a offensive security tool for testing against system prompt leakage and capability discovery of an AI application exposed through API. Built for AI engineers, security researchers and folks who want to know what's going on inside the LLM-based app they use daily
This repository contains various attack against Large Language Models.
Ultra-fast, low latency LLM prompt injection/jailbreak detection โ๏ธ
AI Agent Security Middleware โ 8-layer defense, DLP data flow, prompt injection detection, zero dependencies. SDK + MCP server for Claude Code, Cursor, LangChain, Hermes Agent & more.
Security scanner MCP server for AI coding agents. Prompt injection firewall, package hallucination detection (4.3M+ packages), 1000+ vulnerability rules with AST & taint analysis, auto-fix.
AspGoat is an intentionally vulnerable ASP.NET Core application for learning and practicing web application security.
AI Security Platform: Defense (61 Rust engines + Micro-Model Swarm) + Offense (39K+ payloads)
The antivirus for OpenClaw โ approve dangerous actions, scan skills, block secret leaks, and keep humans in control, for safety.
๐ค Curated AI OSINT resources โ Google dorks, Shodan queries, GitHub dorks, and techniques to discover exposed LLM endpoints, leaked AI API keys, misconfigured vector databases, and unprotected AI agents
Open-core AI red teaming and offensive AI security evaluation platform.
The Anti-Virus for AI Artifacts & RAG Firewall. A static analysis tool scanning Models and Notebooks for RCE, Datasets and RAG docs for Data Poisoning, PII, and Prompt Injections. Secure your AI Supply Chain.
Threat modeling and AI-reasoning vulnerability detection harness for Claude Code โ STRIDE + AI + MAESTRO
Code scanner to check for issues in prompts and LLM calls
Prompt-injection guardrail for LLM applications. Compact model that outperforms larger open-source guards. No regex, no signatures. Demo: anton.securelayer7.net
It is a comprehensive resource hub compiling all LLM papers accepted at the International Conference on Learning Representations (ICLR) in 2024.
RAG/LLM Security Scanner identifies critical vulnerabilities in AI-powered applications, including chatbots, virtual assistants, and knowledge retrieval systems.
First-of-its-kind AI benchmark for evaluating the protection capabilities of large language model (LLM) guard systems (guardrails and safeguards)
OWASP Foundation web repository
This project investigates the security of large language models by performing binary classification of a set of input prompts to discover malicious prompts. Several approaches have been analyzed using classical ML algorithms, a trained LLM model, and a fine-tuned LLM model.
This course uses a deliberately vulnerable banking application to demonstrate common security vulnerabilities, their impact, and how to fix them. The application is built with Flask (backend) and React (frontend).
Local control plane for running AI agents with sandboxes, approvals, guardrails, credentials, and runtime health.
Zero-code LLM security & observability proxy. Real-time prompt injection detection, PII scanning, and cost control for OpenAI-compatible APIs. Built in Rust.
A living map of the AI agent security ecosystem.
LLM security and privacy
Autonomous Bug Bounty Hunting Framework powered by Claude Code. 20 AI agents, state-machine orchestration, Burp Suite MCP, credential vault, LLM security track. Type 'hunt target.com' and let AI find the bugs.
Open-source security platform for AI agents -- audits skills before install, monitors 24/7, shares threat intelligence across all users. | AI Agent ้ๆบๅฎๅ จๅนณๅฐ -- ๅฎ่ฃๅๅฏฉ่จ skillใ24/7 ๅณๆ็ฃๆงใ็คพ็พคๅ ฑไบซๅจ่ ๆ ๅ ฑใ
Open-source AI agent red-team engine, SDK, and CLI. Run offline or against the Humanbound Platform.
Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application Development. It provides a robust platform with a user-friendly UI for streamlining the process of building and assessing the performance of your LLM-powered applications.
Fail-closed execution firewall for AI agents: quarantine MCP tools, proxy OpenAI-compatible requests, emit signed receipts, and verify EvidencePacks offline.
Whispers in the Machine: Confidentiality in Agentic Systems
Find and govern AI attack surfaces in application code at PR time. Free, OSS, runs offline.
AI Firewall & LLM security toolkit - protect your AI applications from prompt injection, jailbreaks, PII leakage, and adversarial attacks
The open-source agent firewall. Prevent AI agents from leaking data, using dangerous tools, and importing poisoned dependencies.
Fast local Rust scanner for AI-agent prompt injection, credential leaks, exfiltration, and risky tool calls