awesome-safety-critical-ai

JavaScript

When the stakes are high, intelligence is only half the equation - reliability is the other ⚠️

Last updated Jul 8, 2026

Stars

Forks

Issues

Stars/day

Attention Score

View on GitHub ↗ Open in github.dev

Topics

#ai-privacy #ai-safety #artificial-intelligence #awesome #awesome-list #awesome-lists #compliance #determinism #explainability #genai #machine-learning #reliability #reproducibility #responsible-ai #safety-critical #trustworthy-ai

Language breakdown

JavaScript 57.1%

Python 31.0%

CSS 8.0%

HTML 4.0%

▸ Files click to expand

README

Awesome Safety-Critical AI

last commit

👋 Welcome to Awesome Safety-Critical AI - a curated space for practitioners, researchers and engineers committed to building intelligent systems that are as reliable as they are capable.

Inspired by systems thinking and engineering rigour, this project focuses on how to move from clever prototypes to fault-tolerant, production-grade ML systems.

Whether you're working on autonomous vehicles, medical diagnosis, or airbone systems, this collection offers a blueprint for AI that can be trusted when it matters most.

AI in critical systems is not about polishing demos or chasing benchmarks. It's about anticipating chaos - and designing for it.

This isn't just another (awesome) list. It's a call to action!

AI for Safety Critical

🐇 Introduction

- What is a Critical System? - AI in Critical Systems - The Bottom Line

🌟 Editor's Choice
🏃 TLDR
📝 Articles
✍️ Blogs / News
📚 Books
📜 Certifications
🎤 Conferences
👩‍🏫 Courses
📙 Guidelines
🤝 Initiatives
💡 Patents
📋 Reports
🛣️ Roadmaps
📐 Standards
🛠️ Tools
📺 Videos
📄 Whitepapers
👷🏼 Working Groups
👾 Miscellaneous
🏁 Meta
About Us
Contributions
Contributors
Citation

🐇 Introduction

What is a critical system?

Critical systems are systems whose failure can lead to injury 🤕, loss of life ☠️, environmental harm 🌱🚱, infrastructure damage 🏗️💥, or mission failure 🎯.

| Application | Industry Vertical | Description | Failure Impact | | --------------------------------- | --------------------- | ------------------------------------- | ------------------------------------------- | | Patient Monitoring | Healthcare | Tracks vital signs | Failure can delay life-saving interventions | | Aircraft Navigation | Aerospace / Aviation | Ensures safe flight paths | Errors can lead to accidents | | Power Grid Control | Energy | Manages electricity distribution | Failures can cause blackouts | | Command & Control | Defence | Coordinates military actions | Failure risks national security | | Industrial Automation Control | Manufacturing | Oversees production processes | Malfunction can cause damage or injury | | Core Banking System | Finance | Handles transactions and account data | Downtime can affect financial operations |

These systems are expected to operate with exceptionally high levels of safety, reliability and availability, often under unclear and unpredictable conditions.

They're the kind of systems we rarely think about... until something goes terribly wrong 🫣

| Incident | Year | Description | Root Cause | Industry Vertical | References | |--------------------------------|-----------|----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|-------------------------------|----------------| | Therac-25 Radiation Overdose | 1985–1987 | Radiation therapy machine gave fatal overdoses to multiple patients | Race conditions and lack of safety interlocks; poor error handling | Healthcare | Wikipedia, Stanford | | Lufthansa Flight 2904 | 1993 | Airbus A320 crashed during landing in Warsaw due to thrust reverser failure | Reversers disabled by software logic when gear compression conditions weren't met | Aviation | Wikipedia, Simple Flying | | Ariane Flight V88 | 1996 | Ariane 5 rocket self-destructed seconds after launch | Unhandled overflow converting 64-bit float to 16-bit integer | Aerospace | Wikipedia, MIT | | Mars Climate Orbiter | 1999 | NASA probe lost due to trajectory miscalculation | Metric vs imperial unit mismatch between subsystems | Space Exploration | NASA | | Patriot Missile Failure | 1991 | Failed interception of Scud missile during Gulf War | Rounding error in floating-point time tracking caused significant drift | Defence | Barr Group, GAO | | Knight Capital Loss | 2012 | Trading system triggered erratic market orders causing massive financial loss | Deployment of obsolete test code; no safeguards for live operations | Finance / Trading | Henrico Dolfing, CNN | | Toyota Unintended Acceleration | 2009–10 | Reports of unexpected vehicle acceleration and crashes | Stack overflow and memory corruption in embedded ECU software | Automotive | SAE, Wikipedia | | F-22 Raptor GPS Failure | 2007 | Multiple jets lost navigation after crossing the International Date Line | Software couldn't handle date transition; triggered reboot | Aerospace / Defence | FlightGlobal, Wikipedia | | Heartbleed Bug | 2014 | Security vulnerability in SSL exposed private server data | Improper bounds checking in the heartbeat extension of OpenSSL | Cybersecurity / IT | Heartbleed, CNET | | T-Mobile Sidekick Data Loss | 2009 | Users lost personal data during server migration | Software mishandling during data center transition led to irreversible loss | Telecom / Cloud Services | TechCrunch, PCWorld |

When the stakes are this high, conventional software engineering practices must be complemented by rigorous verification, validation and certification processes that are designed to ensure system integrity.

Critical systems don't forgive shortcuts. Only engineering rigour stands between order and disaster.

TL;DR Critical systems are built on trust - and trust is built on rock-solid engineering.

AI in Critical Systems

So, where does that leave us? Is there room for AI in critical systems?

This isn't just a theoretical question - we're already well beyond the realm of hypotheticals.

From making life-or-death decisions in the ICU to controlling UAVs, performing surveillance and threat detection, and powering self-driving cars, intelligent systems aren't just emerging in these these domains - they're already fully operational.

| Use Case | Brief Description | Industry Domain | References | |----------------------------------------------|----------------------------------------------------------------------------------------------------------|----------------------------------|----------------| | Predicting ICU Length of Stay | AI models use patient data to forecast ICU duration, improving resource allocation and care planning. | Healthcare | INFORMS, Springer | | AI in Radiation Therapy Planning | Optimizes dose targeting using historical patient models, improving treatment precision and safety. | Healthcare | Siemens Healthineers | | Self-Driving Cars | Powers perception, decision-making, and control systems for autonomous vehicles. | Transportation | Built In, Rapid Innovation | | Autonomous Drone Navigation | Enables drones to navigate complex terrain without GPS; supports rescue and defense operations. | Aerospace / Defense | MDPI, Fly Eye | | AI-Based Conflict Detection in ATC | Forecasts aircraft trajectories to alert controllers of potential collision risks. | Aerospace / Defense | Raven Aero, AviationFile | | Remote Digital Towers for Airports | AI interprets visual data to assist air traffic controllers in low-visibility conditions. | Aerospace / Defense | Airways Magazine | | Predictive Maintenance in Nuclear Reactors| Analyzes reactor sensor data to detect early failures, preventing major accidents. | Energy | Accelerant, IAEA | | AI-Assisted Reactor Control Systems | Supports operators by modeling physical processes and recommending safety actions in real time. | Energy | Uatom.org, Springer | | Autonomous Navigation for Cargo Ships | Enables real-time path planning to avoid obstacles and optimize maritime routes. | Transportation | MaritimeEd, ShipFinex | | AI-Based Collision Avoidance at Sea | Detects and responds to high-risk vessel situations using visual and radar data fusion. | Transportation | Ship Universe | | AI-Driven Fraud Detection | Identifies anomalous financial transactions and flags potential fraud in real time. | Financial Systems | Upskillist, Xenoss | | AI for Compliance Monitoring | Uses NLP to parse documents and logs for regulatory breaches, supporting audits and governance. | Financial Systems | Digital Adoption, LeewayHertz | | AI in Wildfire Early Detection | Processes satellite and sensor data to detect hotspots and alert emergency services. | Environmental Safety | NASA FireSense, PreventionWeb |

Building these systems is no walk in the park. ML brings powerful capabilities, but also adds layers of complexity and risk that need to be addressed through careful engineering.

While its ability to learn patterns and make data-driven decisions is unmatched in some domains, the adoption of AI in high-stakes environments must be tempered with caution, transparency, and a sharp understanding of its limitations.

Let's briefly recap some of the most important...

1. Models can and will make mistakes

Better models may make fewer mistakes, but mistakes are generally unavoidable.

Mistakes are not a sign of poor engineering - they are an intrinsic feature of intelligence.

Working with AI means accepting this uncertainty and designing systems that can handle it gracefully.

2. Mistakes can be strange and unpredictable

AI doesn't always fail in ways that make sense to us.

It might misclassify a stop sign with a sticker as a speed limit sign or switch treatment recommendations based on the user's language.

Unlike traditional software, which follows explicit rules, AI learns from data and generalises.

Generalization allows models to make predictions beyond what they've seen so far, but it's ultimately imperfect because the real world is messy, ever-changing, and rarely fits nicely into learned patterns.

3. Model outputs are often probabilitic

Traditional software is predictable: identical inputs yield identical outputs.

In contrast, ML models, especially those involving deep learning, can break this rule and exhibit probabilistic behavior.

Their outputs are a function not only of the input features, but also of things like model architecture, learned weights, training data distribution, hyperparameters (e.g. learning rate, batch size), optimization methods, and more.

That said, inference is often deterministic. Once trained, most models are capable of producing consistent outputs for a given input, assuming fixed weights and no funky runtime randomness.

This determinism means that systematic errors and biases are reproducible - models will consistently make the same mistakes.

Moreover, models trained on similar datasets often converge to similar representations, leading to shared failure modes and blind spots. So while ML systems may appear dynamic and random, their behavior can be quite predictable.

4. Data and models can change over time

Data and models are not static things. They'll evolve continuously due to changes in the environment, user behavior, hardware, regulations and more.

Imagine you're building a supervised learning system to detect early signs of pneumonia in chest X-rays.

Over time, several factors can cause both the data and the model to evolve:

Data Drift: the original training data may come from a specific hospital using a particular X-ray machine. As the system is deployed in new hospitals with different imaging equipment, patient demographics, or scanning protocols, the visual characteristics and quality of the X-ray images may vary significantly. This shift in the input distribution without an accompanying change in the task can reduce the model's diagnostic accuracy. This kind of drift doesn't actually crash the model, it just makes it quietly wrong*.

Concept Drift: clinical knowledge and medical understanding can also evolve. For instance, new variants of respiratory diseases may present differently on X-rays, or diagnostic criteria for pneumonia may be updated. The relationship between image features and the correct diagnosis changes, requiring updates to the labeling process and model retraining.

Model Updates: the model is periodically retrained with new data to improve diagnostic performance or reduce false positives. These updates might involve changes in architecture, training objectives, or preprocessing steps. While performance may improve on average, these changes can introduce new failure modes and even regressions in certain edge cases. Model changes must be managed and monitored carefully, with rigorous testing and rollback plans.

External Factors: regulatory changes or clinical guidelines may require the model to provide additional outputs, such as severity scores or explainability maps. This requires collecting new types of annotations and modifying the model's output structure.

In safety-critical domains like medical imaging, the evolution of data and models is inevitable. As such, systems must be designed with this in mind, embedding mechanisms for monitoring, validation, and traceability at every stage.

By proactively addressing data and model drift, automating model updates and defining processes for dealing with external influences, teams can ensure that AI systems remain not only accurate but also trustworthy, transparent, and robust over time.

5. Zero-error performance is expensive and often impossible

Here's an uncomfortable truth: no AI system will ever be perfect.

No matter how sophisticated your architecture, how clean your data, or how rigorous your testing - your system will eventually encounter scenarios it can't handle.

The pursuit of perfection isn't just futile; it's dangerous because it creates a false sense of security. Perfection is a mirage.

Instead of chasing the impossible, safety-critical AI demands a different mindset: assume failure and design for it.

This means embracing design principles that prioritize resilience, transparency, and human-centered control:

Graceful Degradation: When AI fails - and it will - what happens next? Does the system shut down safely, fall back to simpler heuristics, or alert human operators? The difference between a minor incident and a catastrophe often lies in how elegantly a system handles its own limitations.

Human-AI Collaboration: AI doesn't have to carry the entire burden. The most reliable critical systems often combine AI capabilities with human oversight, creating multiple layers of validation and intervention. Think of AI as a highly capable assistant, not an infallible decision-maker.

Monitoring and Circuit Breakers: Just as electrical systems have circuit breakers to prevent dangerous overloads, AI systems need mechanisms to detect when they're operating outside their safe boundaries. Confidence thresholds, anomaly detection, and performance monitoring aren't nice-to-haves - they're essential safety features.

Failure Mode Analysis: Traditional safety engineering asks what could go wrong? and designs accordingly. AI systems demand the same rigor. What happens when your model encounters adversarial inputs, when data quality degrades, or when edge cases compound in unexpected ways?

The goal isn't to eliminate failure - it's to make failure safe, detectable, and recoverable. This isn't just good engineering practice; it's an architectural requirement that separates safe systems from disasters waiting to happen.

TL;DR When failure costs lives, AI must be engineered like a scalpel, not a sledgehammer.

The Bottom Line

The challenges we've outlined aren't insurmountable obstacles; they're design constraints that demand respect, discipline, and thoughtful engineering. Each limitation - from unpredictable failures to shifting data landscapes - represents an opportunity to build more robust, transparent, and trustworthy systems.

The question isn't whether AI belongs in critical systems - it's already there, making life-and-death decisions every day. The real question is: Are we developing these systems with the rigor they deserve?

This collection exists because we believe the answer must be an emphatic yes. It's an open call to build AI systems that don't just perform well in the lab, but earn trust where it matters most.

In critical systems, good enough isn't good enough. The stakes are too high for anything less than our best engineering.

"Do you code with your loved ones in mind?"

― Emily Durie-Johnson, Strategies for Developing Safety-Critical Software in C++

🌟 Editor's Choice

🔼 Back to top

If you're just starting, here's our recommended reading list:

- ML in Production by Christian Kästner // Chapters 2 (From Models to Systems) and 7 (Planning for Mistakes) - Building Intelligent Systems by Geoff Hulten // Chapters 6 (Why Creating Intelligent Experiences is Hard), 7 (Balancing Intelligent Experiences) and 24 (Dealing with Mistakes) - MJ's The world and the machine, Kiri Wagstaff's Machine Learning that Matters and Varshney's Engineering Safety in Machine Learning