Date and time: Tuesday 9 September 2025, 14:15-15:00 CEST
Speaker: Prof. Philip Koopman, Carnegie Mellon University
Title: Re-imagining Safety Engineering for Embodied AI Systems
Where: Online on Zoom: https://kth-se.zoom.us/j/69560887455
Host: Martin Törngren, martint@kth.se
This seminar is part of the “Safety Next” workshop program taking place at the 44th SafeComp conference in Stockholm, where we will delve into the new frontiers of safety-critical computing systems!
Bio: Prof. Philip Koopman is an internationally recognized expert on Autonomous Vehicle (AV) safety whose work in that area spans almost 30 years. He has also been involved with AV policy and standards as well as more general embedded system design and software quality. His pioneering research work includes software robustness testing and run time monitoring of autonomous systems to identify how they break and how to fix them. He has extensive experience in software safety and software quality across numerous embedded system, transportation, industrial, and defense application domains including conventional automotive software and hardware systems. He is faculty emeritus at the Carnegie Mellon University ECE Department where he taught software skills for mission-critical systems until 2025.
In 2018 he was awarded the highly selective IEEE-SSIT Carl Barus Award for outstanding service in the public interest for his work in promoting automotive computer-based system safety. He originated the UL 4600 standard for autonomous system safety issued in 2020. In 2022 he was named to the National Safety Council’s Mobility Safety Advisory Group. In 2023 he was named the International System Safety Society’s Educator of the Year.
In 2024 he received the Industry Legend award at the Self-Driving Industry Awards 2024 event. He is the author of the books: Understanding Checksums & Cyclic Redundancy Codes (2024), How Safe is Safe Enough: measuring and predicting autonomous vehicle safety (2022), The UL 4600 Guidebook (2022) and Better Embedded System Software (2010).
Abstract: Embodied AI (eAI) uses artificial intelligence based on machine learning to interact with the physical world. We are already seeing eAI deployed in the real world in robotaxis, smart medical devices, household robots, and other applications. However, everyone is struggling with the safety of these devices: how to design for safety, how to evaluate safety, and how to think about whether any particular eAI system is acceptably safe.
This talk provides an overview of my new book on this topic, with robotaxi safety as a concrete example. Anyone working in this area needs a basic understanding of four core areas: safety engineering, cybersecurity engineering, machine learning technology, and human/computer interaction. The talk also discusses eAI safety challenges in the wild, the complexities of establishing what risks might be acceptable, and open challenges in eAI safety. A proposal for reimagining safety engineering responds to the huge disruption that eAI technology creates when applying traditional computer-based system safety approaches. The talk finishes with a call to build justifiable trust in eAI safety.
