Title: Analyzing Hazards for LLM-based Path Planning

Abstract: Agentic AI has rapidly grown in popularity, primarily because of the promise of no need for extensive coding and software development experience to develop task-based applications. More evaluation is needed to assess the risks ofusing such tools. This is especially true for safety-critical applications such as path planning which is a cornerstone of allrobotics applications and many navigation applications like those found in smartphones and vehicles. To this end, we first illustrate problems found in simply using large language models (LLMs) for navigation, both in vehicle and walking wayfinding scenarios. Then we assess the quality of path planning code generated from LLMS, as this is the supposed sweet spot for LLM utility. We then use these case studies to develop a hazard analysis of the use of LLMs for path planning application, especially those in safety-critical applications. Our results suggest that LLMs should not be used as a plug-and-play capability in agentic AI systems because they lack spatial reasoning. LLM-generated code also presentsserious hazards for path planning applications and should not be applied in safety-critical contexts without rigorous testing.

Bio: Professor Mary (Missy) Cummings received her B.S. in Mathematics from the US Naval Academy in 1988, her M.S. in Space Systems Engineering from the Naval Postgraduate School in 1994, and her Ph.D. in Systems Engineering from the University of Virginia in 2004. A naval officer and military pilot from 1988-1999, she was one of the U.S. Navy's first female fighter pilots. She is a Professor in the George Mason University College of Engineering and Computing and directs the Mason Responsible AI program as well as the Mason Autonomy and Robotics Center (MARC). She is an American Institute of Aeronautics and Astronautics fellow and a member of the Virginia Academy for Science, Engineering and Mathematics. Her research interests include the application of artificial intelligence in safety-critical systems, assured autonomy, human-systems engineering, and the ethical and social impact of technology.