In May 2024, the European Union adopted the world's first set of regulations for artificial intelligence. Among other things, it stipulates that high-risk AI systems - for example in critical infrastructure (KRITIS) or medical diagnoses - must operate under human supervision. The systems should also work transparently for users. As part of the "AI Cockpit" project, the Institute of Human Factors and Technology Management (IAT) at the University of Stuttgart examined exemplary approaches to implementing the European AI Regulation in operational practice across different sectors
Field studies in different organizations
The project consortium has developed AI cockpits together with Caritas, Starwit Technologies and Chemistree. "The three partners couldn’t be more different, yet the cockpit is highly relevant to all of them," says Dr. Nektaria Tagalidou, project manager at Fraunhofer IAO. Through field studies in various real-world scenarios, the project team explored how people can maintain an overview of different AI systems in their daily work. Good situational awareness - in other words, an understanding of what is happening and what the consequences could be - is very important in order to be able to act correctly.
Starwit Technologies is developing AI-supported traffic management planning for municipalities. The AI can recognize dangerous situations such as objects on the road or help to avoid traffic jams by cleverly directing traffic. In the AI cockpit, a human should be able to understand and control the AI's decisions. "Only then can he quickly intervene manually if an error occurs," says Tagalidou.
Caritas Dortmund is looking for ways to use AI systems in the care sector to support employees in their daily work. For example, systems that monitor the fluid intake of residents in a retirement home or algorithms that customize food orders to meet individual customer preferences are conceivable. "Control is particularly important here too, because people can be harmed if mistakes are made," says Michael Bui, a scientist in the Applied Neurocognitive Systems team and research associate at the IAT.
The researchers focused not only on the general design of the AI cockpits, but also on developing systems that adapt to the mental abilities of the users in order to avoid over- or underchallenging them. This is particularly important in areas where quick and correct decisions are required, for example in safety-critical situations. The result is a system that analyzes the user's current mental abilities and adapts information flows accordingly.
"AI cockpit" for companies: Open source for download
All AI Cockpit software applications and accompanying literature can be viewed and downloaded free of charge from the project website. Companies already working with AI or planning to introduce AI systems can benefit from the findings of the project. The Fraunhofer IAO and IAT at the University of Stuttgart offer companies comprehensive advice on this, including support with the introduction of a customizable AI cockpit.