Researchers from TU Graz, Pro²Future and the University of St. Gallen have developed methods that enable IoT devices to run AI models with minimal memory – for example, to correct positioning errors.

Artificial intelligence is considered to be computationally and energy-intensive – a challenge for the Internet of Things (IoT), where small, embedded sensors have to make do with limited computing power, little memory and small batteries. In the E-MINDS project, a research team from the COMET K1 centre Pro²Future, Graz University of Technology (TU Graz) and the University of St. Gallen has found ways to run AI locally and efficiently on the smallest devices – without having to rely on external computing power. For example, it has been possible to run specialised AI models on an ultra-wideband localisation device with only 4 kilobytes of memory, which calculate sources of interference from location data.

Applying a few tricks

“Of course, these small devices do not run large language models, but rather models with very specific tasks, for example to estimate distances,” says Michael Krisper, head of the project at Pro²Future and scientist at the Institute of Technical Informatics at TU Graz. “But you also have to get these models small enough first. This requires a few tricks and it is precisely these tricks that we have been working on as part of the project.”

The result is a kind of modular system consisting of various methods which, when combined, deliver the desired result. One of these is the division of the models and their orchestration. Instead of one universal model, several small, specialised models are available. In the localisation technology investigated in E-MINDS, this means that one model works in the event of interference from metal walls, another in the event of interference from people and yet another in the event of interference from shelves. An orchestration model on the respective chip recognises which interference is present and loads the appropriate AI model from the server within around 100 milliseconds, which can calculate the interference factor from the data. This would be fast enough for industrial applications such as warehouses.

Fold, adjust, trim

Subspace configurable networks (SCNs) are another method within the modular system. These are models that adapt to the data input instead of having a separate model for each input variant. These SCNs were used for image recognition tasks such as object classification and proved to be extremely productive. For image changes or fruit classifications tested on IoT devices, it was possible to calculate images up to 7.8 times faster than using external resources, even though the models were smaller and more energy-efficient. Further reductions are achieved by folding the mathematical structure of the model without losing too much accuracy.

The same applied to the quantisation and pruning techniques. During quantisation, the researchers simplified the numbers used by the model. Instead of floating-point numbers, integers were used, which again saved energy and computing time with an acceptable loss of accuracy for the desired applications. Pruning, on the other hand, involves scrutinising a finished model and removing those parts that are not important for the desired end result. This is because the model will still be capable of fulfilling the core task, even when many (insignificant) parts are dismissed. It was important for the researchers to find the right balance between miniaturisation and remaining accuracy for all techniques. In addition to the successful miniaturisation, the project team also conducted research into the efficient deployment of the AI models so that they can be transferred to the small devices more quickly.

Results transferable to other areas

While the focus of E-MINDS was on wireless ultra-wideband (UWB) localisation in order to determine the exact position of drones, shuttles or robots in industrial automation despite obstacles and interference, the researchers see numerous other fields of application. For example, as an additional security measure for keyless car openers to determine whether the key is really near the car and someone is not just copying the radio signal. With efficient AI models, smart home remote controls would have a much longer battery life and libraries could track their books.

“With new expertise and new methods, we have laid a foundation for future products and applications in the E-MINDS project,” says Michael Krisper. “Our project team complemented each other perfectly here. At Pro²Future, we focused on embedded systems and implementation on the hardware, Olga Saukh worked with colleagues at the Institute of Technical Informatics at TU Graz to develop important scientific foundations in the field of embedded machine learning and contributed to model optimisation methods, and Simon Mayer contributed important research work in the field of localisation at the University of St. Gallen.”