An international team involving TU Graz has investigated how heavy machinery can be operated more safely using extended reality applications. In the process, a disco laser outdid VR headsets.
When it comes to snow groomers, excavators or crane vehicles, how can their operation be optimised even in difficult conditions and made safer for people in and around the vehicle? An international research team, including the Institute of Visual Computing at Graz University of Technology (TU Graz), investigated this question as part of the THEIA-XR project. The researchers aimed to improve human-machine interaction through the use of extended reality technologies. The focus was on the operator, whose field of perception was to be expanded without negatively affecting control performance. When working with snow groomers, for example, the team from TU Graz found that data or VR headsets tend to be counterproductive, while information projected via a repurposed disco laser proved to be a great help.
Drawings in the snow
Wearing a VR headset for long periods was too strenuous for the neck muscles; moreover, due to the constant jolting movements on rough terrain, they quickly caused nausea in many people. Laser projection onto the terrain in front of the vehicle, on the other hand, proved to be a viable solution. This allows not only speed information but also tracks and orientation aids to be projected onto the snow, enabling a snow groomer to be steered more efficiently and safely. Virtual barriers and warning indicators for people in the vicinity of the vehicle further enhance safety. And in poor visibility, due to fog or snowfall, the laser beams become visible in the aerosols in the air, allowing structures to be visualised that lie beyond the available line of sight.
In addition to improving data visualisation for drivers, the team also carried out research on ways to better capture the surrounding environment. It became apparent that the researchers’ intentions and the practical benefits for vehicle occupants were not always the same. “The exciting thing about this project was that our considerations from a research perspective and the drivers’ needs first had to be reconciled,” says Clemens Arth from the Institute of Visual Computing at TU Graz. “For example, we developed what is probably a globally unique prototype for a 360-degree thermal imaging camera to show snow groomer drivers whether there were people or animals around the vehicle. Ultimately, however, the much greater benefit was that they could groom the slopes more accurately, as it became clear where the snow was well-compacted and where it wasn’t.”
Foundations for future remote control
The project also provided important foundations for the increased remote control of heavy machinery in the long term. The rationale behind this is to protect the health of the people in the vehicle, as physical injuries can occur rapidly, particularly when working in rough terrain due to constant vibrations. An important step towards reliable remote control was the improved depth perception in simple camera transmissions, enabling distances to be better assessed via a monitor.
In addition to TU Graz and the snow groomer manufacturer Prinoth, Dresden University of Technology was also involved in the project, conducting research on excavator vehicles in collaboration with Stuttgart Media University. The VTT Technical Research Centre of Finland, in collaboration with the Kalmar company, focused on forklifts and loading machines in port environments. The University of Luxembourg was responsible for the anonymisation of personal performance data and the non-discriminatory identification of individuals by sensors. The consortium was led by TTControl, while the Creanex Oy and Haption companies contributed simulator and control technology.
Video: A snow groomer with a disco laser out in the field