A research team from China Agricultural University has published a study in the international open-access journal Engineering, introducing a multimodal feature representation mechanism designed for three-dimensional (3D) agricultural obstacle detection under limited or zero labeled sample conditions, aiming to ease data bottlenecks restricting reliable autonomous navigation of farm equipment.

Growing food demand and shrinking rural labor pools push the adoption of autonomous agricultural machinery, yet safe field operation relies heavily on robust obstacle detection. Camera–LiDAR fusion deep learning schemes deliver competitive 3D perception results, but mainstream models rely heavily on large annotated datasets, a critical limitation for unstructured farmland. Agricultural obstacle samples are scarce and heterogeneous, and labeling multimodal image and point cloud data requires professional labor and lengthy workflows. Though few-shot and zero-shot learning frameworks exist for vision or LiDAR single-modality detection, three core barriers block their agricultural deployment: heavy computation load from redundant multimodal data impairs real-time performance, terrain-induced sample gaps hinder knowledge transfer across field environments, and inconsistent spatial structure between images and point clouds complicates cross-modal feature alignment.

The proposed architecture addresses these constraints via three core technical modules. Image and point cloud attitude adjusters integrate BeiDou Navigation Satellite System (BDS) and inertial measurement unit (IMU) data to conduct coordinate transformation, eliminating yaw, pitch and roll attitude deviations caused by uneven farm terrain and standardizing multimodal data quality. A voxel-based point cloud feature encoder combined with ResNet image encoder adopts feature-level fusion, reducing redundant calculations on non-target regions and endowing unordered point clouds with regular spatial structure to lift computational efficiency. Semantic feature encoders extract color-related semantic attributes from panoramic camera images, while geometry–intensity encoders capture 3D structural and reflective features from LiDAR voxels; both streams are projected into a unified Bird’s Eye View (BEV) space via cross-modal projection modules. A semantic–geometry–intensity fusion representation space is constructed using pre-trained BERT and 3D feature descriptors including FPFH and SICC, with the convolutional block attention module (CBAM) weighting feature vectors to amplify intra-class similarities and inter-class distinctions, bridging category gaps with sparse annotations. The BEV fusion decoder concatenates aligned multimodal features and feeds them into a multi-task positioning head, which predicts heatmaps, center offsets, bounding box dimensions and yaw trigonometric values to output obstacle category and 3D location results. The overall loss function combines cross-entropy classification loss and smooth L1 regression loss for end-to-end joint optimization.

Field experiments were carried out at the Zhuozhou Experimental Station across cement road, nontilled soil and wheat field scenarios, covering ploughing and harvesting seasons. The test platform was a Lovol Euro-leopard M904-D tractor equipped with a 128-line 3D LiDAR, Ladybug 5 panoramic camera, BDS RTK base and mobile stations, and an MTi-300 IMU, with data synchronized through the ROS communication framework using UTC time from BeiDou satellite clock sources. The team pre-trained the model on the public KITTI multimodal dataset to cut reliance on field-specific annotations, and divided collected farm data into training, validation and test subsets with data augmentation only applied to training frames to avoid data leakage. Comparative tests against BEVFusion, MVXNet, PV-RCNN++ and MonoFlex show the new mechanism lowers training sample demand by 30%–40% while maintaining stable detection metrics. Under full training set usage, the system reaches a precision rate of 95.03%, recall rate of 97.01%, F₁ score of 96.01%, and runs at 16.56 frames per second on a mobile workstation, exceeding the 10 Hz sampling frequency of both sensors to meet real-time perception requirements. When facing fully unseen obstacle categories with zero corresponding training samples, the method still achieves an F₁ score of 81.63%, delivering usable safety perception for unfamiliar field conditions. Detection performance varies slightly across obstacle types due to object scale differences: harvesters and tractors carry more LiDAR points per instance and register above 95% across precision, recall and F₁ metrics, while human targets with fewer point samples show marginally lower scores, yet all categories maintain a 3D IoU higher than 75%.

The paper acknowledges remaining limitations, noting fixed pre-calibrated extrinsic sensor parameters lack adaptability for multi-scale obstacles, and indicates follow-up research will explore scale-adaptive multimodal registration strategies to further improve cross-scenario generalization capacity for agricultural autonomous navigation systems.

The paper “Multimodal Feature Representation Mechanism for 3D Detection of Agricultural Obstacles with Few or Zero Samples,” is authored by Tianhai Wang, Ning Wang, Shunda Li, Zhiwen Jin, Jianxing Xiao, Yanlong Miao, Yifan Sun, Han Li, Man Zhang. Full text of the open access paper: https://doi.org/10.1016/j.eng.2026.01.030. For more information about Engineering, visit the website at https://www.sciencedirect.com/journal/engineering.