In the wave of agricultural automation, fruit picking has long been a technical challenge. Traditional manual harvesting is inefficient and costly, while fully automated robots often struggle with inaccurate recognition and clumsy operation in complex environments. How can machines adapt more flexibly to orchard conditions while lowering operational barriers and achieving “human-robot synergy”?
A research team led by Associate Professor Pei Wang from Southwest University has addressed this question with an innovative study. They developed a gesture-controlled human-robot collaborative harvesting robot that can precisely locate and pick fruit with a simple wave. This technology not only significantly improves efficiency but also offers a new approach for small-scale orchards to transition toward intelligent operations. The study has been published in Frontiers of Agricultural Science and Engineering (DOI: 10.15302/J-FASE-2024588).
The core innovation of this robot lies in its “human-machine division of labor”. Researchers found that humans excel at identifying fruit locations and determining picking paths, while robotic arms outperform in repetitive motions and force control. Based on this insight, they designed a motion-sensing interaction system: the operator uses a Leap Motion sensor to capture hand movements in real time, directing the robotic arm to the target position, and then triggers the automated picking process with a double-tap gesture. This design combines human “eyes” with machine “arms”, retaining human flexibility while leveraging mechanical stability.
To ensure precise execution by the robotic arm, the team overcame multiple technical hurdles. For instance, inverse kinematics calculations for robotic arms often yield multiple solutions, which can cause sudden jerks or freezes. To address this, researchers proposed a “four-step screening method”, evaluating mechanical interference, verifying solution correctness, assessing motion rationality, and optimizing trajectory smoothness to select the safest joint angle combination. Simulation tests showed that the optimized robotic arm exhibited significantly reduced movement paths and joint rotation ranges, resulting in smoother motions.
Unlike traditional robots reliant on camera recognition, this device achieves “intuitive control” through high-precision motion-sensing technology. The Leap Motion controller captures hand movements at a 0.01-millimeter resolution, maintaining stable performance even under uneven lighting or foliage occlusion. Researchers also implemented intelligent filtering algorithms to eliminate “jittery data” caused by hand tremors or environmental interference, ensuring smooth robotic arm movement. Ingeniously, they dynamically mapped Leap Motion's cubic interaction space to the robotic arm's fan-shaped working area, allowing operators to move their hands within a virtual “box” while the robotic arm responds synchronously in the real orchard—as intuitive as playing a motion-sensing game.
Tests revealed an average system response time of 74.4 milliseconds and a 96.7% accuracy rate in gesture recognition. After brief training, operators reduced single-fruit picking time from 8.3 seconds to 6.5 seconds, marking a notable efficiency gain. Particularly in high-altitude harvesting scenarios, the robot eliminates the need for manual climbing, significantly reducing operational risks.
This technology lowers the technical barrier through a “human–robot collaboration” model—eliminating reliance on expensive vision systems and enabling farmers to operate it with minimal training. The modular design of the robotic arm allows for flexible replacement of joint motors, further enhancing maintainability. Tests confirm the system excels in complex terrains and small-scale orchards, adapting well to challenges like foliage occlusion and uneven lighting.
DOI: 10.15302/J-FASE-2024588