Researchers have demonstrated a technique that geometrically models organic objects and creates photorealistic, three-dimensional (3D) images of those objects. These mathematically precise images can be used to engineer robotic systems capable of identifying and sorting these complex shapes autonomously.
The technique was created to improve robotic systems that sort and identify microscopic marine fossils used in climate research, but could serve as a blueprint for applications in a range of other fields.
“We demonstrated the functionality of this technique in two ways: in a robotic system for 3D imaging of these microscopic marine fossils and in a robotic system for identification of the fossils,” says Edgar Lobaton, co-author of a paper on the work and a professor of electrical and computer engineering at North Carolina State University. “And identifying these fossils is very challenging, which is what led us to this work in the first place.”
At issue are foraminifera, or forams, which have been prevalent in Earth’s oceans for more than 100 million years. Forams are protists, neither plant nor animal, and when they die, they leave behind their tiny shells. These shells give scientists insights into the characteristics of the oceans as they existed when the forams were alive. For example, different types of foram species thrive in different kinds of ocean environments, and chemical measurements can tell scientists about everything from the ocean’s chemistry to its temperature when the shell was being formed.
However, evaluating foram shells and fossils is both tedious and time consuming – imagine sorting through hundreds of similarly shaped objects that are less than a millimeter wide. This is why paleontology researchers want to automate the process. And the nature of the challenge caught the interest of Lobaton.
“We had already developed a fully functional robotic system for identifying and sorting forams, called Forabot,” Lobaton says. “And creating Forabot taught us that the most time-consuming aspect of the process is fine-tuning the hardware and how it is laid out. What size should each component be? What is the best configuration of components? There are a million variations you may want to tweak. The work we’re sharing here was developed specifically to address that challenge, because we wanted to find a more efficient way to improve Forabot.”
By capturing 3D facsimiles of these fossils with incredible precision, the researchers can use those facsimiles in simulations of the robotic system.
“You can make adjustments in the simulation far more easily than when working with actual hardware,” Lobaton says. “And once you have optimized the configuration of the system in the simulation, the process of fine-tuning the hardware in the real world is vastly easier – you already know how it should be set up.”
For this work, the researchers modified a mathematical model so that it can produce detailed 3D facsimiles of the fossils. Lobaton’s team then worked with a paleontologist to ensure the facsimiles corresponded to the characteristics of seven representative species of foraminifera.
The researchers then turned to a simulation of Forabot. Using the newly captured 3D facsimiles to explore modifications to Forabot’s system, the researchers were able to improve its accuracy from 82% to 89% – without having to go through the time-consuming process of repeatedly reconfiguring the hardware in their lab.
“Using our synthetic dataset, we were able to test how state-of-the-art AI models can reconstruct 3D shapes from just a sparse set of 2D images,” says Sanjana Banerjee, corresponding author of the paper and a Ph.D. student at NC State. “These simulations helped us understand the best imaging conditions and are now guiding the development of a new robotic system focused on 3D reconstruction – an essential step toward further automating the identification of these microfossils.
“Our work provides a strong foundation for studying the growth and morphology of a wide range of foraminifera species,” Banerjee says. “It also tackles major challenges in micropaleontology, such as limited data availability and accurate shape recovery.”
“More broadly, the approach we took here could be used to develop or optimize any robotic system that identifies or sorts objects with complex shapes,” Lobaton says. “Potential use cases include microbe and pathogen isolation at the microscopic scale and sorting of agricultural produce at a larger scale.”
The researchers have made the code base used in this work open source, so other researchers can make use of it. That can be found at: https://github.com/ARoS-NCSU/Forams-3DGeneration.
The paper, “Foram3D: A Pipeline for 3D Synthetic Data Generation and Rendering of Foraminifera for Image Analysis and Reconstruction,” is published open access in the journal Marine Micropaleontology. The paper was co-authored by Turner Richmond, a former Ph.D. student at NC State; Michael Daniele, an associate professor of electrical and computer engineering at NC State; and Thomas Marchitto, a professor of geological sciences at the University of Colorado, Boulder.
This work was done with support from the National Science Foundation under grant 1829930.