Researchers led by Takaki Hatsui at the RIKEN SPring-8 Center (RSC) in Japan and collaborators have developed a new approach to compressing X-ray imaging data in real time, reducing the size of data files by more than 8,000 times, while at the same time preserving the detailed X-ray intensity information required for quantitative analysis. Because even small distortions in data can compromise experimental results, achieving high compression ratios has been difficult for large sets of scientific data—particularly X-ray images—as the precise physical information must be retained. One advantage of the new system is that it operates directly on high-throughput data streams generated in real-time during experiments. Beyond X-ray imaging, this approach could be applicable to measurement and inspection techniques using imaging detectors with diverse radiation such as electrons, gamma rays, neutrons, alpha and beta particles, protons, heavy ions, muons and other charged-particle or ion beams.
Advances in detector technology for imaging have enabled X-ray and electron-based experiments to generate data stream rates approaching terabits per second. However, these rates exceed the capabilities of conventional data transfer, storage, and analysis systems. This mismatch between data generation and system capability has become a major bottleneck that prevents the full exploitation of modern radiation-based experimental techniques.
For the current research, published in the Journal of Synchrotron Radiation, the group worked to resolve this bottleneck by developing and demonstrating a real-time compression system for high-throughput experimental data at the SPring-8 synchrotron radiation facility. The system was applied to a detector producing continuous data streams of 216 Gbps over extended periods—equivalent to 2.3 petabytes per day—and achieved stable operation at compression ratios exceeding 8,000 times. Using this system, users can now perform quasi-elastic scattering measurements—which can non-destructively probe the dynamics of atoms, molecules, and nanoscale structures in a wide range of materials over sub-nanosecond to nanosecond timescales—at high data rates without being constrained by the underlying data volumes. This capability is expected to accelerate materials development and our understanding of biological phenomena.
To achieve this real-time compression, the team utilized data-processing boards equipped with Field-Programmable Gate Arrays (FPGAs), which they developed in their previous work. Unlike conventional CPUs, which execute software instructions on a fixed hardware architecture, the logic of an FPGA can be configured at the hardware level to implement application-specific digital circuits. This enables massive parallelism and pipelined processing tailored to the data flow. The team implemented the computationally demanding components of the compression algorithm on the FPGA, while preserving the scientific information contained in the data. This approach allows the processing to be executed efficiently in hardware, enabling real-time operation at high data rates.
According to first author Haruki Nishino, “While this work was performed using synchrotron radiation at SPring-8, the approach is broadly applicable to measurement and inspection techniques based on radiation sources equipped with fast imaging detectors.” He continues, “By enabling real-time compression of high-throughput data, and also preserving scientific information, this method addresses a fundamental limitation in handling modern experimental data and opens a new paradigm for data-intensive measurement and inspection.”