A significant technical pain point in distributed storage systems is the massive overhead incurred during node recovery. Traditional Reed-Solomon (RS) codes offer optimal storage efficiency but suffer from high network bandwidth consumption during repair. While Minimum Storage Regenerating (MSR) codes and Locally Repairable (LR) codes have been proposed to mitigate this, they often introduce heavy disk I/O burdens or compromise storage density. This persistent conflict between bandwidth, I/O performance, and storage overhead limits the rapid recovery capabilities of large-scale data centers facing frequent hardware failures.
In response to these challenges, the research team from Shanghai Jiao Tong University developed a novel construction of MDS array codes. This innovation shifts away from single-metric optimization, focusing instead on a holistic matrix design tailored for systematic node repair. The core of the framework utilizes an ingenious geometric mapping to achieve a very small sub-packetization level of l=O(r) while maintaining Maximum Distance Separable (MDS) properties. This design ensures that only a fraction of data needs to be read from disks during single-node failure recovery, effectively eliminating I/O bottlenecks and reducing the amount of data transmitted across the network.
Research indicates that in large-scale simulated storage environments, the proposed scheme offers superior overall performance compared to traditional encoding baselines. Data analysis suggests that the framework significantly reduces the number of disk accesses during repair without increasing storage overhead, while its low sub-packetization level eases the memory pressure during practical implementation. This work provides a reliable and flexible paradigm for erasure coding theory, offering a robust technical roadmap for building high-performance, cost-effective data storage infrastructures in the era of big data.
DOI:10.1007/s11704-024-3581-7