Stream computing systems are critical platforms for real-time processing and analysis of data streams, widely applied in handling diverse workloads from Kubernetes, Apache Kafka, and web services. However, as workload types increase and hardware environments become more complex, resource management faces rising challenges in meeting service quality, load management, elasticity, and cost-efficiency requirements. Accurate workload identification is foundational for effective resource management in stream computing systems, enabling the design of optimization mechanisms such as priority scheduling, elastic scaling, and caching strategies. Yet, existing workload identification algorithms struggle to achieve sufficient classification accuracy in handling diverse workloads and complex environments, limiting their support for performance optimization in stream computing systems.

To address this issue, a research team led by Nong Xiao published their new research on 15 May 2026 in Frontiers of Computer Science co-published by Higher Education Press and Springer Nature. This paper proposes workload identification algorithms designed for multi-scenario application. By analyzing I/O data, they developed three feature extraction methods tailored for workload identification using both fine-grained and aggregated I/O data.

Existing workload identification methods struggle to accurately categorize diverse workloads in complex environments due to overly simplistic feature extraction. To address this, they conducted an in-depth analysis of fine-grained I/O trace from 3,049 workloads on Alibaba Cloud, revealing key feature distribution patterns: substantial differences in read/write ratios among workloads, a broad range of I/O access spans, and limited distinctiveness in certain features across workloads. From this analysis, they derived three optimization conclusions: separate extraction of read and write data features, mapping offsets to data blocks to enhance feature effectiveness, and focusing on high-distinction temporal and spatial features.

Building on these insights, they designed two types of workload identification algorithms to meet various application requirements. For high-precision needs, fine-grained I/O traces are utilized, with Cleanlab used for data preprocessing, followed by feature extraction based on the outlined conclusions, including basic, time, and spatial features. After sequencing the temporal features, CatBoost is applied for classification. For low-overhead requirements, minute-level aggregated I/O data is used, focusing solely on temporal feature extraction and classification to minimize resource usage.

Experimental results show that the proposed algorithms considerably outperform existing methods in classification performance, achieving notable improvements in accuracy, precision, recall, and F1 score. Additionally, the classification effectiveness remains consistent over time, demonstrating strong stability. Tests conducted on 600 clusters and 100 workloads further indicate that the algorithm presented in this study offers high scalability.
DOI:10.1007/s11704-024-40710-5