Artificial intelligence can help identify cells in digital pathology images, but densely packed tissue remains a challenge: neighboring cells may be merged, counted twice or missed entirely. Researchers at Koç University have developed a training strategy that improves detection by teaching AI to consider how cells are arranged within the tissue.
Published in Medical Image Analysis, the study introduces topology-guided hard example mining (TG-HEM). The approach helps models concentrate on image regions where their predictions fail to capture the spatial organization of cells, rather than relying only on errors at individual pixels.
The research was conducted by Professor Çiğdem Gündüz Demir, PhD student Onur Çakı and colleagues from Koç University’s Department of Computer Engineering and KUIS AI Center, working with collaborators in mathematics and pathology.
Learning from the most difficult regions
Cell counts and spatial distributions provide information that pathologists use when assessing disease and planning treatment. Yet a single tissue image can contain tens of thousands of cells, making manual counting time-consuming and subject to variation between observers.
Deep learning systems can automate this task, but local visual cues can become ambiguous when cells lie close together. The researchers’ approach brings information about the surrounding cell arrangement into the training process.
During training, TG-HEM compares predicted cell distributions with annotated reference distributions in sampled image regions. It uses topological data analysis—a mathematical approach for characterizing the structure of data—to quantify differences in their organization.
Regions with larger structural discrepancies receive greater weight during learning. Within those regions, the method also places additional emphasis on difficult pixels. This allows the model to learn from both local detection errors and broader differences in cell arrangement.
The strategy changes how the model is trained without requiring a larger or more complex network.
Improved counting and localization
The researchers evaluated the approach across eight AI models using their KUCell dataset and the publicly available BRCA-M2C breast cancer dataset.
TG-HEM improved cell counting and localization compared with the baseline models and existing hard example mining approaches. The gains were achieved without additional computational overhead when trained models analyzed new images, and with only a negligible effect on training time.
The team also released the KUCell dataset and implementation code, enabling other researchers to evaluate and build on the approach.
Supporting digital pathology research
More accurate cell detection could strengthen automated measurements used in digital pathology and support more consistent tissue assessment.
The study demonstrates improvements in cell detection on research datasets. Its potential contribution to clinical workflows lies in providing more reliable image measurements; improved diagnostic accuracy or patient outcomes were not established by these experiments.