With the advancement of multimedia internet, the impact of visual characteristics on the decision of users to click or not within the online retail industry is increasingly significant. Thus, incorporating visual features is a promising direction for further performance improvements in click-through rate. However, experiments on our production system revealed that simply injecting the image embeddings trained with established pre-training methods only has marginal improvements.

To solve the problems, a research team led by De-Chuan Zhan published their new research on 15 July 2025 in Frontiers of Computer Science co-published by Higher Education Press and Springer Nature.

The team analyzed that the main advantage of existing image feature pre-training methods lies in their effectiveness for cross-modal predictions. However, this differs significantly from the task of CTR prediction in recommendation systems. In recommendation systems, other modalities of information (such as text) can be directly used as features in downstream models. Even if the performance of cross-modal prediction tasks is excellent, it is challenging to provide significant information gain for the downstream models. They argued that a visual feature pre-training method tailored for recommendation is necessary for further improvements beyond existing modality features. To this end, they proposed an effective user intention reconstruction module to mine visual features related to user interests from behavior histories, which constructs a many-to-one correspondence. They further propose a contrastive training method to learn the user intentions and prevent the collapse of embedding vectors. They conduct extensive experimental evaluations on public datasets and our production system to verify that our method can learn users’ visual interests. The proposed method achieves 0.46% improvement in offline AUC and 0.88% improvement in Taobao GMV (Cross Merchandise Volume) with p-value < 0.01, which is significant considering the large active user volume of the Taobao App.
DOI： 10.1007/s11704-024-3939-x