Open-Vocabulary Object Detection through Saliency Transfer and Structured Visual Reasoning
Keywords:
Open-Vocabulary Detection, Saliency Transfer, Visual Reasoning, Vision-Language Models, Scene UnderstandingAbstract
The paradigm of open-vocabulary object detection aims to identify and localize objects beyond the restricted set of categories observed during the training phase. While recent vision-language models provide robust semantic priors, transferring these representations to dense prediction tasks often suffers from background noise and a lack of structural understanding of the visual scene. In this paper, we propose a novel framework that integrates saliency transfer and structured visual reasoning to overcome these limitations. By leveraging pre-trained saliency models, our approach isolates foreground objects from complex backgrounds, thereby providing a strong inductive bias that guides the alignment of visual regions with textual embeddings. Furthermore, we introduce a structured visual reasoning module that constructs dynamic scene graphs, enabling the model to utilize contextual relationships between candidate regions to refine classification scores for unseen classes. Extensive experiments on standard benchmarks demonstrate that the proposed method significantly outperforms existing open-vocabulary detection baselines, particularly in identifying rare and novel categories. We analyze the contribution of each component through comprehensive ablation studies, revealing that the combination of spatial attention via saliency and semantic context via relational reasoning leads to highly generalizable object representations. Our findings suggest a promising direction for developing robust, scalable perception systems capable of operating in unconstrained environments.
References
1. Zhang B, Wang X, Kao H, et al. Consensus Matrix: A Role Specialized Multi Agent Framework for Structured Collaborative Decision Making in Agentic Visual Media Workflows[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 4613 4622.
2. Qi, Z., Yuan, Y., Ruan, X., Wang, S., Zhang, W., & Huang, Q. (2024). Collaborative debias strategy for temporal sentence grounding in video. IEEE Transactions on Circuits and Systems for Video Technology, 34(11), 10972-10986.
3. Xia, Y., Lu, Y., Gao, Y., & Ma, J. (2024, March). Locality preserving refinement for shape matching with functional maps. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 6, pp. 6207-6215).
4. Qu, D., & Ma, Y. (2026). Fourier–Transformer Mixer Network for Efficient Video Scene Graph Prediction.Engineering Proceedings,120(1), 16.
5. Xia, Y., Ye, T., Huang, J., Mei, X., & Ma, J. (2026, March). Probabilistic deformation consistency for unsupervised shape matching. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 13, pp. 10951-10959).
6. Xia, Y., & Ma, J. (2025). Grid-guided sparse Laplacian consensus for robust feature matching.IEEE Transactions on Image Processing,34, 1367-1381.
7. Zhao, Haoyu, et al. "LSTD: Long Short-Term Temporal Diffusion for Video Generation." IEEE Transactions on Multimedia (2026).
8. Wang, H., Xu, Q., Wang, C., Xue, T., Peng, C., Chen, W., & Lin, F. (2026). Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning.arXiv preprint arXiv:2605.14054.
9. Qu, W., Wang, J., Gong, Y., Huang, X., & Xiao, L. (2025, June). An end-to-end robust point cloud semantic segmentation network with single-step conditional diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 27325-27335). IEEE.
10. Li, Q., Liu, Z., Luo, W., Luo, T., & Hou, C. (2026). Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory. arXiv preprint arXiv:2605.24602.
11. Lee, Y., Gao, P., Xu, Y., & Fan, W. (2025, October). How Do Optical Flow and Textual Prompts Collaborate to Assist in Audio-Visual Semantic Segmentation?. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 23342-23352). IEEE.
12. Yang, Y., Bortins, I., Baldwin, D. P., & Zhang, S. (2024). Calibration of dual resolution dual camera structured light systems.Optics and Lasers in Engineering,182, 108472.
13. Li, X., Yang, F., Chen, L., & Cai, H. (2016, July). Saliency transfer: An example-based method for salient object detection. In IJCAI (pp. 3411-3417).
14. Lv, Q., Kong, W., Li, H., Zeng, J., Qiu, Z., Qu, D., ... & Pang, J. (2025). F1: A vision-language-action model bridging understanding and generation to actions. arXiv preprint arXiv:2509.06951.
15. Li, Z., Bai, X., Lu, J., Shen, P., Lam, E. Y., & Peng, Y. (2026). EventTracer: Fast Path Tracing-based Event Stream Rendering. IEEE Transactions on Visualization and Computer Graphics.
16. Zhang, W., Huang, M., Zhou, Y., Zhang, J., Yu, J., Wang, J., & Xu, L. (2024). Both2hands: Inferring 3d hands from both text prompts and body dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2393-2404).
17. Yang, Y., & Zhang, S. (2024). Pixelwise calibration method for a telecentric structured light system.Applied Optics,63(10), 2562-2569.
18. Liu, C., Ma, L., Zhang, X. T., Zhang, Y., Zhang, H., Yang, X., & Tian, F. (2026). Boosting omni-modal language models: Staged post-training with visually debiased evaluation. arXiv preprint arXiv:2605.12034.
19. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems.
20. Ding, Y., Li, Z., Huang, D., Zhang, K., Li, Z., & Feng, W. (2022). Adaptive range guided multi-view depth estimation with normal ranking loss. In Proceedings of the Asian Conference on Computer Vision (pp. 1892-1908).
21. Zhang, Y., Zhang, Y., Wu, H., Liu, Z., Zang, R., Zhu, X.,... & Ma, F. (2026). Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning. arXiv preprint arXiv:2606.29984.
22. Li, B., Zhang, R., Liang, H., Zhang, J., Zhang, J., Chen, X., ... & Wang, J. (2026). Interagent: Physics-based multi-agent command execution via diffusion on interaction graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15253-15265).
23. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2014). OverFeat: Integrated recognition, localization and detection using convolutional networks. In Proceedings of ICLR.
24. Huang, H., Zhang, J., Zhang, J., Xu, J., & Wu, Q. (2020). Low-rank pairwise alignment bilinear network for few-shot fine-grained image classification.IEEE Transactions on Multimedia,23, 1666-1680.
25. Wang, Y., Fang, Y., Wang, T., Feng, Y., Tan, Y., Zhang, S., ... & Xu, R. (2025). Dreamnav: A trajectory-based imaginative framework for zero-shot vision-and-language navigation.arXiv preprint arXiv:2509.11197.
26. Qi, Z., Yuan, Y., Ruan, X., Wang, S., Zhang, W., & Huang, Q. (2024, March). Bias-conflict sample synthesis and adversarial removal debias strategy for temporal sentence grounding in video. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 5, pp. 4533-4541).
27. Chen, L., Jiang, X., Liu, X., & Haardt, M. (2022). Reweighted low-rank factorization with deep prior for image restoration. IEEE Transactions on Signal Processing, 70, 3514-3529.
28. Peng, Qucheng, et al. "3d vision-language gaussian splatting." International Conference on Learning Representations. Vol. 2025. 2025.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Software Engineering and Artificial Intelligence Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
