Realistic Sketch-based Face Photo Synthesis using Generative Adversarial Networks (GANs)

Authors

  • Hnin Ei Ei Cho Faculty of Information and Communication Technology, University of Technology (Yadanarpon Cyber City), PyinOoLwin, Myanmar
  • Aye Min Myat Faculty of Information and Communication Technology, University of Technology (Yadanarpon Cyber City), PyinOoLwin, Myanmar

Keywords:

hand-drawn sketch, image to image translation, sketch to photo synthesis, generative adversarial network, sketch-based face recognition

Abstract

Facial photo-image synthesis and sketch-based face recognition are highly advantageous, particularly in the fields of security forces and forensics. Furthermore, it makes it more feasible for law enforcement to reduce the number of possible suspects in criminal identification operations. However, since pencil drawings and photographs have different properties by nature, creating a synthesis of photographs based on sketches presents a difficult topic. In the last few decades, generative adversarial network-based systems have achieved enormous advances towards improving the performance of image synthesis. It can speed up identification times while improving matching outcomes by reducing gaps among sketch and photo representations. We perform investigations on the well-known photo-sketch pair database CUHK. First, we demonstrate how a generative adversarial network transforms hand-drawn sketches into realistic photos. Secondly, we employ suspect identification by using the pre-trained VGG16-based feature extractor network and KNN classifier. Our technique focuses on the use of deep learning-based networks, which are well-known for their capacity to process data and extract hierarchical features. The presented image-to-image translation framework minimizes the modality differences between hand-drawn face sketches and color images while improving visual quality. Tests on sketch-photo matching demonstrate significant improvements over current state-of-the-art methods on the challenging task of matching sketches with corresponding photos.

References

Xiaogang Wang and Xiaoou Tang, “Face Photo-Sketch Synthesis and Recognition”. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(11), pp.1955-1967.

Wang, Mingrui Zhu, Jie Li, Bin Song, Zan Li, “Data-driven vs. model-driven: Fast face sketch synthesis”, Neurocomputing,257, pp. 214-221. https://doi.org/10.1016/j.neucom.2016.07.071.

H. E. E. Cho and A. M. Myat, "Colorizing Face Sketch Images for Face Photo Synthesis", 2023 IEEE Conference on Computer Applications (ICCA), Yangon, Myanmar, 2023, pp. 119-123, doi: 10.1109/ICCA51723.2023.10182157.

Xiaoou Tang and Xiaogang Wang (2004), “Face Sketch Recognition”, IEEE Trans. CSVT, 14(1), 50–57.

Qingshan Liu, Xiaoou Tang, Hongliang Jin, Hanqing Lu andSongde Ma, “A nonlinear approach for face sketch synthesis and recognition”, IEEE CVPR, pp.1005-1010.

Hao Zhou, Zhanghui Kuang and Kwan-Yee K. Wong, “Markov Weight Fields for face Sketch Synthesis”, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, Rhode Island, June 16-22,2012, pp. 1091-1097.

Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li, “Transductive Face Sketch-photo Synthesis”, IEEE Transactions on Neural Networks and Learning Systems,24(9), pp. 1364-1376.

Jonathan Long, Evan Shelhamer and Trevor Darrell, “Fully Convolutional Networks for Semantic Segmentation”, IEEE Conference on Computer Vision and Pattern Segmentation, Boston, Massachusetts, June 7-12, 2015, pp. 3431-3440.

Gustav Larsson, Michael Maire, and Gregory Shakhnarovichm “. Learning representations for automatic colorization”, In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds) Computer Vision – ECCV 2016. Lecture Notes in Computer Science, vol 9908. Springer, Cham, pp. 577–593.

Liliang Zhang, Liang Lin, Xian Wu, Shengyong Ding and Lei Zhang, “End-to-end photo-sketch generation via fully convolutional representation Learning”, Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghal, China, June 23-26, 2015, pp. 627-634.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozairy, Aaron Courville and Yoshua Bengio. “Generative adversarial networks”, In Advances in neural information processing systems, pp. 2672-2680.

Mehdi Mirza and Simon Osindero, “Conditional Generative Adversarial Nets”, arXiv preprint arXiv:1411.1784, 2014.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei A. Efros, “Image-to-image translation with conditional adversarial networks”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967-5976.

Jun-Yan Zhu, Taesung Park, Phillip Isola and Alexei A. Efro, “Unpaired image-to-image translation using cycle-consistent adversarial networks”, IEEE International Conference on Computer Vision, Venice, Italy, October 21-29, 2017, pp. 2242-2251.

Zili Yi1, Hao Zhang, Ping Tan, and Minglun Gong, “Dualgan: Unsupervised dual leaning for image-to-image translation”, IEEE International Conference on Computer Vision, Venice, Italy, October 21-29, 2017, pp. 2868-2876.

Lidan Wang, Vishwanath A. Sindagi, Vishal M. Pate, “High-quality facial photo-sketch synthesis using multi-adversarial networks”, IEEE International Conference on Automatic Face & Gesture Recognition, Xi’an, China, May 15-19, 2018, pp. 83-90.

Xiaogang Wang and Xiaoou Tang, “Face Sketch-Photo Synthesis and Recognition”. Third International Conference on Image Processing and Capsule Networks, Bangkok, Thailand, May 20-21, 2022, https://doi.org/10.1007/978-3-031-12413-6_7.

Irfan Azhar, MuhammadSharif, Mudassar Raza, Muhammad Attique Khan and Hwan-SeungYong, “A decision support system for Face Sketch Synthesis using deep learning and artificial intelligence”, Sensors 2021, 21(24),

Downloads

Published

2024-06-15

How to Cite

Hnin Ei Ei Cho, & Aye Min Myat. (2024). Realistic Sketch-based Face Photo Synthesis using Generative Adversarial Networks (GANs). International Journal of Computer (IJC), 51(1), 17–32. Retrieved from https://ijcjournal.org/index.php/InternationalJournalOfComputer/article/view/2224

Issue

Section

Articles