Poetry4painting: diversified poetry generation for large-size ancient paintings based on data augmentation

Figure 1: The goal of our work is enhancing the diversities when generating poems based on Chinese ancient paintings. Four kinds of data augmentation are proposed, including quantity(red), shape(green), surrounding(blue), and object augmentation(yellow). They are employed to enlarge our dataset as well.

Abstract: Chinese painting poetry is an extraordinary art form, which not only describes the painting contexts but also grasps the sentiment of the painters. In this paper, we propose an automatic poetry generation method Poetry4painting, which enhances the poetry diversity for large-size ancient paintings. The basic framework is based on multiple modern sentences, that are first captioned from the ancient painting and then used to generate a poem using CNN and LSTM. To solve the repeatability issue of this framework, four kinds of data augmentation are employed during online processing, including quantity, shape, surrounding, and object augmentation. In offline training, data augmentation is also used to create an image caption dataset with over 1500 painting images and 7500 captions. Through ablation studies, evaluations of poetry qualities and diversities, and comparisons with other methods, we demonstrate the validity of the proposed method.

Recommended citation: Jiazhou Chen*, Keyu Huang, Xinding Zhu, Xianlong Qiu, Haidan Wang, Xujia Qin. " Poetry4painting: diversified poetry generation for large-size ancient paintings based on data augmentation." Computer & Graphics. 2023, volume 116, pages 206-215.