Figuring it Out: Exploring AI-Generated Figurative Expression Across Modalities
PhD project of Johanna Rockstroh
Figurative language is a central feature of human communication, enabling abstract or complex ideas to be conveyed in vivid and memorable ways. With recent advances in large language models (LLMs) and multimodal generative AI, there is growing potential for producing figurative expressions not only in text but also in images, comics, and GIFs. This dissertation examines how such systems can support comprehension, creativity, and engagement across domains including education, health communication, and journalism.
Three guiding questions frame the work: How can figurative language make abstract concepts more accessible to non-expert audiences? Which evaluation methods best capture the creativity and appropriateness of AI-generated figurative content? And how do multimodal figurative expressions in form of text, images or GIFs shape user experiences in conversational systems? To address these questions, the project combines the generation of artefacts with systematic evaluation and user-centered studies.
Initial findings indicate that the effectiveness of figurative language depends both on audience background knowledge and on the degree of conventionalization. While familiar expressions tend to be more readily understood, novel or less conventional forms can lead to ambiguity. Figurative language can enhance comprehension in educational contexts and raise awareness in health communication, but across domains it also carries the risk of confusion or misinterpretation when meanings are unclear.
Current projects focus on developing evaluation metrics for LLM-generated artefacts, comparing modalities in chatbots for idiom comprehension, paraphrasing conceptual knowledge in STEM education, and studying user responses to GIFs in dialogue systems. Future work extends these approaches to sexual health communication, journalistic writing support, and social companions, with the aim of refining evaluation methods, examining the role of metaphor and idioms in user perception, and designing strategies for integrating figurative expression into applied systems. Together, these efforts advance methods for assessing computational creativity while offering domain-specific insights into the role of AI in human communication.
Contact
Contact:
Prof. Dr. Andreas Hepp
ZeMKI, Center for Media, Communication and Information Research
University of Bremen
Phone: +49 421 218-67620
Assistent Mrs. Schober: +49 421 218-67603
E-mail: andreas.hepp@uni-bremen.de
