by Davide Ceolin (CWI), Delfina Sol Martinez Pandiani (University of Amsterdam), and Erik Tjong Kim Sang (NL eScience Center)
Memes are extremely viral information items that, in many cases, are used to spread hateful messages. CWI, in collaboration with the University of Amsterdam and the Netherlands eScience Center investigates the effectiveness of semi-automated methods in analysing toxic memes.
This research is conducted by CWI, in collaboration with the University of Amsterdam and the Netherlands eScience Center, in the context of the Eye of the Beholder project, which is funded by the Netherlands eScience Center.
In the era of rising multimodal content consumption, the interplay between visual and textual elements, especially within computer science, is a key research focus. Notably, there's a growing interest in automatically discerning semiotic relationships between images and text. Simultaneously, a growing body of research delves into potentially problematic multimodal content in social media, particularly due to users' ease of engaging with and believing misinformation in multimodal formats. This underscores the heightened interest in automatically detecting hateful, propagandistic, fake, or otherwise misleading or manipulative content in multimodal memes and social media posts.
To this aim, using the PRISMA methodology, we surveyed 158 papers and identified 30 datasets that focus on the problem of hateful meme classification [1]. From this analysis, we observe that in the literature, meme toxicity is characterized in rather diverse and misaligned manners. To this aim, we propose a harmonization of the terminology used, and we also introduce a framework to characterize such terminology. In particular, we identify three content-based dimensions of meme toxicity: target, intent, and conveyance tactics. The framework that we propose illustrates the relationships between these dimensions and meme toxicities. This allows characterizing the current classes of toxic content, as well as provides the means to represent new classes in case these emerge.
Having defined such a framework, we investigated the possibility of using this harmonized classification to automate the task of meme analysis. In particular, we identified in LLMs a potential tool for this goal, because these models can leverage large amounts of contextual information and can help discern the multiple layers of information that memes often portray. We performed preliminary analyses on using LLMs for this aim [2]. In particular, given the amount of resources required to run LLMs, we tested whether locally run LLMs can perform satisfactorily well in assessing the toxicity of memes. The results we obtain are promising: while it is true that larger LLMs perform better in assessing the toxicity of memes, it is also true that locally run models still show promising performance. In particular, locally run LLMs show promising results both in the assessment of the toxicity of memes and in the identification of toxic symbols that these contain.
Several factors can affect performance when analyzing toxic memes with LLMs. In particular, memes contain several layers of often implicit semantics, which may be hard to grasp automatically because of their complexity or because of the amount of specific background information needed to understand them. Moreover, guardrails introduced to avoid the generation of toxic content may impede the analysis of toxic items. The next work we developed is propaedeutic to addressing the former problem.
An important source of knowledge for toxic content detection is represented by The Global Extremist Symbols Database, which contains those symbols that were classified as toxic by The Global Project Against Hate and Extremism (GPAHE). In fact, while memes are complex objects composed of multiple semantic layers and contain implicit messages, many toxic memes portray their toxicity through the explicit use of toxic symbology. For this reason, in a subsequent work [3], we propose OnTox and OnToxKG, an ontology and a knowledge graph to characterize potentially toxic symbols from the Global Extremist Symbol Database. We demonstrate that the use of this symbolic representation of these symbols, often used in toxic memes, is beneficial for the task of toxicity detection. OnToxKG contains multimodal data of almost 800 potentially toxic symbols, and the use of Linked Data to represent them allows to easily link them to generic sources like Wikidata (https://wikidata.org/) and WordNet (https://wordnet.princeton.edu/), to facilitate disambiguation and contextualization of the symbols, as demonstrated also through a set of SPARQL queries. Via such queries, we show that it is easy to use the ontology and the knowledge base to retrieve, for instance, symbols representing similar concepts, as well as retrieve memes containing specific symbolic representations (assuming that the meme itself was previously analysed to identify the relevant symbols).
In the future, we plan to further advance the analysis of memes in a (semi-)automated manner, exploring the possibility of automating this task by OnToxKG in combination with Linked Open Data sources and LLMs, considering also the potential subjectivity of this task.
References:
[1] D. S. M. Pandiani, E. T. K. Sang, and D. Ceolin, “Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities,” arXiv preprint arXiv:2406.07353, 2024. [Under review].
[2] E. T. K. Sang, D. S. M. Pandiani, and D. Ceolin, “Evaluating Locally Run Large Language Models on Toxic Meme Analysis,” 2025. [Under review; to be uploaded to arXiv].
[3] D. S. M. Pandiani, E. T. K. Sang, and D. Ceolin, “OnToxKG: An Ontology-Based Knowledge Graph of Toxic Symbols and Their Manifestations,” 2025. [Under review; to be uploaded to arXiv].
Please contact:
Davide Ceolin, CWI, The Netherlands
