Violent extremists are increasingly exploiting artificial intelligence to bypass content moderation systems on major social media platforms, a leading European violence prevention expert has warned.
The Changing Face of Extremism
Judy Korn, Managing Director of the Violence Prevention Network, revealed the alarming trend during TikTok's Trust and Safety forum in Dublin. She stated that while Islamist and far-right extremism continues to rise, the most significant increase is occurring in what she termed 'unclear violence' and nihilistic violent extremism.
This nihilistic extremism presents a particular challenge for technology companies because it isn't driven by any specific political or religious ideology. "It is driven not by a specific ideology, but simply 'destruction and chaos'", Ms Korn explained. This fundamental lack of recognisable ideological markers means this content often fails to trigger traditional automated content filters designed to catch more conventional extremist material.
How Extremists Are Exploiting Technology
The threat is compounded by extremists' adoption of generative AI tools. Ms Korn highlighted that "Generative AI is much more clever than violent extremists, because it learns faster than a human being how to produce content that conveys the desired message without violating regulations and without violating the platform guidelines."
This technological arms race is occurring alongside another worrying demographic shift. According to UK counter-terrorism police data:
- More than 50% of people referred to the Prevent deradicalisation programme are under 18
- One in five people arrested for terrorism offences is legally a child
Platforms Fight Back With New Measures
In response to these evolving threats, TikTok announced several new safety initiatives at the Dublin forum. For users in Germany searching for terms related to extremism, the platform will now display new educational prompts designed to steer people away from harmful content.
To address the specific challenge of AI-generated content, TikTok is implementing an invisible watermark system to help users identify when content has been created using artificial intelligence. Jade Nester, TikTok's Director of Data Public Policy in Europe, explained the industry-wide challenge: "With these methods, the labels might get removed if you download the content re-uploaded or re-edited elsewhere. These invisible watermarks help us address this by adding a robust technological watermark that only we can read."
The company also revealed plans for a gamified "wellness hub" focused on mental wellbeing, featuring meditation guides, affirmations, and tools to help users manage their screen time more effectively.
