The Bluesky Data Analyzer
Authors AI-CODE partner: Bahia Albrecht, Andy Giefer, Kay Macquarrie – Deutsche Welle
Hateful speech is labeled into four categories: Racism, Religious, Sexism and Sexual Orientation. In addition, non-hateful and neutral posts are represented. The example shows the categorized messages from a single account.
The analysis focused on understanding the overall tone of interactions, with a particular interest in distinguishing between friendly exchanges and harmful posts. The good news from our findings is that, by and large, most posts on Bluesky appear to be non-hateful, with friendly interactions far outweighing instances of hate-filled content. However, while these results are encouraging, the process of categorizing hate speech is still in development. We relied on an external model to label and classify content along axes such as religious, racial, sexist, and sexual orientation-based hate, but the model is still being refined. As such, we must approach the results with a degree of caution, acknowledging that the accuracy of these classifications may improve as the model matures. Also, this analysis is based on a limited data sample collected over a 6-hour period in May 2024, which may not be representative of broader trends or patterns.
Nevertheless Bluesky’s rising numbers in users and their growing numbers in interactions were worthwhile looking into it. Wondering what methods we used to explore this? Let’s take a closer look at our approach.
How we did it
- We created a tool which allows us to organise a large number of posts and label them with a model we used from our project partner ATC in Greece. We named the tool: The Bluesky Data Analyser (beta)
- A bit more than 260,000 messages were recorded using the Bluesky firehose API in a 6 hour-long session
- Next, we excluded those posts that were identified as neutral – these are messages that did not contain strong enough signals to allow the model a clear classification. This leaves us with 165,000 messages
- Then we excluded those messages that the model labeled as distinctly non-hateful. 13,000 messages remain. So every 20th post was labeled as hateful. And we looked deeper into them with the tool to detect hate speech posts and accounts.
The proof of concept: The Bluesky Data Analyser
The UI enables an immediate overview of the data and its automatic categorization. Above a quick YouTube Walkthrough
The Bluesky Data Analyser enables us to look at a considerable number of posts from Bluesky. After filtering out posts which are neutral or non-hateful, 13.689 posts remain. These can be further analyzed by scrolling through “Top Posts” or by clicking into the list of “Top Accounts”. The respected posts are shown in the window above including their likeliness of being hateful on a scale from 0,00 (not very likely) to 1,00 (very likely).
On the right-hand side a five-dimensional Venn diagram is shown. It provides a good indication of where the post-categories are automatically grouped: Non-hateful, racism, religious, sexism and sexual orientation. The pie chart at the bottom helps to put the data into perspective.
What we found out
- The proof of concept helps us to connect technology to journalistic needs and further improve the work
- It detects hateful posts and identifies accounts that publish a significant number of hateful posts
- The test shows that more development is needed. Both into the UI for better usability (e.g. detailed interaction and data display) but also into the labeling of content, as the hate speech detection tool is still under development.
What comes next?
The ultimate goal is to identify superspreaders of hate speech. We expect the hate speech detection model to become more accurate with further training. At the same time, with inspiration from the work of our project partner Universidad Politécnica de Madrid, we intend to combine hate speech detection with an analysis of message propagation. This way, we can identify those hate speech messages that show a high popularity and therefore can have a meaningful impact on the Bluesky community.