Introduction

In my previous blogpost I outlined an idea for a semantic query tool that does not blindly feed semantic retrieval into a larger generative model, but offers a human the tools to look back at the source texts and make sense of them. This is my Orthosphere dashboard.

What is Orthosphere?

The 'Orthosphere' is a term used around the Internet to describe a space of traditionalist Christian content online. It is a particularly Internet phenomenon in that it is a community of forums, blogs, users, influencers, YouTube channels all referencing each other, but also actively interacting with other corners of the Internet (often alt-right spaces and the so-called 'manosphere'[Ribeiro et al., 2021]). It is thus a great example of a the messiness of online discourse and a great test for the capabilities of a language model to explroe large text corpora. Furthermore, it offers more challenges than regular web spaces, as, on the one hand, alt-right adjacent actors tend to use dog-whistling and cryptic language (Quaranto, 2022), while on the other the Orthosphere is actively trying to resolve discursive conflicts between interpretations of religious history, scriptures, modernity, and modern conservatism.

The dashboard

But the Orthosphere itself, being a web space, is vast and I don't have the budget to generate embeddings for an actual chunk of the Internet itself. So, I decided to start with a single blog, the aptly named Orthosphere, which has been actively running since 2012. It offered anything from open bigotry to detailed analyses of philosophers and composers, thus posing a veritable challenge for any LM to navigate this landscape.

Query

I scraped the whole blog, split large paragraphs so they fit into BERT's context, and generated embeddings. I then tested a simple semantic query, and, to my surprise, it performed great out of the box: I could search beyond keywords. Simply writing 'gender-based discrimination' yielded specific examples of derogatory speech towards women, including attempts dressed in philosophical language. I immediately worked out a basic UI and this tool is now available, albeit a bit slow (I am running the whole dashboard, query embedding generation, and vector database on a pretty minimal Fly.io machine, so please be patient). It shows a shart of how the results are distributed over authors over time and then a 2d UMAP reduction of the vector space colored by author (an attempt to perhaps find some relationships between texts in the 2d space). The UMAP graph also offers a selection tool that filters the results below. Finally, as mentioned, there are the query results, ordered by proximity to the query (i.e. assumed relevance).

Topic Model

To offer a more bird's eye view of the vector space, I also used BERTopic to organize my embeddings into a topic model. BERTopic essentially runs HDBSCAN on the UMAP reductions and then applies TF-IDF to extract keywords. It offers a graph of all topics and how they are distributed across the UMAP projection. The user can select a set of topics from there and will see their respective keywords. Below, the dashboard shows a timeline of topic prevalence over time for the selected topics, as well as the most representative texts for each topic at the bottom.

Have fun!

I am happy with how the dashboard turned out and I hope it makes for an interesting thing for qualitative researchers to play around with. I definitely will do so and will make sure to upload my findings in this blog. I also hope to expand the project in the future, allowing it to map multiple websites and webspaces and, who knows, maybe even upgrade to multimodal embeddings to include images in the tool.

If you are interested in the source code, you can find it on my GitHub.