Retrieval is All You Need

For the last few years we have shifted from AI models to AI systems. Any commercial product branded as AI goes much beyond its core Large Language Model and into a Frankenstein of 'tools'. The industry has, thus, given up on emergent capabilities as a path to AGI, at least for the time being. This begs the question: if we need Retrieval Augmented Generation for informed outputs (that is, do a semantic search, then feed it to an AI to rephrase), than what good is the generation part? In other words, a good search engine that yields accurate and succinct information to unstructured inputs is already most of the way there (indeed, arguably better, as it is not vulnerable to hallucination).

Furthermore, there is evidence that BERT is already powerful enough to classify text just as successfully as GPT-4 (Wang et al., 2024). While semantic retrieval differs from classification, the paper demonstrates that BERT's vector space already contains enough detail to reflect complex textual relationships and meaning. This means that an increased focus on improving more 'basic' LLM functionalities, such as classification and retrieval can be a more powerful sphere of development than simply scaling up generative models while hoping they develop consciousness.

Quantifying Qualitative Analysis

It is in this line of though that I decided to test my hunch against real world data. In-depth qualitative analysis has been the de-facto way of understanding large text corpora since the existence of such corpora. However, this process does not scale well: a single researcher has to compile in their head (and/ or notes) a whole corpus and then gradually make sense of it, while also comparing it to the literature. This process becomes more unfeasible that larger the corpus gets. Nowadays, we have the ultimate corpus of text: the Internet. Its sea of text is so vast, that it is impossible for a human to cover even a part of it in her lifetime. Thus, analysing the Internet already presupposes quantitative methods as the default. This has naturally pushged in-depth analysis to the sidelines, as any serious exploration of a blog or a website fundamentally leaves open the question to what extent is this space at all relevant to the whole network. In a way, we have very practically faced a fundamentall pitfall in Latour's ANT theory: any node of a system is already a system in itself and any system is part of a wider system.

What LLMs offer to this field is an (assumed) way to vectorize meaning. We don't yet know how in-depth that meaning is, but we do know that it is there in some form. This, therefore, provides an opportunity to work towards large-scale in-depth analysis.

My attempt

Thus, I began work on my proof-of-concept for a tool that allows researchers to collaborate with LLMs to analyse large text corpora. We possess a powerful retrieval tool that we are not paying enough attention to, feeding its result directly into a generative model's context without diving deeper. Thus, I started work on a dashboard that allows a researcher to explore a textual corpus through its embeddings, gradually making sense of large scale data and, hopefully, gainign a better understanding of how high-dimensional spaces are constructed. You can find more about this here.