An AI that simply knows all your company documents and can answer any question about them sounds like the perfect solution for every company with accumulated knowledge. Yet there is a surprisingly large gap between that vision and the reality of ChatGPT, Claude and the like. Anyone who understands where generic AI solutions fail with large volumes of company documents makes better decisions for their own digital infrastructure.
Introduction
Imagine a mid-sized company: a machine builder with 150 employees. Over the years it has produced thousands of documents – maintenance logs for every delivered system, technical data sheets, field service reports, internal emails about customer problems, training materials and contracts with suppliers. Altogether that quickly adds up to tens of thousands of files, spread across file servers, an old wiki and countless mailboxes.
Management hears about ChatGPT and Claude and has an obvious idea: why not simply give the AI access to all these documents? Employees could then ask questions like "Which spare parts were replaced most often on system X over the last three years?" or "Show me all open complaints from the last quarter" – and the AI would distil the right answer from the company's entire knowledge.
In theory, that sounds like the perfect use case for generative AI. In practice, it quickly becomes clear that cloud AI services like ChatGPT or Claude are not built for exactly this use case – thousands of your own documents as a reliable basis for answers. The reasons lie in four areas: the limited context length, the ongoing API costs, the necessary foundation of storage and index (RAG), and fundamental limits of this technology for complete or statistical analyses. This article goes through each of these points with concrete examples – and shows why, in the end, an application built specifically for your own use case is often the more practical path.
The context window: a hard limit
Language models like ChatGPT or Claude process text in what is called a context window – a limited "working memory" that must hold all the information of a request: the question itself, the conversation so far and all documents supplied with it.
Even very large context windows of 100,000 to 200,000 tokens correspond to roughly 150 to 300 pages of text. That may be enough for individual reports or contracts – but with thousands of documents it is far too little.
To get around this problem, a special AI database is built that makes your knowledge efficiently usable. You can read more about this topic here (in German).
Example: The machine builder from our example has over 3,000 maintenance logs alone, each about two to three pages long. That is already more than 6,000 pages – around twenty times a large context window, for this one document category only. Add data sheets, emails and service reports and you quickly reach tens of thousands of pages. This volume cannot fit into a context window at the same time under any circumstances, no matter how cleverly you shorten or summarise the texts.
This challenge is made worse by the limited capabilities of generative AI. To understand the risks, it is important to know that ChatGPT can give wrong answers.
There is also a more subtle problem: models do not treat every part of a long context equally well. Information that sits "in the middle" of a very long prompt is overlooked or weighted worse more often than information at the beginning or end. Even if you managed to squeeze all relevant documents in somehow, answer quality would suffer – individual important details could get lost in the "noise" of the mass.
This is also why an LLM should not be asked: "How many contracts do we actually have with company XY?" The LLM does not know all your documents. Instead, the documents are pre-selected in an AI preliminary stage via a vector database, and only a few of them are used for the AI's answer. For a counting question to be answered correctly, a suitable AI system (not the same as an AI model!) has to be built that supports this kind of question.
API costs: you pay per token, anew with every request
Cloud AI providers charge per token – both for input (everything you send to the model) and for output (the generated answer). The problem: with every new request, the relevant context has to be sent again, because the model has no memory between two requests.
Example: Suppose that with every request you want to send at least the 20 most relevant document excerpts to enable a well-founded answer – roughly 15,000 to 20,000 tokens of context per request. With 150 employees making an average of five requests per working day, that adds up to around 750 requests a day. The input context alone then totals more than ten million tokens per day – day after day, month after month. Depending on model and provider, this results in ongoing costs of several hundred to over a thousand euros per month, just for repeatedly sending context – without any "new intelligence" being created, merely the same background being processed over and over.
These costs scale linearly with usage. The more successful and popular such a system becomes in the company, the more expensive it gets – an economic model that can quickly spiral out of control if it is not thought through from the start.
Storage, index and RAG: the invisible foundation
To make large volumes of documents usable at all, you need a technical foundation that cloud AI does not bring along itself: Retrieval-Augmented Generation, or RAG for short.
My name is Klaus Meffert. I have a doctorate in computer science and have been working professionally and practically with information technology for over 30 years. I also work as an expert in IT & data protection. I achieve my results by looking at technology and law. This seems absolutely essential to me when it comes to digital data protection. My company, IT Logic GmbH, also offers consulting and development of optimized and secure AI solutions.
