Semantic Search: Finding Knowledge in Company Documents with AI

Thanks to AI, semantic search finds information and knowledge far better than before. Finding instead of searching is the new solution, and it also offers the highest level of data security. Instead of simple comparisons, it uses the meaning of search terms and company documents. This works not only very well, but also very fast.

Introduction

An intelligent search engine for company documents, with AI and full data control. Microsoft and OpenAI do not get your data.

Search engines have long been able to do fuzzy search on text. Search always rests on the following basic mechanism:

1. A number of given documents (texts) are read in. From them, a so-called search index is built. The search index is a database and has nothing to do with an "index" in the sense of a ban.
2. A user enters one or more search terms.
3. The search engine now compares every document in the search index with the search term.
4. The documents that scored best in the comparison are returned first as hits.

Users often make typos when entering search terms. Many queries also use different words than those that appear in the documents in the search index (synonyms).

Sometimes a user also enters a short question, or a statement that is meant as a question. Examples: "What highest mountain" or "highest mountain". You can already guess that the capitalization is often off.

Search engines have long met these challenges with sophisticated mechanisms. Searching by n-grams in particular is successful. An n-gram is a word fragment of length n. The variable n can take any value and sensibly starts at 2.

This compensates not only for spelling mistakes but also for word variants with slightly different spelling. A synonym dictionary ensures that terms with the same meaning are recognized correctly.

Questions asked by the user can usually simply be trimmed down. The input "What is the highest mountain" can thus be converted to "highest mountain". The answer is then found for that.

However, such methods, however sophisticated they may be, fail at searches for knowledge.

Here is an example:

Ideally, a similarity comparison of these four texts returns the following pairs as most similar:

This pair similarity is defined here as the gold standard. Further below, it is described how a different standard can be defined.

A character-by-character or word-by-word comparison would have identified "Hello mother" and "Hello Mother Nature" as most similar. From a semantic point of view, that is wrong according to the defined standard (the standard can be defined by each AI operator themselves). What is correct is that "world" and "Mother Nature" have a much more similar meaning than "mother" and "Mother Nature".

Semantic search with AI

This is exactly where so-called semantic search comes in, which has been possible since the discovery of the transformer in 2017 and the arrival of powerful hardware.

The principle of semantic search is completely different from that of previous search engines. To find texts for a search query, no characters are compared, but meaning is!

But how does that work? How can meaning be compared? Only humans could do that. True, but since 2017 at the latest, the function of human intelligence has been decoded (see Definition of intelligence and artificial intelligence). Many see it differently but cannot explain why that should be so. See also the common misconceptions about AI.

The question of how meaning can be compared can be reduced to the following question:

That is already easier than the original question, but still not a concrete solution.

The concrete solution is this: an AI learns which pieces of information (texts) are very similar, somewhat similar, or little to not at all similar. This happens, surprise, just like with humans, namely through upbringing. Upbringing is based on exemplary behavior, on information and on correction.

Applied to AI, this means that examples are provided. These examples contain both cases of information that are very similar and cases that are less similar or not similar at all. For each case, it is stated how great or small the similarity is. This can be purely qualitative, like a parent raising a child: "That was very good!" or "No, you will be punished for that!".

This reward and punishment mechanism (when parents raise children) can also be found in AI training. The AI model is created by using a so-called loss function. The loss is the part that is missing for perfection.

Turning information into numbers

After a few million or billion examples found on the internet, an AI model for semantic search emerges. This model produces so-called embeddings for information. An embedding is a series of numbers.

Here is an example of the numbers the AI model produces from the input "Hello Mother Nature":

[0.05151367 -0.03955078 0.08984375 … 0.02124023 -0.03076172 0.0100708],
…
[0.06396484 -0.04321289 0.09472656 … 0.02441406 -0.028442380.01434326]

The square brackets indicate groups of numbers. Each group consists of several numbers. This is also called a vector. The three dots between the numbers indicate that some numbers and groups were left out because there are too many. Overall, a vector therefore consists of a great many groups and numbers.

Why are there several groups of numbers and not just one? Quite simply: because it makes sense. A sentence can consist of several words and not just one. After all, semantic search is meant to compare words and their meanings.

A group contains several numbers because that allows more information to be represented. To put it figuratively: if only one number were used for the word "mother", the context-dependent meaning of the word "mother" in the phrase "Mother Nature" could not be represented well.

Back to the question of how knowledge can be found in company documents or on the internet.

The answer: knowledge is found by converting the meaning of the search query into a long series of numbers and doing the same for all the knowledge documents to be searched. Then the series of numbers for the search query is compared with the series of numbers for all knowledge documents. To make this fast, vector search engines were introduced. These are databases that precompute vectors for knowledge documents and can then compare them very quickly with another vector (for the search query).

How are vectors compared?

There are several ways to do this. Cosine similarity is one of them. A vector can be seen as a directed line in space. The similarity of two vectors is determined by their distance in space. The space here is not three-dimensional but multi-dimensional. What cannot be imagined in real life can be handled by mathematics.

If you compare the four example texts above with each other (every text is compared with every text, including itself), the following similarity matrix results:

                   Hello world  Hello mother  Hello Mother Nature  Hello Hans
Hello world        1.00         0.59          0.73                 0.61
Hello mother       0.59         1.00          0.63                 0.47
Hello Mother Nature 0.73        0.63          1.00                 0.46
Hello Hans         0.61         0.47          0.46                 1.00

A note on this, showing that a new IT age has begun (just not in Germany):

The highest similarity value is 1.0 = identical. This value appears when a text is compared with itself. Comparing "Hello mother" with "Hello Mother Nature" gives a similarity value of 0.63. By contrast, the similarity between "Hello world" and "Hello Mother Nature" is 0.73, which is higher than 0.63.

The higher similarity in meaning between "world" and "Mother Nature" versus "mother" and "Mother Nature" was recognized. This example is only meant as an illustration. Someone may see the similarity of these example pairs differently. In that case, it is possible to adjust the AI model and carry out fine-tuning. That way the AI model learns (if needed) that the similarity between "mother" and "Mother Nature" is higher than between "mother" and "world".

Features of an AI search

Searching for knowledge with the help of AI offers numerous advantages.

The problem of synonyms is solved automatically, because learned meanings are compared, not words, letters or n-grams.

Anyone who wants to use AI search can draw on a mature semantic AI model. It is as if you hired a fully trained new employee for your company.

The execution speed of such a search is very high. Millions of documents can be searched almost in real time. This is possible because the documents are vectorized once, that is, converted into numbers. This lengthy process only has to be run once.

A short test on a laptop showed that a similarity search called through a self-programmed API over the network was done within a fraction of a second.

The quality of the results is very good. Companies and public authorities often use special vocabulary and turns of phrase. These can be taught to the AI model with very few examples. Very few examples really are enough, even though more examples are better. Again with the help of AI, your examples can be expanded synthetically. So 10 examples quickly become 200.

Fine-tuning a semantic AI model can already take place on the laptop on which this text is being written.

Semantic search can be run locally on your own company servers, or on an inexpensive rented server from a German provider with a German data center and a German data processing agreement.

Use cases

AI search opens up many possibilities. For example:

• Better online search for websites (currently being finished by the author for a data protection publisher).
• Searching company knowledge and above all finding it!
• Detecting duplicates. Also very exciting for scanned documents, because a scan can potentially distort characters, or some scans are cut off.
• Also possible for images! Which images are very similar to the current image?
• Classifier for user input or documents: is it a question about the weather or about company knowledge? Is a document a construction plan, a technical specification, a legal treatise or a contract?

The possibilities are endless. AI search can be combined with other capabilities that AI offers. The synthetic generation of examples was mentioned above.

Conclusion

This article showed how powerful AI search is. The advantages are enormous. AI opens up undreamed-of possibilities.

Finding knowledge instead of searching for it is possible now. Not as a prototype, but as a mature technology.

For further use cases such as duplicate detection in scanned or digital documents, semantic search is also a godsend.

You can find more guides, model comparisons and practical examples in the overview of local AI.