AI systems deliver unpredictable results. The problem cannot be solved for AI systems with a general purpose (ChatGPT), but it can be solved for company-owned AI systems with a specific purpose. A transparency obligation can be derived from the GDPR alone. Operators and providers of AI systems must fulfill additional obligations under the AI Act.

Introduction

How can you make an AI system transparent? The answer to this question for general AI systems is: not at all. This is because these general systems, including ChatGPT, work on the basis of neural networks. How this network works is well known. If you were to write down a formula describing the network, nobody would understand it, let alone be able to read it properly.

The GDPR prescribes in Article 5 the obligation of transparency when processing personal data. This applies to all AI systems that process personal data. These are all systems into which personal data have flowed during training or at user input (often via a prompt). This is a fact that the Hamburg Data Protection Officer has (only?) negated in a dangerous way.

In Art. 5 Sec. 1 lit. d GDPR it is required that data be factually accurate, i.e., correct. This applies to all personal data in AI systems. At the latest at the time of inference, i.e., when an AI system generates an output, this legal provision should be fulfilled.

The AI-Regulation (AI Act) defines obligations that are particularly incumbent on providers of AI systems. Special obligations are imposed for high-risk AI. This type of system should be an exception in practice.

Most companies that use AI systems are operators. For operators, there are far fewer obligations than for providers. An operator is a company or organization according to Art. 3 No. 4 AI-VO, if one “uses an AI system at their own responsibility.” Everything beyond that falls under the provider term (Art. 3 No. 3 AI-VO).

An idea for increasing the transparency and documentation of AI systems came to the author at a meeting of the AI expert group of the State Data Protection Commissioner of Lower Saxony, of which the author is a member. The author has also previously published a book on test-driven software development.

On the one hand, transparency is an external presentation of AI results. However, internal transparency, i.e. for the operator of an AI, is almost more important: How does the AI work? What results does it produce?

Proof of the correctness of AI outputs

In general, it is not possible to completely ensure that an AI only spends correctly. However, it is possible to come close. Before a suggestion is made in this regard, an example is given by the very good DEEPL translator (from Germany!), who uses AI himself and, just like any other AI system, sometimes makes mistakes:

DEEPL was asked to translate a text containing a monetary amount. DEEPL translated €1,050.00 in such a way that the euro figure was replaced by a pound figure. This is obviously wrong. For anyone who wants to try it out for themselves: It depends on the overall text! This has been partially obscured in the screenshot above because it was semi-sensitive information. You will probably get a correct result if you only enter the last sentence in DEEPL. But if the preamble text is different, the error may occur. This alone shows how non-transparent AI systems work.

Errors can therefore not be avoided. How can you still fulfill your duty of transparency and ensure the correctness of AI outputs as much as possible?

The answer is: Through test cases.

Test cases are pairs of actual inputs and target outputs. A test case consists of an actual input and an actual output that is accepted as good. The AI Regulation (AI-VO) has apparently even taken this into account:

This is because Art. 3 No. 53 of the AI Regulation defines the term “plan for a real-life test” as “a document describing the objectives, methodology, geographical, population and time scope, monitoring, organization and conduct of a real-life test”.

The No. 56 of the same article defines AI competence as “the skills, knowledge and understanding that enable providers, operators and those affected, taking into account their respective rights and obligations within this regulation, to use AI systems competently and become aware of the chances and risks of AI and possible damage it can cause

With the help of test cases, operators (and even more so providers) can become more aware of the opportunities and risks of the AI they operate or offer.

These Deepfakes can also be created in accordance with Article 3 AI-VO No. 60. Here it is about a “content of images, sound or video generated or manipulated by AI that resembles real persons, objects, locations, institutions or events and would falsely appear as authentic or true”. When using image models, one would ensure that inputs targeting real people and aiming to defame them are best possible identified and prevented. In any case, it can already be documented with the help of test cases where (still) the weaknesses of the AI system lie.

The obligation for providers of non-high-risk AI systems to evaluate their own system, as set out in Art. 6 (4) of the AI Regulation, can also take place via test cases.

The risk management system referred to in Art. 9 (1) of the AI Regulation can be underpinned very well with the help of test cases.

Numerous other provisions in the AI Act impose obligations on providers and operators of AI systems that can be served by documented test cases. These include:

• Art. 11 (1) AI Regulation: technical documentation of a high-risk AI system
• Art. 17 AI-VO: Quality management
• Art. 53 AI Regulation as a whole: Obligations for providers of general purpose AI models
• Art. 91 and 101 of the AI Regulation can have negative consequences for AI providers if their documentation does not appear to be sufficient.
• Art. 4 of the AI Regulation also requires operators to ensure that their employees have sufficient AI competence.

Examples of test cases

What does a test case look like? Here is an example of a language model designed to answer questions:

These two test cases alone make it clear that it is not a good idea to want to operate a universal chatbot. Nobody will be able to write a sufficient number of test cases to test all the questions in the world, i.e. to ensure quality.

An AI system should therefore be tailored to a use case or specialist domain. This not only makes it easier to fulfill the obligations of the AI Regulation, but also improves the quality of the results. The quality of specialized chatbots, for example for the construction industry, is significantly better than anyone will be able to achieve with ChatGPT.

The number of test cases should be reasonably high. Additional test cases can be added gradually. In particular, if an AI response to a user question was not satisfactory, it is advisable to include a test case for this. The test case then serves at least as documentation, but preferably, as a basis for optimizing the AI system and using the test case to check the success of the optimization.

When building a knowledge system (as one of many possible AI systems), there is a trick to significantly increasing the quality of results. The so-called RAG approach only leads to limited success and to the top. What this is all about will be described in a later article,

How can test cases be run through?

Once test cases have been set up, they must be run through. In concrete terms, this means:

1. The defined “actual” from a test case is presented to the AI as input.
2. The AI responds.
3. The AI response is compared with the “target” from the test case.

There are several options for comparing the AI output with the expected optimum from the test case:

1. AI analysis with semantic similarity comparison
2. AI analysis via a language model (or several!)
3. Conventional analysis (example: “No” in the target and “Yes” in the AI output contradict each other)
4. Mixture of all methods (recommended)

The alternative mentioned in case two of using several language models simultaneously to analyze the test results works very well with open source models. The costs are always the same, namely zero (plus fixed operating costs for the server). If ChatGPT were used, the costs would be quite high in the long term.

With these analysis methods, test cases can be analyzed to a large extent automatically. The human then checks the result and can write a conclusion in the documentation.

Conclusion

The functionality of AI systems can be documented with the help of test cases and thus made transparent. Of course, transparency also includes information about the architecture of the AI system. This can easily be done if you operate the AI yourself. In the case of third-party systems such as ChatGPT, you have to rely on the information provided by the provider (OpenAI or similar).

Test cases can also be used to check and improve the correctness of AI outputs.

Test cases therefore have several advantages and great benefits. They are often created quickly. With AI support, test cases can even be derived automatically. The human test case creator thus receives a very good template for test cases and can fix them with a fraction of the manual effort that would otherwise be required.