The rapid rise of AI chatbots such as ChatGPT has revolutionized the business world. But while the technology offers impressive possibilities, AI chatbots for business also carry considerable risks regarding data security, data protection and legal requirements that companies should know about. The AI Act adds further obligations.
Introduction
This article complements Datenschutz Deluxe podcast episode #34: Using chatbots: what for, and what better not for? (podcast in German).
Understanding the limits of AI technology
Artificial intelligence has undoubtedly passed the Turing test – in many use cases there is hardly any perceptible difference between human and machine. Yet AI systems share one decisive trait with humans: they are unreliable.
The key difference lies in specialization. While highly specialized human experts can work extremely reliably in their field, publicly available chatbots are general intelligences. They can do many things – but nothing perfectly.
The problem with legal texts
Use in legal applications is particularly problematic. Only a small percentage of ChatGPT’s training data is German text, and legal documents are again significantly underrepresented within that share. Legal texts are characterized by specific vocabulary, much longer sentence structures and highly nested clauses – properties general AI models are not optimally trained for.
Precision is indispensable in legal texts. The vague wording AI systems often produce is in direct contradiction to the precise demands of legal documents.
Data protection challenges
The core problem: personal data
When companies use chatbots like ChatGPT, massive data protection problems arise as soon as personal data is involved:
In statements of claim, for example, the parties’ names appear not only in the introduction but can show up indirectly in many places – through place names, regional particularities or specific phrasing. A surname like “Müller” can be an occupation, a company name or a person’s name – how is an AI supposed to tell them apart reliably?
In cold calling with AI voice assistants, it has been documented how people share their personal data with a chatbot without being told that:
- They are not talking to a human
- Their data is transferred to US companies
- This data may be used as training data
- The provider itself admits it cannot reliably filter personal data out of training data
The illusion of anonymization
So-called anonymization solutions achieve 90–95% reliability at best. That is not enough, because:
- Personal reference often arises from combining several pieces of information
- Regional narrowing, specific swear words or contextual information can make people identifiable
- Even humans do not readily recognize all personal data
Supposed anonymization solutions also do not work, as even moderately critical tests easily show. Anonymization cannot work reliably in principle, because general reasons speak against it (see the linked article, in German).
Neither humans nor AI can fully guarantee that all personal data is detected and removed.
Enterprise versions: no guarantee of legal certainty
Microsoft and OpenAI offer enterprise versions of their products that are said to be “compliant”. But – as always with advertising promises – caution is advised:
The enterprise version does not automatically mean legal certainty – it is merely the prerequisite for compliance to be possible at all. Conversely, the provider thereby publicly admits: Without the enterprise version, compliance is not guaranteed.
Reality shows: companies that unlawfully used Microsoft products to process customer data have been fined in recent years, with fines starting at EUR 50,000 and going up.
The data leak risk with custom GPTs
Recently, users can create their own GPTs with specific training data. Note that other users can extract this supposedly private information from the system through clever questioning. Intimate data uploaded in good faith is not as protected as many assume.
The better alternative to public AI chatbots for business: local AI systems
For companies and organizations there is a much safer and often more effective solution: your own, local AI systems.
Advantages of local systems:
The advantages of your own AI system are immense:
- Data security & data protection: No data transfer to third parties, full control over sensitive information
- Better results: Through fine-tuning on specific use cases, local systems can often deliver better results than general chatbots
- Trade secrets stay protected: No risk of proprietary information falling into the wrong hands
- Legal certainty: Complying with the GDPR becomes much easier
Data protection refers to personal data, whereas data security covers all types of data. In companies, sensitive data is not only data of employees or customers, but also trade secrets, confidential data (NDA) or privately held data such as study results or internal work instructions.
By the way, your own AI does not have to cost much. You will find more information in the article on AI apps.
Sensible use cases for public chatbots
Not all applications are problematic. Chatbots can certainly be used sensibly:
- In the private sphere for general queries without sensitive data
- For creative tasks such as advertising copy or image generation (with subsequent human review)
- To generate training data for your own models (with non-sensitive input texts)
- For synonyms and text variants for general content
Important: The result must always be checked – general chatbots are creativity tools, not reliability tools.
With a general chatbot, however, the question is how a company can generate significant added value with it. Specialized AI apps that solve concrete problems concretely are better. Examples of AI apps:
- Knowledge assistant: Query your (!) company knowledge and get reliable results, always with source references and the ability to jump straight to page 139 of your PDF document, where the AI took the knowledge for its answer from.
- Audio transcription: Yes, this already exists. Try the general solutions. Would you rather make 50 corrections or just 3? Is your surname or your colleague’s recognized correctly? This works better with your own AI app that uses open-source AI models.
- AI database: Link the knowledge from several IT systems that cannot talk to each other so far.
The possibilities are endless. Many problems can be solved very well and economically with AI! Get an assessment of the feasibility of your project.
Conclusion
Data security only exists when data stays where it belongs: with you. In addition, the purposes of data processing must be legitimate and legally sound. With local AI, all of that is no problem. Local AI systems only store data (such as user input) if you want them to.
At its core, data protection is a question of respect for customers, employees and business partners. Companies that carelessly feed personal data into public AI systems risk not only dissatisfied customers but also a data scandal.
The consequences can be serious:
- Fines under the GDPR
- Damages claims from data subjects
- Loss of trade secrets
- Reputational damage
The recommendation can therefore only be: anyone who wants to use AI in a business context should rely on local, self-controlled systems. They offer not only more security but often also better results for specific use cases.
Before you use external AI services, consult not only lawyers but also technical specialists in AI development. Legal documents do not solve technical problems, and the fundamental challenges of handling personal data in AI systems are still unsolved.
My name is Klaus Meffert. I have a doctorate in computer science and have been working professionally and practically with information technology for over 30 years. I also work as an expert in IT & data protection. I achieve my results by looking at technology and law. This seems absolutely essential to me when it comes to digital data protection. My company, IT Logic GmbH, also offers consulting and development of optimized and secure AI solutions.
