Personal Data in AI: How Systems “Learn” It – and Why That Is a Problem

Artificial intelligence is no longer a topic of the future. Language models answer customer enquiries, image recognition systems check ID documents, algorithms decide on creditworthiness. What seems routine in everyday life rests on a technical foundation that raises considerable data protection questions: AI systems learn from data – and very often from personal data. This makes personal data in AI one of the central compliance topics of our time.

This article explains how modern machine learning works, where personal data enters the process, what requirements the GDPR sets and what this means in concrete terms for companies, developers and users.

What “learning” means for AI systems

The term “artificial intelligence” is broad. In practice today it mostly means machine learning (ML) – and above all a subfield called deep learning, which is based on so-called neural networks.

The basic principle: an algorithm analyses very large amounts of example data and recognises statistical patterns in it. Based on these patterns it can later classify, predict or generate new, unknown inputs. A language model such as GPT-4 has processed billions of texts in this way and learned how language works – which words follow each other, which concepts belong together, how questions are typically answered.

What matters: the model does not store texts in the classic sense. It compresses the patterns gained from the data into millions or billions of numerical parameters – the so-called weights of the neural network. These weights are the actual “knowledge” of the model.

Three phases in which data plays a role

Training: The model is trained on a large data set. Personal data contained in this data set influences the model's weights – potentially permanently.

Fine-tuning: An already trained base model is refined further with more specific data, for example internal company documents, customer data or technical literature. Here too, data flows into the model parameters.

Inference: The finished model is used in operation. Users submit queries, upload documents or send images. These inputs are processed – and, depending on the architecture of the system, possibly stored, logged or used again for training.

All three phases are relevant for data protection. Often, however, only inference is noticed, while training and fine-tuning run in the background – frequently without sufficient documentation.

Personal data in AI training data: what is hidden in it

Anyone training large language models needs large amounts of text. The common practice: scraping the public internet. Common Crawl, Wikipedia, books, Reddit, GitHub, news archives – all of it flows into the training data.

That sounds like public data. And yet it contains massive amounts of personal information:

Names and contact details from forums, comment sections, legal notices and social networks.

Health data from public experience reports, patient forums or scientific case studies. See also Article 9 GDPR

Biographical details from CVs, job application pages, personal blogs.

Email addresses and phone numbers that were published somewhere on the web.

Court judgments and official documents that contain the names of data subjects.

Particularly delicate: data that was publicly accessible at a certain point in time may have been deleted since – but live on in the weights of an AI model.

The problem of data minimisation

Art. 5(1)(c) GDPR requires that personal data be limited to what is necessary for the purpose of the processing (data minimisation). This principle is in fundamental conflict with the approach of building training data sets that are as large and diverse as possible. The bigger the data set, the better – that is the empirical principle behind modern large language models. Data minimisation and “more data = better results” can hardly be reconciled as long as personal data is not reliably filtered out of the training data.

Legal basis for training: an unsolved problem

The GDPR requires a legal basis under Art. 6 GDPR for every processing of personal data. For training AI models on publicly available data, the legal position is anything but clear.

Consent (Art. 6(1)(a) GDPR)

Consent from the data subjects is, as a rule, not available in web scraping. Nobody who writes a forum post thereby consents to the use of their text for training a commercial AI model. Even if the data is publicly accessible, that does not mean that consent was given for use in machine learning.

Legitimate interest (Art. 6(1)(f) GDPR)

Many AI providers base their training on legitimate interest. The argument: developing useful AI systems outweighs the interests of the data subjects. Whether this holds is disputed among data protection authorities. Legitimate interest is no free pass – it requires a serious balancing of interests that takes the data subjects' reasonable expectations into account.

Scientific research exemption (Art. 89 GDPR)

The GDPR provides eased conditions for scientific research. However, this exemption is limited to genuine research projects and does not apply to commercial product development.

Is public availability a legal basis?

There is no “publicly available = free to use” in the GDPR system. Publishing a text on the internet does not constitute consent for every conceivable processing purpose. Data subjects keep their rights even if their data is public.

Memorisation: when AI models learn data “by heart”

A particularly critical data protection problem is so-called memorisation. This describes the phenomenon that a language model does not only internalise certain training data as an abstract pattern but – in rare cases – can reproduce it almost verbatim.

Studies have shown that large language models can reproduce personal information from their training data in response to targeted queries: names combined with addresses, phone numbers, medical details or excerpts from private emails that accidentally ended up in the training data set.

This is problematic for several reasons:

Violation of the right to erasure (Art. 17 GDPR): If a language model can reproduce personal data, that data can no longer be effectively deleted – at least not without retraining the model.

Data breaches: If a model in productive use outputs one user's data to other users, this is a reportable personal data breach under Art. 33 GDPR.

Unpredictability: Neither the provider nor the user can know in advance what information a model may have memorised. This makes a reliable risk assessment difficult.

Machine unlearning: the technical answer – not yet mature

Machine unlearning is discussed as a solution: methods intended to make it possible to “remove” certain information from a trained model without having to retrain the entire model. First approaches exist, but a practical, legally secure solution is still missing. The GDPR right to erasure can only be fulfilled incompletely with today's AI technology.

Fine-tuning with company data: special risks

Many companies today do not only use ready-made AI models but adapt them with their own data – through fine-tuning or by connecting them to internal knowledge bases (so-called RAG systems, retrieval-augmented generation).

This considerably increases the usefulness of the AI. But it also multiplies the data protection risks:

Customer data in the model: If customer communication, contract documents or CRM data are used for fine-tuning, personal data flows directly into the model parameters – without the customers concerned knowing about it or having agreed.

Data transfers to AI providers: Fine-tuning with cloud providers such as OpenAI, Google or Microsoft usually means that training data is transferred to servers in the USA. This triggers the requirements of Art. 44 et seq. GDPR on international data transfers.

Missing data processing agreements: If an external AI service is used to process personal data, a data processing agreement (DPA) under Art. 28 GDPR is generally required. Many companies use AI tools without checking whether such an agreement has been concluded.

Confidentiality of inputs: With cloud AI services, user inputs are often used for model improvement by default. Anyone who enters personal or confidential data into such systems risks that this data flows into future models.

Automated decisions and profiling: Art. 22 GDPR

Another critical area: AI systems that make automated decisions about people. Art. 22 GDPR regulates the right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects.

Classic use cases:

• Credit scoring by ML algorithms
• Automated applicant selection (CV screening by AI)
• Insurance premiums based on behavioural data
• Price personalisation in e-commerce

Where Art. 22 GDPR applies, data subjects have the right to human review, to an explanation of the decision logic and to object. This confronts AI systems with a fundamental problem: many modern ML models – in particular deep neural networks – are black boxes. Their decisions cannot be explained in a way humans can follow. This contradicts the GDPR's transparency principle (Art. 5(1)(a)) and the information obligations under Art. 13/14 GDPR.

Explainable AI as an answer

The research field of Explainable AI (XAI) deals with methods that make the decisions of AI systems easier to understand. Approaches such as LIME (Local Interpretable Model-agnostic Explanations) or SHAP (SHapley Additive exPlanations) can provide explanations for individual decisions. But they are no panacea and do not replace fundamental transparency of the model.

The data protection impact assessment (DPIA) for AI systems

Art. 35 GDPR obliges controllers to carry out a data protection impact assessment (DPIA) before processing that is likely to result in a high risk for data subjects. AI systems often trigger this obligation – in particular when:

• systematic profiling of individuals takes place,
• special categories of personal data (Art. 9 GDPR) are processed (health data, biometric data, political opinions),
• automated decisions with significant effects are made,
• data is processed on a large scale.

A DPIA must assess the necessity and proportionality of the processing, identify risks for data subjects and describe measures to reduce risk. Many companies skip this step – often out of ignorance, sometimes under time pressure. This is a considerable compliance violation.

EU AI Act: the new regulatory layer

Alongside the GDPR, the EU AI Act has been coming into force step by step since 2024 – the first comprehensive AI regulation worldwide. It classifies AI systems by risk class:

Unacceptable risk: Systems that manipulate people, enable social scoring by public authorities or real-time biometrics in public spaces for surveillance purposes – prohibited in principle.

High-risk AI: Systems in areas such as education, employment, critical infrastructure, law enforcement, lending or healthcare. Strict requirements apply here: risk management, data governance, transparency, human oversight, accuracy and robustness.

Limited risk: Transparency obligations, for example for chatbots – users must know that they are communicating with an AI.

Minimal risk: No special requirements.

The EU AI Act and the GDPR complement each other but also overlap. Companies have to keep both sets of rules in view – and this poses considerable challenges, especially for smaller companies.

What companies concretely have to do

Anyone who uses or develops AI systems faces a clear catalogue of obligations. The most important measures at a glance:

Take stock of all AI uses: Which AI tools are used in the company? What data flows into them? Often even this basic overview is missing.

Check the legal basis: For every AI-supported processing operation, a reliable legal basis under Art. 6 (and where applicable Art. 9) GDPR must exist.

Conclude a DPA with AI providers: Anyone using a cloud AI service that processes personal data generally needs a data processing agreement. Many providers offer standard contracts – but they must actually be concluded and documented.

Safeguard data transfers: For US AI services it must be checked whether a reliable transfer basis exists (DPF certification of the provider, standard contractual clauses).

Carry out a DPIA: For high-risk AI applications, a data protection impact assessment is mandatory.

Ensure data subject rights: Access, rectification, erasure – companies must be able to fulfil these rights also with regard to AI-supported processing.

Train employees: Who may enter which data into AI tools? Clear policies and training are indispensable.

Update the privacy policy: Every AI-supported processing operation involving personal data must be made transparent in the privacy policy.

Special categories of personal data in AI systems

Art. 9 GDPR protects particularly sensitive categories of data with increased requirements: health data, biometric data, genetic data, data on political opinions, religious beliefs, trade union membership or sexual orientation.

AI systems touch these categories more often than one might first think:

Facial recognition is based on biometric data – in principle a special category under Art. 9 GDPR, for which explicit consent or a specific legal basis is required.

Health AI naturally processes health data. This applies not only to clinical AI systems but also to wellness apps, fitness trackers or AI-supported symptom checkers.

Sentiment analysis and emotion detection: Systems that infer emotional states from speech, facial expressions or text can indirectly allow conclusions about health conditions or political beliefs.

Inference of sensitive attributes: Even if a model was not explicitly trained on health data, it can infer sensitive attributes from other data. An algorithm that infers a pregnancy from purchasing behaviour is in effect processing health data – even if it does not do so explicitly.

Conclusion: AI and data protection – not an unsolvable conflict, but real effort

Artificial intelligence and data protection are not fundamentally contradictory. But privacy-compliant AI requires more than good will. It requires that personal data be treated as an asset worth protecting from the very design of an AI system – following the principle of privacy by design (Art. 25 GDPR).

This means: data minimisation already when selecting training data, clear legal bases for every processing step, transparent documentation, a DPIA where required, DPAs with service providers and a privacy policy that describes AI-supported processing completely and understandably.

Anyone who follows this path consistently benefits not only from legal certainty. Privacy-compliant AI strengthens the trust of customers, partners and authorities and is therefore also a competitive advantage in the long run.