Privacy and data in AI products: what to ask your team for
Privacy in AI products depends on which data enters the model, how long it is kept and whether users know. A design guide with GDPR and AI Act basics.
What data does an AI product really generate?
When someone writes a prompt, the visible data is the text. But it is almost never the only data. An AI product usually generates, at a minimum:
| Data type | Example | Typical risk |
|---|---|---|
| Prompt | “Summarize Ana Pérez’s contract” | Contains names and data about third parties |
| Response | A summary with figures | Gets copied into other documents |
| History | Every conversation in the account | Piles up sensitive data nobody notices |
| Attachments | PDFs, screenshots, audio | Metadata and hidden content |
| Retrieved context | Internal documents the system searches | Can show things the user should not see |
| Telemetry | Which button was pressed and how long it took | Can reveal intimate behavior |
Build this inventory with engineering: if you do not know what is stored, you cannot explain it.
What happens to that data afterwards?
Users ask three questions, and the product should be able to answer them: who sees it, what it is used for and how long it lasts. A 2026 study of five consumer chatbots (ChatGPT, Gemini, Claude, Grok and DeepSeek) describes a structural opacity: providers can reuse chats for training, have human annotators review them and share them across other channels. The author notes that every major provider now trains on consumer chat data by default, and that most users cannot tell who may access their conversations or for how long (IAPP, March 2026).
For design, this means the training setting cannot be hidden in a settings menu three levels deep. It has to sit where the decision happens.
How do you design a notice nobody skips?
The most common mistake is solving privacy with a long text at the bottom of the page. A 2025 experiment with 104 participants on customer service chatbots found that static privacy statements and seals did not significantly reduce users’ concerns (Javadi and Kordzadeh, AIS eLibrary). Another study, from 2026, of interview chatbots compared three conditions: no editing, free editing of the transcript and AI-aided editing at the end. People overshare sensitive information in these chats, and AI-aided editing reduced the disclosure of personally identifiable information while keeping data quality and engagement intact (arXiv:2602.01387). It involved 188 people on a sensitive topic: treat it as evidence of direction, not a guarantee for any product.
A practical pattern that follows from these findings:
- Notice at the moment of sharing. One line next to the input field: “What you write may be saved in your history”.
- Review before sending. If the app detects sensitive data (a card number, a diagnosis), it shows a warning and offers to edit it.
- Review after the conversation. A summary the user can correct or delete, as in the interview study.
- Visible controls. A button to delete history and another to opt out of training use, both outside the terms.
What belongs to design and what to the legal team?
Design does not decide the legal basis for processing, but it does decide whether the user understands what we are asking. It helps to separate three decisions in the interface:
- Use to provide the service. Needed for the feature to work. Explain it in one line.
- Use to improve the product. Ask with a clear choice, not a pre-ticked box.
- Use to train third-party models. Requires a specific, reversible consent. See training data consent.
What do the GDPR and the AI Act require?
If your product processes data about people in the European Union, the GDPR sets rules that turn into screens. This is not legal advice; it helps you know what to ask the privacy team:
| Rule | What it says | What it means in the interface |
|---|---|---|
| Art. 5(1)(b) and (c) | Data is collected for specific purposes and limited to what is necessary | Do not ask for or keep more than the feature uses |
| Art. 5(1)(e) | Data is kept only as long as necessary | Show and respect a retention period |
| Art. 13(2)(a) | When collecting data you must say how long it will be stored, or the criteria used to decide | The period appears where the user types, not only in the terms |
| Art. 7(3) | Withdrawing consent must be as easy as giving it | If it is accepted with one click, it is withdrawn with one click |
| Art. 17 | Right to erasure, for example when data is no longer necessary or consent is withdrawn | The delete button exists and works |
The AI Act adds, in Article 50(1), that AI systems intended to interact directly with people must inform them they are dealing with an AI, unless that is obvious from the context. The obligation applies from 2 August 2026 (artificialintelligenceact.eu). The concrete design is covered in letting people know an AI made something.
Rules differ by country: for concrete obligations, check your country’s data protection authority and the person responsible for privacy on your team.
What data policy does the model provider have?
Before designing the notice, find out what happens to data in the model you use. One documented case: on 28 August 2025 Anthropic announced that on its consumer plans (Free, Pro and Max, including Claude Code on those accounts) the person chooses whether their chats may be used for training. If allowed, data is kept for five years; if not, the 30-day period applies. Services under commercial terms (Team, Enterprise, API) are outside that change (Anthropic). The general lesson: the same model can have different rules depending on the plan, and those policies change. Read each provider’s current terms.
Example
Illustrative example: a medical notes app for physiotherapists includes an assistant that summarizes sessions. Before the redesign, the notice lived in the terms. Now a line appears under the field: “This note is saved in the patient record and is not used to train models”. A separate toggle, “Improve the assistant with my anonymized notes”, is off by default and explains what anonymized means. When a summary finishes, the app highlights names and dates so the professional can review them.
To see what a well-made consent looks like, read consent to train on user data.
Common mistakes
- Leaving training consent on by default inside a bundle of terms acceptance.
- Promising “we don’t store your data” when an audit log or a history exists.
- Hiding the delete-history button in a submenu.
- Showing other users’ data in the context the system retrieves.
- Treating privacy as a notice instead of a flow that changes with the data being shared.
- Designing deletion with no clear end: users need to know whether deleted data leaves backups, and within what time frame.
Sources
- IAPP, “You Trust Your Chatbot With Everything. Should You?” (summary of the study by Théodore Christakis, March 2026): https://iapp.org/news/a/new-study-maps-the-privacy-gap-in-consumer-ai-and-proposes-a-fix
- Javadi and Kordzadeh, “Anthropomorphic Design and Privacy Assurance in Customer Service Chatbots” (Pre-ICIS SIGDSA Symposium, 2025): https://aisel.aisnet.org/sigdsa2025/64
- Li, Xiao and Li, “Disclose with Care: Designing Privacy Controls in Interview Chatbots” (arXiv, 2026): https://arxiv.org/abs/2602.01387
- GDPR text, Articles 5, 7, 13 and 17: https://gdpr-info.eu/art-5-gdpr/, Art. 7, Art. 13, Art. 17
- AI Act, Article 50: https://artificialintelligenceact.eu/article/50/
- Anthropic, “Updates to our consumer terms” (28 August 2025): https://www.anthropic.com/news/updates-to-our-consumer-terms