On-Device AI: What Changes for Privacy, Latency and UX

On-device AI runs the model on the phone or computer: it gains privacy and offline use, and loses capacity. The common pattern is hybrid, local plus cloud.

info Quick Definition

On-device AI means running an AI model directly on a phone, a computer or another local device, without sending every request to a server. It changes three things for design: the response no longer depends on the network, data can stay on the device, and capacity is limited by the hardware, memory and battery.

What is gained and what is lost?

Android’s documentation sums up the trade-off: running prompts locally removes server calls, keeps sensitive data on the device, allows offline use and lowers inference cost. But speed depends on the device’s hardware, and Gemini Nano has token limits and model capability boundaries. Apple, for its part, describes its local model as a 3-billion-parameter, device-scale model that is not designed for world knowledge or advanced reasoning.

Dimension	On the device	In the cloud
Latency	No network trip, but depends on the chip	Depends on the connection
Privacy	Data can stay on the device	Data leaves the device
Capacity	Small models	Large models
Cost per use	Hardware and battery	Payment per request
Knowledge	Limited to the installed model	Updatable

What options exist in November 2026?

Apple. The Foundation Models framework, introduced in 2025, gives apps access to the local Apple Intelligence model. Apple describes it as free-of-cost inference that is available offline and protects privacy. You must check availability at runtime, because the model only runs on devices with Apple Intelligence enabled in supported regions. In June 2026 Apple introduced a third generation of its models (AFM 3), still in a beta period.
Android. Gemini Nano runs in AICore, a system service that manages distribution and updates of the model. The ML Kit GenAI APIs offer ready-made tasks such as summarization, proofreading, rewriting, image description and speech recognition.
Chrome. According to Chrome’s documentation, the Prompt API is stable on the web since Chrome 148 and uses Gemini Nano. It works only on desktop, needs at least 22 GB of free storage and more than 4 GB of VRAM, and downloads the model the first time. It accepts English, Spanish, Japanese, German and French. The translation, language detection and summarizer APIs have been stable since Chrome 138.
ExecuTorch. PyTorch’s runtime for taking your own models to phones, embedded devices and computers. Version 1.0 shipped in October 2025, and Meta uses it in production in its apps.

A market research firm agrees with the picture: in September 2026 it rated assistants, summarization, translation and dictation on phones and PCs as in production, and expected most workloads to be hybrid. Check the specific versions before you commit to one: the market moves fast. To see how sizes compare, review small vs. large models.

How do you design a hybrid product?

The device handles the routine and sensitive work, and the cloud handles the hard work. A practical design:

Classify the tasks. Which are frequent and sensitive, and which are rare and complex.
Define an escalation threshold. For example, when the local model has low confidence, or when the task exceeds a certain length.
Tell the user when something leaves the device. It is a privacy decision, not a technical detail.
Design the offline mode and the no-model case. Not every device has the model available; define what the app does then.
Measure separately. Latency and quality locally and in the cloud, because results can differ.

Example

Illustrative example: a medical notes app for a clinic wants to summarize the doctor’s dictations. It decides the summary runs on the device, so the audio never leaves the machine. When a dictation goes past a certain length or the local model has low confidence, the app asks for permission before sending the text to the cloud, and shows it on screen. The doctor always knows what left the device. The case is invented to illustrate the method.

How do you choose the size of the local model?

Start from the budget, not from the largest model that fits. Define three numbers before choosing: the memory you can occupy without affecting the rest of the app, the maximum acceptable latency for the task, and the battery use the user tolerates in a typical session. With those numbers, test two or three sizes on low-end, mid-range and high-end devices, because the experience on an old phone can be very different from a new one.

Measure response quality on those same devices. A model that answers fast but summarizes a clinical dictation badly is no use, and quality only shows up with real cases from your product. Document the chosen size and the minimum supported device, so the team does not change it without measuring again.

How do you communicate what leaves the device?

Communication is part of the design, not a legal text at the end. Tell the user, in a sentence they understand, what is processed on the device and what leaves it, in which cases and for what purpose. A visible indicator, such as an icon or a label next to the answer, helps more than a notice accepted without reading. The option not to send should be as easy as the option to send.

Think about history too. If the app stores local summaries, say where they are kept and how they are deleted. A user who understands the flow trusts it more, and that trust is the main advantage of running the model on their own device.

Common mistakes

Promising total privacy without explaining what leaves the device in escalation cases.
Assuming every device has the model: without a fallback, the feature fails on those that do not support it.
Using a large local model that drains the battery.
Measuring only latency and not the quality of the response.
Hiding the switch between local and cloud.

When NOT to run it on the device

When the task needs recent knowledge or long reasoning, when the device is very old, or when maintaining several models costs more than the privacy benefit. In those cases, the cloud with strong data controls is often the simpler choice. To think about the data that enters the model, review privacy and data in AI products and cost per token.

Sources

Frequently asked questions

What is on-device AI?

Running an AI model directly on a phone, laptop or other local device, without sending every request to a server.

What are its advantages over the cloud?

Data can stay on the device, answers do not depend on the network, and it works offline. The cost is lower capacity and a dependence on the device’s hardware, memory and battery.

Which models and tools exist today?

On Apple platforms, the Foundation Models framework gives access to the local Apple Intelligence model. On Android, Gemini Nano runs in the AICore service and is used through the ML Kit GenAI APIs. In desktop Chrome, the Prompt API uses Gemini Nano. PyTorch’s ExecuTorch is for deploying your own models.

What is the most common pattern?

Hybrid: the local model handles routine and sensitive tasks, and the cloud handles the hard cases. Google documents that combination with Firebase AI Logic on Android.