The definition, without the marketing
On-device AI means the model runs locally. The file that does the work sits in your phone's storage, is loaded into its memory, and produces its result there. Nothing is sent anywhere and nothing is waited for.
The alternative, which is still how most of it works, is server-side. Your device packages up the input, sends it over the network, and displays whatever comes back. The model might be enormous and constantly improving, and you will never see it.
Same category of software, two places to put it. That is the entire distinction. "AI" is doing a lot of heavy lifting in most sentences it appears in.
Where you have already used it
Plenty of it is old news and does not get called AI at all.
Your camera decides where the faces are before you press the shutter. Your photo gallery groups pictures by who is in them. Your keyboard predicts the next word from the last few. Some phones transcribe speech without a connection, and some translate text with a downloaded language pack.
None of those ask permission to send anything, and most people have never wondered where they run. They were built locally because latency mattered, or because it would have been absurd to upload every camera frame.
What running locally buys you
It works without a connection. No signal, airplane mode, a basement, a country where your data plan does not apply. If nothing is being requested, none of those are events.
Nothing is transmitted. For anything sensitive, this is the property that matters, and it is a fact about how the software is built, not a promise in a document. There is no upload to intercept, no server holding a copy, and no retention policy to read.
Latency is predictable. A local result depends on your hardware. A remote one depends on your connection, the distance to the server and how busy it is. Local is not always faster, but it is consistent, and it does not fail quietly when the network is poor.
Often no account. Server-side processing usually implies knowing who is asking. Local processing does not need to.
What it costs
Five costs, and none of them go away with a better app.
Memory and storage. The model has to fit on the device and be held in memory while it runs. On a phone that competes with everything else you have open.
Capability. A model that fits on a handset is smaller than one running on a server with specialist hardware, which usually means somewhat lower accuracy, particularly on hard inputs like heavy accents or noisy recordings.
Battery and heat. Real work generates heat, and a phone has no fan. Anything sustained has to be designed carefully or it becomes unpleasant to hold.
Updating. A server-side model improves for everyone at once. A local one improves when you download a new version.
Device variation. A model that runs comfortably on a recent flagship may struggle on a five-year-old phone. Server-side processing hides that difference; local processing exposes it.
Why it became practical
Nothing about the idea is new. Running software locally is how computing worked for decades. What changed is that the useful models became small enough to fit.
At roughly the same time, model architectures got more efficient, so a given level of capability needed fewer numbers to express. Techniques for compressing a trained model, broadly known as quantisation, allowed those numbers to be stored with less precision and therefore less space, usually at a modest cost in accuracy. And phone processors gained dedicated hardware for exactly this kind of arithmetic.
Alongside all that, some useful models were released with their weights available, which meant developers could put a capable recogniser inside an app instead of building a client for someone else's service.
None of that made local processing better than server-side in raw capability. It made it good enough for a large class of everyday tasks, which is a different and more useful thing.
Where it does not make sense
The pattern is being applied to places it does not fit.
Anything that needs knowledge of the world beyond your device, current information, other people's data, the contents of a service, cannot be answered locally no matter how good the model is. Anything requiring a very large model will not run on a handset in a reasonable time. And anything shared between people is a coordination problem, not a processing one.
The sweet spot for on-device is narrow and specific: a bounded task, on input you already have, where latency matters or the input is sensitive. Photography. Text prediction. Translation of a page in front of you.
And dictation, which happens to be all three at once.
The one caveat about privacy
On-device does not automatically mean private.
An app can run its model locally and still send analytics, still require an account, still back its data up to a cloud, and still upload the result even if the processing happened on the handset. "The model runs on your device" and "nothing about you leaves your device" are two different claims, and only the second one is what people usually mean.
So the useful checks are about behaviour. Forget the architecture and watch what the app does. Does it request network access at all? Does it work in airplane mode? Does its data survive a move to a new phone, and if so, where was it kept?
How to tell whether an app really does it
Two minutes, no expertise needed.
Install it, let any first-run download finish, then turn off mobile data and Wi-Fi and use the feature. If it works, it works locally. If it stalls or shows a connection error, it does not.
The first-run download is itself a good sign: a local model has to arrive somehow, so an app that fetches a few hundred megabytes and then goes quiet is behaving exactly as you would expect. An app that never downloads anything and responds instantly is using someone else's hardware.
Speech is the clearest example
Dictation is where the trade-off is easiest to feel, and for that reason it tends to be the first thing people move on-device.
The input is private by nature, since you are speaking sentences you would not necessarily publish. The work is short and bounded, so it suits a phone. And the failure mode of the server-side version is obvious: no signal, no dictation.
LocalType is that example in product form. It is an Android keyboard where the speech model sits on the phone and does the recognition there, with no cloud path in the app at all. It ships three model sizes, 60, 190 and 539 MB, so the memory and accuracy trade is a decision you make, and we do not make it for you. The middle one converted 6.9 seconds of speech in 4.1 seconds on a test phone with 4 GB of memory. It is fair to ask any on-device app for a figure like that. It also covers eighteen languages from a single multilingual model, and asks for the network exactly once, to fetch the model you chose.
Every cost listed above applies to it. It occupies storage, its models are smaller than a server would run, accuracy varies by language, and older phones want the smaller model. Those are not caveats bolted onto the pitch. They are what on-device means.