Every mobile AI feature request eventually hits the same fork in the road: should this run on the device, or should it call out to the cloud? Teams often treat this as a technical detail to figure out later. It shouldn't be. It's one of the first decisions that shapes everything else about the feature.

What's actually different between the two

On-device inference runs the model directly on the phone, using something like Core ML on iOS or LiteRT (formerly TensorFlow Lite) on Android. No network round trip, no server cost per request, and the feature keeps working with no connection.

Cloud inference sends the request to a server running a larger, more capable model, and streams the response back. More power, easier to update, but every call costs money and depends on the network.

The instinct is to treat this as purely a technical tradeoff. In practice, it's really three separate tradeoffs stacked on top of each other, and they don't always point the same direction.

The latency tradeoff

On-device wins here, decisively, for anything that needs to feel instant: live camera analysis, text prediction as someone types, or a feature that runs continuously in the background. A 200-400ms round trip to a cloud model is invisible in a chat interface. It's very visible in something like real-time object detection.

The catch is that on-device models are smaller and less capable than what you'd get from a frontier model in the cloud, because phone hardware has real memory and compute limits. You're trading raw capability for speed.

The cost tradeoff

On-device inference is effectively free per request once the model is shipped in the app. Cloud inference bills per call, and that adds up fast at scale. It's easy to underestimate during a demo with ten test users and very visible once you have a hundred thousand daily active users.

We've had more than one client engagement where the deciding factor for going on-device wasn't latency or privacy at all. At the client's projected usage, the cloud-inference version of the feature would have cost more per month than the rest of their infrastructure combined.

The privacy and offline tradeoff

Some features genuinely can't send data to a server, whether that's a compliance requirement, a user trust issue, or a product promise ("this works even in airplane mode"). On-device is the only real option here. Sending health data, financial records, or private messages to a cloud API for every interaction is a real liability, and for some categories of app it's a non-starter regardless of how good the cloud model is.

A pattern worth knowing
A lot of production mobile AI features aren't purely one or the other. A small on-device model handles the fast, frequent, low-stakes cases, and only escalates to a cloud model for the harder cases that actually need it. This hybrid approach gets you most of the latency and cost benefits of on-device with most of the capability of cloud, at the cost of more engineering complexity.

The decision framework we actually use

  1. Does this feature need to work offline or in a poor connection? If yes, on-device is mandatory, not optional, at least for a fallback path.
  2. Is this triggered continuously or on every keystroke/frame? High-frequency triggers push hard toward on-device, both for latency and to avoid a runaway cloud bill.
  3. Does the task need frontier-level reasoning, or is a smaller specialized model good enough? A lot of mobile AI tasks, like classification, simple extraction, or intent detection, don't need a massive model. That opens the door to on-device even for tasks people assume require cloud.
  4. What's the actual per-request cost at your real projected scale, not your demo scale? Model this before building, not after launch. We do this math explicitly during discovery for every mobile AI engagement.
  5. Does the data involved have real privacy or compliance weight? If yes, that alone can override every other factor in the decision.

What this looks like in a real build

A recent example: an in-app shopping copilot we built for a retail client. Product search and simple Q&A run through a lightweight on-device model, since those happen constantly and need to feel instant. Personalized recommendations that draw on purchase history and inventory data route to a cloud model, because they need context the phone doesn't have locally, and they happen far less often per session. The split wasn't obvious at the start. It came out of profiling real usage patterns during the prototype phase, not guessing upfront.