Two years ago, running AI inference on-device meant accepting significant tradeoffs: slower models, lower accuracy, more battery drain. The default was always to call a cloud API. That's changed, and the change is big enough that "should we run this on-device?" is now a legitimate architecture question instead of a workaround for offline capability.
What got better
Three separate things converged. First, quantized models — models that use lower-precision arithmetic to run faster and use less memory — got a lot better. A Claude Haiku model quantized to 4-bit precision now performs within 2-3% of full precision on most tasks, and runs on devices with 2GB of RAM.
Second, Apple and Qualcomm released actual good ML accelerators built into their chips. An iPhone 15 or a high-end Snapdragon can run a 13B-parameter model in real-time. That's not theoretical, it's shipping in phones people carry today.
Third, the models themselves got better at following instructions in context. An older model might need fine-tuning to specialize. A modern model can be specialized by showing examples in context. That context costs tokens, but it's cheaper than re-running inference on the cloud when you can keep the model resident on the device.
Where this is already happening
A mental health app runs a model locally on-device for initial symptom screening. The model is Claude Haiku quantized to 4-bit, resident in the app. Users submit symptoms, the model runs on-device, and triages them to appropriate resources. No health data leaves the device. The model runs at 97% of full-precision accuracy while using less power than the app's natural language keyboard.
A field service app runs a model on technician phones to analyze photos of damaged equipment, generate reports, and suggest diagnoses. It's faster than uploading to cloud and waiting for a response. The technician's offline by necessity, so running on-device is the only option. But the speed and latency benefits are big enough that they kept it on-device even after the technician's internet came back.
A retail application that needs to identify products from photos for inventory or point-of-sale runs on-device for two reasons: speed (< 100ms latency) and privacy (customer doesn't upload a photo of their home to classify the item they're selling).
The economics
Running Haiku via API: $0.80 per million tokens, 300-400ms latency, 0% user data stays on device.
Running quantized Haiku on-device: $0 per inference, 100-200ms latency, 100% user data stays on device, plus a one-time cost of about 4GB of device storage and 150MB per app download.
If you're running fewer than 10 API calls per user per day, on-device is cheaper and faster. If you're running more than 100 calls per user per day, cloud APIs with better batching might be cheaper. For everything in between, it depends on whether you care about latency and data residency, which you usually do.
What's still pushing to cloud
Long-context reasoning tasks, where you're processing thousands of tokens of input and expecting sophisticated output, still run better on frontier models in the cloud. On-device models are good at local pattern matching and transformation, not as good at holistic reasoning across lots of input.
If you need the very latest model (this week's release), cloud APIs give you that faster. On-device models often lag by a version or two because pushing an update to a mobile app takes time.
If you need to preserve output for auditing and compliance, keeping inference in the cloud where you can log everything is often simpler than trying to reconstruct what happened on-device.
The architecture shift this enables
The default is starting to flip. For consumer applications and mobile-first work, the new default is "run on-device unless there's a specific reason not to." For enterprise work, it's still "cloud unless the use case requires on-device," but that list of use-case requirements is growing.
If you're building infrastructure that depends on "inference must happen in the cloud," you should be rethinking that constraint. It's becoming a legacy assumption, not a requirement.
The hiring picture
If your team is staffed for "cloud AI infrastructure," you have a gap: someone who understands how to optimize models for on-device inference, how to manage model updates across distributed devices, and how to debug performance issues that only show up on end-user hardware. That's a specific skill, somewhere between ML engineering and mobile engineering, and it's in real demand right now.
We cover hiring for mobile and edge AI roles in our career guides, and this shift is exactly why that niche is suddenly worth filling.
