The problem
The client is a fitness startup building a strength-training app with real-time form checking: point your phone at yourself during a squat or deadlift, and the app tells you if your form is breaking down before an injury happens, not after. It had to work in a home gym with unreliable Wi-Fi, during a set, with no perceptible lag, because a half-second delay between a form break and the feedback defeats the purpose.
Cloud-only inference was the default assumption when they came to us, and it was the wrong architecture for this product. Sending video frames to a cloud model and waiting for a response adds exactly the latency that makes real-time form feedback useless. It also brings a data cost that scales badly with usage, and real privacy questions about continuously streaming home workout video to a server.
Why on-device was the right call, not just a nice-to-have
We scoped this from day one as a latency and privacy problem first and a modeling problem second. Based on demos from larger fitness AI products, the founders assumed they'd need a large, powerful cloud model for good form detection. We spent the first two weeks benchmarking whether a much smaller on-device pose-estimation model could reach acceptable accuracy on the narrow set of movements the app needed to check (squat, deadlift, and overhead press form, not general movement recognition).
Architecture
On-device pose estimation as the default path
Core form checking runs entirely on-device, using a lightweight pose-estimation model converted and quantized for mobile inference (Core ML on iOS, LiteRT on Android). It handles the real-time joint-angle analysis that flags a form break during a set, with no network round trip in the critical path.
Cloud as the fallback, not the default
A cloud-backed path exists for what on-device inference doesn't handle well: detailed end-of-session form analysis with richer feedback, and a small subset of supported devices (mostly budget models) where on-device performance fell below threshold in testing. This inverts the typical pattern of cloud first with on-device as an optimization, and it was the right inversion for a feature whose primary use case demands zero perceptible lag.
Native builds, not cross-platform
Given the latency requirements and the need for tight integration with each platform's camera and ML runtime APIs, we built separate native iOS (Swift, Core ML) and Android (Kotlin, LiteRT) apps rather than using a cross-platform framework, which would have added an abstraction layer between the app and the camera and ML pipeline that this feature couldn't tolerate.
Battery and thermal profiling
Continuous camera use and model inference during a workout are demanding on a phone's battery and thermal limits. We profiled across a range of device ages (not just new flagships, because many target users have two- to three-year-old phones) and tuned the inference frequency (how often a frame is analyzed versus skipped) to balance responsiveness against battery drain and heat over a 45-minute session.
Challenges and tradeoffs
- Model accuracy dropped meaningfully in low light. Home gyms are often poorly lit, and pose-estimation accuracy degraded in ways our well-lit test footage hadn't revealed. After this surfaced in beta testing, we added a lighting-quality check that warns the user rather than silently giving unreliable feedback in bad conditions.
- Battery drain was the actual engineering fight, not model accuracy. Getting the core model accurate enough took about three weeks. Keeping a 45-minute session from draining more than 30% of the battery took closer to five, through inference-frequency tuning and profiling across device generations, and it was the harder problem throughout the build.
- The cloud fallback path needed its own privacy design. Even for the smaller share of cases using cloud inference, we send only extracted pose keypoints, never raw video frames, to avoid the privacy exposure of streaming home workout footage. The founders hadn't specified this constraint; we flagged the risk during architecture review.
Results
Across the beta period, 89% of workout sessions ran entirely on-device with no cloud round trip. A median form-check latency of 120 ms was fast enough that testers described the feedback as "instant" in usability sessions, which was the real bar the founders needed to clear. Total inference cost came in 76% below the founders' original all-cloud estimate, because most sessions never touch the cloud, and that mattered directly for unit economics at their target subscription price.
The app shipped to both app stores thirteen weeks after architecture validation began, with the on-device-first decision validated in week two rather than discovered as a problem after a cloud-only version was already built.
What we'd do differently
We'd build low-light testing into the initial device and environment matrix rather than discovering the gap during beta. Home gym lighting varies a lot, and our early validation, done mostly in a well-lit office, gave an overly optimistic accuracy picture that beta testing corrected later than it should have.