Let’s think of the most basic, simple mobile app we could have on our phones: the note-taking app. Due to its simplicity, the tasks we can perform on it are essentially writing, editing and deleting notes. Those features are limited, but essential: no more, no less. But what if we imagine a more powerful tool that relies on AI to expand its capabilities? Something like a note-taking app with “steroids” that can perform tasks such as using prompts to generate text automatically, sorting and categorizing, summarizing notes, and so on (actually, the Notes app in iOS already includes these features thanks to Apple Intelligence). Those characteristics are desirable and can greatly improve our apps, but how can we do that? Some API calls could be the solution: use any cloud-hosted AI model to perform such simple tasks, no big deal. However, we are losing the simplicity that characterizes our note-taking app. Now, we rely on an internet connection to use our new features (unless we restrict them when the phone has an internet connection, it doesn’t sound very good). What about our privacy? We’re deliberately giving all our precious notes to an unknown AI model, and we can’t be sure how that information will be used.
We have barely explored the concerns of using AI models in mobile devices such as availability and privacy. We will shortly come back to that topic, but now let’s take a look at our possible solution: on-device AI models. While cloud-based AI models run on powerful computers that are accessed remotely, on-device AI models perform their data processing locally using only the edge device’s hardware. With these models, we avoid the need to send and retrieve information over the internet.
Note: Edge devices can include IoT sensors, routers, controllers, robots, and more, but in this article, we will focus only on mobile devices, i.e., smartphones and tablets.
Let’s take a closer look at on-device AI models by comparing them with the traditional cloud-based AI. One of the most notable differences is where the models are hosted. As we stated before, the on-device approach is characterized by running models locally, that is, on the mobile’s own hardware. Cloud-based models, on the other hand, are hosted in data centers distributed worldwide. Computational resources are another key differentiator: while cloud-based models can exploit the power of many processors and, therefore, execute complex models, on-device models are limited by the mobile device’s capabilities which, in most cases, can only run limited AI models.
But this limitation is counteracted by three advantages: privacy, availability and latency. To access cloud-based models we must go across the internet, which means exposing our information to many risks and delays caused by the data transmission, but the on-device approach doesn’t suffer from these drawbacks since we have the model stored on our own device, this means that we don’t have to depend on internet access to leverage the benefits of AI models.
In the following table we summarize some aspects that differentiate cloud-based vs. on-device AI models:
Now that we have seen the differences between cloud-based and on-device AI models, let’s review three frameworks available to implement on-device AI models on our mobile apps:
LiteRT is Google’s bet on the on-device AI approach. This framework, which replaces its predecessor TensorFlow Lite, stands out for its simplified development, top‑tier GPU speed, unified NPU acceleration, advanced LLM support and wide ML framework compatibility. Its features include the capability of being implemented on multiple platforms (Android, iOS, Web and Desktop), supports streamlined conversion from PyTorch, TensorFlow, and JAX frameworks to .tflite or .litertlm format, a CompileModel and Interpreter API (this one is equivalent to former TensorFlow Lite), and so on. Also, there are some libraries that allow incorporation of .tflite and .litertlm models into React Native apps such as react-native-fast-tflite. For more information, visit this site.
Apple’s solution for integrating machine learning models into our apps is Core ML. This framework uses its own model format (.mlmodel files) that can be created using the Create ML app bundled with Xcode. Alternatively, Apple provides the Core ML Tools to convert models from other machine learning libraries into the compatible format. According to Apple’s Core ML website, it optimizes performance by leveraging CPU, GPU and Neural Engine while minimizing memory usage. You can find useful information on Apple’s developer site.
ExecuTorch is PyTorch’s solution for AI inference in mobile devices. Its key features include portability, a lightweight runtime with hardware acceleration and using familiar PyTorch tools to increase productivity. Like LiteRT, it supports multiple platforms such as Android, iOS, Desktop and Embedded systems. Their APIs can easily be implemented on Android and iOS as well. All the documentation can be found on PyTorch’s official website.
How can we exploit the benefits of on-device AI models? Many opportunities can be derived from the use of this approach, and mobile platforms offer solid solutions to implement them in our applications. Extended features such as personalized recommendations based on user data, image recognition using the mobile’s camera, improved responses on voice assistants, text generation and enhancement, users’ health monitoring, etc., are boosted by the advantages of on-device AI models: real-time responses, speed, enhanced data privacy and availability, with the help of the growing development of powerful hardware such as NPUs, mark the beginning of a trend that is likely to become part of our everyday experience.
But until then, we have some obstacles that need to be overcome to fully realize the potential of on-device AI models. Most of the hardware in mobile devices still lacks computational resources to deal with AI. The complexity of the models, storage/memory limitations and maintainability are challenges that still bound the expansion of this AI approach.
Bringing AI models directly to our mobile devices opens up a wide range of possibilities. Although on-device AI still faces many obstacles such as hardware and maintainability limitations, the applications related to it represent an opportunity to level up our mobile apps and explore the benefits of using this approach. Thanks to their portability, the three frameworks presented in this article are suitable for use on practically any platform; characteristics such as efficiency, support for multiple model formats and ease of use can help the implementation of on-device AI models in our apps. As hardware continues to evolve, on-device AI will likely become a cornerstone of mobile innovation.
