Running Models on the Device Comes Down to Cost and Moat

Originally published in Chinese on February 29, 2024. Most of the compute in a phone's processor sits idle while apps send every request to the cloud. Whether a product should run a model on the device instead is a question I have been working through. Cost and moat Only one factor decides whether to use a local model, and it is cost. The decision logic also includes a vendor's wish to build its own moat. A vendor afraid of becoming a mere pipe that can be replaced needs some capability on the local device. And if the local device is something users must have anyway, the vendor may as well integrate model capability there too.