AutoGLM Operates Your Phone for You: What It Does Well, and Where It Breaks

Jiang Wu · 2024-10-27

*Originally published in Chinese on October 27, 2024.*

Zhipu AI is a Chinese foundation model company, and AutoGLM is its agent that operates a phone on the user's behalf. I tried it over the weekend.

The product's core idea and how it works

The product idea is very clear. It recognizes the user's intent by voice, splits it into steps, and finds the corresponding apps to deliver the service. Technically it uses the phone's accessibility features. The route is clear and the implementation is not very hard. The key lies in recognizing the user's intent from the voice input and breaking it down.

Three steps of delivering a service

There are three steps: first voice input, second intent recognition, third carrying out the delivery through the user's own apps. It is currently limited to a few scenarios, so the basic implementation is not difficult.

A strength: working across platforms

Start with a strength, one that shows how a product can truly run across platforms. A model product that completes its delivery only inside the model is somewhat thin. Being able to call other products in the system to collaborate on a service may be the first step toward opening up AI services. The technical route has also been adapted to some degree. The team has at least analyzed users' common behaviors, done some user research and matched usage scenarios. As I understand it, the engineering workload behind this is not especially large: pull data from some mainstream apps and you have it.

Accuracy of imitating user behavior, and how it is tuned

What remains is accuracy and the process of imitating user behavior. The system or the app itself might judge the activity to be machine behavior, or trigger a user verification mode. So the product has to imitate real users' habits and scenarios as smoothly as possible, and it may have been optimized for that.

Doubts about where this goes

Personally I am doubtful about the future. On the technical route, there are several ways to meet this user need:

Path one: link accounts through the AI assistant and connect the service system

The first is to bring all user data into the assistant's own system. In effect, the AI assistant's back end links the user's accounts and connects to the user's service systems to complete the service. This requires users to link their accounts on other platforms inside the assistant, and requires those platforms to open API interfaces. Services such as ride-hailing and business travel have largely opened APIs that can be called, and navigation apps can be jumped to. Other e-commerce products have not fully opened up so that another platform can call an API directly to place an order. So if an AI assistant wants to close the loop on a service, some providers will support it (vertical services) and some will not (platform services). For the user, though, this is where the experience is complete.

Path two: AI imitates operations on the phone (the current approach)

The second is the current one, where AI imitates how a person operates a phone. Many earlier products tried this technical route. RPA robots, and the assistants some gray-market phones used to imitate user behavior and perform operations, could all do it. There is no technical difficulty. But it is plain that the way this path is carried out, or who ends up in charge of it, is not the assorted assistant products. The most effective way is the phone's own voice assistant, such as Xiaomi's Xiao Ai or Siri, which lets the system's built-in voice assistant connect and plug into services. That is a more effective approach for the phone. So most likely the entry point will be completed by the phone assistant at the system level, and it is not very likely to be handed to third-party tools. It is also unlikely that users will trust a third party with those permissions, and accuracy would suffer.

The two paths compared, and what comes next

The first path is clearly very hard, but it is meaningful to users and gives the product a barrier. The second will most likely not run for long, although it can show phone makers what the company can do. But phone makers are themselves building their own intelligent assistants now, so whether this scenario works broadly is in doubt.

A note on voice recognition accuracy

Note: AutoGLM's voice recognition was far too inaccurate. Almost not once did it get what I said right. (Perhaps my accent is just too strong?)

Home · All articles · Topics · 简体中文