CHI 2025, the top-tier academic conference in human-computer interaction, released its accepted papers list last month, and Tencent's AppAgent paper is on it (arXiv:2312.13771). After reading it through, our judgment: this is the latest systemic bet by a Chinese tech giant on the "mobile AI assistant" direction — but the core innovation, its "learn-then-use" design, leaves significant doubts about real-world usability on production apps.
What this is
AppAgent is a framework that lets multimodal large models — LLMs that can simultaneously see images and read text — operate mobile apps the way humans do. Instead of calling an app's backend APIs, it looks at screenshots, taps buttons, and swipes screens to complete tasks.
The core design splits "using an app" into two stages. The first, called exploration, lets the AI autonomously probe an app's interface and write natural-language notes describing what each button does. The second, called deployment, retrieves those notes when a real task comes in and operates according to them. The critical difference: notes are reusable, so new apps don't have to be learned from scratch every time.
On the input side, the model sees a numbered screenshot — the system marks every tappable element in red with an index, and the AI only outputs "tap number X," no pixel coordinates needed. This effectively outsources the hardest positioning problem to the system's view hierarchy (a structured document the OS auto-generates for each screen).
Industry view
Supporters argue that the staged "learn-then-use" design fixes the pain point of prior approaches having to learn from scratch every time they hit a new app; having the model just look at images, pick a number, and outsource positioning accuracy to the system's view hierarchy also lowers the training bar.
But the counterarguments deserve equal airtime. First, this approach is deeply tied to Android's accessibility APIs — apps heavy on custom widgets can't have their elements captured by the view hierarchy, and the workflow simply breaks. Second, the exploration phase relies on extensive trial and error; the paper openly admits that "actions execute without changing the UI" is common, and real-world app complexity far exceeds test scenarios. Third, CHI 2025 is a top-tier HCI venue, with most of its impact in academia — scaling to production is still a stretch.
Impact on regular people
- For enterprise IT: don't treat this as a procurement list anytime soon. It's more a technical signal — stable commercial mobile-side AI operation is still far off.
- For individual professionals: what mobile AI can currently do is light tasks like "image-based Q&A." Complex operations still need to be done manually for now.
- For consumer market: in the next year or two, phone makers or super-apps may embed similar capabilities, but early versions will likely get stuck often on complex apps.