Qualcomm's Snapdragon 8 Elite and Running Models Locally on the Phone

Jiang Wu · 2024-10-21

*Originally published in Chinese on October 21, 2024.*

The Snapdragon 8 Elite is a high-performance flagship mobile platform that Qualcomm introduced in 2024. This piece first goes through its main specifications, then looks at what running models on the device could change.

CPU performance

Architecture

It carries Qualcomm's second-generation custom Oryon CPU in an all-big-core "2+6" design: 2 prime cores and 6 performance cores. The prime cores run at up to 4.32GHz, suited to intensive apps that need faster response. The performance cores run at 3.53GHz and handle the most demanding applications, and each one is tuned for high efficiency.

Performance gains

Compared with the previous generation, single-core CPU performance is up 45% and multi-core performance is up 45%. With the new architecture and technical optimizations, the chip can stay cool all day in compact designs such as phones. Its L2 cache reaches 24MB, the largest in mobile today, which supports the chip's high performance.

GPU performance

Architecture upgrade

The Adreno GPU uses a sliced architecture and runs at 1.1GHz, a major upgrade in graphics processing. Each slice gets dedicated memory for graphics data transfer, which reduces the graphics data sent to chip memory and raises the efficiency and performance of graphics processing.

Performance

For graphics-intensive games, performance rises 40% while power consumption falls 40%. Ray tracing performance is up 35%. Together with Snapdragon Elite Gaming's technical solutions, this gives users an excellent gaming experience. The chip also supports Unreal Engine 5.3 for the first time, which makes cinematic 3D environments in mobile games possible.

AI performance

NPU upgrade

It carries the new-architecture Hexagon NPU. AI performance is up 45%, energy efficiency is up 45%, and token generation on base large language models is up by as much as 100%. On some of the popular large language models in the industry, processing speed exceeds 70 tokens/s, a strong response speed.

Intelligent experience

Users can build a personalized multimodal AI assistant. Underneath, the assistant relies on several advanced AI models, including automatic speech recognition (ASR), a large language model (LLM), a large vision model (LVM) and a new large multimodal model (LMM). Through heterogeneous computing, these models run on different cores of the Qualcomm AI Engine, so every task runs smoothly and efficiently. For example, a user can point the phone's camera at something they want to know about and ask the phone a question. The phone analyzes the live camera image and gives an answer, which improves the interaction experience.

Imaging

AI-ISP

The chip brings a new AI-ISP. On top of autofocus, auto white balance and auto exposure, it adds AI-assisted enhancements that improve image quality across the board. Pixel throughput is up 33% to 4.3 billion pixels per second, which lets the phone record video at 30fps with zero shutter lag from three 48-megapixel sensors.

Video features

It supports real-time AI enhancement at 4K 60fps, and AI magic erase on video: select the object to remove, and it is removed. The whole process happens locally on the phone without uploading to the cloud, which protects user privacy.

Connectivity

Modem

It integrates the Snapdragon X80 modem and RF system. By combining sub-6GHz and mmWave, 5G download speeds reach 10Gbps and upload speeds reach 3.5Gbps.

Wireless

It supports Qualcomm FastConnect 7900 and Qualcomm aptX Lossless, delivering Bluetooth music at 24bit 48kHz and Wi-Fi music at 24bit 96kHz.

The highlight: on-device models opened up

Of all these elements, the biggest highlight is the opening up of local on-device model capability. In effect the chip prepares its own interfaces and capabilities, and other applications simply call them. In cooperation across different fields, the examples chosen so far are Tencent's Hunyuan, Zhipu's GLM-4V on-device vision model, and an AI teammate trial by NetEase Games.

Official cooperation cases

Tencent Hunyuan

By supporting hardware-based INT4 quantization, the Snapdragon 8 Elite enables on-device deployment of the 7B and 3B versions of Tencent's Hunyuan model, which greatly improves how efficiently Hunyuan runs on the device. For example, in the smart SMS recognition feature of Tencent's phone manager, massive data combined with deep neural networks and pretraining lets the model understand the intent of a text message more accurately, raising SMS recall by nearly 200% and recognition accuracy by 20%.

Zhipu

Qualcomm worked with Zhipu AI to adapt and optimize the GLM-4V on-device vision model for the Snapdragon 8 Elite, supporting rich multimodal interaction. The on-device multimodal app ChatGLM supports three ways of interacting on the device: real-time voice conversation using the camera, conversation about an uploaded photo, and conversation about an uploaded video. With rich multimodal input and output, the assistant ChatGLM can be used across home, travel, work and education. Examples include getting outfit advice through the phone camera, using visual information for smart navigation, identifying items left behind using an in-car camera, and photographing a painting to analyze its style.

What on-device AI could change

A large drop in the service cost of AI apps

Most AI runs in the cloud today, and all the service is completed on the server side. The phone becomes only an input and output interface for AI apps, and the hardware itself has no service capability (some models do have local packages). The drawback of the cloud is that each service output must be sent there, which hurts response speed and experience, and more importantly the service costs a lot. If part of the service's input and output can be done on the chip, then future app development only needs to add a local check on whether the service is handled by the local model or must be sent to cloud service capacity. That would cut the service cost of apps by a large amount.

A large increase in response speed

Beyond its own cost, cloud processing has another problem: it depends heavily on external conditions such as network speed, response speed and cloud service speed. The result is that services can only be relatively asynchronous. Services with little information and low concurrency may output in real time, but services with a lot of information and concurrency may see some delay. If everything is local, response speed rises sharply, which basically means some simple AI services can be provided in real time, such as everyday conversation and recognition of content intent.

Some AI hardware can move its AI locally

At present some AI hardware serves only as a port for data input and service output, and some can only take input, with the service itself hosted in the cloud. If the hardware has no communication ability of its own and is an accessory to the phone, then those services also put pressure on the phone's power consumption and data transfer. If the phone itself has on-device processing in the future, an AI accessory only needs to call the phone's port, and network delay and power consumption are no longer problems. That would free up a batch of new hardware, with new capabilities and scenarios, such as smart glasses or smart earbuds.

Real-time multimodal vision input and output

A more important advantage of going local is that multimodal capability used to be almost impossible to do in real time because of the demands on transmission bandwidth and response speed, so in most scenarios it was limited. If the chip itself can do the processing locally, real-time multimodal capability is almost within reach. Examples are the local video interaction, input and output, that Zhipu provides, and the new experiences that lower interaction costs and delay will bring in real-time video scenarios.

Fully offline deployment lowers the cost of using models

One idea for the future is that the chip deploys a model purely offline, with the scenario optimized very specifically and precisely. Then the cost of using the model, and its integration in hardware, would be largely freed up. Examples are early childhood education, companion toys, AI classrooms in schools and AI coaches in gyms. The more concrete and clear the scenario, the more exact the requirement for local deployment, the better the service works, and the lower the cost can go.

Home · All articles · Topics · 简体中文