What Multimodal AI Needs Before It Takes Hold

Jiang Wu · 2024-12-09

*Originally published in Chinese on December 9, 2024.*

Multimodal AI, meaning models that handle text, voice, images and video together, is still waiting for the moment it takes hold. By my count there are three elements. The first is the model's base capability. The second is demand from scenarios, whether everyday use or vertical use. The third is the extension of where it gets used. Each can be looked at in turn.

Model capability

Start with the base capability of the models. Seen from the multimodal side, past progress has been slow. For one thing, the training logic used so far runs into a heavy data bottleneck. Text data on the internet is plentiful. Image and video data exist too, but compared with the volume text needed for a qualitative jump, the demand for image and video data far exceeds today's supply. To a degree, a shortage across the whole data system is what has made the models slower than expected, and it will take time to supplement their base capability. Until there is large-scale data supply, or a newer solution, capability still needs a period to mature.

Value added by application scenarios

Multimodal applications are already being tried in some dimensions, whether text, voice, images or video, but everyone's solution is scenario-based, so each company is optimizing scenarios and improving capability inside that logic. Yet the value added in the different scenarios is not high enough today, or does not look prominent enough in commercial value on the consumer side, so no one has found a good breakthrough path. When the value added is high enough, more resources and time go into research in the field. When it is not, most players will choose to watch. While multimodal technology is still not mature enough, applying it in high-value scenarios is probably the meaningful direction. But we are still exploring, and no one can yet say they have found a scenario with high enough added value.

Looking for high-value scenarios

More products will probably have to try more user scenarios before more possibilities appear. Today there are too few attempts. Other factors may also be at work. Once capability expands, moderation pressure and service costs rise. Infrastructure is probably sufficient, but optimizing the service and keeping it controllable still need more targeted solutions, and if the added value is not high, companies may not commit fully. In certain high-value fields there has been progress, for example education and adult content.

Integrating technologies

There is a finer problem too, which is how different technical capabilities and models relate and get applied. In concrete scenarios they may be integrated in many different forms, and applications will have to deal with that next.

Interaction carriers

The third element is the carrier. Multimodal applications offer rich forms of interaction that are closer to natural ones, such as voice, images and video, and the hardware carrying them is at an early stage. Whether it is earbuds, glasses, headsets or watches, fast hardware adoption and broad use of scenarios can speed up the spread of multimodal applications. When the core multimodal capabilities are integrated with hardware, scenarios fit better, and the value added is larger than a product living on a phone or a computer alone. So to reach broader maturity, multimodal needs a push from hardware. This is more than a new way to present and display. It involves redefining the logic of interaction, or of communication between people and machines, and hardware plays a relatively key role in it.

What AI glasses may change

A dense wave of AI glasses and similar products is coming in 2024, and it may solve part of the data-input problem. We will see in 2025 whether a better product appears that fits a scenario more closely or is more tightly tied to the hardware.

When each element will be ready

Looking at the three dimensions, namely base model capability, maturity of application scenarios, and the extension of scenarios together with the hardware ecosystem, they are not yet all at the 60-point mark. Either one element becomes mature enough to pull the other two along, or several mature at the same time and bring a new step change. At best that takes six months to a year, depending on where the key variable comes from. At the moment it looks like models and hardware are the key variables, while applications and scenarios still need some time.

What if none is mature? Then we wait a while, or we build today.

Home · All articles · Topics · 简体中文