A model never processes text, images or sound. It processes numbers. Text, image and sound are three different ways of arriving at the same numbers. Once you have understood that, omnimodality stops being a feat of engineering and becomes a logical consequence.
Yesterday we reported on a model that natively handles four modalities. Here is how that is possible.
Everything becomes a sequence of vectors
We explained the mechanism for text in our article on tokens and for images in the one on how they are billed. The principle is identical in all three cases.
Text is split into fragments of around four characters, each converted into a list of numbers.
An image is split into squares of a few dozen pixels, each converted into a list of numbers.
Sound is split into slices of a few hundredths of a second, each described by its distribution of frequencies, then converted into a list of numbers.
At the end of these three operations, the model receives exactly the same thing: a sequence of vectors. It does not know where they came from, and it does not need to.
Why the core mechanism does not care
The heart of these models works by comparing each element of a sequence with all the others to determine what matters. This mechanism makes no assumption about the nature of the elements.
That is why a single architecture has been able to absorb text, then images, then audio in turn, with no conceptual change. No new way of processing images was invented: a way was found to turn an image into something the existing mechanism already knew how to handle.
The consequence is that the model can mix sources within a single sequence. A video clip becomes an alternation of visual tiles and sound slices, in temporal order, and the model relates them to one another just as it relates words.
Picture a recorded meeting. An assembled system transcribes the audio, describes the slides separately, and feeds both texts to the model. It will never know that the sentence "this figure looks optimistic to me" was spoken while slide number seven was on screen, because that simultaneity disappeared during conversion. An omnimodal model receives both streams interleaved in their real order. It can therefore link what is said to what is shown. That is where the gain lies, not in the quality of the transcription.
What it costs
Two practical limits, and they explain why not everyone does it.
Volume. A minute of video produces a considerable number of vectors — far more than a page of text. That is why very long contexts, in the region of a million tokens, are the precondition for any serious video use, and why it is expensive.
The training trade-off. A model trained on all four modalities has to divide its capacity between them. Nothing guarantees that it will be as good at each one as a specialised model. This is the compromise we described in our article on generalist versus specialist.
Key takeaways
Omnimodality is not a feat: it is what happens when you realise that the core mechanism is indifferent to the nature of what it processes.
It is also a good illustration of a point we often make. The spectacular advances of these systems rarely come from a new idea. They most often come from an old idea applied to something nobody thought to feed it.