Research
This interview is part of a broader editorial project by Imminent, featuring conversations with expert professionals collaborating on Experience AI — which begins when machines learn and interact with the real world in real time — within the DVPS project.
DVPS is among the most ambitious projects funded by the European Union in the field of artificial intelligence, backed by an initial investment of €29 million. It brings together 20 organizations from 9 countries to shape the next frontier of AI — one rooted in the interaction of machines with the real world. Building on the success of large language models, DVPS explores the future of AI through multimodal foundation models. Unlike current systems, which learn from representations of the world through text, images, and video, these next-generation models are designed to acquire real-time empirical knowledge via direct interaction with the physical world. By integrating linguistic, visual, and sensor data, they develop a deeper contextual awareness, enhancing human capabilities in situations where trust, precision, and adaptability are critical. The overall initiative is led by Translated, which coordinates the project’s vision and implementation. The team brings together 70 of Europe’s leading AI scientists. The potential applications span across several domains, including language, healthcare and environment.
Jan Niehues is a Professor at the Karlsruhe Institute of Technology leading the “AI for Language Technologies” group.
Jan Niehues
Professor at the Karlsruhe Institute of Technology leading the “AI for Language Technologies” group.
Jan Niehues is a professor at the Karlsruhe Institute of Technology leading the “AI for Language Technologies” group. He received his doctoral degree from Karlsruhe Institute of Technology in 2014 on the topic of “Domain Adaptation in Machine Translation”. He has conducted research at Carnegie Mellon University, LIMSI/CNRS and Maastricht University. His research has covered different aspects of machine translation and spoken language translation. He has been involved in several international projects on spoken language translation, e.g. the German-French Project Quaero, the EU H2020 project QT21, EU-Bridge and ELITR. Currently, he is one of the organizers of the International Conference on Spoken Language Translation (IWSLT).
Can you describe your work and how it connects to the DVPS project? Could you describe your area of work, the research questions guiding you today, and the main challenges that have shaped your path?
We currently work on AI for different language technologies and look at a range of use cases. One is human-to-human communication, for example, supporting it through translation. Another is human-computer and human-robot interaction: how can we build systems that allow people to interact naturally with robots? We also work on information access. Across all these areas, we take a full-stack approach, from developing prototypes and studying individual tasks such as machine translation and dialogue to building AI systems and, in particular, large language models (LLMs).
Today, LLMs dominate the AI debate. Within DVPS, however, we’re working on something different: multimodal foundation models that learn from the physical world. In your domain of language, what difference does this paradigm shift make, and what do current models still lack, based on what you see in your everyday work?
The really interesting thing is that we can integrate much more context. As humans, we don’t just look at text or listen to language; we also look around and understand the scene. The hope is that multimodal foundation models will work in a similar way: they will understand the broader context in which something is said and, as a result, make better-grounded decisions. They can draw on all the information available to provide better answers.
In real-world language use, a single source of information is often not enough. Audio can be noisy, text can be ambiguous, and gestures matter. Have you seen cases where adding a modality has radically changed a system’s understanding? Could you give us some examples?
We recently published work on speech translation in which we also took lecture slides into account. There are several ways in which this can help. For example, if a system needs to recognise my surname, which has an unusual spelling, it might make a mistake. If it can also see my name on a slide, it can use that information to recognise it correctly. Another example is the English word “bug”, which can refer either to an insect or to an error in a computer program. When translating it into German, the two meanings require very different words. Italian has similar distinctions. To get the translation right, a system needs more context and should make use of all the information available.
Let’s talk about Europe. We live in a multilingual and inclusion-oriented context, with 24 official languages. This makes Europe a natural stress test for your field of research. What specific innovations or techniques emerge from this multilingual environment that English-first labs might not discover? Are there advantages hidden in what may seem like a constraint?
I think there are many opportunities. First, people understand the need for multilingual technologies. In large, predominantly monolingual societies such as the US or China, the importance of multilingualism may not be as immediately apparent, whereas in Europe there is much greater awareness and interest. There is also more data available, and we know that data is the fuel for much of AI research. We have been working in machine translation for nearly 20 years, and the proceedings of the European Parliament, which are translated into many languages, were an important resource that helped researchers get started in this field. I think this is a great opportunity for Europe: we have the need, and we have researchers working on it, so we can develop solutions that genuinely help us preserve the different languages, which I see as a valuable resource.
This is also a challenge, though. Have you observed patterns of bias or significant performance disparities between high- and low-resource languages? In a European context, where high-quality parallel datasets do not exist for many language pairs, what strategies are emerging as the most promising for building robust multimodal models?
Of course, it is a challenge, but it is also an opportunity for researchers. We see large differences in the performance of LLMs and other translation systems across languages. At KIT, we’re even working with very low-resource languages. For example, in speech recognition, we have worked on a language spoken by around 2,500 people in Azerbaijan, for which only two hours of data are available. This creates many research opportunities. We have been working on multilingual models for quite some time; our first publications in this area date back ten years. There are commonalities between languages, and by training on many languages, the effort required for each one becomes much smaller. We can transfer knowledge across languages by training on increasingly diverse language sets. I think this is one of the great opportunities offered by deep learning and LLMs: instead of developing tools dedicated to a single language, we can build multilingual tools and then adapt them where necessary. Some data may still be needed, but the process is much more efficient and makes it possible to cover more languages.
In this paradigm, DVPS is particularly interesting because it gives us an opportunity to access more data. Would you agree?
Yes. With DVPS, we’re extending this approach in several directions: we have more languages, more domains, and more modalities. The hope is that we can transfer knowledge between them. We can build systems that cover different modalities, infer information from them, and then produce and understand content in many languages, which adds another dimension. As you said, this is particularly important in Europe. What makes DVPS so interesting is that it brings together many different conditions and sources of information, allowing us to use them to develop new solutions.
Looking ahead to the next five years, what concrete impacts can we expect from these systems, for example in education, accessibility, or institutional communication? And what limitations remain particularly challenging today?
I think we will definitely see AI being used more and more in these areas. Through DVPS, we can try to build solutions that are genuinely helpful. In education, for example, AI systems could help students learn new material, understand it, and enjoy a more personalised learning experience. These are exciting opportunities, although the technology still needs to improve and many challenges remain. One of the biggest is hallucination and confidence: models are often overconfident. On the one hand, we need to make models better at estimating whether something is correct and providing reliable answers. On the other hand, we also need to educate the general public about how to use AI models. They can be very useful in many ways, but, just as with humans, we should remain critical, question what they tell us, and check whether the information is correct. I think this ability will become even more important.
Looking ahead to 2028, when the project comes to an end, what legacy do you hope DVPS will leave—not just in terms of the models developed, but also for the European research ecosystem? What capabilities, infrastructures, and networks should remain active after the funding ends?
I think EU projects are a great opportunity, especially in Europe, where we have many excellent research institutes, companies, and research centres that can work together to achieve something significant. This may be different from the situation in the US, where one large company can sometimes accomplish a great deal on its own. In Europe, the ecosystem is more distributed. These projects can help extend and strengthen networks, giving us the opportunity to build important things by bringing together the expertise and strengths of different partners.