Today we interview a developer aiming to revolutionise video games and museums with AI characters. The hottest topic in tech right now is AI. Imagine game characters that can interact with the player creatively and intuitively. That’s the dream of Alexandre Folliot, the CEO of Paris-based developer X&Immersion, who plans to make such an experience possible.
The company works with video games – and other media, like museum exhibits – by providing studios with tools to create engaging experiences while cutting production costs. One of the company’s flagship features is its character creation tool, which includes dialogue, voices, and facial animation for immersive interactive stories. In addition to a recent historical experience, currently on tour around Europe, X&Immersion is preparing a new game, Crash Investigation, set to launch in the next year.
In this Q&A, Folliot opens up about his journey as an entrepreneur, his vision for the future of AI, and the impact his company is making on the gaming and museum industries.
Westworld, Mona Lisa and games: the X&Immersion interview
BeyondGames.biz: Please tell us a little bit about your background.
Alexandre Folliot: I’m freshly out of university. I studied in France and in Germany – machine learning and artificial intelligence. It was for image and sound.
I was a huge fan of roleplaying in video games, but I was not a player who played a lot; I was more of a quality player than a quantity player.
After my time at university, I quickly started a PhD in a similar area. And then, I decided to start my company on this theme.
So I’m pretty fresh on the market. Only two working years on it.
Please give us an overview of the company that you’ve founded. What is X&Immersion’s purpose?
The purpose is to bring narrative immersion into video games.
I’m convinced that today we have so many cool things going on with 3D objects and 3D worlds – and sometimes, I have the impression that the goal is to achieve the most beautiful game. Still, it’s not always about the most interesting game or what the story tells! I think newer games can sometimes have a really great story, but too often, the graphics seem more important than the narrative. Sometimes I have stronger feelings about games from the 2010s! I’m a huge Skyrim and Oblivion fan, the Elder Scrolls games, and love The Witcher games and so on.
And that was the idea. It’s what I think is going to be the future of the gaming industry. Considering dialogue, narrative, and story – adapting to what you just did inside the open-world game and the characters responding to that. It’s immersion and having the impression that you’re playing a tabletop RPG game, like pen-and-paper games.
That was the motivation to found X&Immersion. We will bring the impression of immersion and free will into gaming – in a way, I’m going to say, like Westworld! That was the motivation.
You have a logo on your site and business card, which is reminiscent of the Westworld maze map. What does Westworld mean for you? How has it influenced what you think of AI?
It’s very personal for me because, studying in Germany, I was on the path to mechanical engineering. I had some programming, but basically, it was mechanical engineering in my second year there. I was fascinated with Westworld.
AI was known, but it was a time in 2014 when the first real applications came along, for instance, Yann Lecun, if I’m going to cite one name in the machine learning research field.
I thought: you can do some interesting things in this area. There is so much to do there. And then, I changed my subject from mechanical engineering to artificial intelligence! I went to France because I thought there were some interesting schools there to learn that.
The goal of X&Immersion right now is to develop smart non-player characters. NPCs that understand what’s happening inside the game and can talk with the player, or between themselves, with dialogue generation. And they can speak this dialogue in a natural voice.
And then the last part is the animation of the face – it’s facial animation with emotion, depending on what the character wants to communicate, and to have emotion inside of this loop of communication.
We sell that as a software and a service, depending on the size of the company.
If someone’s developing a game right now, and they want NPCs who can interact with the player – do they come to you, and your software plugs straight in?
Completely. What’s really interesting is the training of the model.
I can tell you about one company. Someone had an investigation game, and you’re a pet – a smart pet – investigating inside a museum. And depending on what you notice inside the museum, you’re going to have all of the characters responding in a different way. If you notice that a painting is a fake, that will change everything in the storyline. You’re going to have all the characters talk in a completely different way than if you notice that there are non-edible hamburgers in the cafeteria. Then they’re going to have a completely different storyline.
This kind of thing, it’s just possible with an AI helping the narrative designers to help achieve million – literally millions – of dialogues.
Usually, the companies we’re working with come with the storyline, the basic idea. They come with context, like: “I want basic information about greetings or information about what’s happening inside the quest.”
Then you have some situations – like a stolen painting or a situation about hamburgers. And then you have the characters themselves. Depending on if it’s the director of the museum, or if it’s the cook or whoever. If you have a cook who is a dog, and he’s asking a lot of stupid questions. Or the director is maybe arrogant. So you have different dialogue situations. It’s very interesting.
The artificial intelligence model for these conversations: are you using GPT-3 or something similar that’s already available?
It would be great just to use that and have everything done! But the problem with GPT-3 is it’s a huge tool. It’s great, and ChatGPT has been really amazing in the last few months. But if you want to have a lot of custom information inside your game, you will be limited. And you cannot customise ChatGPT to do all that you wish. It’s going to be, at some point, very complicated.
So GPT-3 is really cool to start playing with. But then, inside your own game, you want more information. You want more context. And what we do is we use an open-source language model, and we retrain it for dialogue – and then for the situation and for the character. So if it’s a nice character, that character is supposed to be happy, and so on.
So that work we do – we select that asset. We retrain them and have them respond like an NPC inside the game.
Once you’ve got the dialogue, the character has to have a text-to-speech model. Then the emotional model as well, the character’s got to have the right facial expressions. Without giving away the secrets of your software, how do you interpret the language model into an emotional performance?
You’re right. It’s not the easiest part. What we do is, use the game itself. We use the mechanics inside the game to detect emotion. If you have a traditional game with three or four choices for the player, you’re going to have one nice choice, one bad choice, and two middle options.
That will feed inside the model because detecting the emotion is very complicated, and we are not working on this part itself. But we can use the emotion to generate a different output of the text in lots of ways.
That’s why it’s always custom-made, because we have to adapt, and I think it’s a good thing that each game is also different and has its own specifications. And we use the information provided inside the game to do that.
Is AI going to change the way we play games?
I hope so, and I hope in a good way because I’m a fan of the human input inside games. The first thing I want to say when I talk to a studio is that it’s not to replace narrative designers or animators. It’s more like: it’s a smaller amount of money, and you can do more things.
You have your ideas. You have your characters, and you design your characters. It’s not going to change. I’m not saying you have to build AI-developing-itself games. It’s more like you can have AI helping you generate millions of lines when you would have time to write only 1,000 of them, or something like that, depending on every different context. Then you can read them again and change everything you want to change.
So it’s still a human in control of it, and that’s pretty important. If it works well in the next year, I think we will see that in the same way, every AAA game now seems to be an open world. It’ll be the same with narrative. The next games are going to give the impression that you can do everything and that the game is responding to you.
You’ve also worked on an exhibition that’s started in Paris and is now on tour. What do people experience in that exhibition?
It’s a mini-game inside the exhibition. The exhibition is about Venice in the 19th century, called Venice Revealed. Our job was to create eight smart NPCs with different jobs in this Venice, from the courtesan to the shipbuilder to the dragoman (the person working as a translator and interpreter between different cultures in Venice).
We had a historian working with the museum. We trained the AI to respond as the different kinds of characters. And we did some facial and 2D animation for the characters. And we made a mini-game where you had three questions to ask the characters and then guess what their job is. You can ask them anything! It was not a very difficult game, but the idea was to transmit a bit of knowledge about the lifestyle and how people lived in Venice at that time. It was more like entertainment, like what you would find in some games, more than in a museum!
How did you come to be working on this project?
My associate Oriane works a lot with the culture industry to experiment and to see what’s interesting to do with museums. It was a good opportunity because they had one spot left in the exhibition. We told them what we could do, and they were pretty interested.
We had really good feedback from the visitors inside the museum because you had several thousands of discussions every day! We found 80% of the visitors stayed for at least one minute in front of the screen. It was really, really great when you compare that with the other interactions with the screens inside the exhibition. In a museum, people will stay between five and 10 seconds in front of a painting! So it’s 20 times longer. You have the impression you’ve done something inside the museum; you’ve learned something.
So that was the goal: to do it in a fun way. And it was also cool for kids. We had really good feedback from both kids and seniors because it was pretty appealing to all kinds of categories.
We’ve also seen your Mona Lisa “Convince Me” talking painting online. Was that part of the same thinking about how interactive museum exhibits could work?
Exactly. That’s from the idea of talking paintings, like in Harry Potter [laughs]. It could be precisely the same thing. It’s the “wow” effect. When you have a painting talking, like in the movies, and you see that for the first time, you’re like, “It’s cool.” And it’s really possible to do that.
That was our demo for the museum. And now, we’re also working on several prototypes and mini-games with clients to demonstrate how it works on the videogame side. It’s a bit different – it’s difficult to show games studios how effectively the tool works inside the game. You have to do something outside it, so they see: “Oh, okay, it’s fun. It’s working like this. I could use it this way.”
Do you think that the general public, or even gamers, understand the potential of AI yet? What is needed to educate people?
I think cool applications like we’re seeing with OpenAI or Midjourney can help. People are using them.
I saw many cool games coming out with 2D animation or 2D pictures. It’s pretty beta. People can use [those tools] and put an experience around it. But I’m not sure that you can have a lot of games only relying on this kind of application because you still need the touch of a human to really make characters like you want them to be inside a game.
I think for the public, it’s integrated enough to understand that it’s cool and that it can sometimes make mistakes, but sometimes that’s also the fun of it. Interesting times.
I think this process will be a bit longer, taking the next two to five years. And when you see the first games coming out with AI in, I would say, between next year and 2025, that’s going to be the big revelation. You’re going to have everyone talking about it, like what we have now with ChatGPT. If you have a game where you can have that chat thing with characters, it’s going to be a great game. The first [big IP] coming out from the games industry that has a great AI dialogue system is going to be like when we first saw a great game mechanic like the roof-jumping in Assassin’s Creed – amazing, suddenly we’re going to see it in all games!
With things like ChatGPT and various image tools, people are able to experiment with AI themselves. So unlike, say, the metaverse, a person can easily play around with some of these tools…
Yes. What was a great game – a great idea, at least! – and the first one to do that was AI Dungeon, using GPT-2, I think, or maybe even just GPT. It’s very fun.
But I’m not sure that there are going to be a lot of games doing the same thing as AI Dungeon because you were, at the time, limited in the options you had. But it’s a great idea and use case of this technology.
What advice do you have for a business looking to incorporate AI technology into its work? Where should they start?
What we try to do with our clients first is [identify] problems in production. What is the scope of your game? Where’s your pain? And sometimes, it’s going to be animation. Sometimes it’s going to be dialogue generation. Sometimes it’s going to be text-to-speech.
I hear some studios saying, “Okay, we don’t want voices because it’s too expensive.” So they sometimes limit themselves. They don’t achieve the game they would have wanted because they’re unwilling to use some tools already out there. Sometimes the tools are not good enough for what they want to do. But I would say that if you have the right data set, and the right time and money, you can pretty much achieve everything with AI. It’s automatic: you can make any tool you want to do.
I imagine it’s going to be step-by-step. But if a game wants more dialogue, with more options, animation, and voice, you can use an AI tool to help you with that.
Find out more about AI
We appreciate Alexandre Folliot taking the time to talk to us about X&Immersion’s vision and about the possibilities of AI in general. You can find out more about the company at its official website. There’ll be more conversations about AI in games at Pocket Gamer Connects Seattle in May.