
AYE AI!
How we use AI in our VR training applications, Driving Simulators, Holographic experiences, and more.
At MXTreality (aka MXT) we use Artificial Intelligence extensively for a variety of products, solutions, and industries.
Using AI: How, Where, Why, and When
How and why does MXT use AI?
MXT uses AI in various ways to create the most immersive experiences. For example, we use AI to:
Define the story
We leverage many common AI applications to help ideate, create initial imagery, develop timelines, and generate storyboards. The use of AI allows for fast and iterative design concepts that accelerate our ability to prototype.
Improve your experience
We use AI to track your movements, change your voice, and understand the reasons for your actions and behaviors. AI helps us understand why you do the things you do, and improve the experience based on this knowledge.
Create intelligent and responsive NPCs
AI is used to create the NPCs (non-player characters) that populate MXT's experiences. These NPCs are able to interact with the player in a realistic and intelligent way. Each NPC (albeit a person, bird, car, or other vehicle is goal driven, meaning it has a purpose e.g. get from point A to point B. We can also give each NPC sub-directives like follow NPC#1, or only cross the road on sidewalks. For added complexity, we will often apply some randomness, so NPCs behave more humanlike, e.g. only use the crosswalks 95% of the time.
Personalize the experience for each user
AI is used to personalize the experience for each user of MXTreality.com's experiences. This can include things like adjusting the difficulty level, helping understand if you need help, changing the language, adding virtual voices, or acting as virtual concierge guiding you through an experience.
Where & why does MXT use AI?

AI assistant bot
In our EuropaPrime experience, we have created a helpful assistant social bot. This bot can answer any question about your specific location, the environment, the tasks to complete, and even offer advice on how to solve the puzzles (yet only helpful, it won’t solve them for you). Additionally, users can choose from either a factual bot or a social bot, allowing customization to the preferences.

Translation while training
An exciting recent development of AI, is the ability to efficiently handle multiple languages, both on the text to speech and speech to text approaches. Specifically in our training and learning experiences, we are now able to offer non-native english speakers the ability to ask questions or receive answers in their native language, thereby increasing their immersion, enjoyment, and expediting their learning.

Behavior Identification
For the AugmentedEF neurodiversity research project, we used custom created Large Language Models (LLM) to identify on and off task behavior. The core aim is to help students study more effectively, by identifying their behavior and choosing from an array of intervention methods if they become distracted.

Virtual Voice Assistants
The Mars society has a new initiative called the MTI (Mars Technology Institute). We have created virtual characters that replicate real world people, sounding like them, and even mimicking nuances and inflections in their knowledge, tone and speech structure

Virtual Concierge
For the AugmentedEF neurodiversity research project, we used custom created Large Language Models (LLM) to identify on and off task behavior. The core aim is to help students study more effectively, by identifying their behavior and choosing from an array of intervention methods if they become distracted.

Holographic Avatars
To create realistic body tracking for our magic mirror experience at the Seattle Space Needle, we leveraged AI to track the various joints of the human body. We then take thost movements and replay them on a different virtual character, so that when users look into the mirror, they see an avatar version of themselves (either a Sasquatch or Astronaut)

Europa jelly ecosystem
In the underwater world of Europa, we have created a new species of deep sea animal that resembles a jellyfish. These creatures tend to move together and essentially follow a hive-mind collective intelligence. To achieve this goal, we crafted a follow-the-leader AI algorithm that makes the movements interesting and realistic
_JPEG.jpeg)
Pedestrians & Cars in DoT Drive Sim
For our driving simulator, we added a level of intelligence to our NPCs (non-player characters) like vehicles and people. Each had a mission to accomplish, yet each had a mind of its own on how to accomplish that mission. This leads to some very interesting and realistic scenarios, like people jaywalking, or cars following too closely.

Cycle flock in DoT Drive Sim
For a special training module of our driving simulator, trainees must adapt to driving a bus amid a passing cycle race. We implemented a special system that gave all cyclists the same goal, while also offering them the ability to have a nuanced perspective of adapting to their environment.
What are the benefits of using AI in our experiences?
There are a number of benefits to using AI in immersive experiences, including:
Improved interaction
AI can be used to create intelligent and responsive NPCs that can interact with the player in a realistic way. This can make the experience more challenging and rewarding.

Increased immersion
AI can be used to add immersiveness to our virtual environments through virtual assistants, knowledgeable helpers, or more realistic environmental details. This can make the experience more enjoyable and engaging for the user.

Personalized experiences
AI can be used to personalize the experience for each user, to help keep focus, answer questions, or offer you the option of companionship. This can make the experience more relevant and engaging for the user.
What are the challenges of using AI in immersive experiences?
There are a number of challenges associated with using AI in immersive experiences, including:
-
The need for powerful hardware. AI requires a lot of computational power, which can make it difficult to run AI-powered immersive experiences on mobile devices or other low-powered devices.
-
The need for large datasets. AI requires large datasets to train on. This can be difficult to obtain, especially for niche applications like the motion capture movements of children vs adults (most models are trained on adult body sizes and movements).
-
The need for expertise. AI is a complex technology that requires specialized expertise to use effectively. This can make it difficult for companies to develop AI-powered immersive experiences in-house
AI CONCLUSION
AI is set to revolutionize the way we experience immersive applications. By using AI, MXT (and companies like us) can create more realistic, immersive, and personalized experiences for users. However, there are a number of challenges associated with using AI in immersive experiences that need to be addressed. Luckily, MXT is at the forefront of AI implementation, and we will continue to ride this wave as it helps us, help you!


