top of page
artificial intelligence, holograms, vr, metaverse.jpg

AYE AI!

How we use AI in our VR training applications, Driving Simulators, Holographic experiences, and more.

At MXTreality (aka MXT) we use Artificial Intelligence extensively for a variety of products, solutions, and industries. 

Using AI: How, Where, Why, and When
 

How and why does MXT use AI?

MXT uses AI in various ways to create the most immersive experiences. For example, we use AI to:

Define the story

We leverage many common AI applications to help ideate, create initial imagery, develop timelines, and generate storyboards.  The use of AI allows for fast and iterative design concepts that accelerate our ability to prototype.

Improve your experience

We use AI to track your movements, change your voice, and understand the reasons for your actions and behaviors.  AI helps us understand why you do the things you do, and improve the experience based on this knowledge.

Create intelligent and responsive NPCs

AI is used to create the NPCs (non-player characters) that populate MXT's experiences. These NPCs are able to interact with the player in a realistic and intelligent way.  Each NPC (albeit a person, bird, car, or other vehicle is goal driven, meaning it has a purpose e.g. get from point A to point B.  We can also give each NPC sub-directives like follow NPC#1, or only cross the road on sidewalks.  For added complexity, we will often apply some randomness, so NPCs behave more humanlike, e.g. only use the crosswalks 95% of the time.

Personalize the experience for each user

AI is used to personalize the experience for each user of MXTreality.com's experiences. This can include things like adjusting the difficulty level, helping understand if you need help, changing the language, adding virtual voices, or acting as virtual concierge guiding you through an experience.

Where & why does MXT use AI?

AI assistant bot.jpg

AI assistant bot

In our EuropaPrime experience, we have created a helpful assistant social bot.  This bot can answer any question about your specific location, the environment, the tasks to complete, and even offer advice on how to solve the puzzles (yet only helpful, it won’t solve them for you).  Additionally, users can choose from either a factual bot or a social bot, allowing customization to the preferences.

speech bubbles in front of a holographic

Translation while training

An exciting recent development of AI, is the ability to efficiently handle multiple languages, both on the text to speech and speech to text approaches.  Specifically in our training and learning experiences, we are now able to offer non-native english speakers the ability to ask questions or receive answers in their native language, thereby increasing their immersion, enjoyment, and expediting their learning.

Europa UniVRsal Terc Minos uw.png

​Intelligent non-verbal communication

In a special underwater observatory on the moon of Jupiter, we created an intelligent being that communicated through light pulses.  We created an AI algorithm that was able to receive player messages and choose a response based on user interactions and timing.

use virtual reality and ipads to show a lady sitting at a desk seeing virtual holographic

Behavior Identification

For the AugmentedEF neurodiversity research project, we used custom created Large Language Models (LLM) to identify on and off task behavior.  The core aim is to help students study more effectively, by identifying their behavior and choosing from an array of intervention methods if they become distracted.

Prof.png

Virtual Voice Assistants

The Mars society has a new initiative called the MTI (Mars Technology Institute).  We have created virtual characters that replicate real world people, sounding like them, and even mimicking nuances and inflections in their knowledge, tone and speech structure

Charlie the AI octopus.jpg

Virtual Concierge

For the AugmentedEF neurodiversity research project, we used custom created Large Language Models (LLM) to identify on and off task behavior.  The core aim is to help students study more effectively, by identifying their behavior and choosing from an array of intervention methods if they become distracted.

SpaceNeedle Proto sasquatch.jpg

Holographic Avatars

To create realistic body tracking for our magic mirror experience at the Seattle Space Needle, we leveraged AI to track the various joints of the human body.  We then take thost movements and replay them on a different virtual character, so that when users look into the mirror, they see an avatar version of themselves (either a Sasquatch or Astronaut)

Jelly Flame 23.png

Europa jelly ecosystem

 In the underwater world of Europa, we have created a new species of deep sea animal that resembles a jellyfish. These creatures tend to move together and essentially follow a hive-mind collective intelligence.  To achieve this goal, we crafted a follow-the-leader AI algorithm that makes the movements interesting and realistic

Octopus 4.jpg

AI based interactions

We used AI to improve interactions like the mimicry of movements of hairs on our sasquatch and legs of our octopus, so that as these creatures move, their body’s realistically respond to their environments.

​

We also use AI to help create special drawing experiences.

Space Needle 360 Bungee.mp4 screenshot 2019-04-06 17-25-02.jpg

Seattle Bungee Jump

One of our first implementations of AI, was when we applied virtual brains to cars, birds, trains, boats, and pedestrians.  This meant that if you experienced the application more than once, you would never have exactly the same experience.

DoT MXT sim (9).JPEG

Pedestrians & Cars in DoT Drive Sim

For our driving simulator, we added a level of intelligence to our NPCs (non-player characters) like vehicles and people.  Each had a mission to accomplish, yet each had a mind of its own on how to accomplish that mission.  This leads to some very interesting and realistic scenarios, like people jaywalking, or cars following too closely.

Bike race.jpg

Cycle flock in DoT Drive Sim

For a special training module of our driving simulator, trainees must adapt to driving a bus amid a passing cycle race.  We implemented a special system that gave all cyclists the same goal, while also offering them the ability to have a nuanced perspective of adapting to their environment.

Want to know the AI Terms?

Learning AI is like learning a new language. Click this link to learn all about the key terms you need to know.

What are the benefits of using AI in our experiences?

There are a number of benefits to using AI in immersive experiences, including:
VR interactions, black background white iconogrpahy.jpg

Improved interaction

AI can be used to create intelligent and responsive NPCs that can interact with the player in a realistic way. This can make the experience more challenging and rewarding.

virtual robot assistant, black backgroun

Increased immersion

 AI can be used to add immersiveness to our virtual environments through virtual assistants, knowledgeable helpers, or more realistic environmental details. This can make the experience more enjoyable and engaging for the user.

personalization, black background, white

Personalized experiences

AI can be used to personalize the experience for each user, to help keep focus, answer questions, or offer you the option of companionship. This can make the experience more relevant and engaging for the user.

What are the challenges of using AI in immersive experiences?

There are a number of challenges associated with using AI in immersive experiences, including:

  • The need for powerful hardware. AI requires a lot of computational power, which can make it difficult to run AI-powered immersive experiences on mobile devices or other low-powered devices.

  • The need for large datasets. AI requires large datasets to train on. This can be difficult to obtain, especially for niche applications like the motion capture movements of children vs adults (most models are trained on adult body sizes and movements).

  • The need for expertise. AI is a complex technology that requires specialized expertise to use effectively. This can make it difficult for companies to develop AI-powered immersive experiences in-house

AI CONCLUSION

AI is set to revolutionize the way we experience immersive applications. By using AI, MXT (and companies like us) can create more realistic, immersive, and personalized experiences for users. However, there are a number of challenges associated with using AI in immersive experiences that need to be addressed.  Luckily, MXT is at the forefront of AI implementation, and we will continue to ride this wave as it helps us, help you!

bottom of page