Back in July of 2016, developers Niantic, Inc., an American software development company, released Pokémon Go, one of the most popular augmented reality mobile games for iOS and Android. The game went on to be downloaded more than 1 billion times.
Using the GPS system in the player’s phone, the game layers virtual Pokémons on to the terrain the device’s camera is focused on so as to appear as if the player and Pokémon are occupying the same real-world location at the same time.
At this point in time, virtual reality was nothing new and augmented reality systems were beginning to become more numerous. Today, however, we’re seeing these technologies merge and give rise to mixed reality applications.
So, what is the deal with Mixed Reality?
Mixed Reality (MR) is the use of both Virtual Reality (VR) and Augmented Reality (AR) technologies to create an environment where physical and virtual objects can exist and interact in real-time. MR overlays images or videos over a screen showing reality through a mobile camera, smart glasses or headsets.
This ability to interact with both physical and virtual objects gives mixed reality technologies a huge number of potential applications.
According to SuperData Research’s Virtual Reality Market and Consumers report, the investment in AR/MR for companies is set to reach $4 billion by the year 2020. With this in mind, it’s not hard to see how mixed reality devices like Microsoft’s HoloLens 2, a wearable holographic computer, could potentially become commonplace in schools, colleges, hospitals, and used in a multitude of other professions.
But MR will also be seen in the retail departments like e-commerce and fashion. As shown in the picture below, the first augmented reality dress designed by Marga Weimans, uses 3D-tracking and occlusion effects to allow the augmented additions to spin around the user.
In another market research on Augmented and Mixed Reality, performed by Juniper Research, it is believed that VR/AR apps will reach 10 billion installations by 2024. It is also predicted that the market will reach over $43.8 billion, for the same year.
The most popular uses for AR/VR apps, for now, is for Social Media and Gaming. Snapchat, Pokemon Go, and the new Harry Potter: Wizards Unite game are some of the biggest interests for the general public.
In this article, we’ll look at a few of the ways mixed reality technologies are being applied today and where this use could possibly lead future iterations of mixed reality technologies.
Common Mixed Reality Applications Today
So how is mixed reality being used today?
At present, Mixed Reality (MR) systems are still being developed and fine-tuned. This technology is still not as widely available as separate VR and AR devices or software, although there are some real-world examples of mixed reality technologies in use today.
Mixed reality devices like the Microsoft HoloLens 2 aren’t found everywhere just yet, although, with a steadily increasing number of applications and growing use in certain industrial and commercial fields, they certainly could be in the not-too-distant future.
As we’ll see, most of the current use cases of mixed reality technologies are bespoke to a specific task or objective and have different applications within different fields or sectors.
Mixed reality technologies are being used within the education industry to both enhance students’ ability to learn and take in information. It also gives the students the opportunity to personalize the way they learn.
Using 3D projections and simulations, students can interact with and manipulate virtual objects in order to study them in a way that is relevant to themselves and their studies. By inserting three-dimensional objects into a classroom as a means of gauging the size, shape, or other features of a defined “virtual” object, students can gain a deeper sense of understanding as to what it is they’re studying.
Some ways that MR can help in the classroom?
- Interact with the environment in an immersive experience.
- Touch and manipulate objects.
- It is an engaging and fun way of learning.
- MR can teach any kind of subject.
An example of MR technology in education is when the Students at Case Western Reserve University in Ohio were able to take advantage of the Microsoft HoloLens 2 in order to learn anatomy. This device also enabled professors to teach and interact with students despite being hundreds of miles away.
The following video shows these lenses in action in the Univerity of Ohio. It is really amazing to see the kind of interaction and knowledge that this device can give to students of anatomy.
Mixed reality in engineering is slowly but surely becoming a game-changer. From 3D modeling and virtual sculpting to remote repair guidance and project monitoring apps. There are various ways in which the engineering sector has begun to take advantage of mixed reality devices.
Some benefits in Engineering?
- Real-time simulation of engineering processes.
- Use MR with an industrial IoT device to monitor services.
- Engineering training.
For example, using 3D modeling apps on mixed reality devices, professionals are able to build their projects up in a shared virtual environment. This type of detailed 3D modeling + collaboration gives engineers the best chance for spotting errors while also allowing real-time manipulation of their designs. The collaboration environment allows supervisors to evaluate and check their 3D designs in real-time.
Project monitoring apps featuring MR, are also growing in popularity due to their ability to project 3D or 4D design models over structures as they are being built. This can then help engineers and construction workers visualize progress as well as to inspect the quality of what has already been built.
A popular player of VR/AR in engineering is Daqri, a company that creates augmented reality glasses for the industry. In one application, they used AR in an assembly of a gas turbine power plant. A step in the assembly that used to take 480 minutes, they got it done in less than 45 minutes with AR. The engineers didn’t require any training and they got it right away.
VR entertainment has been around for decades and because of this, it’s the most acclimatized and most widely available of the three technologies. Augmented reality games and apps are coming out thick and fast since the success of Pokémon Go and there’s little reason to suspect they’ll disappear out anytime soon.
But the mixed reality in entertainment is not only reserved for gaming.
AR and VR technology is improving so much that the film and big screen industry are after the vision of incorporating them into their movies. The idea of interactive storytelling which is a mix between a game and movie will open up lots of new doors for mixed reality. An example is the new Netflix series, You vs. Wild with adventure survivalist Bear Grylls, which lets you make decisions and change endings.
Mixed reality entertainment is already here with companies like Magic Leap, Lucasfilm, and Industrial Light And Magic all looking to delve into mixed reality entertainment. For example, Magic Leap’s is embracing mixed reality in cinema by using what they call a Dynamic Digitised Lightfield Signal. Magic Leap’s tech projects images directly into the eye, without the need for it to bounce off an object and then head towards the eyes. This tricks the brain into thinking the object is there when in reality, it is a projection.
The following picture shows Magic Leap in action, showing a whale in a school gymnasium.
When it comes to healthcare, mixed reality technologies have many potential applications. The most obvious is training and education. An example is the over-the-shoulder surgeries, where surgical students can be taught remotely by experts as they perform surgeries in real-time.
Another example is interactive learning. Topics like anatomy with mixed reality technology can be used to map the different layers of the human body. Being able to produce three-dimensional models of the anatomy complete with information accessible by just a simple gesture could change the way health care and medicine is taught.
MR will also transform the way in which medical students learn, using three-dimensional holograms in a virtual environment rather than two-dimensional diagrams from medical textbooks in base reality. As shown in the video below, this is how learning anatomy looks like with an AR/VR system.
There is also the potential in healthcare for mixed reality and Machine Learning (ML) to combine a doctor’s skills and create healthcare experiences. This allows doctors to take advantage of the intelligence and data provided by the ML and the visualization and interactivity capabilities of mixed reality technologies.
The potential for mixed reality applications in healthcare and medicine cannot be overstated. Aside from medical education, healthcare can also benefit from pre-procedural planning and enhanced visualization during critical surgeries.
Although it seems that mixed reality is all about the headset, it actually involves lots of high-performance infrastructure. Some MR technologies require multiple sensors and cameras. These sensors can be deployed externally in the environments while others like the Microsoft HoloLens include everything internally in the headset.
Usually, the MR technology headsets are composed of a set of:
- Infrared detectors
- Gaze trackers
The headset provides the interface between the viewer and the virtual world. But most of these MR applications will likely be deployed in the cloud. Running data from the cloud which could be stored thousands of miles away puts a lot of pressure on the bandwidth.
Fortunately, mixed reality technology can leverage key mobile broadband technologies such as 5G and MEC to provide ultra-low latency speeds. All the immersive VR/AR content stored in the cloud can be easily accessed and interacted with the right equipment.
An example of these IoT appliances that can support this kind of content delivery is the NCR-1510 which is compatible with Certified LTE modules like PGN-300 and PGN-600. Both of these modules have acquired AT&T certifications. The PGN-600 is compliant with FirstNet and CBRS LTE network bands.
As we’ve seen, mixed reality applications are now slowly emerging into various different industrial and commercial sectors. As development continues on mixed reality devices and technology, it seems sensible to expect even further integration of MR tech into our lives.
However, to fulfill its promise in commercial or industrial setting, VR will require a better network to operate in. The currently available bandwidth, latency, compute, and storage resources in most environments are bound to saturate or fail unless properly upgraded to meet the requirements for implementing a wide-scale VR or MR application.