HDG Explains : How Does Augmented Reality Work?

Everyone has some idea of what VR (virtual reality) is, but it’s cousin AR (augmented reality) is much less famous. Yet, it’s likely that AR will be a much more commonplace technology than VR in the future.

So, what is AR and how does augmented reality work? We’ll answer both of these questions for you right now.

AR, VR and MR Compared

Augmented reality is one technology in a family of technologies. It’s joined by virtual reality and mixed reality as what’s now known as “XR” or extended reality. Extended reality refers to the entire spectrum of reality-altering technologies, from fully-virtual worlds, to full integration between the real and the virtual. The dividing line between these three points on the spectrum isn’t clear cut, but the basic definitions go something like this:

• Virtual reality completely immerses you in a virtual world, replacing what you see and hear
• Augmented reality overlays digital information over what you see in real life
• Mixed reality integrates digital sounds and imagery completely with the real world.

The dividing line between AR and MR is the fuzziest of all. For example, a heads up display in a car or inside a motorcycle helmet technically qualifies as augmented reality. Then again, most of what’s labeled as augmented reality these days is actually firmly in the mixed reality part of the spectrum. As such, in this article, we’ll be including mixed reality as part of the discussion.

Marker-based, Markerless and Location-based AR

Augmented reality applications usually need some way to sense the outside world, so that there’s a reference point to render visuals and audio. There are three approaches to this:

• Marker-based AR uses a predefined object in the real world and maps the AR content to it
• Markerless AR can use arbitrary objects in the real world as anchor points, or doesn’t need to be anchored at all.
• Location-based AR is triggered by the user being in a specific place,such as various locations in a zoo or museum

Plenty of early AR experiments need markers. For example, games such as The Eye of Judgement used a camera attached to the Playstation 3 to bring life to paper playing cards. You’d also see marker-based AR with objects such as movie posters, or QR codes in books.

The real advancement in AR/MR technology comes from markerless methods, which deserve their own more detailed discussion.

AR and MR Tracking Technologies

If you give a computer system an accurate, real-time map of a space such as a room, you can render graphics mapped to that space. Making it appear as if those virtual objects exist in the real world.

At first, the main way developers tried to achieve the ideal of markerless augmented reality was to integrate sophisticated sensors into devices like smartphones. These include laser systems that scan objects and surfaces, stereoscopic cameras, infrared sensors, sonar and more.

A notable example of this approach was Google’s Project Tango. An augmented reality hardware platform that made its way into several phones and tablets. The technologies developed in projects like Tango were crucial to the AR we use today, but the expense and complexity of these specialized devices turned out to be their demise. However, Apple computers might have had something to do with it.

Mobile AR and MR Technology

While Google was working on Tango, Apple dropped a bombshell in the form of ARkit. A software platform which could create markerless, persistent augmented and mixed reality experiences using nothing more than the standard cameras on existing iPhones andiPads.

Thanks to sophisticated machine vision algorithms, iOS app developers could easily build AR functionality into their software. This rendered the specialized hardware approach of Project Tango effectively obsolete overnight.

Google did eventually respond with the ARCore API, which allows Android app developers to do more or less the same thing as they can on iOS using ARKit. Both of these programmer resources have created a stable and reliable base for mobile augmented reality to enter the mainstream. This lowered the bar for both users and developers.

AR and MR Hardware Products

Though ARKit and ARCore both make quality AR possible on just about any modern smartphone, that doesn’t mean the idea of dedicated augmented- or mixed-reality hardware is dead.

First of all, phones are now coming with sensors that make AR better. Apple’s latest iPad Pro models sport a LiDAR sensor. This is the same 3D-sensing technology that robots and self-driving cars use. Presumably future iPhones will also get this new miniaturized version of the technology.

Several virtual reality headsets now use head-mounted cameras for tracking and also for mixed reality applications. Windows Mixed Reality headsets are explicitly designed for this purpose. There are also persistent rumours of an AR headset from Apple, supposedly the final goal of their ARKit development, LiDAR technology and acquisition of many AR specialist companies.

We also have high-end AR headsets that already exist, such as the much-delayed Magic Leap and the Microsoft Hololens. However, these impressive devices have a price tag where if you have to ask how much, it’s probably too much. For now, those headsets are aimed at enterprise and educational contexts. However, you can be certain that consumer-grade versions will come to market eventually, just as they did with high-end VR.

Awesome Augmented Reality Apps You Can Try Right Now

The really wonderful thing about augmented- and mixed- reality is that you don’t just have to read about it. Most of our readers have access to a smartphone or tablet, which means you can head over to your respective apps stores and try them right now!

Whether you’re on an iPhone or Android handset, we suggest you try Pokemon Go first. This addictive take on the Pokemon formula has you trekking through the real world and encountering wild pokemon, which you must then catch. These cute critters are projected into the real world world using augmented reality. So it feels like they’re present in your space.

Ikea Place is a neat application that lets you see what that Ikea furniture you’ve been eyeing will actually look like in your home. A genuinely useful application of AR!

On iOS, MeasureKit is a paid AR utility that lets you quickly determine the dimensions of objects in the real world. Over on Android you can get a similar tool, Measure, for free. Either way, no need to guesstimate things any more!

Augment Your Life!

The future of augmented reality may be much more important than we think. If head-mounted AR devices become small enough and can produce imagery with enough quality, they could replace every other display type that exists. Imagine a world where televisions or computer monitors are no longer needed, because your personal AR glasses can simply project a virtual TV on the wall.

In the long term, augmented reality technology could remove the need to decorate your home or for companies to put billboards or other physical visual material in the world. A world with affordable, advanced AR technology may very well look very different to the one we know today!