Over the last few years, Microsoft has staked out an interest and position in augmented reality (AR) as opposed to virtual reality (VR). As the name implies, “augmented” reality is about showing the user additional information or cues over and above what they can see in the real world, while virtual reality is used to project an entirely different environment from wherever the user is actually standing. Microsoft’s HoloLens is one of the first high-profile AR headsets, while games like Pokemon Go have become smash hits in the real world thanks to the way they leverage augmented reality to project virtual Pokemon into every day life.
Apple has stayed out of either market to-date, but that may be about to change. In a recent interview with the Washington Post, Cook was asked what he thought of augmented or virtual reality. He responded:
What’s interesting about this isn’t the fact that Cook is aware of or interested in AR — plenty of companies undoubtedly are, especially after the success of Pokemon Go — but that he’d describe it as a core technology when Apple, to date, has done absolutely nothing with the feature.
Apple’s greatest strength, historically, has been its ability to capture new interfaces and adapt them for easy use. Apple didn’t invent the mouse or the modern GUI, but it popularized both with its initial Mac 128K in 1984. The iPhone wasn’t the first smartphone or the first device with a touchscreen, but it was the first smartphone to treat touch as the primary means of interacting with a device without a need for a stylus or keypad.
Augmented reality has the potential to rewrite the rules of interface design like few products before it. AR has been a mainstay of science fiction for decades — a tricorder is an AR device with a sophisticated sensor package that can scan a local area for a wide range of measurements, then report those measurements back to the user. While Star Trek rarely showed precisely what the user saw as output, multiple Star Trek episodes feature the device being used to scan for a general anomaly before honing in and following it in a specific direction. Deep Space Nine established that the bridges of Jem’Hadar warships don’t have forward-facing viewscreens — they have wearable devices that project a heads-up display to each individual. While this isn’t exactly the same as AR technology, there’s nothing about the system that implies it can’t be used to communicate other information as well.
But — and this is key — in order for AR to become more than a minor feature, it’s going to have to evolve and grow up. This has significant ramifications for every aspect of system design (and a few for the real world as well). Pokemon Go already has a reputation as a battery killer, precisely because it leverages multiple sensors and devices within your smartphone at the same time.
We don’t know how Apple envisions its AR system working or whether its part of the company’s car project or if it would roll out on an iPhone. But integrating AR into software is difficult, and demonstrating its use case is even trickier. Niantic had a hit with Pokemon Go because it was able to trade on an established game franchise and use expertise gleaned from its previous AR title, Ingress. Any push to bring AR into the real world as a general use case will have to learn from previous high-profile attempts that have gone awry (Amazon’s short-lived Fire Phone, for example, could use the device’s cameras to show you products you could buy online).
It’ll probably still be a few years before we start to see AR showing up in devices and software as a rule, but it’s not too early to plan for it — and Apple has good reason to try and position itself to catch that wave. That’s probably what Apple had in mind when it bought AR software developer Metaio last year as well as Flyby Media in January 2016.