What has Apple got planned for Augmented Reality (AR)? Here are all the rumors, rounded up!

Augmented Reality (AR) or Mixed Reality is different than Virtual Reality (VR). Rather than being immersed in a fully-rendered world, AR/MR layers data on top of the real world. Rumor has it, Apple's special projects group is exploring AR for a variety of applications.

Apple doesn't see AR as a single product, like Microsoft's HoloLens or Snap's Spectacles, but as a core technology that will be part of many products coming in the future. Just like LCD and OLED are part of many products today.

So, what could Apple be doing with AR? Let's round up the rumors!

June 5, 2017: Apple announces ARKit at WWDC

Apple announced ARKit at WWDC 2017. It's a framework available as part of iOS 11 for iPhone and iPad that allows for impressive on-device AR. How about that!

From Apple:

iOS 11 introduces ARKit, a new framework that allows you to easily create unparalleled augmented reality experiences for iPhone and iPad. By blending digital objects and information with the environment around you, ARKit takes apps beyond the screen, freeing them to interact with the real world in entirely new ways.

Everything you need to know about ARKit

June 3, 2017: Rumor claims Apple AR glasses postponed, maybe canceled

Google had glass, which was more personal display than AR. Microsoft has HoloLens. Snapchat has spectacles. And Apple? While rumors have suggested Apple was working on glasses of its own, a new supply chain AMA suggests that product might have been pushed back — or even killed entirely.

From Reddit

65% Project Mirrorshades / Iris will be cancelled. It's a very unique design, if it can be done properly it will be like what the iPhone 10 years ago.

Rough bill of materials of Project Mirrorshades - Apple Iris. Note Iris is Siri backwards. Kopin NED Acetate frame Polarisized or prescription lens with Zeiss smart optics Bone conduction modules Microphones (noise cancellation) Light sensor Accelerometer for step tracking and head movement app navigation Magnetometer for navigation Capacitive Pavel Ceramic battery Apple chipset Charging circuit BL5 Induction module

Apple will likely introduced AR through iPhone or iPad, perhaps using Pokémon Go as a demo, long before we see glasses. But, AR is a long game for Apple, so we'll also have to wait and see where the technology takes them. And us.


Wait, how do we even know Apple is working on AR?

Apple has the intelligence and resources to work on anything they want to. They can explore, prototype, and if they feel something has merit, move into production. Not every project makes it that far. The Apple Watch did. The Apple television set did not. The Apple Car project? We'll see. Same for Apple and AR.

First off, Apple bought Mataio, a well known and highly regarded AR company, a few years ago. According to TechCrunc:

Apple has acquired Metaio, an augmented reality startup that launched way back in 2003 as an offshoot of a project at Volkswagen. The company's site said it stopped taking new customers, and now a legal document shows Apple has bought it. The document confirms a transfer of shares of the startup to Apple on May 21st/22nd.

When asked by TechCrunch, Apple responded with its standard reply it gives as confirmation of acquistions, "Apple buys smaller technology companies from time to time, and we generally do not discuss our purpose or plans."

Secondly, Apple CEO Tim Cook has commented on AR a few times already.

Talking to The Independant on February 10, 2017:

I'm excited about Augmented Reality because unlike Virtual Reality which closes the world out, AR allows individuals to be present in the world but hopefully allows an improvement on what's happening presently. Most people don't want to lock themselves out from the world for a long period of time and today you can't do that because you get sick from it. With AR you can, not be engrossed in something, but have it be a part of your world, of your conversation. That has resonance.

I regard it as a big idea like the smartphone. The smartphone is for everyone, we don't have to think the iPhone is about a certain demographic, or country or vertical market: it's for everyone. I think AR is that big, it's huge. I get excited because of the things that could be done that could improve a lot of lives. And be entertaining. I view AR like I view the silicon here in my iPhone, it's not a product per se, it's a core technology. But there are things to discover before that technology is good enough for the mainstream. I do think there can be a lot of things that really help people out in daily life, real-life things, that's why I get so excited about it.

On October 13, 2016, while speaking to BuzzFeed:

"There's no substitute for human contact," Cook told BuzzFeed News. "And so you want the technology to encourage that." It's not the first time Cook has indicated that Apple might favor AR. "We are high on AR for the long run," Cook said during an earnings call this past summer. "I think AR can be huge."

"VR has some interesting applications, but I don't think it's a broad-based technology like AR," Cook explained. "Augmented reality will take some time to get right, but I do think that it's profound. We might … have a more productive conversation, if both of us have an AR experience standing here, right? And so I think that things like these are better when they're incorporated without becoming a barrier to our talking. … You want the technology to amplify it, not to be a barrier."

During Apple's Q2 2016 conference call on July 26, 2016:

In terms of AR and the Pokémon phenomenon, it's incredible what has happened there. I think it's a testament to what happens with innovative apps and the whole ecosystem and the power of a developer being able to press a button, so to speak, and offer their product around the world. This certain developer has elected not to go worldwide yet, because of the pressure on their servers, etc. because of the demand, but I'm sure that they will over time. It also does show, as you point out, that AR can be really great and we're— we have been and continue to invest a lot in this. We are high on AR for the long run. We think there's great things for customers and a great commercial opportunity. So we're investing, and the number one thing is to make sure our products work well with other developers kind of products like Pokémon. And that's the reason you see so many iPhones out in the wild right now chasing Pokémon.

You know, I know there's people that want to call it a new computer platform, and we'll see. I think there's a tendency to call everything new the next computer platform, however, that said, I think AR can be huge. So, we'll see whether it's the next platform. Regardless, it will be huge.

Recently, Patently Apple noticed that Apple has updated its trademark for the Mac Pro to include a mention of "augmented reality displays," though without referring to a specific product.

Yet some of the entries in their filing definitely stand out, if not scream out, as real possibilities. The most interesting one of all is mentioning 'augmented reality displays," because Apple has a patent for such a device and we know that Apple's Augmented Reality team just keeps growing with all-star engineers. The displays could also be a part of a headset or smartglasses that could be powered by the Mac Pro.

Stop. Pokémon?

Pokémon Go was a huge success last year. We can argue about whether it's really Augmented Reality or just clever live view integration, but it succeeded in immersing millions of people in a hyper-real experience. And Apple — and everyone else — took notice.

It just so happens that everything Apple has been adding to iPhone over the years, including motion and camera sensors and screaming fast processors, are the same core technologies that can enable AR.

You have to be able to pull in reality, after all, before you can augment it.

<

div class="closeup bg-image" style="background-image: url('https://www.imore.com/sites/imore.com/files/styles/xlarge_wm_blw/public/field/image/2016/09/ios-10-hero-camera-outdoor-shot.jpg')">

<

div class="extra" markdown="1">

What about that transparent iPhone rumor? RIGHT?

Ha. You're referencing Robert Scoble:

More on iPhone 8. Just had a source that I trust tell me it is a clear piece of glass. With the compute in a strip at the bottom. Translation it does Mixed Reality. VR, AR, and AI.

He also talked to me about Siri. Asks "why did Viv.ai sell to Samsung? I know the founders didn't want to do that." So Apple showed them the new Siri for mixed reality and the game was over.

Scoble followed up with:

The next iPhone will be, I am told, a clear piece of glass (er, Gorilla Glass sandwich with other polycarbonates for being pretty shatter resistant if dropped) with a next-generation OLED screen (I have several sources confirming this). You pop it into a headset which has eye sensors on it, which enables the next iPhone to have a higher apparent frame rate and polygon count than a PC with a Nvidia 1080 card in it.

The clear iPhone will put holograms on top of the real world like Microsoft HoloLens does. Eye sensors will also bring a new kind of user interface.

It's less likely Apple will have an actual transparent iPhone, complete with transparent battery inside. It's more likely a great camera on the back combined with an edge-to-edge display on the front, will make it seem "transparent" under AR-like circumstances.

What about Apple Glass / Snaptacles? Will we get a headset?

That's one of the rumors! At some point having to hold a phone in your hand becomes less convenient than simply putting on a pair of glasses or contacts or otherwise intercepting and altering the photons going into your eyes.

Gizmodo got its hands on an Environmental Health and Safety report in April 2017 that seemingly implies employees have been testing some sort of AR or VR unit.

One report on February 21 that included "medical treatment beyond first aid," involved a prototype unit at Apple's De Anza office in Cupertino. "After BT4 user study, user advised study lead, that she experienced discomfort in her eye and said she was able to see the laser flash at several points during the study. Study lead referred her to optometrist and secured prototype unit for analysis." In another report, an employee working at Apple's Vallco Parkway office in Cupertino reported eye pain on March 2. "Employee reported eye pain after working with new prototype, thought it may be associated with use. He noticed that the security seal on the magenta (outer) case had been broken and had thought the unit may have been tampered with." A source inside Apple speculated that this injury may have something to do with an augmented reality product Apple may be testing, something like glasses with an overhead display.

Injuries or no, what, if anything, Apple might do with visual wearables is interesting to imagine. The Financial Times, on March 27, 2017:

Apple first began to build a team to examine the feasibility of a head-worn device more than a year ago. Now, it is devoting more resources to its augmented-reality efforts, with the aim of taking it from a science project towards a consumer product, according to people familiar with the company's plans. However, any launch is still at least a year away, perhaps much longer. Apple declined to comment.

As its engineers have become more adept at miniaturisation technology with products such as its AirPods wireless headphones and the iPad's Pencil, AR seems to have overtaken Apple's secretive car project as the company's top priority for its next big launch, beyond the iPhone.

Spoiler: Apple's special project group works on multiple projects at once and priorities change depending on how far along a project is and how Apple feels it can make a difference in the market.

Bloomberg, November 14, 2016:

Apple Inc. is weighing an expansion into digital glasses, a risky but potentially lucrative area of wearable computing, according to people familiar with the matter.

While still in an exploration phase, the device would connect wirelessly to iPhones, show images and other information in the wearer's field of vision, and may use augmented reality, the people said. They asked not to be identified speaking about a secret project.

When should you expect AR from Apple?

Same as artificial intelligence (AI), we're already seeing AR from Apple. And we'll continue to see more of it as time goes on. Every product release will likely use more and more of AR and AR-related technologies.

As to Apple mixed reality glasses, we can only ever expect those when we see them. Or see through them.