Cuseum, a company focussed on implementing augmented reality to improve museum experiences, is working with the Isabella Stewart Gardner Museum in Boston. In March 1990, thieves broke into the museum and stole 13 pieces of art worth an estimated $500 million. Today, the works have still not been recovered.
This year, on the 28th anniversary of the heist, Cuseum decided to use augmented reality powered by Apple’s ARKit to digitally put the stolen paintings back into their frames. Visitors are now able to see the pieces exactly where they originally were in the museum.
The museum is also offering a $10 million reward for any information that aids in the return of the stolen works.
Why it’s hot: This is a simple way to combine digtial and traditional ways to experience a museum, and leveraging AR to (somewhat) bring these paintings “home”.
Details are a bit scant, but eBay announced this week it will soon be integrating AR functionality into its app.
Per Fortune, “The San Jose, California-based marketplace said it’s working on an AR kit that, for example, will let car enthusiasts see how the images of new wheels would look on their vehicles before making a purchase. Another feature will help sellers select the correct box size for an item by overlaying an image of the box on the merchandise.”
Why It’s Hot:
While not the newest kid on the block (eg, Ikea has used AR for years), eBay is a massive marketplace where millions of people globally buy and sell things. With physical retail integrating technology to fight back against the convenience of e-commerce, this is an example of e-commerce trying to bring elements of physical retail to the digital world. One of the big disadvantages of e-commerce is usually you’ll only see a bunch of images of a product, which in eBay’s case may or may not be of the actual product you’re buying. The ability to see what something looks like in a virtual 3 dimensions is a major new advantage.
Instead of spending money on an ad, Ally Bank created an Augmented Reality game for this year’s Super Bowl. As explained in the video above, while other brands spent big money on the big game, Ally’s “Big Save” app allowed users to compete to see who could grab the most virtual money, after identifying what they were saving money toward. The game allegedly only activated during commercial breaks, and users would tap and drag AR dollar bills into a small AR piggy bank. The user with the highest score / most virtual money saved got a real cash money prize to be used toward their real-life savings goal.
Why It’s Hot:
Without knowing how successful it was, their different approach to the Super Bowl as a marketing moment is interesting. On one hand, it’s nice to see them trying to do some good with their budget. It also gave them specific insight into the things users were saving for that they could use later for marketing or to create products addressing those things. On the other, it seems a very noisy time to try and get people to ignore friends, family, and entertainment to play a game, albeit one with a nice prize. Either way, you can appreciate their attempt to hijack peoples’ attention during the commercial breaks coveted by other marketers.
Fortuitously timed, a genius developer has created an app that lets you appear to wield a Star Wars styled Light Saber using Augmented Reality. Per its creator:
“It’s an iPhone app that turns a rolled up piece of paper into a virtual lightsaber. I think the best thing about it is that it brings a special effect that has typically been reserved for advanced video editors to a mass audience.”
Why It’s Hot:
Augmented Reality has of course seen many new uses since becoming a widely available capability on iOS. Some are useful, and some just let you live out childhood fantasies like this. In either case, it’s amazing the digital layer of the world we are building on top of the physical one we have known for our entire lives.
Brain Power is a suite of AR and VR-like apps that work with Google Glass, to help people with autism learn crucial social interaction skills.
A few examples – “Emotional Charades” teaches them to identify emotions in real peoples’ faces. “Transition Master” helps them get comfortable with new circumstances before entering them. And “Face2Face” teaches them to make eye contact with others.
It makes it all a game, but unlike other types of teaching moments on an iPad, they’re always experiencing things without the artificial interference of a device screen, where really they’re just interacting with themselves.
Why it’s hot:
Earlier in the year, we talked a lot about the augmented self – how technology was helping us become almost super-human. But, it’s not just that. As Brain Power shows, it’s even helping learn basic human skills we might have a hard time with otherwise. While this is for autistic children to learn social skills, there’s no reason any child couldn’t learn through digital technology that feels like real life.
Plus, it’s another example of hardware getting out of the way. The only device needed here is the wearable Google Glass, which makes the experience feel like real life with a digital layer, rather than an artificial, screen-based experience.
Amazon joins the augmented reality scene with a feature on their application called AR view. This allows customers to virtually view how an item would look in their home prior to their purchase.
When a customer gets on the Amazon application they access the feature through the small camera icon located at the top right of the screen, and then choose the AR view option. From there, they can locate thousands of products to virtually place into their home to see how they would look. The customer viewing the item can rotate it around in a 360 degree fashion to see how it would look from multiple different angles in their home. This feature was announced alongside Amazon opening their Black Friday Deals Store.
The feature comes exclusively to Amazon application users who have an iPhone with the iOS 11 update. Amazon plans to make the feature available for Android phones sometime in the future.
Amazon announced a new augmented reality (AR) functionality for the Amazon App that will give shoppers a chance to envision real-world products around their homes before deciding to buy them.
Amazon didn’t specify exactly which of its offerings will be optimized for the app, but it claims that “thousands” of items across multiple product types will be viewable in AR. You can check out exactly how the tool works in the video below.
The app update is now available for the Amazon app on iOS 11 via Apple’s ARKit, so for now AR View is strictly for shoppers that have iPhones dating back to the 6S. Amazon didn’t share any plans to expand to Android phones.
AR visualization is a growing trend as the tech becomes more common, thanks to new efforts from Apple and Google. Home goods giant Ikea offered one of the first apps using the new ARKit for its customers back in September, while Google teamed up with Wayfair to show off a similar functionality for Tango phones on a mobile version of Chrome at the I/O conference in May.
Amazon is ramping up the tech offerings, giving us voice ordering with Alexa, AI style guidance with the Echo Look, and now AR functionality. The services are all cool shortcuts to make shopping easier than ever — which is exactly what Amazon wants to drive sales.
Why It’s Hot
For someone who is 15 steps ahead of the tech game, this is quite a lag for Amazon
Though late, Amazon continues to extend its world-class UX experience
This is yet another big ripple made created by iPhone’s ARKit
Quartz News, the digital-focused arm of Atlantic Media, has launched a nifty AR feature in their news stories thanks to the new updates to iOS11. Now in a select number of the daily stories featured in the Quartz News mobile app you will find augmented reality to help illustrate objects featured in a particular story. For instance, its coverage of the demise of the Cassini spacecraft is joined by a 3D model of the ship that users can examine as if it was physically in the same room with them.
Why It’s Hot:
Of all the emerging technologies that companies have their eyes on, augmented reality seems to be the easiest to scale by way of mobile phones with no need for extras like headsets or glasses. Apple CEO, Tim Cook believes that many people will “have AR experiences every day, almost like eating three meals a day. It will become that much a part of you,” Cook predicted at a tech conference last year.
Quartz News sees this as an opportunity to bring news stories to life in ways that users have never experienced before. “In the same way we can use images and emoji and gifs to bring alive the stories we’re sharing, we think we can use AR to help people understand objects in the news,” John Keefe, head of Quartz’s Bot Studio, adding that the tech could also be used to illustrate stories with 3D landscapes, models of landmarks and historic structures, or even data visualizations.
Kabaq is an augmented reality menu that will let you preview a 3D version of each dish before you commit to it. Users who want to see their order before they are served can the menu item and interact with a 360-degree simulation. This helps visualize ingredients, portion sizes, and side dishes. The app is meant to inspire cautious eaters to try new dishes.
Currently, 15 restaurants have signed on to test out Kabaq, with around 150 food items onboard. The AR service costs between $150-$200 each month. Aside from menu previews, the 3D models can also be used on websites, social media, and marketing materials, as well as accessible in the Kabaq AR app. The startup is working on an API to make the 3D food library accessible for food delivery, cookbooks, catering and menu prep services in the future.
This might help some restaurants from a ‘wow’ factor perspective, however I do think this is not a great thing for a few reasons.
This has the risk of increasing the time patrons spend at restaurant tables, increasing the turnaround time for new customers – which might threaten overall restaurant revenue and user experience (longer wait times)
Unless the restaurants have some great, photo worthy food, this might deter some users from ordering what they might have previously.
…you could interact with pretty much anything your mind can dream up.
Disney Research developed a somewhat lo-fi solution for mixed reality that requires no special glasses or singularity type of stuff. Its “Magic Bench” allows people to interact with things that aren’t there, watching the action in 3rd person view, on a screen broadcasting them. It even provides haptic feedback to make it feel like the imaginary character or object truly is on the bench with you.
Why It’s Hot:
1) It’s a great example of technology enabling a physical experience without getting in the way. Historically, augmented/mixed reality required some type of personal technology like glasses/headset, or a phone. This requires nothing from the user but their presence.
2) It shows how Disney is using technology to create experiences that extend its “magical” brand into the digital age.
Thanks to the recent release of Apple’s ARKit, front end developer Frances Ng has created a point-and-translate app. That’s right, simply point your phone at the item you want to translate and if the item is recognized, associated language options will display.
Whether you’re looking to learn a new language in the comfort of your own home or in a foreign land looking for a helping hand, the app is a great example of often-too-rare AR utility.
The UX magic is possible through a combination of Apple’s ARKit and an existing database of about 1,000 learned objects that Ng ported into the app. While Ng says her app is just a demo and she has no immediate plans to take it to market, what’s so remarkable is that while companies like Microsoft spent many years and dollars on mastering object recognition, Ng was able to build her app in a weekend, simply because it’s building off so much past work that’s now freely available and baked into platforms like Apple’s.
Six months ago, Omega Ophthalmics did a small trial of seven patients outside of the US. Their goal was to test for adverse effects of a surgery similar to lens replacements that often accompany cataract removals. The difference? Rather than replacing the cloudy lens with a normal artificial lens, surgeons instead implanted a lens that could be used for augmented reality, interactive sensors, or drug delivery.
Why it’s hot
Although widespread adoption of this technology is unlikely in the near future, scientists, entrepreneurs, and venture capitalists hope that there is a market for such implants in an aging population that wants to be independent for longer. Whether this small trial is successful may pave the way for larger trials to test additional possibilities and risk.
Apple released ARKit, a mobile AR platform that uses Visual Inertial Odometry (VIO) to accurately track the world around it. VIO fuses camera sensor data with CoreMotion data, allowing the device to sense how it moves within a room with a high degree of accuracy, and without any additional calibration. It’s open to all developers who want to come create and play…and it’s also probably luring them away from FB/Snapchat/Googs.
By having the experience live in an app on millions of iOS devices, it’s giving creators (developers) a much wider audience (and incentive) to test and socialize.
Already, ARKit’s sharing platform is seeing some really awesome crowdsourced examples of the tech in action.
See more examples of what developers have already been toying with, go here.
Why It’s Hot:
Democratizing AR to both users and developers (HoloLens who?)
Positioning it’s self the Go-To AR platform
Low-Overhead cutting edge technology for brands to explore and own
A developer named Abhishek Singh recreated the first level of Super Mario Bros. as an augmented reality game on Microsoft HoloLens, giving himself a first-person perspective into a unique 3D look inside the classic setting.
This demo was done in NYC’s own Central Park.
Why It’s Hot
Seeing the game as Mario sees it is totally hot. Watching a guy in Central Park acting like he’s slowly losing his mind: not so hot.
Nike built an augmented reality application called SNKRS for users to gain access to limited-edition sneakers available for purchase. The first sneaker to debut through the app was the Nike SB Dunk High Pro Momofuku, a collaboration with David Chang, creator and owner of the Momofuku restaurant group.
For a user to gain access to the shoe, they have to open the app and point their camera at the menu at Fuki East Village Momofuku in New York. People can still gain access to the shoes elsewhere, as an online menu works as well. Users need to look for a special ‘SNKRS’ label for the app to work properly. Once scanned, the shoes are unlocked and users have an opportunity to purchase a pair, as long as they’re in stock.
Right now, the SNRKS application only works on iOS phones, but Nike plans to release a version for Android soon.
How do brands use augmented reality in a way that engages their core audience? What’s interesting about this is (1) the audience understanding — sneaker freaks DO care about insider, unique, unlocking-type tasks and (2) the localization factor + partnership factor. However, I have to wonder what the reach is on something like this — is it a lot of effort for a little engagement?
Air New Zealand has partnered with Microsoft to begin beta testing HoloLens augmented reality headsets on flights to help their crews better serve their passengers.
Flight attendants using headsets on their faces might look really strange and scare little children, but the practical applications are pretty cool. Being able to know, for example, which passengers have dietary restrictions or are in a certain mood can enhance the customer experience.
It’s hot because while this might not be a solution that gets mass adoption with every airline, it is nice to know that there is an airline out there that is trying to improve the travel experience (*cough* unlike United *cough*).
UC Berkeley Art Museum and Pacific Film Archive—or BAMPFA—wanted to give their new exhibition titled “Hippie Modernism: The Struggle for Utopia” a little bump.
But Goodby Silverstein & Partners, their agency, does not know how to give “bumps”. Instead, they shot this out of a bloody cannon by developing a geolocation-based augmented-reality mobile app that lets people pepper the virtual space over the Bay Area with digital balloon hearts containing hippy-like messages such as, “Love Is in the Air” or “Free sex”. Okay, okay, to be transparent, I TRIED to upload this last message but the app wouldn’t let me. Apparently, you have to be in the Bay Area. Bummer.
Anyway, think of it as a Pokémon, except instead of catching the little trolls, you are making and popping virtual hearts that release lovey-dovey messages.
In addition to allowing you to spread the virtual love, the app also plays ’60s and ’70s songs like John Lennon’s “Imagine” and features a virtual tour of Bay Area locations significant to the hippie movement’s artistic and political history.
Why It’s Hot:
NAILS a very unique and cool use of AR
Provides a gamified AR experience ala Pokémon
App is fun and practical – i.e. virtual tours tied to the thematic of the exhibition
Subtly hints at a larger cultural sentiment of acceptance and hippy rebellion
Earlier this week, Facebook’s Oculus Rift, the clunky VR headset, shipped to the market. But Facebook has hopes for the tech and form factor to be reduced to the size of a normal pair of glasses.
During a recent developer conference, CEO Mark Zuckerberg demonstrated a concept pair of smart glasses that he envisions being able to view both virtual reality AND augmented reality. It would be similar to what Google Glass has tried to establish.
To distinguish, AR provides you overlays of data and information while you view the real world through the glasses vs VR, which is a virtual rendering of worlds.
There may be a race to market heating up as Microsoft just shipped its HoloLens headset to developers and Snapchat is apparently working on its own augmented reality glasses.
Why It’s Hot
AR and VR show true integration of the real world with information and data that can enrich our experiences. Particularly with augmented reality, overlaying information into your field of vision that is contextually relevant to what you are experiencing, can augment and enrich the experience.
Perhaps the only thing more confusing than the buttons, displays and lights on a new car is its manual — that 100+ page booklet of details, photos, and instructions that are enough to make you dizzy and vow never to read it, unless you need to know what that strange glow on your rearview mirror is.
Here comes Hyundai to the rescue. In the first of its kind, the car manufacturer is introducing virtual manuals that rely on augmented reality to explain your car. AR is first being introduced on the 2015 Hyundai Sonata, but will soon be compatible for other models.
Using a smartphone or tablet, car owners will be able to recognize “more than 45 major features of the Sonata and contains 82 how-to videos, six 3D overlay images that appear once users scan different areas of their vehicle, and more than 50 informational guides,” according to the post in psfk.com. Users position their device over the part they want to learn about and AR will provide information about warning lights, Bluetooth phone pairing, brakes, fuse box, etc.
Why it’s hot
As cars become more complicated, augmented reality can take the mystery out of key features of your car and educate the owner on conditions, DIY repairs and an overall understanding of how their car works. Eventually, AR technology will be used in other electronics manuals, such as household appliances and electrical devices, helping to make wordy, poorly written manuals a thing of the past.
Alien landscapes with hot breezes and falling rain, or an Aztec Temple where you feel the heat of your torch. This is the Star Trek holodeck come to life, a true mapping of virtual world over physical world and its being developed by a small group of enthusiasts– setting the bar for companies like Oculus to beat.
Why It’s Hot
The most advanced of the true VR experiences and its moving rapidly toward deployment worldwide. But its also setting the bar for the category in timing and sophistication.
The SmartSpecs device is simple to put on, sits comfortably on the eyes and is very portable. Despite its size, the device has three camera sensors, a processor and a display.
For people with difficulty in seeing colors and contrast, the glasses can convert scenes into simple, high-contrast image fields. This can help legally-blind people see object locations in relevance to other items in the scene.
The depth camera, the one which can record the distance of an object from the wearer, also functions in the dark. With the use of the Smart Specs, a legally blind person may easily navigate independently both during day and night.
The SmartSpecs run on Android and microprojectors display the processed images onto the transparent lenses. The wearable also has zoom and pause functions for viewing scenes in greater detail. Customizable to user needs, the SmartSpecs can display various amounts of detail.
At SXSW, I focused on wearables, and there was a lot of discussion as to the breakthrough medical uses they may have. This is a great example of that type of application — and makes you wonder what advances may be in store in the near future
There is now a new and exciting way to search and interact with the world around us. The Blippar app, which was first created as an image recognition platform, has been reinvented. The re-launched app focuses on non-brand visual search, allowing users to delve more deeply into an array of both products and real world objects. Blippar uses the camera on your smartphone to look at an object. This image activates Blippar’s digital search and serves up relevant information directly from the local area. Blippar’s CEO and founder Ambarish Mitra seeks to expand the app’s focus into an all-encompassing and universal visual search engine to complement text- and link-based engines like Google and Bing.
Why It’s Hot
The service is being launched in waves and for now it is only available for use on English language album covers, movie posters, fiction books, and DVD covers. Eventually, any object in the physical world will be fully searchable. Our ability to find out more about the world around us will be limitless – all from your smartphone screen. As the app is developed, it will incorporate artificial intelligence and location-based technology to personalize and refine results for the user. It will be faster than the traditional web search, so instead of scrolling through pages of search results, you can receive key information in an instant.
Blippar brings out the curiosity in all of us. For example “when you blip a puppy” said Mitra, “you know what puppy it is, its history and genealogy, how to look after it, if there’s a vet nearby, the dog sitters and dog walkers — all the connecting points of information. It’s almost like that puppy becomes its own portal.”
Blippar is available for iOS, Android and Windows Phone.
Modiface’s Mirror combines augmented reality and facial recognition so shoppers can see what they’ll look like with different shades of makeup before making a purchase decision.
The Mirror combines a 3D live-video feed with a Photoshop-style sampler of more than 2,000 existing products and that maps the shades and textures to a movable, rotatable image of the user’s face.
Mirror’s core format is a sales kiosk, a robust and stand-alone version that automatically grabs a browser’s image and attracts their attention with recommended shades and brands. Once interested, the shopper can experiment with various products until the shopper becomes a buyer.
The concept is also available as a less robust, but considerably less expensive, app that uses a smart phone’s camera to capture the image. The apps are available for iPhones, Android and most tablets.
Mirror also includes an anti-aging mirror that shows the long-term benefits of using various moisturizing and age-reversing products. The featurette shows these effects as a slow progression, demonstrating the gradual effects.
ModiFace began in 1999 with initial research into facial analysis at Stanford University. The work continued through 2006 when ModiFace, Inc was born. Modiface technology has gone on to hold the leading patent portfolio on skin and facial analysis, powering over 150 web and mobile apps with a total of over 50 million downloads as of March 2015.
I’ve been paying close attention to retail trends that (1) enable creative sampling in a digital world and (2) connect the digital & physical in useful ways that can encourage purchase. This technology is a great example of both of these trends, and could open the door for even more impactful usage — think pre-surgical previews, impacts of UV exposure and more.
The technology of Google Glass combined with the fun of the ski slopes is a recipe for success. RideOn, new Israeli crowdfunding campaign, claims to be the first augmented reality ski goggles.
The ski goggles enable wearers to see projected ski paths via AR on their goggles, as if the paths are being projected on the snow 15 feet in front of them. Wearers can also play games and challenges with friends, and make phone calls and video messages.
The technology uses sensors, a video camera, and an Internet connection (all built into the goggles) to make the experience happen. But the goggles also still include the anti-scratch and anti-fog features of most ski goggles. In addition, there will be a free mobile app which wearers can use to track their progress, interact with other nearby users and friends, and instantly upload videos of their skiing.
Why It’s Hot | Augmented reality is increasingly in popularity, but many brands are struggling with ways to use it in a sensible way, beyond being a novelty. This usage enhances a sport that many already enjoy, by increasing its competitiveness (via the built-in games), safety (via the projected ski paths), and shareability (via the video creation and upload feature). It will be interesting to see if, in the future, AR is more accepted in targeted, niche environments, as opposed to broad everyday uses.
First launched softly on Twitter last August, the viral campaign for Universal’s Jurassic World movie went into high gear after the Super Bowl. Two incredibly immersive websites launched– one for the fictional park which will have you believing you could book a trip there tomorrow (take a tour through the resort rooms, check the restaurant menu, look at activities, live webcams of the park, even buy merchandise), and the other for the equally fictional company that supposedly owns and built the park, Masrani Global.
Neither site acknowledges they are fictional. Rich in video, imagery and detail; updates are now being released that have fans poring through the work for clues about the film.
Why It’s Hot
Viral marketing is often a “leap of faith” as we cant predict or forecast success, it is completely in the hands of the user to determine whether the content is shareable. It is impressive how much detail and depth went into these websites, and is continuously being released without a shred of paid promotion. That’s the definition of faith in your idea, your instincts and the resulting content is so rich and compelling i am waiting to see when this gets real traction.
USPS’ new mobile app features an AR experience that enables users to enhance the real world via their smartphones. By using the app to scan any of the 156,000 blue collection boxes throughout the country, users receive a “magical mailbox” holiday message.
The app is meant to compel more business from direct-mail marketers by showcasing innovations they can use in their own direct mail pieces via the USPS.
Why It’s Hot | What sets augmented reality apart from other mobile ad units is its ability to compel users to engage in a more meaningful way. It can be used to provide information about a user’s surroundings, or provide video extensions to print ads. As the world is becoming more mobile, innovation that integrates direct mail with the digital world is helping to bridge the gap between traditional channels and new audiences.
even with the same or similar product available in different places, we will gravitate toward the superior retail experience– as brands are now shaped by experience as much as ideas, the ubiquity brought by technology continues to push the importance of brands in a new direction.
This is a pair of articles that together drive home the need to understand people as we build multi-channel, multi-device, multi-format experiences.
First, a new study that confirms mobile commerce has not upset or displaced brick-and-mortar shopping– in fact, it drives in-store sales by extending the experience. The way we use smartphones is so fundamentally different from desktop or tablet; we use smartphones for couponing, reviews, location-finding but not for actual commerce. This harkens back to a “scholarly disagreement” I had with a Harvard Economics professor 12 years ago, who predicted that ecommerce would be frictionless and drive product sales on price alone, that brands would be irrelevant.
Use of Force is a VR simulation, that places the viewer as a witness to a real-life incident where border guards beat a non-resisting man to death. It uses actual cell-phone audio, 3D imaging of the location and witness accounts as the basis. It was created by Nonny de la Pena, a former Newsweek journalist to change the way we perceive events being broadcast to us. It mirrors the way television changed our perceptions of the Vietnam War.
While most applications now talked about are gaming or virtual meetings (social and/or business), this kind of VR experience has the potential to radically change our perceptions– good or bad. Studies are showing our perceptions of VR experiences are fundamentally different than other immersive even 3D experiences.
British Airways ‘Magic’ Billboards has recently won some awards, and it’s clear why.It’s informative, intriguing, and well executed! According to Ad Age, these digital billboards are located throughout London, and serve to highlight the company’s various destinations by encouraging passers to look up into the sky, to spot the airplane passing over.
Why it’s hot | The idea behind the campaign is quite simple, but because of the technological complexity and accuracy, it’s pretty brilliant. It involves mounting an antennae on the roof of each board, which picks up data from British Airways aircraft transponders. This information is then sent to a server which detects the aircrafts origin or destination. A trigger zone was also included, providing accuracy of the aircraft passing, based on cloud altitude which showed whether the plane could actually be seen.
For a campaign which serves the purpose of drawing potential customers to their website, the agency behind the idea (Ogilvy) did a pretty good job of making it enjoyable. The only call to action in the advertisement includes the hashtag #LookUp, as well as the html, ‘ba.com/lookup’ which drives the user to a website listing of the companies cheapest flights. Well done!