Augmented and virtual reality go to work Seeing business through a different lens

The future of mobile is tilting increasingly toward wearables, especially as augmented reality and virtual reality solutions hit the market. Long the objects of sci-fi fascination, the looming potential of AR and VR technologies lies in the enterprise with capabilities that could potentially reshape business processes, or fundamentally recast customer experiences. While the consumer world waits for the dominant AR and VR players to emerge, the enterprise can fast-track adoption—and begin the process of fundamentally reimagining how work gets done.

Until recently, augmented reality and virtual reality (AR and VR) technologies have served primarily as inspiration for fiction writers and Hollywood special-effects teams. Yet increasingly, both are finding more practical application in the enterprise. And while the hype surrounding each—particularly in the realms of entertainment and gaming—makes for good headlines, the real story in the coming months will likely be AR and VR’s disruptive potential to recast long-standing business processes and tasks while opening a door to fundamentally new experiences.

VR makes it possible for users to immerse themselves in manufactured surroundings that depict actual places or imaginary worlds. Meanwhile, AR overlays contextual information on the immediate physical environments users see before their eyes, thus blending digital components and experiences with real life. Both allow us to deploy technology in ways that would have been previously infeasible or even impossible.

The transition from client-server and web-based technologies to mobile has been transformative, partly because it has made it possible to deploy solutions at the actual point where business takes place and decisions are made. It also represented a long-overdue move toward more simple, intuitive interactions: Point-click-tab-type gave way to touch-swipe-talk. AR and VR take this a step further by offering experiences built around natural modes of interaction such as posture, gesture, and gaze, thus shifting attention from a glass screen in our hands to the real or simulated world around us.

Already, the disruptive impact of AR and VR is being felt across consumer technologies as dozens of new products enter the market. More broadly, AR and VR are introducing new opportunities to transform the enterprise, particularly in the areas of communication and collaboration, training and simulation, and field and customer service, as well as in the reinvention of employee and customer experiences. Device costs continue to decline, standards are being defined, and app ecosystems are beginning to emerge. The combination of these influences—along with a spate of high-profile acquisitions that are shining klieg lights on AR and VR possibilities—may represent a tipping point for AR and VR’s business and technical implications—and, more importantly, for how we rethink the role of the individual in the workplace.

The process of rethinking begins by understanding the affordance of new interfaces and interactions, and their impact on business process and legacy enterprise technology. Through this world view, the ways in which companies exchange data, execute tasks, share culture, and build the core of the business change dramatically.

A job with a view

Momentum around virtual and augmented reality grows with each new deployment. In particular, noncommercial prototypes are sparking curiosity across a wide spectrum of applications. For example, the Los Angeles Philharmonic immerses audiences in the world of Beethoven.¹ The British Museum invites visitors into a Bronze Age roundhouse containing both real and virtual artifacts of the period.²  Psychologists at the University of Louisville are creating exposure therapies to help phobia patients confront and learn to contain their fears.³ Filmmakers are crafting first-person POV documentaries that place viewers in the middle of a Syrian refugee camp or an African village beset by Ebola.⁴

Meanwhile, businesses are taking the same technology and interaction paradigms to new heights across many industries, including construction, health care, and manufacturing. For example:

• Communication and collaboration: Virtual reality and augmented reality may soon accomplish what static and flat mediums for knowledge exchange failed to do: Replace real, one-to-one human interactions. AR and VR both offer IT opportunities to change how the business and its employees report and share information and take action. Marketing managers are already using AR to view retail shelf inventory and sales data. Engineering teams across the globe are deploying VR to collaborate in real time to test and refine a single design. What’s more, virtual reality is transforming simple productivity tools like videoconferencing and live chats, enabling immersive face-to-face interactions that feature real facial expressions, physical gestures, and subtle nonverbal cues that are replicated in real time.
• Training and simulation: AR and VR will make it possible for IT to play an active role in retooling high-cost training and simulation environments, many of which exist to rehearse critical scenarios without the risk of real-world consequences. For example, manufacturers can replicate maintenance and repair scenarios in virtual environments. In fact, by creating parallel processes that leverage remote controls and robotics, they may be able to remove employees from dangerous, real-world analogs altogether. Executive teams are using simulated high-resolution stages to rehearse and refine their presentation skills. In the construction industry, commercial developers can now walk through complete, full-scale computer-rendered structures—getting a sense of the width of a hallway or the impact of detailed design decisions—before touching shovel to dirt.
• Field and customer service: It is the IT department’s responsibility to determine how AR and VR will be used in tandem with existing and other emerging technologies. Therefore, CIOs can lead efforts to redefine how field and customer service workers approach their jobs. For example, deploying augmented interfaces that pair with connected devices, sensing objects, and relational data can deliver task-specific information to workers in the field in context and on demand. Augmented solutions can overlay a jet engine’s service hours, component temperature, and service panel details into an aircraft mechanic’s field of vision. Likewise, virtual solutions can immerse customer service agents in collaborative scenarios featuring perceptive conversations and problem-solving. Remote experts can see what field reps see and provide guidance as they perform maintenance or mechanical tasks. Think of a sportscaster explaining a replay with diagrams superimposed on the screen; the same technique can be used as an overlay to the field rep’s view of the task at hand.
• Customer experience and interactive marketing: AR and VR offer potential new ways to interact with products and services. Moreover, they offer companies opportunities to raise awareness, promote features, and inspire desire for their suites of goods. Travel, hospitality, and leisure firms are offering immersive, interactive samplings of cruises or hotel stays that allow potential guests to explore properties and preview amenities virtually. Some of these samplings go so far as to use wind machines and olfactory stimulants to replicate not just the sights, but also the sounds and smells one might experience during a day at the beach.

Shifting focus

Designing user experiences for immersive environments is a fundamentally different process from creating experiences for flat screens. Immersive environments leverage cues derived from ambient sounds or a simple glance to drive both intentional and reflexive movements. In both AR and VR, the clicks and swipes that animate flat screen experiences are replaced by spoken words, gestures, fidgeting, grabbing, pushing, a nod, or even a blink.

Consider the notion of focus. Naturally, people have notoriously short attention spans. In the context of computing devices, we have dealt with this by shrinking, reflecting, and curving the displays. But in the context of behavioral interaction and productivity, focus becomes a different obstacle altogether. In virtual or augmented environments, what happens to objects when a user looks at them is as important as what the other objects are doing even when the user is not looking at them. A gaze becomes the new hover state, directing user intent and presenting options not previously visible. Likewise, a gesture (for example, the snap of a finger or the blink of an eye) could be used to change the perception of both time and scale, pausing or stopping time, accelerating outcomes, or even changing the position and relationship of objects not bound by physical realities. This creates an opportunity for the enterprise to design environments that offer empathetic, personalized responses. For example, in a virtual environment, an avatar could act as a performance coach that analyzes the body language and speech patterns of individual employees to help them enhance their presentation skills.

Through AR and VR, organizations can create environments that can react to changes in posture, mood, and attention. For example, dynamically reordering how tasks are presented to account for a user who is sleepy or distressed can change the relationship between technology, behaviors, and outcomes, and compensate for a higher cognitive load in decision making. This, in effect, puts the enterprise at the core of human-centered design: design emphasizing comfort, health, safety, happiness, productivity, and growth.

LESSONS FROM THE FRONT LINES

The AR/VR consumer market heats up

Over the next 18 to 22 months, we expect to see augmented reality and virtual reality technologies transition from the science fiction ether to the more earthly, practical realms of business and government. However, enthusiasts will not have to wait for solid use cases to emerge before they can begin to enjoy AR and VR at home. The consumer AR/VR market is heating up as offerings from Samsung, Microsoft, Facebook Google, HTC, Motorola, Sony, and other leading technology brands near completion. Likewise, start-ups such as MagicLeap, Lensar, and NantMobile, among others, plan to launch their own compelling offerings in the near future.

Early use cases are focusing on familiar consumer scenarios: gaming, video entertainment (Hollywood, adult, and other programming), and social media/collaboration. Meanwhile, new product categories are emerging that focus primarily on the technology footprint required to make the virtual or augmented realities tick. Some solutions require a high-end PC to function; some use a smartphone as the processing core. Many are tethered to batteries, controls, or control units, while others are truly wireless. Several benefit from baked-in or, in some cases, locked-in development partners to help expand the breadth and utility of the platform. Others are either nascent plays or have closed-garden content models. Regardless of the approaches, expect to hear more about consumer augmented reality and virtual reality devices in the coming months. Existing products will evolve. New product categories will emerge. Welcome to the future.

AR in construction: The next best thing to building there

In a remote corner of west Texas, a field technician wearing a geotagged helmet equipped with AR technology gazes up at a 270-foot-tall telecom tower. Using hand gestures, he pulls a data overlay into his field of vision containing the technical and design data he will need to perform a thorough equipment review on this tower.

Connecting field workers to data in this way is one of many potential uses global engineering and construction company Black & Veatch envisions for AR technologies in the near future, says Dan Kieny, B&V’s senior vice president and CIO. “In our more than 100 years of building critical human infrastructure, we have seen a lot of technology advancements, and AR has a compelling value proposition in our industry right now. We are looking at AR applications that provide individual operators with data they need to perform specific construction and maintenance tasks remotely.”

Wearables are nothing new in the construction industry. Workers in the field regularly don protective goggles, vests, and helmets, along with tool belts and other items that help them perform specific tasks. Smart wearables, such as augmented and virtual reality tools, therefore, represent a natural progression. Black & Veatch is currently exploring applications of AR technologies such as helping to train unskilled labor remotely to perform highly technical tasks; providing mobile monitoring capabilities that display system-status details in real time; and using smart helmets that are geotagged to provide location-relevant information to field workers.

The company is also looking for ways in which VR tools can be utilized to create immersive environments, providing visibility to large-scale designs. This capability could make it possible for owners and operators to vet design decisions and consider the operational implications to layout, equipment placement, and other factors that impact maintenance. Longer term, artificial intelligence and machine learning can help Black & Veatch refine the information that field workers receive, and enhance the AR interface between people and data.

Behind the scenes, Black & Veatch is already laying the foundation for these and other scenarios. For example, it is deploying sensor and beacon technologies at construction sites to provide a backdrop of tool, supply, and personnel data. Efforts are under way to capture and contextualize these new data sources for use in AR and VR experiences, as well as to enable exploration and analysis of hidden trends and business implications. “Data will never be fully structured, and that’s OK,” says Kieny, emphasizing the shift in focus from aggregation and stewardship to harnessing increasingly dynamic data to enhance human interaction in a number of ways. These include creating more intuitive interfaces with systems and data, and enabling more engaging dialogues with customers and partners.

According to Black & Veatch CTO Brad Hardin, the company is initially focusing on AR opportunities. He also sees eventual opportunities to use VR technology in areas like remote robotic welding and providing security training simulations for power plants and other vulnerable infrastructure. “In exploring opportunities to use smart wearables, we are ultimately trying to create more value for the company and our clients,” says Hardin. “But we are also trying to disrupt our business model before we get disrupted.”⁵

Can virtual reality help deliver the goods?

Even as automation increasingly disrupts long-established operational models throughout the parcel delivery industry, the process of sorting packages for delivery worldwide remains labor-intensive. At one global package-delivery company, training workers to operate and maintain massive pieces of sorting equipment that can be half a football field long traditionally required flying them to remote training centers where they would receive several weeks of intensive instruction. The problem with this approach is that many workers don’t retain learned skills unless they use them regularly. In a high-velocity performance environment in which equipment must run at top speed 24/7, the inability to address all mechanical problems quickly and efficiently can cost the company dearly.

The organization is currently prototyping a 3D simulation solution that has been designed to be leveraged via VR to provide virtualized worker training on an ongoing basis, in any location. In this solution, workers wearing VR headsets would be immersed in a virtual 3D production environment that features simulated versions of equipment in use. A training program, using both visuals and sound, would take users step by step through detailed maintenance and repair processes.

The company envisions several ways in which the VR training solution could be deployed. In addition to providing just-in-time instruction on how to perform specific maintenance and repair tasks, it could also embed 3D simulations into mid-level e-learning programs for experienced workers. So, for example, a user might click on a prompt to bring up a new page that includes a 3D simulation depicting how to complete a specific task. The company could also create VR training courses in which new hires could learn five basic tasks in a virtual environment. When they complete those five tasks, they can advance to the next five, and so on, until they complete an entire entry-level course.

MY TAKE

Chris Milk Co-founder and director, Vrse and Vrse.works

At Vrse and its sister company, Vrse.works, we create fully immersive 360º video and VR cinematic experiences. But really, we tell stories. And every story should (and does) dictate how it’s to be told. Naturally, we’re staying up to date on all the new advancements in technology and the great work other people in the field are doing. But we can’t wait around until all the bumps are smoothed out, and neither should you.

Virtual reality as an artistic medium—and, increasingly, as a tool for innovation in business, health care, and other areas—is in its first growth spurt, and we’re proud to be adding to the innovations. We create and pioneer a lot of the technology we use, and every progressive iteration is inspired by a storytelling choice. We like to take on challenges and find creative solutions. That’s how cinema got from the proscenium wide shot to where the art form is today. Mistakes tell us as much as successes about the future of VR.

Our first foray into VR was the Sound and Vision experience I did with Beck a few years back. We wanted to reimagine the concert and create something organic and inclusive. Traditional concerts are a battle: The audience faces one way, the band another; sound clashes in the middle. Video captured this brilliantly for years, but we wanted to try a different shape—the circle. The concert-going experience is so round and immersive that we needed to try our hands at a new technology if we were going to effectively capture Beck and the musicians’ magic. The event was being billed as an experiment in immersion, so my ultimate goal was to capture and preserve the moment for a later broadcast in VR. This was more than three years ago, though, and VR mostly only existed in research labs. Luckily, this was right around the time that Palmer Luckey and the guys at Oculus were making waves, so we started a conversation. The result is what the viewer experiences in Sound and Vision: fully immersive 360º virtual reality, captured from various perspectives, painting the full portrait of the experience and not just tightly squeezed snippets.

I’ve always been interested in the intersection between emotion and technology. Studying people’s experiences while inside VR gave me the confidence and curiosity to bring like minds to Vrse. We’ve found that VR, when exercised with precision, can tap into a viewer’s sense of empathy. In short, VR is a teleportation device. It can take you into a conflict, instead of just showing you one. It can bring you face to face with a child in a refugee camp or a band on a stage, and the emotional response has been measured to be similar to actually experiencing those interactions. For the UN experiences we’ve created, UNICEF has taken to the streets with VR headsets in an effort to raise money for faraway causes. When people on the street experienced VR, they were twice as likely to donate. And we’re talking monthly donations, not just one-offs.

We’re continuing to create stories in VR that mean something to us. We’ve had the great fortune of caring a whole lot about every experience we’ve put out, and we want to keep that going. We want to keep reevaluating how people experience familiar stories.

And you? Now is the time for exploration. All previous art forms were built on mounds of trial and error, and VR is no different. Sometimes storytellers need to travel down the long and winding road a hundred times in order to find the highway.

CYBER IMPLICATIONS

Even with the “virtual” nature of augmented and virtual reality, these two technologies introduce very real cyber risk concerns. Though quite different, AR and VR share several common security and privacy considerations.

The devices themselves need to be tracked, managed, and hardened to control access to underlying data and applications and to entitlement rights to the gear. As they do with mobile devices and wearables, companies should mitigate different risk scenarios involving data and services at rest, in use, or in flight. They should also consider adopting existing cyber protocols from mobile-device, application, and data management programs to create the necessary management and controls around AR and VR efforts.

Controlling the associated digital assets should be a priority. Virtual reality and augmented reality introduce new and different intellectual property that may contain sensitive information requiring controls for security and privacy, regulatory and compliance issues, and competitive advantage. High-definition 3D renderings of facilities, detailed tracking of property and equipment location and controls, and associated beacons, sensors, and connected footprints all need appropriate protection, from encryption and access controls to rights and asset management. Likewise, protecting the information being presented in an AR/VR world and ensuring its integrity becomes critically important. What if data appearing in a cockpit AR display were to become compromised, and consequently, a jet veers off course? The possible implications could be tragic.

Yet beyond these considerations, each technology presents its own unique cybersecurity and cyber privacy challenges. For example, AR requires many more data points than VR to drive content for GPS and positioning, tagging, shared metadata, and facial recognition. Moreover, to enhance and tailor the augmented experience of an individual user, AR systems may also integrate data from a host of sensors tracking that person and from other personal sources like health and fitness monitors. This raises a number of security and privacy concerns about the data sources to which individuals have access, and whether combinations of data being aggregated by AR may compromise personally identifiable information (PII) or payment card industry (PCI) data.

With VR, when you recreate the physical world in a VR environment, you also recreate its problems and vulnerabilities. Virtual representations of your company’s assets may add another layer of risk by providing detailed blueprints to potential weaknesses. How will you protect these soft spots? And what new governance approaches will be needed to protect user identities in the virtual world?

The flip side of the AR/VR cyber risk coin is that these two technologies show promise as tools that may help organizations boost their overall security and privacy strategies. VR, for example, can be used in disaster recovery efforts and war room simulations. Scenario planning around incident response can be taken to another level with experiences closely resembling real-life events. Likewise, AR may help companies better visualize the cyber threats they face.

WHERE DO YOU START?

The world of AR and VR should be an extension of an organization’s digital strategy, applying new technologies to transform customer engagement and employee empowerment. While AR and VR may reek of “shiny object syndrome,” in both cases, the underlying promise is exceptionally grounded. AR can help to seamlessly integrate technology with the real world; VR provides immersive simulated environments that help model complex, unsafe environments that are infeasible to explore in real life. Both can potentially create efficiencies and drive innovation that impacts shareholder value. As an added benefit, harnessing these tools at the bleeding edge of consumer hype can also help cement a CIO’s reputation in the C-suite and throughout the enterprise as a purveyor of futuristic solutions that are grounded in business realities.

Consider the following points as you begin your AR/VR journey:

• The time is now: The consumer market is becoming crowded in both categories. It will take time for dominant players to emerge, but once they do, third-party developers will be able to focus their efforts on building surrounding ecosystems. For now, companies can begin to justify their AR/VR use cases around single purposes with measurable impact and value. Expect the market to evolve; as it does, companies can then move forward on point decisions that have self-contained positive ROI, design to allow portability, and reevaluate the field with each new initiative to determine where to place the next bet. Volatility will be the name of the game for some time. However, this volatility may also drive rapid feature expansion, lower price points, and more creative arrangements with vendors eager to partner with leading organizations.
• Behind the looking glass: Designing for AR and VR requires embracing new patterns and perspectives along with a wholly different design vocabulary. It also requires new enabling tools and services to bring the experiences to life and make them work in the real world. High-definition 3D image capture and mapping equipment are emerging, accelerating developers’ abilities to recreate real-world physical environments within new AR/VR tools. Gaming engines are gaining a new purchase in the enterprise, with Unreal, Unity, and others being used to create simulations and virtual environments for AR and VR interaction.
• Side jobs: As companies deploy AR/VR solutions, they may need to install beacons, sensors, or even QR tags around facilities and equipment to guide the context of augmented scenarios, especially for equipment on the move. Likewise, they might also need to construct wireless and cellular infrastructure to support AR/VR connectivity in remote areas. Finally, emerging middleware platforms can help abstract device-specific interaction from the underlying data and rules.
• New horizons: As many enterprises learned during the first mobile technology wave, systems designed around previous-generation technology can’t simply be ported to a new form factor. In fact, it took years for many organizations that were constrained by incrementalist thinking to evolve from a “mobile maybe” to a “mobile first” mind-set. The most compelling examples of this transition have been “mobile only,” centered on experiences that would not have been possible without smartphones and tablets. Luckily, AR and VR lend themselves to more imaginative thinking. Begin with ambitious scenarios that look beyond yesterday’s use cases.

Given that these tools are brand new, accept that experimentation is not only necessary, but essential to help everyone—IT, business executives, and end users alike—understand what they can do and how they should be applied to drive value.

Bottom line

Emerging technologies that invoke futuristic, seemingly fictional realities can spark a backlash within the enterprise. Executives raised on Gene Roddenberry, George Lucas, and Michael Crichton may be at once curious and dubious about what augmented reality and virtual reality are and, more importantly, how they might impact business. But know that AR and VR are here; their benefits to the enterprise will likely outpace consumer adoption cycles, which is notable given that the market may swell to $150 billion annually by 2020.⁶  It’s time to put AR and VR to work—and bring enterprise IT back to the future.

Deloitte Consulting LLP’s Technology Consulting practice is dedicated to helping our clients build tomorrow by solving today’s complex business problems involving strategy, procurement, design, delivery, and assurance of technology solutions. Our service areas include analytics and information management, delivery, cyber risk services, and technical strategy and architecture, as well as the spectrum of digital strategy, design, and development services offered by Deloitte Digital. Learn more about our Technology Consulting practice on www.deloitte.com.