Mixed reality: Experiences get more intuitive, immersive, and empowering Tech Trends 2017

Advances in AR and VR promise to change the way we interact with technology. Though consumer buzz surrounding devices designed for gaming and entertainment is growing, AR’s and VR’s enterprise potential is proving to be the real cause for excitement. Across sectors, use cases and concepts are emerging, and pilot programs are ramping into production.¹

Meanwhile, the Internet of Things (IoT) is attracting more business investment as attention begins shifting from underlying sensors and connected devices to real-world scenarios driven by advances in IoT technology. Pioneering applications are emerging areas of personal health and wellness, supply chain, and in the civic infrastructure of smart cities, among others.²

Mixed reality (MR) represents the controlled collision of the AR/VR and IoT trends. With MR, the virtual and real worlds come together to create new environments in which both digital and physical objects—and their data—can coexist and interact with one another. MR shifts engagement patterns, allowing more natural and behavioral interfaces. These interfaces make it possible for users to immerse themselves in virtual worlds or “sandboxes,” while at the same time digesting and acting upon digital intelligence generated by sensors and connected assets. For example, as a worker wearing smart glasses examines a system in a remote location, diagnostic information appearing in his field of vision indicates the system is malfunctioning. If the worker can’t fix the problem himself, skilled technicians in another location would be able to transmit detailed digital instructions for repairing the malfunction and, then, walk him through the repair process quickly and efficiently. In this and similar scenarios across industries and operating models, MR makes it possible to deliver actionable information to any location where work is done—on site, on the shop floor, or in the field.

The mixed-reality trend is being fueled by investments in platforms, devices, and software ecosystems. The ultimate goal of these investments is to replace keyboards and flat displays with entirely new paradigms for communication and collaboration. If successful, this would represent the biggest fundamental shift in user engagement we have in seen in the modern technological era. Each successive improvement to user interface patterns—from function keys and typing, to point and click, to touch and swipe, and now to talk—offers a chance to move “beyond the glass” of static displays, and to reimagine engagement around gestures, grasps, and glances.

MR advances are already profoundly affecting how we relate emotionally to these novel ways of working. The ability to grab information from things around us—and for that information to respond to our visual and physical feedback—changes the ways in which we consume and prioritize our work. By stripping away the intermediaries and barriers to cognitive recognition, processing, and understanding, the enterprise may be able to transform worker effectiveness and engagement.

The where is the what

Mixed-reality use cases and patterns have emerged around early investments. Though specific priorities differ by industry, many land somewhere within the following areas:

Training, education, and learning: AR and VR can be used to onboard new employees and develop existing talent by immersing them in highly realistic, virtual work environments that feature both educational information and interactive problem-solving challenges. Immersive environments offer a number of advantages over traditional teaching methods. For example, they provide trainees with safe (virtual) exposure to complex and potentially dangerous equipment and scenarios. They also make it possible for supervisors to review video recordings of training sessions to monitor progress and tailor lessons to specific employee needs.

Operations: By providing field service technicians, warehouse pickers, assembly-line workers, and others with IoT applications and digital system content tailored to their unique tasks, companies may be able to boost employee productivity and streamline work processes. On the production floor, for example, job aids can guide workers performing kitting tasks to locations of shelved items. In the field, engineers could access the service history of specific equipment, guidance on triage and repair steps, and any real-time information that intelligent devices may be generating from a site. They would review this information in a hands-free, heads-up manner that maintains their autonomy and supports worker safety.³

Communication and collaboration: As organizations become “unbounded” by eliminating functional siloes, multidisciplinary teams will be able to work seamlessly together within and across company walls. Mixed reality can support this kind of next-generation interaction by replacing shared productivity tools and videoconferencing with immersion and a sense of presence. Workers in disparate locations can interact with the same digital artifacts, just as if they were in one conference room manipulating the same physical objects. Automotive manufacturers are applying these methods to visualize design improvements of existing components—dramatically accelerating the concept-to-manufacturing process among globally distributed teams.⁴ Likewise, research and development functions, construction and engineering firms, and even some services organizations are exploring this new style of collaboration, removing geo-temporal constraints from both creative processes and more tangible operations.⁵

Marketing and customer service: From high-tech experiential marketing to virtual branches, MR—specifically leveraging VR capabilities—can provide experiences that not only replicate aspects of the real world but build compelling engagement patterns predicated on the convergence of technologies. For example, you will be able to put a virtual product in your customer’s hands and then guide her experience, gauge her responses, and personalize content based on this customer’s transaction history and preferences.⁶

Shopping: Virtual reality stands poised to revolutionize the way we shop. Consider, for example, “walking” through a virtual cruise-ship cabin or hotel suite before booking it or immersing yourself in a virtual jewelry store where you try on necklaces that catch your eye. Mixed-reality capabilities could enhance these virtual experiences by providing pricing or descriptive information on the products or services you are considering, along with suggestions for similar offerings.⁷

MR technology: Enablers and obstacles

As it often does with emerging technologies, tremendous hype surrounds AR’s and VR’s promise—for good reason. These platforms offer new ways of framing and delivering content, experiences, and interactions. They also bring with them opportunities to redefine the tools, models, and business processes that they could potentially replace. But as MR devices, software, and standards evolve, enterprises are discovering hidden challenges beyond the management of technology.

AR, VR, and IoT represent new categories of devices that need to be managed and secured. Learning from the workplace adoption of smartphones and tablets, organizations can begin their MR journeys on the right foot by developing appropriate controls and policies to monitor and enforce enterprise needs. Security and privacy are important considerations—at the device level, the data and supporting content level, and the application level. Safety and regulatory compliance implications are paramount, especially since many potential scenarios involve critical infrastructure and sensitive operations.

To process event streams, render mixed-reality experiences, and capture and respond to the movements of an individual user, platforms will need several enabling environments. They must be immediately aware of that user’s role, the context in which he works, and information he needs to complete his tasks. MR’s ability to simultaneously track an environment and an individual’s relationship to that environment is the key to allowing virtualized objects and information to respond realistically to what that individual is doing. This requires the kind of complex signal processing and response generation found in some advanced videogame designs. It’s no surprise that early experiments have been built on top of popular gaming development engines such as Unreal.⁸

MR must also navigate interfaces that allow for the blend of voice, body, and object positioning to open up a world of business process opportunities in every industry. Recognizing subtle expressions requires precise techniques, from eye tracking to interpreting a wink or a smile correctly. MR design patterns should make it possible for digital content to react intuitively to signals. New vocabularies for design patterns are required, along with solutions to manifest in both the AR/VR systems, as well as the surrounding connected devices and sensor-enabled environments.

MR should bring together relevant data to provide insightful interactions. That could mean integrating MR with the core in order to access information residing in ERP, CRM, HR, marketing, and other systems. Conversely, MR can be an important tool to digitize work—potentially automating underlying workload, updating data with supplemental supporting information, or allowing MR steps to be a part of a longer-running business process.

The context of content

Perhaps most importantly, natural interfaces make it possible to simultaneously process the meaning, intent, and implications of content in context of how the world works—and how we behave in it.

A digital object’s weight, scale, angle, position, and relationships with other virtual and real-world objects allow designers to fashion truly visceral experiences. Sound and haptics (or sensory feedback) can replace graphical predecessors, allowing for the enterprise to rethink alerts, warnings, or the completion of tasks. Built-in object and facial recognition gives us the ability to map spaces and people to accurately render in the virtual or augmented experience, and to insert purely digital enhancements in a way that seems both natural and realistic.

MR requires an entirely new set of digital content and context. High-definition, 360-degree renderings of facilities and equipment are necessary to translate the real world into virtual environments or to marry augmented physical and digital experiences. Sensors and embedded beacons may also be required to track devices, equipment, goods, and people. Likewise, metadata describes not just an asset’s base specifications but also its composition, behavior, and usage—all necessary to simulate interactions.

All together now

Even as we elevate mixed reality above its piece-parts of AR, VR, and IoT, the underlying technologies themselves are continuing to advance. Individually they represent very different solutions, domains, and potential applications. However, if companies pursue them in isolation, their full potential will likely never be realized. The goal should be evolving engagement—building more intuitive, immersive, and empowering experiences that augment and amplify individual users, leading to new levels of customer intimacy, and creating new solutions to reshape how employees think and feel about work. If done correctly, mixed reality may open floodgates for transforming how tomorrow’s enterprises are built and operated.

Lessons from the front lines

Using immersive technologies to protect child welfare

Each year, newly graduated social workers enter child welfare agencies determined to make a difference. Unfortunately, many of them may be unprepared, both practically and emotionally, for what awaits. It can take years of on-site visits to homes and childcare centers to help a social worker develop the deep observational skills and attention to detail required to accurately assess a child’s living situation and, then, to determine whether further investigation and action is necessary.

Immersive technologies may someday offer child welfare agencies an efficient way to accelerate that learning process. Using serious games—games designed specifically to teach skills—and 3D simulation, social workers would be able to practice real-time engagements designed to help develop sensitivities and nuanced evaluation skills previously achieved only after years on the job.

For example, one training module might place a trainee social worker in a virtual home setting and ask her to identify all potential risk factors in a few minutes. After the initial scan, the trainee returns to the virtual room, where risk factors she overlooked during the initial assessment are blinking. When the trainee points at a specific signal, a description of the specific risk factor and an explanation of its importance appear in her field of vision. As part of this same process, the trainee could also practice reacting to difficult situations and documenting what she sees.

3D training models could be customized and refined for use in any environment or scenario. Not only could this expand the breadth of training available for new recruits—it would likely help veteran social workers further their professional development. When used over time, these capabilities might also help agencies assess and improve their overall effectiveness and teach critical thinking and decision making. In this light, MR capabilities are not just technical game changers but behavioral solutions, creating experiences that potentially benefit social workers and—even more importantly—help those they serve.

AR meets the IoT on the shop floor

In 2016, two innovative companies introduced to each other at the MIT Media Lab convened at Jabil Blue Sky, an innovation center in Silicon Valley, to kick the tires on a new digital manufacturing process technology. The companies involved were Jabil, a global provider of engineering, manufacturing, and intelligent supply chain solutions, and Tulip, which offers a cloud-based platform featuring shop-floor apps, industrial IoT, and real-time analytics.

The solution being tested? A new cloud-based operating system that feeds IoT production-line data in real time to workers on a shop floor through their smartphones and tablets. By monitoring this information stream as they perform their production tasks, workers can respond on the fly to process changes. Eventually, the system could also integrate the power of mixed reality into manufacturing environments. The goal is to improve manufacturing flow through the shop floor. With real-time information enabled by the latest improvements in digital technology, companies may be able to reduce worker pauses or idle machine cycles that typically accompany changes in production conditions.

After initial testing at the Blue Sky innovation center, Jabil and Tulip deployed the system in a production environment used by workers executing highly specialized work processes. For a period of six months, engineers monitored cycle and step-times data to further optimize manufacturing processes through continuous time studies and root-cause analysis exercise aided by the new digital tools. The results? Production yield increased by more than 10 percent, and manual assembly quality issues were reduced by 60 percent in the initial four weeks of operation, which exceeded customer-accepted yields and predictions for the current design.

According to Tulip co-founder Natan Linder, in the near future, AR capabilities will likely amplify the power of IoT manufacturing solutions such as the one his company tested with Jabil. The delivery of contextual information to workers without requiring a screen is already providing significant benefits to global manufacturers, says Linder, citing increased product and service quality, increased worker productivity through reduced rework, and higher throughput, as well as reduced training time. “Increasingly, we’re seeing deployment of light-based AR in manufacturing, which uses lasers and projectors to layer visual information onto physical objects. This approach doesn’t have many of the disadvantages of other AR interfaces; most importantly, it doesn’t require workers to wear headsets.

“The real power of augmented reality comes into play,” he continues, “when it is combined with sensors, machines, and data from smart tools. These IoT data sources provide the real-time information that the hands-on workforce needs to get work done and optimize processes, with augmented reality delivering the information at the right time and in the right place.”⁹

Yeah, but can you dance to it?

By adding production capabilities to data collected from sensors and multiple cameras in the field, mixed-reality solutions can transform how humans interact (visually and socially) with the world around them and the events they “attend.”

In the last few years, we’ve seen VR broadcasts of sporting events as diverse as NASCAR, basketball, golf, and even surfing. Those initial broadcasts gave viewers a 360-degree view of the playing field and allowed them to choose their own vantage point throughout the game or race, supplemented with fully mixed 3D VR audio, announcer commentary, VR-like graphics, and real-time stats.

The use of sensors can enable broadcasters to provide additional value to remote viewers. Spanish start-up FirstV1sion, a wearables company that embeds video and radio transmission equipment in athletes’ uniforms so VR viewers can watch a play or a game from a specific player’s view point, outfitted players for European soccer matches and basketball games. In addition to the video feed, the electronics include a heart monitor and accelerometer so viewers can track players’ biometric data as the game progresses. The hope is that fans will be more emotionally involved in the game if they can see a player’s heart rate increase as the action heats up.¹⁰

Musical events are obvious settings for social interaction, and virtual streaming of concerts is becoming more common. For example, the Coachella festival provided a cardboard headset with each ticket sold last year in case concertgoers didn’t want to leave the hospitality tent to watch a band. One VR company is taking it a step further to let music fans be the performers: TheWaveVR has developed a virtual reality concert platform that not only allows users to watch musical performances—when paired with the HTC Vive, it enables them to DJ their own set in a virtual venue.¹¹ Viewers can listen and dance to the music in the “club” while talking to other attendees.

On the political front, NBC News used mixed reality to encourage dialogue between American voters during the recent presidential campaign. It virtually recreated the real-life “Democracy Plaza” it had erected at New York’s Rockefeller Center so viewers located anywhere could enter the plaza, view live programming, access real-time viewer opinion polls, interact with its newscasters and pundits, and, most importantly, engage with other audience members.¹²

It seems the dire predictions of virtual reality’s isolationism have been much exaggerated. By adding sensors, voice recognition, and data overlays to create a mixed reality in which humans can interact more naturally, the future looks quite engaging for playtime as well as industry.

My Take

Brian Ballard, CEO and founder
Upskill (formerly APX Labs)

A longstanding technology gap separates employees on the shop floor, in the field, or on job sites—places where work actually gets done—from the data solutions that are driving the work. However, investments in wearables and mixed reality are finally providing a form factor that is bridging this divide. At Upskill, our mission is to provide workers with better information, more understanding, and more empowerment to help them to amplify their expertise. We started building software for smart glasses in 2010. Today, we provide our clients with a platform for extending the power of augmented and virtual reality into manufacturing, field services, and logistics.

When discussing mixed reality’s business potential, we start with the problems people are trying to solve and then try to identify opportunities to drive real value through the technology strategies a given company can employ. For example, a company might need to solve a logistics responsiveness problem. It can do this by overlaying real-time information on inventory levels or the shelf location of products in the vision fields of warehouse handlers wearing smart glasses. Or deploy the same tactics in kitting processes in manufacturing—picking parts, adding to carts, and bringing to the proper workstations for just-in-time manufacturing orders. These are real-world challenges we have seen firsthand—in both scenarios, some of our clients have achieved up to 50 percent improvements in delivery times.

Broadly speaking, the results across industries are very exciting. We recently performed experiments with workers from several industrial customers. In each, we filmed these workers performing complex assembly tasks in two different circumstances: one in which task-related instructions were paper- or tablet-based, the other in which the same instructions were delivered via smart glasses. The results were dramatic: With digital instructions, aerospace workers worked 32 percent faster using wearables; likewise, the energy workers worked 35 percent faster with wearables—and this was their first time using a new technology. What other solutions have instantly delivered a skill increase to the workforce?

In both situations, neither the workers nor their job function changed. The transformative factor was the way the workers accessed the information they needed. Visualizing a task at hand with an overlay of contextual or instructional information can be very powerful. It eliminates the need to stop working when reading instructions, and it can improve safety, letting workers stay hands-free and heads-up while completing tasks. At the same time, it enables workers to augment their own expertise, which can lead to better decision making. The confusion that can occur when dealing with complex information is significantly reduced and the need to memorize training materials is virtually eliminated. It’s like a GPS for your job. Never getting lost or having to memorize a map has certainly changed the way the world gets around. I think the parallel between mixed reality and GPS is strong. Can you imagine paper maps ever becoming the norm again?

Wearable solutions also make it possible for companies to digitize, analyze, and optimize organizational behavior. For example, in a complex assembly environment, a manufacturer may able to capture information about each of the 100,000 steps being performed—something impossible to track before mixed reality brought work data to the edge. By applying machine learning to the resulting data, the manufacturer could experiment with different ways to group and break down the work to enhance productivity and provide coaching and training to individual workers based on understanding at which tasks they excel, and where there are development opportunities. Pulling the GPS analogy forward, this is the equivalent to real-time traffic alerts.

People often ask me how their companies can get started with mixed reality. Whether framing pilot initiatives or full-blown rollouts, best practices start with specific ROI goals. For example, “I will onboard new employees 20 percent faster,” or, “First-time quality on our assembly line will improve by 35 percent.” Once a tangible use case focused on measurable impact is identified, avoid hypothetical integrations and simulated environments. Wherever possible, try to deploy to a production environment to understand real-world behavior, benefits, and opportunities to improve. Of course, safety, security, privacy, and operational continuity are essential and can be addressed while rolling out the pilot. I think process owners would be surprised by the number of deployment success stories the wearables industry can already tell.

Here in the midst of the Fourth Industrial Revolution, we hear a lot about the interconnectedness of products, supply chains, technology, and data; the list of connections that run through industries and economies is endless. More than ever, companies have a long-term connection to the products they produce, and this is neither a fluke nor a passing trend. Mixed reality and wearables represent just one more connection linking people to products and, perhaps more importantly, to data that lives on with these systems. When production problems arise, data suggests a solution. Internet of Things sensors generate performance and contextual data in real time, and data-driven insights drive strategy and product redesign. At the center of it all are connected workers, consuming information and providing it back to the system, upskilling into their roles more productively than ever before.

Cyber implications

Companies exploring mixed reality use cases and experimental applications should take into account several cybersecurity and risk considerations inherent to MR’s components—virtual reality, augmented reality, and the IoT. Consider these questions:

1. When you are immersed in an MR environment, can you trust the integrity of the digital data on display? Can you confirm its authenticity? Are you confident that it—particularly the data sourced through IoT technology—is not vulnerable to malicious hacking?


2. With mixed-reality environments, can you confirm that your view has not been manipulated? There may be distinct MR environments used for training purposes, and variations of these same environments deployed for live business purposes. Can you trust that you are immersed in the correct environment? Could the digital assets, experience, and supporting context be manipulated to mislead? Or worse, could they trick users into performing digital corporate sabotage by encouraging wasteful, brand-damaging, or even dangerous actions?


3. If you are engaged with another individual within an MR environment, how certain can you be that that person is actually who she says she is? How can that trust be maintained?

These and similar considerations highlight one of the major risk-related challenges surrounding MR environments: controlling and securing associated digital assets. Virtual reality, augmented reality, and IoT technology introduce new and different intellectual property that may contain sensitive information requiring controls for security and privacy, regulatory and compliance issues, and competitive advantage—for example, high-definition 3D renderings of facilities and detailed tracking of property and equipment location and controls. Associated beacons, sensors, and connected footprints need appropriate protection, from encryption and access controls to rights and asset management.

In an MR environment, this can be challenging because each technology presents its own cyber risk challenges. For example, AR requires many more data points than VR to drive content for GPS and positioning, tagging, shared metadata, and facial recognition. Moreover, to enhance and tailor an individual user’s augmented experience, AR systems may also integrate data from a host of sensors tracking that person and from other personal sources such as health and fitness monitors. This raises a number of security and privacy concerns about the data sources to which individuals have access, and whether combinations of data being aggregated by AR may compromise personally identifiable information or payment card industry data.

Regarding IoT technology, each new device introduced in an IoT ecosystem adds a new attack surface or opportunity for malicious attack, thus adding threat vectors to a list that already includes protecting devices, data, and users. IoT ecosystem structures that organizations often deploy typically depend on the closely coordinated actions of multiple players, from vendors along the supply chain to clients, transport agencies, the showroom, and end-use customers. Vulnerabilities exist within each node and handoff seam between sensors, devices, or players. It should not be assumed that vendors or other third parties—much less customers—have broad mechanisms in place to maintain data confidentiality and guard against breaches.¹³

The flip side of the MR cyber risk coin is that VR, AR, and the IoT show promise as tools that may help organizations boost their overall security and privacy strategies. VR, for example, can be used in disaster recovery efforts and war-room simulations. Scenario planning around incident response can be taken to another level with experiences closely resembling real-life events. Likewise, AR may help companies better visualize the cyber threats they face.

Where do you start?

The promise of customer, partner, and employee engagement moving beyond the glass is powerful. The mixed-reality trend could usher in a new world of simulated experiences grounded in the ways business gets done and customers actually use products. Yet because MR’s component parts are still proving their worth in an enterprise setting, pursuing their full promise could present challenges, at least for now. To short-circuit potential false starts, consider taking the following initial steps:

• Do try this at home: Words cannot describe how powerful virtual reality’s immersion experience can be. Likewise, potential stakeholders should see augmented reality’s many possibilities firsthand not only to build credibility for the journey MR journey ahead but to light a spark for ideation.

• Anchors overboard: MR exploration should start with how something could or should be done, rather than with existing processes or interactions. MR represents a sea change from static displays and clumsy user interface techniques. Don’t waste time trying to improve today’s realities incrementally—instead, begin with bold scenarios in mind.

• Debunk the science fair: Given that many associate MR with science fiction, some organizations’ institutional inertia may naturally dismiss it as yet another shiny object. In reality, many companies and government agencies are actively piloting the technologies, with many aggressively moving into broad production investments. A wait-and-see attitude will only put you further behind others in your industry.

• Avoid a device derby: Recognize that existing devices will evolve and new categories will emerge. And while the broader consumer market will likely deliver an oligopoly of “winners,” consider focusing your investments in areas that offer potential today, and design an architecture that is as loosely coupled as possible. Let the market shake out as it will. Don’t let long-term uncertainty distract from immediate benefits, especially as the shelf life of any given device needs to be only long enough to support its original purpose.

• OT meets IT: For some industries, MR may force a new level of collaboration between operating technology and information technology. Especially as many of the best MR solutions will have deep hooks into core applications, there is a good possibility that integration with microelectromechanical systems and machine-to-machine systems will be necessary. This may also require a new governance and architecture that enables orchestration between the broader OT and IT stacks—recognizing IT’s desire to innovate new services and solutions, while fiercely protecting OT’s requirements around reliability, security, performance, and availability.

Bottom line

Mixed reality elevates the potential of AR, VR, and IoT technology by combining the best of our digital and physical realities. Instead of removing users completely from the real world, or simply layering flat content on top of our immediate view, MR adds intelligence—physics, gravity, dimension, even personality—to digital content relative to the space around us. As a result, we are able to blur the lines between what is real and what is imagined while stripping away the barriers that interfere with our ability to make decisions quickly, absorb and process critical information, visualize possible scenarios before acting, or share knowledge and tasks between individuals and groups. Science fiction no longer, the future of engagement is here, and enterprises will likely be the first to embrace it.