More real than reality: Transforming work through augmented reality Deloitte Review, issue 21
31 July 2017
It's hard to wire a wind turbine while juggling a thick technical manual, or to ask offsite experts for help without being able to show them exactly what you see. Augmented reality can help, overlaying digital data onto the real world to give workers immediate access to vital information.
Seeing information in a new way
LEARN MORE
Subscribe to receive Future of Work content
Visit the Future of Work collection
Read Deloitte Review, issue 21
Create a custom PDF or download the issue
The first tools used by humans were little more than sticks and small rocks. Later, as tasks became more complicated, tools did as well. More complicated tools, in turn, allowed for new types of work previously undreamt of. Imagine Galileo looking through his newly constructed telescope and seeing clearly for the first time that the uneven spots on the moon were, in fact, shadows from mountains and craters. He had built his telescope to meet the demands of his scientific studies, but in doing so, also created new fields. Little did Galileo know that within three and a half centuries of his sketches, workers from an entirely new career field—astronauts—would be walking in those exact craters.
While simple tasks require only simple tools, today workers are increasingly asked to do much more: to sift through troves of data, and to perform complex, variable, and often unpredictable tasks that require an ability to access and understand that data, often quickly while juggling heavy workloads. Tasks such as diagnosing an almost invisible crack in a jet engine turbine or finding the optimal route for a delivery truck can require workers to access, aggregate, analyze, and act on vast amounts of information—more than any human could possibly memorize—that changes constantly depending on real-world conditions. To avoid overwhelming workers and allow the future of work to actually . . . well, work, workers need the ability to sift through it all and determine what is relevant to the task at hand. This means that modern workers will commonly need an entirely new set of tools that affords them a new way to interface with information and tasks.
That new toolset can be found in the promise of augmented reality (AR), enabled by the Internet of Things (IoT). Like the telescope before it, AR can offer an opportunity to see and use information in a new way. AR presents digital information to workers by overlaying it on their view of the real world (figure 1). For example, with AR, technicians who wire control boxes in wind turbines can see exactly where each wire goes in their field of view rather than wasting time flipping pages in a technical manual. In one experiment, eliminating even this seemingly minor inconvenience resulted in a 34 percent faster installation time.1 By marrying digital and physical information in this way, AR can offer more realistic training, speed up repetitive tasks, and even introduce entirely new forms of work.2
By reimagining how humans relate to digital tools, AR can offer fresh insights about how work gets done as well as new opportunities for collaboration and remote work. In this sense, AR can be seen as a tool that can work alongside people, with humans and digital technologies working together, leveraging their inherent strengths to achieve an outcome greater than either could accomplish alone.
What is augmented reality, really?
For many, the term “augmented reality” may conjure images of slick presentations of data—digital images overlaid on live video or projected on glasses, for example. But that is only one facet of AR; it has the potential to provide far more value to today’s workplaces. AR can integrate digital information into the ways in which workers perceive the real world, enabling them to seamlessly use that information to guide their choices and actions in real time, as they accomplish tasks.3
Three key elements underpin AR (figure 2):
- The source of the data
- The ways in which that data are presented
- The interaction with, and use of, that data as an impetus for action
Together, these three elements combine to make AR a unique tool with powerful potential.
Source of the data
Starting at the beginning, where information is created, takes us outside of the realm of pure AR and into another connected technology: the IoT. Put simply, the IoT creates flows of information from connected tools, systems, and objects—information that, when aggregated, can be used to create a more holistic view of the world and illuminate new insights. Information can drive the workday; workers use information in one form or another, from some source or another, to accomplish their tasks. Much of this information can easily be pulled from databases or reference materials, but in the fast-paced world of the modern workplace, it is not helpful to know the as-designed pressure in a hydraulic pump or what the pressure was last month. Workers need to know what the pressure in that specific pump is today—right now—if they are to accurately use or maintain that pump. Gathering digital information about the world from sensors, and communicating that information so it can be aggregated, analyzed, and acted upon is what the IoT is all about.4
Presentation of the data
Simply having the right information isn’t always enough. Workers can be quickly overwhelmed when presented with too much information, which can actually lead to poorer performance.5 Instead, try to present information when it is relevant, and in a manner that workers can easily absorb. Much of the current research into AR focuses on how to present digital information in increasingly natural, contextual ways. For example, while early systems had to rely on specific markers or cues, such as lines or bar codes telling computers where and how to display information, current development focuses on marker-less systems that can more seamlessly weave digital content into a user’s field of vision.6
Interacting with, and using, the data
Even having the right data, presented in the right way, does not create any new value if it fails to result in action. Value is created only when a worker can use this information to do something new—find the right part faster or get help from an expert. This means that AR sits at the end of a long trajectory of not only displaying digital information, but controlling it in increasingly natural ways.
Early computers displayed data via tape printouts; later versions progressed to screens via command line interfaces. Workers controlled these machines with keyboards or punch cards, but could not easily “edit” or control the data once it had been printed. Later, the graphic user interface and the mouse made consuming and controlling digital information easier. But AR can take this trajectory still further; it not only incorporates the display of information in a way that people naturally perceive the world, but also increasingly allows workers to control that information through movements such as gestures or gazes.7
AR is fundamentally about allowing humans and machines to team together to achieve results neither could alone. That teamwork can be the key to success in the complex, data-rich environment of the 21st century.
AR is a prime example of how optimally leveraged new technologies can change the future of work. After all, work is, at its foundation, an interaction between people and tools. New tools introduce new capabilities that can generate measurable improvements in work performance. Freestyle chess exemplifies this. Instead of asking, “Which is better, human or machine?” it takes the question one step further and asks, “What happens if the humans and the machine team up?” In freestyle chess, competitors can use any technical tool or reference aid to help select their moves; this often results in large teams of people and computers working together to try to win a game.
In 2005, playchess.com hosted a freestyle chess tournament. Armed with the best computers, several grandmasters entered the tournament as heavy favorites. But none of the grandmasters took home the prize. Instead, it was awarded to two amateur players who used three home computers.8 How did they beat the odds? It turned out the most important thing was not technology or the skill of the players, but rather the quality of the interaction between them. As Garry Kasparov later explained, “Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.”9 Similarly, AR is fundamentally about making the human-machine team work as naturally as possible.
AR is fundamentally about allowing humans and machines to team together to achieve results neither could alone.
Why AR? Why now?
While AR may seem cutting-edge, it is actually not a new technology. Its roots stretch back to World War II, when British engineers combined RADAR information with a gunsight, enabling fighter pilots to attack targets in the dark.10 But in the decades that followed, AR failed to catch on in the workplace, likely because it was not required to complete tasks. But as the nature of work in the 21st century is transforming, tasks are changing; in the future, human-machine relationships will likely become increasingly critical to organizational success. To those of us bombarded daily with hundreds of emails, social media posts, and texts, it is perhaps no surprise that the volume of information in the world is increasing every day.11 In fact, in 2003 alone, the amount of information contained in phone calls alone was more than three times the amount of words ever spoken by humans up to that point.12 As more companies derive value from this information, the demands of sifting through mountains of information to find the right pieces of data for a complex task will be beyond the capabilities of most people.13 The result is that AR will likely be increasingly necessary for tasks with high volumes of data or highly variable tasks.
Research from psychology, economics, and industrial design indicate that there are two main factors that determine how we process information to accomplish tasks: the volume/complexity of data and the variability of the task.
Volume and complexity of data. Data is an invaluable asset to decision making and task performance, but it can have diminishing returns: While a little information is good, too much information can actually reduce performance. This is because information overload often distracts workers from key tasks and causes them to miss relevant details. Highway accident statistics illustrate this principle: As car manufacturers continue to make safer vehicles, highway fatalities actually rose in 2015. According to the National Highway Traffic Safety Administration, this was at least partially due to an increase in distracted driving; more drivers are now using phones and other devices when behind the wheel.14 And while smartphone apps featuring turn-by-turn directions can be useful, they can also cause drivers to miss even more important information, such as the brake lights of a truck ahead.
Variability of task. When each iteration of a task is different, it may also become difficult to sift out the relevant pieces of data.15 In this case, humans may have the advantage over computers. Computers do a better job handling large volumes of data, but humans are much better at dealing with variation. For example, human language is rich in variation and context. So while a person would quickly detect the sarcasm if a friend said how “great” the weather was on a rainy vacation, computers would struggle to detect anything but praise for the precipitation.16
Both of these factors can negatively impact job performance, and both are increasingly inherent in the tasks asked of modern workers. In order to accomplish today’s tasks, we likely need a new way to interact with digital tools. We cannot rely on ourselves alone because humans cannot process or remember enough information. But neither can we rely solely on automation, because it can only do what it was programmed to do and cannot deal well with variability. And so, it seems clear that increasingly, we will need teaming between human and machine, with each playing to its strengths. In short, for many modern tasks, we would benefit from AR.
What does this mean for business?
Today’s work environment often asks workers to perform tasks that are both increasingly data-intensive and increasingly variable. These two attributes determine the value that AR can bring to an organization. Large organizations will continue to offer a wide variety of jobs, falling across multiple categories, and AR can bring value to each in different ways. So understanding the type of tasks each job requires is the first step to understanding how AR can help. While concepts such as task variability and volume of data can seem abstract, organizations can work through these two questions to make this process a bit more intuitive:
- What do I need to know to accomplish this job successfully (complexity of information)?
- Where, and how often, do judgment and intuition come into play in this job (variability of task)?
Because these questions can be answered yes or no independently, the result is that AR can bring benefits and improve work along four main categories (figure 3).
Evolution I: Equilibrium
In this scenario, both data complexity and task variability are low. Here, the employee can use AR largely to do what he or she already does today, but perhaps just a little bit better. In this stage, AR can be leveraged to provide insights the typical worker may not easily have at his or her fingertips, which can result in more efficient, more productive, and even more accurate work. This can involve using AR to make up for deficiencies in human senses or abilities, to uncover the temperature of an object via superimposed heat maps, to view three-dimensional visual terrain models, to be guided by other perceptual enhancements, or to provide an overlaid measurement scale that enables greater precision in construction, assembly, or repair.17 For example, AR has been used to help historical reconstruction efforts painstakingly reassemble Roman vaulting by precisely guiding the placement of each piece of the vault, providing feedback on when a portion had been placed incorrectly.18
AR can also be used to help “discover” new information, such as detecting when a machine or device might be emitting excessive heat or radiation, or providing enhanced visibility of terrain in conditions (fog, fire, darkness, etc.) in which humans might not be able to see or navigate on their own.19
In other cases, AR can log data and information automatically for the user, transforming how a workforce captures, reports, and shares information. This can, in turn, increase productivity, reduce errors in documentation, and streamline audit or accounting processes. It can also more accurately track physical tasks and labor to help optimize assignments and scheduling based on worker availability and capacity. All of these uses of AR represent a streamlining and potential improvement of current work processes, rather than an evolution of capabilities.
Implications of Evolution I—a new mind-set. While Evolution I does not significantly change the tasks workers are asked to do, it does significantly impact how they are asked to accomplish them. Whether inspecting pipelines for leaks or setting ancient Roman vaults, workers will be asked to do familiar tasks in new ways. The rationale for this shift must be clearly communicated and workers must see some benefit or they may simply revert to older, more familiar techniques.
Evolution II: Infinite mind
Workers are increasingly being asked to handle high volumes of data—in many cases, a greater load than the human mind can possibly handle. For scenarios in which the volume of information is high while tasks are relatively predictable, AR can be used to provide workers with a data overlay in a consumable way that still makes it possible to accomplish tasks. Here, AR can begin to enable workers to accomplish new tasks, or address old tasks in new ways.
Maintenance crew on an aircraft carrier, for example, must maintain and repair a wide variety of extremely complex machinery, from fighter jets to helicopters. This requires highly technical skills, but also the use of bulky manuals; crewmembers often find themselves stopping and starting as they scroll through documentation to find the correct set of instructions to accomplish a task. AR can free the maintenance crew from the need to remember large lists or carry around bulky manuals, by overlaying instructions in the crew member’s field of vision in real time, as needed. This makes the work faster and more accurate—and frees both hands to accomplish tasks. In fact, this is already becoming a reality with a beta test from Siemens, which has equipped its Vectron series of train locomotives with AR manuals.20 These manuals allow workers to pull up CAD drawings or even repair instructions for the exact part they are looking at, offering them immediate and easy access to several thousand pages of information.
Implications of Evolution II—new skills. Evolution II makes huge volumes of data available to workers. This can allow them to perform previously impossible tasks, but it also requires new skills to navigate vast amounts of information. For example, now train drivers would not only need to know how to operate a train, they would also be required to learn how to inspect and use the AR tablet. Care must be taken in the training, and even hiring of these positions, given the new skills required.
Evolution III: New connections
In the third evolution, new connections can be formed using AR, enabling highly variable tasks with simple information requirements. The majority of tasks of this nature involve human interactions, which differ and can be highly unpredictable. Some, however, can require the user access data that a worker might not have at his or her fingertips. So in this stage, having ready and contextual access to that sort of information can enable higher productivity.
At its simplest, this sort of new connection can simply take the form of “see-what-I-see” sharing. For example, continuing with train maintenance, imagine a train that wouldn’t start and a worker who, after attempting all of the typical troubleshooting steps, could not identify the malfunction. Since the problem is unknown, the worker cannot use AR to call up instructions to fix it. So instead, he or she could contact a small cadre of senior maintainers at a central facility. With AR showing those maintainers exactly what the on-site worker sees, they can help to diagnose the issue.
AR can also be used to capture and disseminate specialized knowledge. For example, a surgeon who just developed a novel, potentially lifesaving technique can use AR to easily share information and instructions with colleagues, spreading the word more quickly and effectively than a journal article would. By using AR, colleagues would then be able to access this information quickly during surgery, should that specialized knowledge be needed at any given moment. In another scenario, engineers and designers could use AR to make the design process more efficient and less wasteful. Rather than printing or manufacturing physical prototypes to test product ideas, they could use AR to improve designs by planning and testing product assemblies or working with virtual prototypes during the design process.21
Implications of Evolution III—untethered work. Evolution III offers the opportunity to break free of the constraints of location. Now maintainers do not need to be in the same location as the machinery; workers can collaborate on designs or share notes across the globe. Much like the tele-work revolution enabled by the Internet, this use of AR will require some care to create cohesive teams that can work together effectively despite the loss of direct contact.
Evolution IV: Full symbiosis
This final evolution represents the culmination of AR’s use in the workplace. In assisting workers with highly variable tasks that also require a great deal of information to complete, AR can augment and complement the human strengths of intuition, creativity, and adaptability with those of computing—the ability to handle, access, and analyze high data volumes while connecting with other resources in real time—to enable new capabilities and maximize performance. This can bring the best of both humans and machines together, with machines able to deal with more complex data than any human could, and workers able to adjust to variability faster and more reliably than any computer. In this way, Evolution IV describes the future of human-machine interaction and the future of work.
In these scenarios, AR can link a human worker to, for example, a digital supply network, overlaying data about supplies, expected shipment times, production schedules, external data, and machine functioning over a field of vision, enabling planning processes or re-routing troubled shipments in real time to reach the production site on time.22 In this way, AR can bring together a full, complex network of constantly changing information and provide it in a contextual, visual manner to enable decision making in the moment.
In these data-rich, fast-paced uses of AR, human-machine interaction goes beyond a simple interface between worker and tool; the human and machine become a true team. Research from NASA offers a glimpse of what these future human-robot partnerships may look like, using AR for space exploration. Through research into joint human-robot teams, NASA is examining ways in which astronauts and scientists can collaborate naturally with robots and computing systems during complex missions via AR. NASA has pointed out that “to reduce human workload, costs, fatigue-driven error, and risk, intelligent robotic systems will need to be a significant part of mission design.”23 The agency points in particular to actions such as “grounding, situational awareness, a common frame of reference, and spatial referencing” as crucial to performing its work effectively, making AR a useful partner to solve these challenges. Using spatial dialog, NASA is looking to AR as a means of facilitating the collaboration between humans and robots as part of a holistic system. Taken back down to earth, similar AR-driven systems can be used to aid humans in highly unpredictable and potentially dangerous situations, such as search and rescue missions.24
In the long run, these uses of AR have the potential to completely reshape how work is done. Imagine it is 2025, and a cybersecurity analyst comes into the office in the morning. Defending a computer network involves sifting through immense volumes of data, but also reacting to the unpredictable variability of human hackers on the other side. After getting his or her morning cup of coffee, the analyst can sit down at the terminal and ask the system, “What is unusual about my network this morning?”25 If the system detected something unusual, not only could it highlight any unusual parameters, it could also identify who the individual hackers might be and what they may be after.26 With this information, the analyst can better respond to the variability of the situation and take appropriate action to deny the hacker’s goals and protect the system.
Far from being the realm of science fiction, the component parts of such a system already exist. What remains is for leaders to combine them in a way that is suitable for their organizations.
Implications of Evolution IV—Pushing the boundaries. More than any other use of AR, Evolution IV pushes the boundaries of human-machine interfaces to uncover previously unknown uses of the technology. As with any exploration into uncharted territory, it is likely to uncover new problems that designers or operators of AR may not have anticipated. As a result, companies electing to try to reap the large rewards of such a massive transformation need a workforce that is ready for the inevitable hiccups and motivated by the sheer challenge of exploring new ground.
Evolving into the future
These four evolutions of AR are not firm categories that restrict how the technology can be used. On the contrary, they are simply guides to help understand how AR can change the work environment. As a result, the evolutions can—and quite likely will—begin to merge together over time. Take the two areas where AR has already been widely piloted: vision picking and “see-what-I-see” expert support. Vision picking is an Evolution II use of AR; warehouse workers use smart glasses to keep track of a pick list and direct them to the proper shelf to find those items. In “see-what-I-see” support, part of Evolution III, workers are able to call upon experts to help them diagnose issues on the fly.
Flash forward a few years into the future, and we can see how both examples have expanded and pushed the boundaries of AR’s potential. The same vision picker now can not only see where the next item to be picked is located, but can also see other workers and their locations, passing items back and forth on lists depending on who is closest (Evolution III). The handful of employees in the warehouse are supplemented by an automated workforce that can take over many of the less difficult and repetitive tasks, such as moving inventory (Evolution IV). In addition, passive capturing of product data can help create records of arriving and departing shipments without the need to stop and answer phones, talk to drivers, or sit at a workstation (Evolution I). The wearable AR device has become a seamlessly integrated tool that allows workers to have maximum flexibility, access to information, and the ability to interface with a wide range of systems, from IoT-enabled machinery to legacy video feeds and communication systems.
A similar story can be told around “see-what-I-see” support. The system continues to offer live video support, but only as a last-ditch effort to solve a problem that has likely been faced before. The field worker is now equipped with a wearable device that has a library of solutions, compiled from a database of previous issues. By simply focusing on a given part within the field, the wearable will be able to identify the specific part and download performance data from sensors on that part. Predictive maintenance algorithms will then be able to show the worker directly when the part will likely fail (Evolution IV).27 If the part needs to be replaced, an overlay of how-tos will provide the field operator with just-in-time, step-by-step information including sequencing, proper tools, and tips/tricks to move through the process (Evolution II). Once completed, the AR device will record the maintenance procedure and the data will be added to better predict future part failures or maintenance needs before they become an issue (Evolution I).
The future of work merges humans and machines into one team so that they can seamlessly accomplish multiple types of tasks quickly and intuitively.
Realizing the future
How will this future of work be realized? Certainly the technology must continue to develop. Currently, AR still has some technical limitations, which include the need for tethering (being wired to a PC or laptop for processing power), an inability to recognize 3D objects, and a lack of actual spatial awareness. Much of AR is currently limited to 2D image recognition, meaning that devices can only recognize 3D objects from within a limited angle. And while the current technology can easily create 2D overlays on 3D objects, without the ability to lock these digital items onto the physical environment, it is difficult to accomplish anything meaningful. Hardware, too, must continue to develop; many headsets are clunky and awkward and have a very limited field of view, which make them seem restrictive and can be dangerous in high-risk environments (warehouses, industrial settings, etc.).
While the above may seem like a long list of shortcomings, they are all well-known and improvements are already being developed. So the real challenge to achieving the future of work promised by AR is not technological; it lies in how AR changes work itself. In other words, the impact of AR can stretch far beyond mere technology and touches how we work as individuals and as teams. This is where the true hurdles to AR lie and, as a result, it is where organizations would likely need to take the critical first steps toward achieving an AR-infused workplace.
Organizational leaders should understand that preparing a workforce for the inevitable onslaught of technologies that will support the emergence of an augmented workplace requires a shift in culture toward innovation and collaboration. Leaders also need to provide an incentivized way to integrate technology and just-in-time learning into the DNA of the organization. Here are some practices leaders could adopt to help build a more innovative and collaborative culture:
- Give credit for explorative and “just-in-time” learning. Employees could earn credit for activities such as watching TED Talks and listening to educational podcasts, as well as sharing solutions to issues and best practices with colleagues through “lunch and learns.” These informal sessions, in particular, can help develop a culture that values active problem solving.
- Promote the use of emerging toolsets (such as Skype, FaceTime, and Speech to Text) to increase the adoption of new productivity tools as they become available. This can be done by making tools readily available and having a rollout plan that includes incentivizing the use of new technologies. For example, encourage staff to use webcasts, screen sharing, and the live chat tool by having more remote meetings or creating work-at-home opportunities.
- Create a culture of technology integration and play. Organizations that adopt technical solutions quickly have established a culture of exploration where play is often encouraged. Having activities such as hackathons, where colleagues are provided a “play time” to identify ways in which tools can be used to solve problems, can create a culture where innovation and problem solving are recognized as important aspects of the organization.
- Cultivate a fast-fail mind-set among your staff. The fear of failing can choke innovation, stifle problem solving, and slow the adoption of toolsets that can make the workplace more efficient. A culture that encourages a “fail fast” mind-set where experimentation is supported and failures are viewed as learning opportunities—and as such, stepping stones on the path to success—can quickly adapt to innovations as they emerge in the marketplace.
By instilling these features in a workforce, an organization can help ensure that it is positioned to take advantage of the benefits of AR, wherever those lead—even to the moon.
