Reshaping human-machine connections Insights In Depth: Tech Trends 2020

15 July 2020

AI-powered solutions may help technology respond to humans more naturally. Deloitte's Amelia Dunlop and Scott Buchholz discuss the possibilities with Sterling National Bank's Daniela Fiumara and Marcelo Theodoro.


Tanya Ott: There’s a bank in the metro New York area that has grown exponentially over the last decade, mostly through acquisitions.

Daniela Fiumara: We went from a US$3 billion bank to a US$30 billion bank within a seven-year timeframe. So, it’s been a crazy time, but exciting as well.

Tanya: They serve about 285,000 clients, most of them on the consumer side. Which means they get the same questions over and over again.

Daniela: The contact center receives a high volume of calls on a yearly basis. And a majority of these calls are related to basic information or requests that can be easily automated. So, they’re asking for things like their balances, transactions, check activity.

Tanya: The majority of their customers are native Russian and Spanish speakers, which can present its own challenges. But the bank is now experimenting with technology that could be a game changer.

I’m Tanya Ott and today we’re talking about human experience platforms on this Tech Trends edition of Insights In Depth.

So, what’s a human experience platform? Well, one version of it is Skye—the new artificial intelligence customer-service worker at Sterling National Bank, where Daniela Fiumara is the director of Automation, AI & Robotics.

Daniela: My role is basically to execute on enterprisewide projects throughout the organization, specifically in the area of [automation, AI, and robotics].

Tanya: She’s one of our guides. We’ve also got Marcelo Theodoro …

Marcelo Theodoro: I’m currently the chief digital officer at Sterling.

Tanya: Scott Buchholz

Scott Buchholz: I’m the emerging trends research director within Deloitte, and that means that I lead our Technology Trends research and report and get to go share it with customers.

Tanya: And Amelia Dunlop.

Amelia Dunlop: I’m the chief experience officer for Deloitte Digital. And that means I get to wake up every day trying to elevate the human experience for our customers, our workforce, and our partners.

Tanya: You’ve probably had a mix of positive and not so positive experiences with customer- service call centers. Maybe the call-center worker stumbled through their script or couldn’t quite understand your question. My guests today say we have the potential to improve that experience through what they call human experience platforms like Skye—the AI interface.

Amelia: The way we think about the human experiences is, we don’t wake up each morning and think, “I am the customer of this amazing cup of coffee I’m drinking right now,” or “I am an employee of this great organization that I’m going to work for.” We start our day as humans. We end our day as humans. And now, even more in this moment, we are really experiencing our humanity. So, the human experience that we really try to focus on, particularly as we see the acceleration of digital in our lives, is how do we be more human at scale? And how do we use the technologies to let us connect with each other in more human ways to demonstrate the empathy that creates loyalty and long-term business results, as well as just a greater sense of our common humanity?

Scott: What people are trying to do in order to achieve the goals that Amelia was talking about is understand, how do we use technology to recognize and respond not just to the content of the requests, but also to the human emotion that sits behind it? Part of what we’re talking about is it’s a combination of different technologies that range from everything from voice recognition and sentiment analysis to cameras and microexpression detection and a variety of other things across the spectrum. In this particular case, we’re talking about virtual call-center agents. As we move forward, we’ll increasingly likely see kiosks that are smarter and other things that enable people to do really interesting interactions in a much more human and humane way.

Tanya: Marcelo, what does this virtual call-center experience sound like?

Marcelo: It connects with what Amelia and Scott said. Because of our client profile, we are being very careful about things like accent. As you can see, my English is far from perfect and I have a strong accent. Our solution needs to be able to support me as well as support Daniela, who is a perfect English speaker. So, how do we bring this human aspect to the interaction so I don’t feel bad because my English is not perfect or I don’t feel bad because my accent is not understood by the solution? Imagine if I call there and I keep seeing our solution reply to me: “I’m sorry, I cannot understand you. I’m sorry. I cannot understand you.” The experience itself becomes so bad that this potential client will likely [not feel comfortable being supported by the virtual agent anymore]. So, bringing this human aspect to the support model and incorporate it within our solution is something that we are very careful [about].

Our first stage, our first deployment will not be as human as Scott said, when you have an image, or you have facial expression being understood by the platform. But I have no doubt that’s the future. When you think about automating the financial center, I’m sure that the kiosk example that he said is extremely helpful. So, we are trying to take care of that. We are trying to guarantee that accent or different ways to express the same word is understood and is properly replied throughout this interaction.

Amelia: I did have a question for you, Marcelo. I love what you said about making sure that the virtual assistant can understand different accents. Have you thought about, will the assistant respond in an accent that is recognizable to the individual or is that just another level of design in the future?

Marcelo: I personally thought about it. The challenge that I have is when I hear myself speak in English, I really think it’s horrible. So, I don’t know how our clients would feel about it. But Daniela, can you tell us about the accent aspects of the project?

Daniela: For [our current] implementation, we’re focusing solely on [General American] English. We were trying to [improve detection of] the Russian accent, but we couldn’t get it in time for this first release. But we will be going forward.

Marcelo: How do Scott and Amelia see this internalization of solutions and how companies can move forward from this point?

Amelia: I’ll take a crack at that. One of the things that I’ve observed is that organizations that are trying to bring in virtual assistants or bring in artificial intelligence treated it as though they’re bringing in a new employee. And that new employee requires acclimation, onboarding, training, mentoring, coaching, and actually a lot of human touch, to be honest, to [be] able to make them functional, valuable contributors. Sometimes when we think AI, we think that’s just as a platform. We’ll kind of put it in and turn it on and it’ll work. And what it sounds like you’re experiencing is it’s not quite that easy as, you know, plug it in and it will work. Anticipating the fact that you actually have some human problems to solve, very similarly if you were to be bringing in a new employee. So that’s [the] observation I have.

Scott: That’s right. It’s important to recognize you’re onboarding a new colleague on some level. On another level, we sometimes forget how many different systems call-center agents and contact-center agents actually have to use [that] to be useful in their daily, day-to-day interactions. And we forget that the AI probably needs a deeper level of interaction and a deeper level of training to be able to use it in the way that seems intuitive to people.

And a lot of this process is taking the discernment and judgment of human beings and embedding it in the technology. And while everybody wants to believe that there’s an easy button for that, I don’t think we’ve yet found it as an industry.

Tanya: Let’s talk a little bit about that training, because the training is so important. Can you guys break down for me how you’ve trained Skye? What goes into that process?

Daniela: In order to train Skye, we needed to understand the client journey when calling into the contact center. There are some questions that we asked ourselves: What are the user goals? How are our calls handled in the contact center today? What ways can we improve this experience? How can we use Skye to overcome any obstacles or specific pain points? To answer these questions, we had to interview our contact-center agents as well as our client experience teams to gain more clarity around the client’s behavior during these calls [and] what they’re asking for. We created journey maps and process flows. We conducted research [on], for example, what factors other banks are using in the authentication process and how to make that a better user experience. We also listened to hundreds, maybe thousands of call recordings to better understand the types of questions clients are asking and how the contact center agents are responding back to the client. Throughout this process, we were consistently reassuring that the voice of the customer was [being] represented by those who know the customer best.

Tanya: What are some of the problems that you encountered as you were going through that process?

Daniela: One of our challenges was related to the architecture of our technology stacks. Parallel to the Skye implementation, we are redesigning our API platform because we want to build out our digitalization strategy based on a reliable and scalable solution. With that being said, while implementing Skye we are also developing the necessary APIs that she needs to consume in order to support the use cases that she’ll be handling now and going forward. Marcelo, do you want to add on to that? I know the API [platform] is your baby.

Marcelo: I think you explained it greatly. In time, I see at least that our solution, Skye, is going to be just part of a society of robots. We are going to have different robots, different virtual agents specializing in different solutions. And they have to interact and play together in a very orchestrated way. As Daniela said, the APIs supporting these different solutions are still under construction and every day we learn something new. It’s kind of an endless journey that we have to prepare ourselves early in the process to go through.

The second point that I would like to add is the UAT process—user acceptance testing. So in order to guarantee that we’re covering all the different scenarios that involve human beings, as we said before, we’ve been expanding this UAT process a lot so we can test all types of interaction, all types of accents, all types of questions in a way that when we deploy it, we are covering everything as much as possible that can happen in production. Combining the evolution of other stuff that are happening in parallel, with the testing that you have to go through to guarantee that your solution is ready to go live, are two additional challenges that I would highlight.

Amelia: Marcelo and Daniela, I just had a question, too, for folks listening. Obviously, you’ve been working on this initiative for many months now, but any particular challenges that you’ve identified or even opportunities, given the current circumstances? Did it have to change any of your timelines, change any of the requirements?

Marcelo: We didn’t change our timeline based on that, but everything that is happening highlighted the importance of digitization for everybody, including ourselves. So if we had the desire, for example, to build 20 use cases with Skye, now our design is going to be to build 100 use cases with Skye. And virtual agents are a piece of this big puzzle, but we are going through a whole review of our digitization strategies that would intensify the virtual agent process, that would intensify any kind of automation, that would intensify the APIs built, that would intensify online banking solutions. But [the work is] going to speed up the digitization process and the desire and the need for a solution like Skye is now growing exponentially.

Amelia: I totally agree. It’s almost as though the present situation has accelerated the need for the future. And you happen to be sort of ahead of that curve, already anticipating it, but it’s only accelerated the need to change how we interact in a digital way.

Marcelo: And there is that joke I think everybody saw online about who sped up the digital transformation in your company—the chief digital officer, the CTO, the CIO, or COVID-19? And through this, in 99% of the cases, it is COVID-19, because I [believe] we are working more from home, we are accessing all our service providers digitally and remotely. If there was any question in any company that digitization was not the future, I think it’s gone. Definitely gone. So it’s a matter of time and how much you want invest to be more digital. But again, each company has its own pace.

Tanya: Scott, I’d love to bring you into this because obviously this is the direction a lot of things are moving. But from the perspective of the consumer and whether or not they’re comfortable with this experience, we have this idea in video of the uncanny valley, when you see that video-created image and the eyes just look a little glassy. Not quite right. So how do you address that potential creep factor when you’re dealing with AI systems like this?

Scott: At the end of the day, the art of doing AI really well and the art of doing a virtual call-center agent really well is a lot of what Daniela and Marcelo have been talking about. You have to deeply understand what the best call-center agents do. It’s actually really easy to emulate the worst experience you can imagine. It takes much more work and care to actually emulate the best experience. Once you’ve figured out how to emulate the best call-center agent, then we get into some more philosophical questions about the degree to which it’s important to let people know that they’re communicating with a virtual agent or a human being. And that is something that different organizations are making different choices about. In the coming years, we will see a lot of different experiments with different organizations trying different things that will help us collectively as a society decide what’s acceptable or not.

Marcelo: I would comment that it brings us back to the humanization aspect of this conversation, right? We tend to judge (or underestimate) the capacity of our clients to adapt or to adjust themselves to some technology and it’s not necessarily true. I will bring a different example, but I think it proves the point. The company had discussed the implementation of end-to-end digital [workflow for origination] and [initially] there were a lot of concerns. Because we needed to launch the PPP program (SBA Personal Paycheck Protection program) really quick (we launched it one weekend) there was no other way for clients to apply for this program, if not through digital solution. And we had thousands of applications and again, despite some challenge through the road, every client did it, did it well and is receiving their funds now on an end-to-end digital journey. So, when we have a need or, to Scott’s point, when you have a great user experience, the channel becomes something secondary. The important part is what is the experience that you’re providing your clients and how you are making their lives better? If we make it better, they’re going to talk with Marcelo, with a robot, with whoever they need to talk, as long as the experience is great.

Amelia: I’m curious, Marcelo, Daniela—how have you thought about the role of empathy and training a virtual assistant? And, you know, how important is it or not? How do you teach empathy to a robot? What does it sound like? It’s important because we’ve all had that experience where we’ve had that best kind of call-center truly empathize with the frustration that we might have, de-escalate the situation, and then help us get to the result.

Marcelo: We thought about it. Maybe this is one of the biggest challenges because empathy, it’s such a human feeling, and it’s so hard to replicate through an AI. My personal belief is that probably in the beginning the solution is going to be a bit colder, a little bit straightforward. I would believe that we are in different spectrums of interaction, without judgment and without any kind of label. Skye has a challenge that you have to play between these two worlds, and you have to adjust yourself. So, I don’t expect Skye to offer empathy enough in the beginning, but I believe that in time, as we learn it, as we reach this platform with more content and more inputs, we are going to get there. It’s a concern, in my opinion, very hard to address in the beginning.

Daniela: In understanding the demographics of our client base we wanted to develop Skye’s persona to act like a more younger, tech-savvy individual, who’s always respectful, caring, patient and doesn’t mind taking her time to explain things. And on top of that we want to make her apologetic when she doesn’t understand a particular request. And basically that she can quickly sense a customer’s frustration and know when to escalate to a live agent if needed. So, we’ve kind of built that in. But like Marcelo said, she might not be able to show the empathy right away. But [once] in production, she’ll get better as she goes and start to better empathize with our clients.

Amelia: That makes sense. It sounds like what we all do, right? We are all learning. We all grow. So, it sounds like she’ll start out and then learn and then hopefully grow in empathy as well.

Tanya: So what’s the timeline for deployment, Daniela? When does Skye go live and what’s the process after that?

Daniela: With this launch, our strategy is based on an agile approach. As we’re one of the earlier adopters of this technology, considering our bank size as well, we want to be mindful of the pace that we’re taking when rolling out into production. We like to think of day one being Skye’s first day on the job. We don’t want to overwhelm her with 100% of the calls coming in. She’ll start with a small chunk of calls in the beginning, and we’re going to see how she handles them. If successful, we ramp up the number of calls each day. But if there are calls that we see that she’s not able to handle successfully, we’ll ramp down the number or not feed her any calls at the moment and fix any bugs or issues. And then we start all over again and give her another small chunk of calls and then ramp up each day as she becomes more and more successful.

Tanya: So if you were projecting forward, then what do you all think the next, say, 24 months or so look like for Skye and this experience that Sterling and Sterling’s customers are going to go through?

Marcelo: Coming back to Amelia’s point about empathy, in time we want Skye to also support our internal employees, our internal colleagues. And the level of empathy, the level of emotional engagement is very different. So, if I need to ask for a new [computer], maybe Skye can help me. I don’t have to go to a human body for that. Human resource request—I want to know my paycheck. I want to know how to get information about my health insurance. The portfolio of possibilities is huge and we really want to embrace the majority of them, or at least the ones that are possible to be embraced by this technology.

Tanya: Scott, what Marcelo and Daniela are talking about is a world where not only clients or customers are going to be interacting with these kinds of AI robots, but pretty much everyone within an organization. What sort of implications does that have for the way we function as organizations?

Scott: It’s more exciting than many people realize. I have personally been the recipient of a number of less than delightful interactions with call-center agents over the years, whether as a customer or occasionally as an employee. What people don’t recognize is a lot of contact-center agents’ bad days come from having to answer the same routine things over and over again. If we now have technology that can deal with the routine things and we leave the really interesting things for the human beings, we leave the challenging things for the human beings, that actually means that not only do we collectively as employees or customers have a better experience overall, but the people sitting in the call centers and contact centers actually get to have better days as well. And their better days will help our days be better. So I’m actually optimistic about how this goes in a virtuous circle over time.

Tanya: Marcelo, Scott, Daniela, Amelia, thank you so much for the conversation today. Fascinating stuff!

Marcelo, Scott, Daniela, Amelia: It’s lovely to be here. Thank you.

Tanya: That was Scott Buchholz, chief technology officer for Deloitte Consulting’s Government practice; Amelia Dunlop, the chief experience officer for Deloitte Digital; Daniela Fiumara, the director of Automation, AI & Robotics with Sterling National Bank; Marcelo Theodoro, Sterling’s chief digital officer.

In this series on Insights In Depth, we’re taking a deep dive into each tech trends from our 2020 report … so stay tuned for more in-depth conversations with the people who are building the future right now in some of the most exciting developments at the intersection between technology and humanity.

You can also connect with us on Twitter at @DeloitteInsight. I’m on Twitter at @tanyaott1. I am Tanya Ott. Be well and stay safe.

This podcast is produced by Deloitte. The views and opinions expressed by podcast speakers and guests are solely their own and do not reflect the opinions of Deloitte. This podcast provides general information only and is not intended to constitute advice or services of any kind. For additional information about Deloitte, go to