Generative AI meets the virtual world: A model for human-AI collaboration
Imagine a future in which virtual meetings with specialized AI agents felt similar to meetings with human coworkers. Could this enhance decision-making and boost performance?
Around the world, enterprises are experimenting with ways to incorporate generative AI into work processes. The future of work is starting to come into focus, but not without questions: We know we’ll be working with AI, but how exactly will the collaboration work? Will humans collaborate with one, all-knowing application (as some users do now), or will workers collaborate with a collection of AI specialists, otherwise known as agentic AI? Either way, how will we interact with these AI coworkers?
One effective approach could resemble collaboration between remote, human teams. Picture a human donning a pair of VR glasses and seeing a table full of coworkers ready to meet. Each is a gen AI program with a face, voice, and distinct personality. Each “agent” can provide status updates, ask questions, strategize, and—unlike humans—instantly present complex data visualizations and adjust them in real time. By combining an AI team-of-specialists approach with spatial computing, each human could be put in charge of a team of AI agents that they can chat, email, and hold virtual meetings with. While still on the horizon, this combination of technologies may enhance efficiency, collaboration, system resilience, and task-specific optimization once it arrives, leading to better performance and problem-solving capabilities.
Agent collaboration models: One brain vs. many minds
Generative AI-enabled agents have emerged as powerful tools capable of autonomously addressing intricate challenges, sometimes with little or no human intervention. These agents are designed to analyze vast amounts of data, identify patterns, and propose solutions, making them effective partners in creative and strategic processes. They could bring a level of efficiency and precision that enhances human decision-making, allowing teams to focus on higher-level thinking and innovation.1
When considering how AI agents collaborate with humans, two models emerge: the single, super-smart agent, and multiple specialized agents working together. Each model offers distinct advantages and challenges.
The single, super-smart agent is a centralized powerhouse capable of tackling diverse tasks with remarkable efficiency. Its strength lies in its ability to see the bigger picture by synthesizing information from various, diverse sources, which enables it to tackle a wide range of tasks. However, it has several drawbacks, including a lack of deep specialization, potential performance bottlenecks, and inflexibility in handling nuanced or industry-specific tasks.2 It increases the risk of system failure, as all tasks depend on one agent, and may lead to user overreliance on the technology.3 Additionally, a universal agent may take longer to reach proficiency in specialized areas, making it less efficient than using multiple agents tailored for specific functions—each with its own team of experts working to evolve its capabilities.4 These limitations can hinder efficiency and adaptability in complex environments. On the other hand, multiple specialized agents bring a diversity of expertise to the table.5 They mimic the dynamics of human teamwork, where each member contributes their strengths to achieve a common goal. This model fosters a sense of collaboration and inclusivity, enabling humans to feel part of a cohesive unit.6 However, coordinating these agents and ensuring seamless communication can be technically challenging.7
Another important aspect is control over task execution. A single, super-smart AI agent offers simplicity in communication, as it handles various tasks under one interface, making it easier to interact with. It also ensures consistency, providing uniform responses across different tasks. However, this can come at the cost of transparency. Since the agent handles everything and the user does not have direct visibility into the process, it’s harder to see exactly how decisions are made, and if something goes wrong in one step of a workflow, the entire workflow could be impacted without the user knowing it.8 Customization is another challenge, as it may be difficult to fine-tune individual task execution with a generalist AI.9
On the other hand, using several specialized AI agents can allow for more targeted control over specific tasks. Humans can adjust each agent individually, making the agent more effective to their specialized domain—and ultimately offering more flexibility and transparency.10 Specialized agents make it easier to track performance because a human can check each agent’s accuracy and correct errors when necessary. This means managing multiple agents can become complex, requiring more effort to coordinate and ensure consistency across tasks. There’s also the risk of overlap between agents, which can create inefficiency.11 However, the concept is powerful: Imagine a meeting with a team of virtual agents, each specialized in different types of analytics, to discuss organizational and market performance, as well as to create projections and recommendations for future corporate strategy.
Here are some potential advantages to using multiple agents:
- Specialization and efficiency: Specialized agents can be tailored to specific tasks making them more efficient and effective in their respective domains.12 For example, a fine-tuned language model trained on medical data would be better equipped to handle medical terminology and generate text related to healthcare, compared to a general language model. This mirrors the way human teams operate, where individuals bring unique expertise to projects. Experiments demonstrate that systems consisting of multiagent groups can outperform a single agent.13 By dividing tasks among multiple agents, each focusing on its specialized role, the system can handle complex, multifaceted challenges more effectively than a single, all-encompassing agent could for the same reason that a team of human specialists with complementary skills and the ability to share knowledge and work in parallel would likely outperform a single, multitalented generalist.
- Emergent collaboration: Studies on multiagent frameworks, such as Deloitte’s “How AI agents are reshaping the future of work”14, have shown that when agents collaborate—just as with humans working together—they can surprise us with new, creative ways to tackle complex problems, otherwise known as emergent behaviors. This collaboration allows for dynamic problem-solving, where the collective intelligence of the agents results in outcomes greater than the sum of their parts.
- Resilience and flexibility: Also, a feature of human teamwork, multiagent systems are typically more resilient to failures. If one specialized agent encounters a problem, others can continue their tasks, reducing the impact on the overall system. In addition, one agent can review outcomes produced by another agent to ensure the work is aligned with a project’s goal.15
Using multiple specialized agents in a multiagent environment can provide greater flexibility and scalability compared to relying on a single, super powerful agent, especially for complex problems. Multiple specialized agents can operate independently or collaboratively, adapting more readily to dynamic conditions and diverse requirements. This decentralized approach enables more efficient resource allocation, parallel processing, and robust problem-solving capabilities. In contrast, a single super powerful agent may struggle to handle the intricacies and varied demands of complex scenarios. The modularity and distributed nature of multiagent systems enable agents to scale effectively, accommodating growth and evolving challenges without compromising performance or efficiency.
Agents are not humans
Despite some similarities between teams of humans and teams of advanced AI agents that possess reasoning capabilities, it’s important to harness the technology’s power while keeping expectations realistic. Though they can process vast amounts of information, detect patterns, and simulate reasoning, AI is still driven by algorithms and data rather than true human understanding or intuition. AI, no matter how sophisticated, currently lacks emotional intelligence, ethical judgment, and creativity, which are essential in many decision-making scenarios.
To effectively collaborate with generative AI, it’s crucial to leverage its strengths—such as data analysis and logic-driven insights—while still recognizing that ultimate responsibility for complex decisions should rest with humans. AI can offer valuable support, but it can’t replace human judgment in situations that require empathy or ethical considerations. Using human judgement to set clear, specific parameters for how AI engages in problem-solving ensures that its contributions remain focused on tasks that fall within its computational abilities.16
While gen AI can generate sophisticated responses, it’s essential to remember that these are not the result of human-like understanding but rather the output of language models trained on patterns in data. Keeping this in mind helps prevent attributing human qualities to AI, allowing for a balanced and productive collaboration where AI enhances human decision-making without overshadowing it.17
Add the virtual universe and spatial computing: A new frontier for collaboration
A virtual universe, or “metaverse,” where digital and physical realities converge, is emerging as a platform for collaboration between humans and AI.18 Spatial computing refers to the technology that blends digital and physical environments using 3D models, augmented reality (AR), virtual reality (VR), and sensors to interact with data and environments in a spatial context. It allows computers to understand the physical world by interpreting the space around them and augmenting reality with digital information. What will be interesting to explore is the sheer number of ways humans can give feedback to these AI agents. Humans could use virtual whiteboards, voice commands, live demos, and even run benchmarks across multiple screens—all within an immersive environment that could make them feel closely connected to their work.
Imagine a brainstorming session. A human sketches out ideas on a virtual whiteboard while the gen AI analyzes their work, suggests improvements, and even visualizes complex data in real time. This isn’t a typical, human back-and-forth; it’s a dynamic exchange between human and machine, where creativity and AI-driven insights come together to produce something innovative.
Imagine also the ability to run live demos and benchmarks. Humans could see the results of their feedback instantly, whether it’s tweaking a design, testing software, or exploring new strategies. The gen AI agents could quickly simulate different scenarios across multiple screens, providing insights that would take hours—or even days—without this technology.
The real magic of the virtual universe lies in this continuous loop of interaction. It’s not just about making things faster or more efficient; it’s about fundamentally changing how we innovate. By integrating diverse feedback mechanisms, the virtual universe transforms collaboration into a rich, responsive process. This is where the future of creative problem-solving may begin.
The concept of human-agent collaboration using spatial computing in the virtual universe is gaining significant traction, particularly with the integration of generative AI.19 For example, research into MetaAgents (agents that are designed to monitor and manage the actions of other AI agents) shows that these AI-driven agents can simulate human-like behavior and collaborate on tasks, improving coordination and effectiveness in virtual spaces.20 This research indicates that such agents can operate as part of a team with humans, filling roles that require specialized knowledge, similar to how human colleagues would in a traditional setting
Would we have to put a pair of VR glasses on every time we communicated with these team members? No. As with human-to-human collaboration, agents could be made available by email, instant messaging, or even voice-only calls if we found that helpful. Agents could be working simultaneously with humans, ready to report their progress to us on our time, across whatever medium worked best for each human.
Practical examples of human-AI collaboration in a virtual environment
The possibilities for synergy between humans, generative AI agents, and virtual technology are vast and varied. Here are a few potential scenarios:
- Discovering new medications: In the virtual universe, researchers and AI agents could collaborate to simulate clinical trials and analyze patient data. Human scientists would bring domain expertise to guide AI in selecting relevant data sets and forming hypotheses. AI agents could simulate potential chemical reactions, quickly iterating through possibilities to suggest promising compounds. For example, two medical researchers working for a pharmaceutical company are tasked with developing a new diabetes medication. They meet weekly with AI coworkers that run simulations and analyze clinical trial data. The researchers guide the AI in selecting relevant datasets and formulating hypotheses, while the AI rapidly simulates chemical reactions and tests compounds for safety and efficacy. Together, they identify a promising compound and prepare for human trials, accelerating the process of drug discovery.
- Software development: AI agents, equipped with advanced programming capabilities, would autonomously generate code based on specifications and user stories provided by human product owners.21 For example, a product owner at a tech company collaborates with an AI programming agent to develop a fitness app. The product owner defines the vision and user stories, while the AI agents autonomously generate code, test for bugs, and refine the app based on feedback. With the technical work handled by the AI, the product owner focuses on strategic development and user experience, delivering a functional app in a fraction of the usual development time.
- Marketing campaigns: AI agents could analyze consumer data and generate creative concepts, while humans refine these ideas to create compelling narratives that resonate with target audiences. For example, a marketing lead at an apparel company works with an AI agent strategist to create a campaign for a new eco-friendly product line. The AI agent analyzes consumer sentiment and generates creative concepts, which the marketer refines into a compelling narrative that aligns with brand values. During the campaign, the AI agent monitors performance and recommends strategy adjustments in real time, while the marketer oversees brand messaging, resulting in a successful and impactful campaign.
- Financial services and banking: In a virtual universe, a team of financial analysts and AI agents could work together to optimize investment strategies. Human analysts would provide insights into market trends and customer needs, while AI agents could conduct risk assessments, simulate investment outcomes, and identify opportunities for cost reduction. For example, a financial analyst at a global bank teams up with an AI agent to optimize investment portfolios for clients. The analyst provides insights into market trends and aligns strategies with client goals, while the AI agent runs simulations, assesses risks, and monitors market changes. Together, they deliver personalized, data-driven investment solutions that enhance client satisfaction and help the bank maintain a competitive edge.
Impact on the workforce: Navigating the transition
As AI becomes more integrated into the workplace, many have concerns about job displacement.22 The shift to a team-of-agents future could easily magnify those concerns. How can organizations and employees transition smoothly into this new era? Shifting how we think about human contributions may go a long way: investing in reskilling and upskilling programs and redesigning work processes so that human qualities such as creativity, empathy, and strategic problem solving are at the core of employees’ contributions.
Leaders can also address the “productivity effect”: Deloitte’s research indicates that the impact of generative AI on the labor market will depend heavily on how well it boosts overall productivity.23 While some jobs may be displaced, the increase in productivity could create demand in other sectors, leading to new job opportunities. For instance, higher productivity in one area could lead to lower costs and increased demand for goods and services, which in turn could create jobs in other industries.
It would be disingenuous to suggest that a future with humans leading teams of AI agents won’t be disruptive. But by focusing on how AI can augment human capabilities rather than replace them, companies may foster an environment where technology enhances human work and aligns with broader goals of employee wellbeing and societal benefit.24
How will this brave new virtual world come about?
We are seeing progress toward the vision of humans collaborating with AI agents in the virtual universe, but several technological developmental hurdles remain.25
Currently, generative AI, like large language models, has made significant progress in terms of understanding and generating human-like responses, which enables real-time collaboration in text and voice-based environments. However, seamless integration of these capabilities into a fully immersive virtual environment, where AI agents can interact in 3D spaces with complex, dynamic tasks, is still emerging.26
The infancy of the virtual universe itself is another factor. While there are platforms offering immersive environments, we are still years away from the level of sophisticated, scalable, and fully interoperable virtual spaces required for true collaboration. As an example, NVIDIA’s Omniverse, which was the culmination of more than 25 years of advancements in graphics, computing, simulation, and AI, could potentially serve as a foundation for a functional metaverse. Combining more advanced virtual spaces with advanced AI capabilities—allowing humans to work alongside AI agents in real-time, interact with virtual objects, and engage in problem-solving across industries—will require further advances in both AI and virtual universe infrastructure.27
To achieve collaboration at scale, we would need:
- More sophisticated AI agents: While AI can currently process natural language, we still need AI that can work with complex, non-verbal tasks, like manipulating objects in a virtual space and working intuitively with humans across various mediums, including visual and spatial inputs.
- Better VR/AR hardware: More immersive, user-friendly, and affordable VR/AR devices will enable widespread adoption and integration with AI agents in collaborative environments.28
- Standardized virtual universe platforms: These spaces will need standardization and interoperability to allow seamless transitions between various virtual environments, enhancing collaboration between AI and humans.
By taking the opportunities and addressing the challenges of the metaverse, and utilizing the potential of generative AI within it, we can move toward a future where people can fully explore and integrate the physical and digital realms, driving a wide range of innovations.29 To make this happen, it will take further advancements in both the AI domain and the underlying infrastructure of a virtual universe.30
Meanwhile, organizations can further realize the potential of AI by embracing and maturing AI agent frameworks powered by generative capabilities and explore spatial computing to integrate the physical and virtual worlds.
Ultimately, the fusion of AI, virtual environments, and human creativity is not just a technological shift but a transformation of work. It offers a chance to reshape collaboration and innovation by embracing our human qualities to achieve business goals in a world full of endless possibilities.
