Scaling Agile for government has been saved
At first glance, Agile approaches may seem anything but suitable for the kinds of large, complex IT projects common at government agencies. But while applying Agile at scale can be challenging, adopting several key principles can help make the effort a success.
Government is increasingly looking to use Agile to quickly deliver technology that meets users’ needs. But what about applying Agile in the kinds of large, complex IT projects so common in government?
Scaling Agile presents unique challenges. A big project can require multiple teams. So how do you fit the pieces together, particularly if the final design isn’t fully envisioned at the start? How can you get the same sort of quick decision making that allows small-scale Agile to rapidly deliver working prototypes? How do you ensure that interdependencies between teams are accounted for without creating the sort of bureaucracy Agile seeks to eliminate?
It is true that Agile often involves small project teams—often under 10 people—working on narrow projects with timeframes measured in weeks or months rather than years. The Agile approach, however, can also be used on large megaprojects within sizable organizations—but it can be tricky. Agile is more than just a way of developing software; at its core, Agile is about creating high-performance teams, and as such can be well suited for use within large government agencies. Agile is about bringing out the very best in people, both on the business side and the technical side, to work together to solve real business problems.
Successfully building software through Agile relies on three elements. First, a clearly defined set of business problems and a vision of what’s needed to solve those problems are important. Second, because business leaders and technologists generally bring different perspectives and working styles to the task, steps should be taken to ensure their viewpoints are integrated. Third, the creation of the solution should be an evolving journey between the business side and the technology side who collaborate to constantly redefine the best available outcome as quickly as possible. As a result, while there are various “flavors” of Agile that employ various methods and implementation approaches, they all share certain characteristics, including:
These Agile characteristics can challenge traditional government systems, particularly on larger projects. In essence, you want to create a space to “Let Agile be Agile” while still satisfying the demands of the larger organization. Some of the key challenges that typically need to be addressed in scaling Agile include:
The basic principle is simple: Create an overall framework that allows small Agile teams to work in time-boxed, iterative sprints in close connection with business users. Agile distinguishes itself in part by ensuring frequent assessment of progress, and these iterative sprints allow for regular demonstrations of business value. The framework should ensure frequent delivery of working software from across the Agile teams to help demonstrate that coordination is happening at scale.
Several examples show how Agile is being applied at scale in government:
The FBI case management system: Following the 9/11 attacks, the FBI sought to build a new virtual case file management system to facilitate information-sharing. The project was initially undertaken using a traditional “waterfall” approach. Between 2000 and 2010, the FBI spent hundreds of millions of dollars on multiple iterations of the project with disappointing results. The agency subsequently adopted an Agile approach for the project, a change that required no small amount of culture change.1 Using the Agile approach, the agency was able to deliver a working system.
Texas Health and Human Services Commission: The Texas Health and Human Services Commission is one of the largest agencies in one of the nation’s largest states, with a budget of roughly $30 billion per year.2 So its decision to embrace Agile as a way of maintaining one of its core IT systems—its integrated eligibility and case management system—required a significant amount of cultural change. After a successful pilot effort, the agency more broadly embraced Agile, including educating staff to deliver better software faster. This investment in people readiness was critical to the effort’s success.3
California Child Welfare Services: The California Health and Human Services Agency had been working for years on a traditional waterfall request for proposal (RFP), and then chose to switch to an Agile procurement approach. According to Stuart Drown, the deputy secretary of innovation and accountability at the Government Operations Agency, “In 2015, we were about to release an RFP for a $500 million-dollar project. We were on the seventh version, and had worked on it for nearly three years. In a very dramatic switch . . . our leaders decided to go to an Agile model or Agile approach.”4 The agency's approach includes breaking the project into sequenced modules and employing multiple vendors. It will take time to see how this effort at massively scaled Agile will play out, but it does indicate a serious shift in public sector thinking toward applying Agile to large, complex projects.
Regardless of the specific Agile approach, as with any large public sector IT problem, the potential exists for challenges, as mentioned earlier. Below we discuss the four key challenges and strategies to address them:
In Agile, the precise features of the final product emerge through a process of joint discovery rather than through a theoretical design vision. One of the most challenging aspects of Agile at scale may be the need for multiyear road maps in light of an uncertain, evolving future end state.
Teams can do all the right things, with all the right controls and following best practices, but get nowhere without a guiding vision. With the help of multiyear road maps, project leads can help teams remain on track to deliver business value, constantly refining the product’s next minimum viable product (MVP) release. Large organization initiatives often involve multiple objectives with long time horizons. Implementations should inform long-term business and product road maps while maintaining basic Agile principles, including the ability to adjust the end vision as new information comes to light during the build. This requires a “loose-tight” long-term road map that ensures the project provides consistent incremental improvements to business value throughout the long and winding journey.
The road map is primarily geared toward framing the work in executive-level terms. It should help teams understand the sequence of work needed, including balancing the user-visible features with the underlying enabling capabilities—a.k.a. the “plumbing” needed to support working software that solves the most pressing business needs.
Large-scale Agile implementations involve multiple Agile teams from disparate functions, often in dispersed locations. These resources are often pulled together from various organizational entities, all of which could have a stake in their use. In scaled Agile, it is critical to coordinate these teams under some form of common governance. For example, additional roles are often needed to facilitate communication and resolve conflicts. And strong leadership via the governance framework is critical.
Multiple Agile coaches and product owners should regularly convene in a “scrum of scrums” to facilitate cross-community communication and provide fast decisions. The Agile community of leaders provides a forum for communicating across teams, reporting, and consolidating tracking. This provides a more integrated and up-to-date message framework that allows senior management and interested stakeholders to receive real-time updates about progress during and after attending in-progress product demonstrations of working system capabilities. The regular cadence of demonstrations, along with direct two-way executive and business stakeholder communication and velocity reports from user and technical story completion rates, becomes the primary metric for quantifying progress, and it helps to build confidence that the project as a whole is on track.
Often, a team requires completed components from another team before proceeding further. Such cross-team dependencies create the need for additional coordination when planning upcoming iterations. The ordering of the sprints becomes important, but there should be a balance between flexibility and adherence to the overarching plan.
In Agile at scale, project leaders need to keep an eye on the big picture while the teams absorb themselves into sprints. The most common approach is to divide groups of teams into an “initiative” and multiple initiatives into a project. With different teams working on different features, someone has to ensure that the user interface—the overall “look and feel” of the various components—is eventually consistent. There is a particular risk here that user experience (UX) teams can get ahead of back-end services to the point that expectations held by the product owner are not achievable. Sometimes, addressing cross-team dependencies simply requires diligent communication and frequent validation through demos that exercise slices of functionality. Each Agile team typically holds a brief (usually 15–30 minutes) daily morning planning session to identify the work completed on stories and tasks, work planned for that day, and what blockers are impeding their progress. This information can then be fed up to a regular feature of scaled Agile where all of an initiative’s Agile coaches meet for a “scrum of scrums” to discuss progress, blockers, and needs.
Traditional Agile relies on functional and unit testing within sprints as well as other quality control processes. In Agile at scale, an additional testing layer is recommended, which provides end-to-end product testing to identify issues between distinctive components built by Agile teams. The increasingly closely aligned practice of “DevOps”5 and micro-segmentation architecture6 are natural partners of Agile management and enable teams to increase velocity. To be sure, “bad code doesn't scale,” but perfect code also doesn’t exist. The goal is to write good code quickly and receive feedback often. Agile at scale is something of a balancing act that avoids overly prescriptive top-down standardization and allows rapid adoption of best practices, but without creating a free-for-all “Wild West” atmosphere that becomes a troubleshooting and maintenance nightmare. Coders typically hate it when someone else writes better code than they do. As a result, when teams are cross-pollinated, their natural competitive tendencies are engaged in a healthy way, resulting in more innovation and opportunity via self-organization. Agile at scale may require more guardrails than a smaller project; however, providing latitude for innovation within reasonable boundaries is important.
When multiple teams are building a large system, it is imperative that the approach accommodate end-of-sprint demonstrations that exercise real capability and functionality (albeit incomplete) from the beginning. This favors an approach where vertical (through the tech stack) slices of functionality are sequenced instead of building the system up in layers from the bottom. Delivering in vertical slices also facilitates the ironing out of kinks related to build, promotion, provisioning, and configuration that are sometimes ignored and can lead to surprises at the end, which is the antithesis of Agile.
While some Agile teams release code into production much more frequently than traditional development, others operate over several sprints before reaching sufficient MVP for a new code release. For example, the teams at Texas Health and Human Services issued functional releases frequently, roughly every 5–6 weeks, rather than in a disruptive “big bang” rollout once every 6–12 months.
The term Agile covers a wide array of approaches, and no single cookbook approach can be applied for Agile at scale. Multiple frameworks exist for scaling Agile, but the first order of business may be setting realistic expectations. Large IT projects are more challenging than small projects. Agile can boost your success rate, but Agile isn’t a magic wand that makes all the challenges of megaprojects disappear. From vendor selection to staffing, Agile at scale—like any IT project at scale—is a management challenge as much as a technical challenge.
With large projects, governments are experimenting with “modular contracting,” which breaks the large task into smaller chunks, as is being done at California’s Child Welfare Services (see sidebar, “Agile at scale in government”). While this enables a broader pool of potential contracting partners, the bad news is that not all these partners may have the wherewithal to capably tackle the coordination aspects of a large project. Moreover, using a slew of small contractors without an overarching “general contractor,” can compound the problem of coordinating multiple work streams. From the government agency’s perspective, even if multiple sub-vendors are involved, there may be a great benefit in having “one throat to choke.”7 This doesn’t mean that the public agency can simply offload the responsibility for success to an external project management office vendor; rather, it means that the government should dedicate needed resources to oversight, and strongly consider a structure that has a central locus for oversight, a role that requires experience in partnering with government on large engagements.
A strong primary contractor can identify issues earlier and help ensure that the government agency and the vendors fulfill their part of the bargain in providing subject-matter expertise, access to needed technology infrastructure, and other elements crucial to success. Agile methods require increased involvement from affected stakeholders and, in some cases, dedicated stakeholder resources. Develop a mutual understanding of the frequency and type of involvement needed to specify requirements, review progress, answer questions, provide feedback, and encourage sign-on. At Texas Health and Human Services, users and subject-matter experts worked directly with the developers, in essence co-designing the software. At the same time, there were strong protocols in place for assignment of developers, coding standards, quality control, and the like.
Much of the success of Agile at scale depends on setting proper expectations and diligently applying project management methods to meet those expectations.