Agile by the numbers A data analysis of Agile development in the US federal government

There is little question that the Agile approach has been making inroads in government. Originally developed as an approach for building software in the private sector, government has also embraced Agile, at least in concept.

In 2012, the US Office of Management and Budget (OMB) issued guidance that “encourages agencies to shift away from the bloated, multi-year projects so common in the past” and to adopt a more modular approach.¹ The US federal government's Digital Services Playbook” urges officials to build the service using Agile and iterative practices.”² Indeed, Agile Software Development Life Cycle (SDLC) textbooks have been seen on employees’ desks at the highest levels of the US government.³

But how deep-seated and fundamental is this shift? Have Agile methods taken root within the federal IT community? How much IT procurement is Agile-based, and how much is still based on the traditional waterfall approach?

These are the questions that the Deloitte Center for Government Insights set out to investigate by analyzing hard numbers. We used a variety of federal data sources to dig deep into what is taking place with respect to Agile and federal IT. (See sidebar, “Data sources: The hard numbers that describe Agile’s reach.”)

What we found is that the Agile movement coincides with a long-standing trend toward less expensive and more nimble IT projects, but most likely did not itself cause those trends. For example, between 2004 and 2015, the duration of major federal IT projects went from an average of nine years to less than two years. But most of this shift took place prior to the embrace of Agile approaches. In fact, in 2011, fewer than 10 percent of major federal IT projects described themselves as “Agile” or “iterative.” This number, however, has grown rapidly in the past few years: In 2017, fully 80 percent of major federal IT projects are now describing themselves as “Agile” or “iterative.”⁴

One explanation for the increase in self-reported adopting of Agile and/or incremental approaches could be a desire to appear compliant with leadership rather than an actual underlying shift in IT procurement approach. While an openness to Agile, incremental IT procurement seems real, there continues to be a strong role for traditional waterfall practices as well. We conducted an analysis of more than 3,000 procurement documents, and found evidence that both waterfall and Agile approaches are currently reflected in federal IT projects.

In the following pages, we’ll review the evidence in more detail and offer some insight about what the future may hold for the Agile movement and federal IT acquisitions.

Major federal software projects have been getting shorter in duration and smaller in cost for more than a decade

The Agile movement is all about flexibility through iterative approaches—which often entails breaking large projects into smaller and shorter chunks. But we find that the trend toward smaller, cheaper, and more iterative projects in federal IT development actually started years ago, as far back as 2004, before “Agile” was widely recognized as a distinct software development methodology in government. According to our analysis of data from the OMB, major software projects⁵ have been getting both shorter and smaller in value every year (figures 1 and 2). In fact, the average cost for line-item projects in constant dollars⁶ fell from $143.5 million in 2004 to only $1.72 million in 2015. Average project duration fell over the same period, from 108 months to 7.9 months.⁷ Shorter IT development projects are widely perceived as less risky,⁸ matching one of Agile’s core tenets, “fail fast.”

Improvements in IT procurement timelines and prices are not tied to Agile methods

To what extent are Agile methodologies responsible for the trends we see towards less expensive and shorter major projects? Self-reported use of Agile or iterative methods is definitely on the rise among major federal IT projects (figure 3). By 2017, some 80 percent of these major systems were reporting using Agile or iterative SDLC methods.

Since they self-report their choice of SDLC method, the owners of these projects may be responding to encouragement from OMB to use Agile methods. Or the reporting may reflect guidance and high-profile success stories from projects by the United States Digital Service (USDS) or GSA’s 18F division.

But it’s not possible to determine with confidence the extent to which Agile methods caused the reductions we’re seeing in IT development times and costs for major IT projects.⁹

Are contracting officers asking for Agile in their solicitations?

Are contracting officers (COs) requesting Agile methods in IT solicitations? Do the procurement solicitations that are now routinely self-described as “Agile” or “incremental” truly reflect a fundamentally different approach?

To find out, we analyzed statements of work from FedBizOpps, the primary source for federal contracts over $25,000. Our analysis shows that the majority of solicitations do not currently specify a particular SDLC, although some solicitations do specifically request Agile methods. We see no evidence of an increasing preference for Agile methods in solicitations at the expense of waterfall.

Using text analysis, a form of artificial intelligence, we studied more than 3,000 solicitations for software procurements between October 2015 and October 2016.¹⁰ Figure 4 shows the similarity between the language in those solicitations and a separate set of reference documents about the Agile method. For comparison, we also measured the similarity of the solicitations’ language to the language in reference documents about waterfall methods. The amount of “Agile language” included in solicitations is stable over time. Importantly, our analysis found no apparent decline in terminology related to waterfall methods.

Simply put, the majority of current solicitations appeared broad enough to cover both Agile and waterfall methods. Few and far between were “pure” Agile solicitations, though they did fit within the Federal Acquisition Regulations (FAR) and our text analysis method could detect them. Our finding is consistent with statements from proponents of Agile methods who point out that the flexibility of the FAR as they stand today allow for the use of Agile approaches.

The Agile movement came to government when IT procurement was already changing

It seems clear that Agile techniques have taken root in parts of the federal IT ecosystem, but they are by no means ubiquitous or exclusive. While we have seen a spike in self-reporting projects as “Agile” or “incremental,” there is no evidence that COs are specifically requesting Agile methods in solicitations, and waterfall continues to play a significant role in IT procurements. How can we make sense of these seemingly contradictory indicators?

The most likely explanation is that high-profile delays, cost overruns, and cancellations¹¹ of major federal IT systems in the 2000s, coupled with the 2008 economic downturn, led to a general avoidance of lengthy projects with huge price tags. Somewhat later, the Agile methodology became popular inside the core of the federal government—the White House and agencies closely tied to its policy agenda, such as OMB, US Digital Service, and 18F. Both policy leaders and federal contracting officers appear to have become more open to new approaches to software contracting. These separate trends coincided, and with encouragement from the White House, more and more owners of major government systems began labeling their methods as “Agile,” even though they had been moving toward shorter, smaller projects since at least 2004. In addition, there has been a proliferation in the “flavors” of Agile, meaning that aspects of Agile methods can be (and are being) incorporated into traditional procurement vehicles.

Regardless of what label is applied, the fundamental trends toward smaller, shorter IT projects is undeniable. As the federal contracting workforce continues to refine more Agile approaches to IT procurement, we are likely to see, at the very least, the fundamental principles of Agile continue to play an important role in federal IT.