Technical debt reversal has been saved
Understanding, containing, and mitigating technical debt can be a platform for a renewed level of trust and transparency with the business.
Technical debt is a way to understand the cost of code quality and the impacts of architectural issues. For IT to help drive business innovation, managing technical debt is a necessity. Legacy systems can constrain growth because they may not scale; because they may not be extensible into new scenarios like mobile or analytics; or because underlying performance and reliability issues may put the business at risk. But it’s not just legacy systems: New systems can incur technical debt even before they launch. Organizations should purposely reverse their debt to better support innovation and growth—and revamp their IT delivery models to minimize new debt creation.
Learn more about technical debt reversal.
Create and download a custom PDF of the Business Trends 2014 report.
Technical debt is not a new term, but it’s gaining renewed interest. Originally coined by Ward Cunningham in 1992, the phrase describes the “not quite right” code typically introduced with initial software releases because of an incomplete understanding of how the system should work.1 Organizations that regularly repay technical debt by consolidating and revising software as their understanding grows will likely be better positioned to support investments in innovation. And like financial debt, organizations that don’t “pay it back” can be left allocating the bulk of their budgets to interest (i.e., system maintenance), with little remaining to develop software that can support new opportunities.
Technical debt is often the result of programmers taking shortcuts or using unsophisticated techniques. It’s typically misfeasance, not malfeasance. For example, a developer may copy and paste code blocks without thinking through the longer-term consequences. If the code ever needs to be updated, someone will have to remember to fix it in each instance.
But sometimes, technical debt is simply the result of dealing with complex requirements. To meet a project deadline, complicated proprietary code may be developed, even though simpler alternatives may have been available. With each such action, technical debt proliferates. This is like high-interest, short-term borrowing. If you don’t pay off the debt promptly, compounding kicks in.
The impact of accumulated technical debt can be decreased efficiency, increased cost, and extended delays in the maintenance of existing systems. This can directly jeopardize operations, undermining the stability and reliability of the business over time. It also can stymie the ability to innovate and grow.
Articulating technical debt is the first step in paying off its balance. With new tools for scanning and assessing software assets, CIOs can now gauge the quality of their legacy footprint—and determine what it would cost to eliminate the inevitable debt. A recent study suggests that an average of $3.61 of technical debt exists per line of code, or an average of more than $1 million per system.2 Gartner says that “current global IT debt is estimated to stand at $500 billion, with the potential to rise to $1 trillion by 2015.”3 While the idea of debt doubling in a year’s time may seem astonishing, we’re in the midst of unprecedented investments in disruptive technologies—often with deep hooks into core systems. The push for rapid innovation in unproven domains is also leading to compounding debt.
These estimates address only the literal definition of technical debt—how much it would cost to fix the exposed code quality issues. But there’s also another dimension, which we call “architectural debt.” Architectural debt refers to the opportunity costs associated with system outages or the inability to deliver new capabilities. In some cases, architecturally complex defects can absorb as much as 52 percent of the total effort spent repairing defects.4 They can also derail new initiatives.
Technical debt is not limited to legacy systems; every new project has the potential to add to the backlog. With that in mind, you should incorporate the cost of technical debt into project management processes and portfolio reporting. This kind of transparency can not only raise awareness of quality among development teams, but can also provide a foundation for talking to the business about the hidden cost of IT delivery. By documenting your debt-decreasing efforts, you can account for those efforts—important progress that would likely not otherwise be visible (or appreciated).5
The ability to quantify technical debt can provide a common point of reference for the C-suite when you are deciding how to prioritize IT projects for an organization. Typically, technical debt should be paid down within the context of delivering against business priorities by incrementally refactoring existing solutions and using improved development processes to minimize new debt accumulation. Incorporating techniques described in our Real-time DevOps chapter6 can help reduce waste generated when software is developed.
Some organizations may also need to spur projects that address especially messy issues such as bolstering performance, preventing production issues, or preparing for future strategic investments. The goal is a sustained, prioritized reduction of the balance sheet, where each project systematically improves on the baseline.
For most organizations, technical debt comes with the territory, an unavoidable outcome of decades of technology spend. The big question is: How will you manage the liability? Understanding, containing, and mitigating technical debt can be a platform, not only for a stronger IT foundation, but for a renewed level of trust and transparency with the business.
To keep up with the over 150 billion pieces of mail delivered each year,7 the United States Postal Service (USPS) depends in large part on the quality and effectiveness of its IT systems. So when quality concerns became apparent during one of its IT modernization projects and the USPS was facing budget concerns, USPS leadership proactively took action to manage the organization’s technical debt.
First, USPS used the SQALE8 method for assessing the quality of its technical debt across four software dimensions: reliability, performance, security, and changeability. With a clearer picture of how much technical debt existed and where, USPS developed a roadmap to remediate the critical software issues and transition to long-term sustainment following CISQ9 standards. For example, USPS instituted automated unit test scripts, minimum code coverage testing levels, and static analysis of the source code. These changes improved application quality and performance.
Going forward, USPS is also applying these same measurable standards to other projects by including them as standard oversight and acceptance criteria in their statements of work. And because incorrect project estimates can introduce technical debt, USPS is revamping its project estimation techniques by requiring the use of both parametric and bottom-up estimating techniques. With these changes, USPS is starting to see both improved quality and more accurately planned IT costs across its portfolio.
DB Systel, a subsidiary of Deutsche Bahn, is one of Germany’s leading information technology and communications providers, running approximately 500 high-availability business systems for its customers. In order to keep this complex environment—a mix of packaged and in-house–developed systems that range from mainframe to mobile—running efficiently while continuing to address the needs of its customers, DB Systel decided to embed processes and tools within its development and maintenance activities to actively address its technical debt.
DB Systel’s software developers have employed new tools during development so they can detect and correct errors more efficiently. Using a software analysis and measurement platform from CAST, DB Systel has been able to uncover architectural hot spots and transactions in its core systems that carry significant structural risk. DB Systel is now better able to track the nonfunctional quality characteristics of its systems and precisely measure changes in architecture- and code-level technical debt within these applications to prioritize the areas with highest impact.
By implementing this strategy at the architecture level, DB Systel has seen a reduction in time spent on error detection and an increased focus on leading-practice development techniques. The company also noticed a rise in employees’ intrinsic motivation as a result of using CAST.10 With an effective technical debt management process in place, DB Systel is mitigating the possibility of software deterioration while also enriching application quality.
Military Health System (MHS), a unit within the United States Department of Defense, provides billions of dollars’ worth of health services to over 9 million beneficiaries.11 Facing enterprise-wide budget cuts, MHS began looking for ways to provide the same level of care with reduced resources. With dozens of IT systems built over 20 years ago, including clinical systems, supply chain, and billing, MHS recognized that reducing its technical debt was one way the organization could reduce its IT budget and improve business efficiency.
MHS embarked on a transformation with portfolio rationalization at the forefront in an effort to streamline its investments. Using an application health grid that removed potential subjectivity, MHS measured the business value, technical maturity, and cost of each of its systems. Business value was determined by how many business processes are supported. The technical maturity analysis focused on four areas: external stability (an evaluation of third-party software, hardware, and associated vendors); internal stability (an architectural evaluation); system availability; and security. The rationalization effort helped MHS identify over a dozen systems with high levels of technical debt that could be decommissioned—saving the organization over $50 million in ongoing operating costs within the first phase of the transformation.
MHS continues to use data-driven analytics implemented through SEMOSS.12 The transformation that began with portfolio rationalization has now moved into optimization and dynamic portfolio planning. Reviewing a system’s technical composition in combination with functional capabilities allows MHS to protect itself against future technical debt and make informed decisions about its overall IT portfolio.
NASA’s Mars Science Laboratory project was classified as a “flagship mission”—the agency’s first in almost a decade. It was a $2.5 billion project to land a car-sized, roving science laboratory, Curiosity, on Mars. The rover launched in 2011 and landed on Mars on August 5, 2012, with the continuing objective of determining whether Mars ever contained the building blocks for life.
Building a roving science lab is an immense challenge. Curiosity is an order of magnitude larger than any rover that had previously landed on Mars: It weighs almost a ton, stands seven feet tall, contains a robotic arm that could easily pick up a person, and includes a laser that vaporizes rocks. Curiosity’s software is essentially the brain of the rover—integrating its many hardware functions to provide mission-critical functionality such as the descent and landing sequence, autonomous driving, avionics, telecommunications, and surface sample handling.
The software initially developed for Curiosity was inherited from previous rover missions. The core architecture was developed in the 1990s on a shoestring budget. The Curiosity project put approximately four years of work into building on top of that architecture for NASA’s most complex mission to date. As the launch date approached, NASA started to see that the project wasn’t coming together: The software had bugs and inexplicably failed tests; there were issues with the hardware and the fabrication of key components.
The project faced a difficult question: Do we push on towards a 2009 launch or delay the mission? The unique aspect of launching a mission to Mars is that the opportunity only exists once every 26 months, when Earth and Mars align. If they delayed the launch two years, there was a risk that the project might be cancelled altogether.
The project team decided to postpone the mission and began an incredible regrouping effort. The software team had to quickly decide whether to fix the current software or to start over completely from scratch. Given the existing software’s technical debt, it was unlikely they could determine the magnitude of the lurking issues, or confidently plan for new project milestones. The decision was made to tear down the foundation and rebuild using the old code as a reference.
The team started from the beginning: revisiting the requirements, software design, coding, and reviews, and testing and implementing standard processes. The team instituted what they called the “Power Ten,” a set of 10 basic rules each developer followed. The team developed coding standards, implemented multiple automated code analyzers and testing tools, and established a cadence of releases—one every four months. They unit tested every line of code and instituted code reviews early in the development lifecycle. Two hundred code reviews produced 10,000 peer comments and 25,000 tool comments—each one reviewed and resolved.
The results were staggering: 3.5 million lines of code, over 1 million hand-written, across 150 different modules. But this time, the numerous bugs and unexplained failures were gone. The standards, though they required additional work, added stability and quality. And with the fresh start, the team were adamant that technical debt be minimized—building a new foundation for future missions.
Though NASA’s approach required a remarkably difficult decision, the results were worth the effort. The world can now watch as Curiosity tells us more than we ever dreamed we might know about Mars. And the achievements of the mission led to the announcement of a new $1.5 billion mission to Mars in 2020.
Technical debt doesn’t just happen because of poor code quality or shoddy design. Often, it’s the result of a series of good or necessary decisions made over time—actions individually justified by their immediate ROI or the needs of a project. There are many examples: skipping a software update or infrastructure upgrade because there wasn’t a clear business benefit; building point-to-point interfaces into a small departmental app to get it into the business’s hands more quickly; choosing a product you already own to build a prototype in order to avoid a drawn-out vendor selection and procurement process.
The path to technical debt can be paved with good intentions, but when combined, can lead you to quality and architectural issues.
But good intentions don’t give you a pass to ignore technical debt. Leading IT organizations can, and should, actively manage and reverse technical debt. These organizations have a vision for robust platforms ready to fuel growth and use nimble business-aligned delivery models to innovate, fulfill unexpected business-driven requirements, and ultimately solve business problems.
There are two aspects that are important to technical debt management. The first is to know where you stand. Reversal starts with visibility—a baseline of lurking quality and architectural issues. Develop simple, compelling ways that describe the potential impact of the issues in order to foster understanding by those who determine IT spending. Make technical debt a metric that your IT organization is conscious of—not just in planning and portfolio management, but in how projects get delivered.
The second is with the actual management of technical debt. There are a couple of ways to approach it: a big bang approach that fixes everything at once (which almost never works) or a selective approach to systematically reduce the backlog. Consider what is needed in the next year or two to assist with achieving your strategic goals. This will allow you to identify the parts of your portfolio that should be upgraded to achieve those goals. When it comes to each of your platforms, don’t be afraid to jettison certain parts.
At Deloitte, we deliberately separate our IT budget into core and business-driven investments so business users can choose investments driven by their priorities. A server upgrade rarely trumps a functional requirement when battling for fixed investment funds. That’s why architecture, platform, and technical debt investments are part of our core investment bucket—with priorities set by the IT organization. My philosophy is: What’s the point of having a CIO if I need a committee to approve every upgrade? By keeping the core investments separate from the business-driven investments, we are able to avoid the technical debt we might otherwise accrue.
Preventing technical debt requires a philosophy that addresses the known and expected requirements with an underlying, agile platform. CIOs need the courage to make the investments that reduce technical debt—and the knowledge and the team to know where and when to make those investments.
Technical debt calculation can begin when you have clear visibility to the quality of code for legacy systems as well as projects on the horizon. Only with both sets of information can you make the trade-offs necessary to manage technical debt effectively. For companies eager to get ahead of the technical debt curve, here are some important steps:
When CIOs operate like venture capitalists,14 technical debt is a big part of the financial picture. Without a clear view of the real cost of legacy systems, CIOs lack the information required to make effective decisions about new initiatives and investments. While it’s important not to get obsessed with technical debt, it’s also critical to understand and plan for it. Every new project automatically comes with technical debt as a cost of doing business. Reversing technical debt is a long-term investment, but if left unaddressed, it can bankrupt your ability to build for the future. Capers Jones, a long-term technical debt specialist, once said: “If you skimp on quality before you deliver software, you end up paying heavy interest downstream after the software is released for things you could have gotten rid of earlier, had you been more careful.”15 He was right.