Technical debt reversal Tech Trends 2014

Technical debt is a way to understand the cost of code quality and the impacts of architectural issues. For IT to help drive business innovation, managing technical debt is a necessity. Legacy systems can constrain growth because they may not scale; because they may not be extensible into new scenarios like mobile or analytics; or because underlying performance and reliability issues may put the business at risk. But it’s not just legacy systems: New systems can incur technical debt even before they launch. Organizations should purposely reverse their debt to better support innovation and growth—and revamp their IT delivery models to minimize new debt creation.

Lowering the IT debt ceiling

Technical debt is not a new term, but it’s gaining renewed interest. Originally coined by Ward Cunningham in 1992, the phrase describes the “not quite right” code typically introduced with initial software releases because of an incomplete understanding of how the system should work.¹ Organizations that regularly repay technical debt by consolidating and revising software as their understanding grows will likely be better positioned to support investments in innovation. And like financial debt, organizations that don’t “pay it back” can be left allocating the bulk of their budgets to interest (i.e., system maintenance), with little remaining to develop software that can support new opportunities.

Technical debt is often the result of programmers taking shortcuts or using unsophisticated techniques. It’s typically misfeasance, not malfeasance. For example, a developer may copy and paste code blocks without thinking through the longer-term consequences. If the code ever needs to be updated, someone will have to remember to fix it in each instance.

But sometimes, technical debt is simply the result of dealing with complex requirements. To meet a project deadline, complicated proprietary code may be developed, even though simpler alternatives may have been available. With each such action, technical debt proliferates. This is like high-interest, short-term borrowing. If you don’t pay off the debt promptly, compounding kicks in.

The impact of accumulated technical debt can be decreased efficiency, increased cost, and extended delays in the maintenance of existing systems. This can directly jeopardize operations, undermining the stability and reliability of the business over time. It also can stymie the ability to innovate and grow.

Articulating technical debt is the first step in paying off its balance. With new tools for scanning and assessing software assets, CIOs can now gauge the quality of their legacy footprint—and determine what it would cost to eliminate the inevitable debt. A recent study suggests that an average of $3.61 of technical debt exists per line of code, or an average of more than $1 million per system.² Gartner says that “current global IT debt is estimated to stand at $500 billion, with the potential to rise to $1 trillion by 2015.”³ While the idea of debt doubling in a year’s time may seem astonishing, we’re in the midst of unprecedented investments in disruptive technologies—often with deep hooks into core systems. The push for rapid innovation in unproven domains is also leading to compounding debt.

These estimates address only the literal definition of technical debt—how much it would cost to fix the exposed code quality issues. But there’s also another dimension, which we call “architectural debt.” Architectural debt refers to the opportunity costs associated with system outages or the inability to deliver new capabilities. In some cases, architecturally complex defects can absorb as much as 52 percent of the total effort spent repairing defects.⁴ They can also derail new initiatives.

Technical debt is not limited to legacy systems; every new project has the potential to add to the backlog. With that in mind, you should incorporate the cost of technical debt into project management processes and portfolio reporting. This kind of transparency can not only raise awareness of quality among development teams, but can also provide a foundation for talking to the business about the hidden cost of IT delivery. By documenting your debt-decreasing efforts, you can account for those efforts—important progress that would likely not otherwise be visible (or appreciated).⁵

The ability to quantify technical debt can provide a common point of reference for the C-suite when you are deciding how to prioritize IT projects for an organization. Typically, technical debt should be paid down within the context of delivering against business priorities by incrementally refactoring existing solutions and using improved development processes to minimize new debt accumulation. Incorporating techniques described in our Real-time DevOps chapter⁶ can help reduce waste generated when software is developed.

Some organizations may also need to spur projects that address especially messy issues such as bolstering performance, preventing production issues, or preparing for future strategic investments. The goal is a sustained, prioritized reduction of the balance sheet, where each project systematically improves on the baseline.

For most organizations, technical debt comes with the territory, an unavoidable outcome of decades of technology spend. The big question is: How will you manage the liability? Understanding, containing, and mitigating technical debt can be a platform, not only for a stronger IT foundation, but for a renewed level of trust and transparency with the business.

Lessons from the front lines

Express delivery of quality

To keep up with the over 150 billion pieces of mail delivered each year,⁷ the United States Postal Service (USPS) depends in large part on the quality and effectiveness of its IT systems. So when quality concerns became apparent during one of its IT modernization projects and the USPS was facing budget concerns, USPS leadership proactively took action to manage the organization’s technical debt.

First, USPS used the SQALE⁸ method for assessing the quality of its technical debt across four software dimensions: reliability, performance, security, and changeability. With a clearer picture of how much technical debt existed and where, USPS developed a roadmap to remediate the critical software issues and transition to long-term sustainment following CISQ⁹ standards. For example, USPS instituted automated unit test scripts, minimum code coverage testing levels, and static analysis of the source code. These changes improved application quality and performance.

Going forward, USPS is also applying these same measurable standards to other projects by including them as standard oversight and acceptance criteria in their statements of work. And because incorrect project estimates can introduce technical debt, USPS is revamping its project estimation techniques by requiring the use of both parametric and bottom-up estimating techniques. With these changes, USPS is starting to see both improved quality and more accurately planned IT costs across its portfolio.

Cleaning up shop

DB Systel, a subsidiary of Deutsche Bahn, is one of Germany’s leading information technology and communications providers, running approximately 500 high-availability business systems for its customers. In order to keep this complex environment—a mix of packaged and in-house–developed systems that range from mainframe to mobile—running efficiently while continuing to address the needs of its customers, DB Systel decided to embed processes and tools within its development and maintenance activities to actively address its technical debt.

DB Systel’s software developers have employed new tools during development so they can detect and correct errors more efficiently. Using a software analysis and measurement platform from CAST, DB Systel has been able to uncover architectural hot spots and transactions in its core systems that carry significant structural risk. DB Systel is now better able to track the nonfunctional quality characteristics of its systems and precisely measure changes in architecture- and code-level technical debt within these applications to prioritize the areas with highest impact.

By implementing this strategy at the architecture level, DB Systel has seen a reduction in time spent on error detection and an increased focus on leading-practice development techniques. The company also noticed a rise in employees’ intrinsic motivation as a result of using CAST.¹⁰ With an effective technical debt management process in place, DB Systel is mitigating the possibility of software deterioration while also enriching application quality.

Combating system complexity

Military Health System (MHS), a unit within the United States Department of Defense, provides billions of dollars’ worth of health services to over 9 million beneficiaries.¹¹ Facing enterprise-wide budget cuts, MHS began looking for ways to provide the same level of care with reduced resources. With dozens of IT systems built over 20 years ago, including clinical systems, supply chain, and billing, MHS recognized that reducing its technical debt was one way the organization could reduce its IT budget and improve business efficiency.

MHS embarked on a transformation with portfolio rationalization at the forefront in an effort to streamline its investments. Using an application health grid that removed potential subjectivity, MHS measured the business value, technical maturity, and cost of each of its systems. Business value was determined by how many business processes are supported. The technical maturity analysis focused on four areas: external stability (an evaluation of third-party software, hardware, and associated vendors); internal stability (an architectural evaluation); system availability; and security. The rationalization effort helped MHS identify over a dozen systems with high levels of technical debt that could be decommissioned—saving the organization over $50 million in ongoing operating costs within the first phase of the transformation.

MHS continues to use data-driven analytics implemented through SEMOSS.¹² The transformation that began with portfolio rationalization has now moved into optimization and dynamic portfolio planning. Reviewing a system’s technical composition in combination with functional capabilities allows MHS to protect itself against future technical debt and make informed decisions about its overall IT portfolio.

Countdown to zero technical debt

NASA’s Mars Science Laboratory project was classified as a “flagship mission”—the agency’s first in almost a decade. It was a $2.5 billion project to land a car-sized, roving science laboratory, Curiosity, on Mars. The rover launched in 2011 and landed on Mars on August 5, 2012, with the continuing objective of determining whether Mars ever contained the building blocks for life.

Building a roving science lab is an immense challenge. Curiosity is an order of magnitude larger than any rover that had previously landed on Mars: It weighs almost a ton, stands seven feet tall, contains a robotic arm that could easily pick up a person, and includes a laser that vaporizes rocks. Curiosity’s software is essentially the brain of the rover—integrating its many hardware functions to provide mission-critical functionality such as the descent and landing sequence, autonomous driving, avionics, telecommunications, and surface sample handling.

The software initially developed for Curiosity was inherited from previous rover missions. The core architecture was developed in the 1990s on a shoestring budget. The Curiosity project put approximately four years of work into building on top of that architecture for NASA’s most complex mission to date. As the launch date approached, NASA started to see that the project wasn’t coming together: The software had bugs and inexplicably failed tests; there were issues with the hardware and the fabrication of key components.

The project faced a difficult question: Do we push on towards a 2009 launch or delay the mission? The unique aspect of launching a mission to Mars is that the opportunity only exists once every 26 months, when Earth and Mars align. If they delayed the launch two years, there was a risk that the project might be cancelled altogether.

The project team decided to postpone the mission and began an incredible regrouping effort. The software team had to quickly decide whether to fix the current software or to start over completely from scratch. Given the existing software’s technical debt, it was unlikely they could determine the magnitude of the lurking issues, or confidently plan for new project milestones. The decision was made to tear down the foundation and rebuild using the old code as a reference.

The team started from the beginning: revisiting the requirements, software design, coding, and reviews, and testing and implementing standard processes. The team instituted what they called the “Power Ten,” a set of 10 basic rules each developer followed. The team developed coding standards, implemented multiple automated code analyzers and testing tools, and established a cadence of releases—one every four months. They unit tested every line of code and instituted code reviews early in the development lifecycle. Two hundred code reviews produced 10,000 peer comments and 25,000 tool comments—each one reviewed and resolved.

The results were staggering: 3.5 million lines of code, over 1 million hand-written, across 150 different modules. But this time, the numerous bugs and unexplained failures were gone. The standards, though they required additional work, added stability and quality. And with the fresh start, the team were adamant that technical debt be minimized—building a new foundation for future missions.

Though NASA’s approach required a remarkably difficult decision, the results were worth the effort. The world can now watch as Curiosity tells us more than we ever dreamed we might know about Mars. And the achievements of the mission led to the announcement of a new $1.5 billion mission to Mars in 2020.

My take

Larry Quinlan, global chief information officer, Deloitte Touche Tohmatsu Limited

Technical debt doesn’t just happen because of poor code quality or shoddy design. Often, it’s the result of a series of good or necessary decisions made over time—actions individually justified by their immediate ROI or the needs of a project. There are many examples: skipping a software update or infrastructure upgrade because there wasn’t a clear business benefit; building point-to-point interfaces into a small departmental app to get it into the business’s hands more quickly; choosing a product you already own to build a prototype in order to avoid a drawn-out vendor selection and procurement process.

The path to technical debt can be paved with good intentions, but when combined, can lead you to quality and architectural issues.

But good intentions don’t give you a pass to ignore technical debt. Leading IT organizations can, and should, actively manage and reverse technical debt. These organizations have a vision for robust platforms ready to fuel growth and use nimble business-aligned delivery models to innovate, fulfill unexpected business-driven requirements, and ultimately solve business problems.

There are two aspects that are important to technical debt management. The first is to know where you stand. Reversal starts with visibility—a baseline of lurking quality and architectural issues. Develop simple, compelling ways that describe the potential impact of the issues in order to foster understanding by those who determine IT spending. Make technical debt a metric that your IT organization is conscious of—not just in planning and portfolio management, but in how projects get delivered.

The second is with the actual management of technical debt. There are a couple of ways to approach it: a big bang approach that fixes everything at once (which almost never works) or a selective approach to systematically reduce the backlog. Consider what is needed in the next year or two to assist with achieving your strategic goals. This will allow you to identify the parts of your portfolio that should be upgraded to achieve those goals. When it comes to each of your platforms, don’t be afraid to jettison certain parts.

At Deloitte, we deliberately separate our IT budget into core and business-driven investments so business users can choose investments driven by their priorities. A server upgrade rarely trumps a functional requirement when battling for fixed investment funds. That’s why architecture, platform, and technical debt investments are part of our core investment bucket—with priorities set by the IT organization. My philosophy is: What’s the point of having a CIO if I need a committee to approve every upgrade? By keeping the core investments separate from the business-driven investments, we are able to avoid the technical debt we might otherwise accrue.

Preventing technical debt requires a philosophy that addresses the known and expected requirements with an underlying, agile platform. CIOs need the courage to make the investments that reduce technical debt—and the knowledge and the team to know where and when to make those investments.

Where do you start?

Technical debt calculation can begin when you have clear visibility to the quality of code for legacy systems as well as projects on the horizon. Only with both sets of information can you make the trade-offs necessary to manage technical debt effectively. For companies eager to get ahead of the technical debt curve, here are some important steps:

• Assess the status of code for all significant investments. Calculate your technical debt. Know the size of the hole you’re in—and whether or not it’s getting deeper. Evaluate the importance of each system to understand whether the technical debt has to be addressed—and in what timeframe. Aim for surgical repairs when possible, but recognize that some aging systems may be beyond incremental fixes. Prevention is preferred, but early detection at least allows for a thoughtful response.
• Find out how future investments are dependent on your legacy systems. Is your architecture ready for new initiatives? Can it scale appropriately? How well are back-end complications understood and fed into planning efforts? Should you launch legacy modernization efforts now to get ahead of impending business demands?
• Think through the availability of talent to support debt remediation. For some aging systems, your resources may not be sufficient for cost-effective updating. Talent should be factored into your analysis. Think of it as a multiplier on top of the raw technical debt calculation—and use it to define priorities and timelines.
• Hold developers accountable. Consider rating and rewarding developers on the quality of their code. In some cases, fewer skilled developers may be better than volumes of mediocre resources whose work may require downstream reversal of debt. Regularly run code complexity reviews and technical debt assessments, sharing the results across the team. Not only can specific examples help the team improve, but trends can signal that a project is headed in the wrong direction or encountering unexpected complexity.
• Spread the wealth (and the burden). Communities are great ways to identify and address technical debt. Peer code reviews are leading practices for informal spot checks. Formal quality assessments by seasoned architects can find issues that would be undetectable with standard QA processes. Learn from open source communities, where quality is continuously refined by the extended pool of developers poring over each other’s code.¹³
• Determine your debt repayment philosophy. Companies have different profiles when it comes to debt for the various parts of their asset pools. Debt is not inherently bad; it can fuel new investments and accelerate product launches. But left unchecked, it can be crippling. There’s no single right answer for the appropriate amount of technical debt, but its accumulation should be a conscious, transparent decision.

Bottom line

When CIOs operate like venture capitalists,¹⁴ technical debt is a big part of the financial picture. Without a clear view of the real cost of legacy systems, CIOs lack the information required to make effective decisions about new initiatives and investments. While it’s important not to get obsessed with technical debt, it’s also critical to understand and plan for it. Every new project automatically comes with technical debt as a cost of doing business. Reversing technical debt is a long-term investment, but if left unaddressed, it can bankrupt your ability to build for the future. Capers Jones, a long-term technical debt specialist, once said: “If you skimp on quality before you deliver software, you end up paying heavy interest downstream after the software is released for things you could have gotten rid of earlier, had you been more careful.”¹⁵ He was right.