DevOps–shift everything left

February 28, 2018

A blog post by Mike Kavis, managing director, Deloitte Consulting LLP.

DevOps has also always been about continuous learning. As enterprises big and small matured their processes and became proficient at automating infrastructure and build processes, they started to address the next bottleneck: testing and security. Instead of handing off builds to testers or scheduling security audits that often resulted in rework, they started shifting testing and security left. Many mature enterprises have now implemented automated tests, security scanning, and analysis tools in their build process. Builds are forced to fail if various performance and regression tests fail or if security and coding standards do not receive an acceptable score from the code scanning processes. These advancements can lead to faster development, fewer meetings, higher productivity, and higher morale.

Organizations that have entered years three through five of their DevOps transformation often shift their focus from IT automation to IT transformation. They begin addressing the next level of bottlenecks: Process, compliance, and support. Shifting these bottlenecks left requires strong leadership because this is where significant shocks to organizational structures and culture occur.

At virtually any major DevOps event today, the conversation focuses largely on culture and leadership. Becoming a high performing organization is not a technology project. It is an exercise in leadership, organizational change management, and culture transformation. Any organization can automate infrastructure and IT processes. Very few enterprises seem to have the chops to undertake effective transformation.

Addressing transformational bottlenecks

Changing processes can be quite a challenge for most organizations. Many legacy processes were put in place over time due to a variety of reasons such as:

• Handoffs between organizational silos
• Lack of trust between silos
• A response to a painful event that happened years before
• A set of leading practices established in the days when deployments occurred only once or twice a year.

One example of the latter are software development review processes that often exist solely to compel developers or testers to meet a standard or adhere to a policy. These reviews are often held extremely late in the SDLC where there is little or no time to take corrective action.

As things shift left, legacy processes can create bottlenecks in the flow of work. Many legacy processes can be automated, performed in a different way, or even eliminated as we move to a world of automation, small changesets, and frequent deployments. Yet doing so is hard for a number of reasons, including:

• Resistance to change because people’s jobs are impacted
• Mismatched incentives between silos
• Lack of understanding of WIIFM (what’s in it for me)
• Disruption of ownership and organizational boundaries

Value stream mapping plays a vital role in identifying process bottlenecks and revealing potential business benefits of change to all stakeholders. Often, organizations can’t describe their end to end processes until all stakeholders are brought together to map out the value stream. Once everyone sees the entire value stream and where the bottlenecks are, they can then start thinking outside of their silos and looking at the system as a whole.

Shift process left

Optimizing legacy processes is key to agility. Automation by itself can only go so far. Process bottlenecks should be removed to improve the flow of work in process (WIP). Many processes are designed within organizational silos and add little or no value to the broader development process. As previously mentioned, these processes are often established as ways to enforce policies—for example, to review functionally before it gets to production. The problem with this approach is that it treats every change equally, regardless of the criticality of the change. In financial institutions, for example, this means that simple web page changes often go through the same rigor as a transaction processing system change.

To get around this, some companies are asking policymakers to define and own their policies, but leave implementation to the individual product teams so it doesn’t interrupt their flow. For example, a common change process involves filling out a series of tickets and holding various review meetings with many stakeholders. This often takes more time than the actual code change. The new thought process is to define the requirements for change management and allow the product team to streamline the process.

Moreover, a number of tasks can be automated, including enforcing architecture and security standards, updating a configuration management database (CMDB), and requesting infrastructure. But doing so requires the culture to change from a “we must check everything” mindset to one of “trust in your automation.” In a model of frequent deployments, people can only hold so many review meetings. Said another way, humans don’t scale in a continuous delivery model. Instead, enterprises should consider adopting a post-mortem review of change management to determine whether the automation is still complying with the policies.

Shift compliance left

After making progress with process improvement, many enterprises can proceed to the next biggest bottleneck, which is often compliance. Taking a page out of the “shift process left” playbook, product teams can ask compliance teams to focus on identifying applicable regulatory and audit controls and requirements. Each product team can then design solutions to meet compliance and audit requirements and yet still support efficient product and service delivery.

Too often I see policymakers forcing suboptimal technology decisions on product teams. Policymakers tend to excel at policy management, not technology decision making. Technology decisions should be more focused on customer satisfaction and technology strategy, not policy. Policies are often nothing more than requirements which should be left to technologists to implement. Policymakers can still bless the technology decisions to ensure that the solutions satisfy the requirements, but they should not dictate the how.

Shift support left

There is also a movement to shift tier one through three support left. Historically, tier 1 support services have often been provided from a separate silo that may know very little, if anything about the actual products that customers are calling about. Those tier one resources typically troubleshoot and solve basic problems by following a decision tree. They farm out anything more complex to a tier two support group that has more product knowledge, tools, and access to developers, or tier three support group, who are typically the actual product developers or operations teams.

The new mindset is to move the issues closer to the people who know the product so there is less shuffling of tickets and meetings across silo organizations. Shifting support left in this way means creating a T shaped organization where the manager owns both development and operations—no more handoffs across organizational silos. This is a swarming model in which the operations team has all the necessary knowledge and tools to solve the issues and developers are embedded within the team to assist.