DevOps with built-in continuous quality on cloud

A blog post by Srinidhi Magge, DC specialist leader, Deloitte Consulting LLP; Josh Moore, DC manager, Deloitte Consulting LLP; and Nitin Kohli, DC specialist master, Deloitte Consulting LLP

Continuous integration and continuous deployment (CI/CD) have changed the way teams develop, test, and deploy software. However, as competition in the market grows, it is no longer just about application functionality and deployment. Poor quality of software has shown to have a negative impact on an organization. 

• Potential loss of customers: Today’s web and mobile application users expect great functionality and user experience. If that experience feels slow, defective, or unintuitive, they will look for other options. Testing and QA can help teams identify issues earlier to reduce churn.
• Damage to the brand: It takes years to build a strong brand, but a brand’s reputation can be damaged if the customer’s experience doesn’t meet their expectations. Great DevOps with CI/CD can help ensure a solid infrastructure.
• Financial loss: From direct-to-consumer products to financial services, if an application can’t deliver the core functionality as an organization’s customers expect, it could face major financial loss.

Source: Deloitte research

The right DevOps strategy that combines agile and continuous testing and continuous quality (CT/CQ) can help minimize downtime and maximize delivered value by deploying tested solutions early and often.

Continuous testing and continuous quality

"In previous years, we found that test automation had a significant impact on continuous delivery. This year, we built upon prior years' research and found that continuous testing positively impacts continuous delivery."

Dr. Nicole Forsgren, Gene Kim, and Jez Humble at DORA, 2018 Accelerate: State of DevOps Report

The 2018 Accelerate: State of DevOps Report states that in previous years, test automation had a significant impact on continuous delivery. Further research pinpointed that continuous testing seems to be the next factor in creating positive impact on continuous delivery.

CT/CQ combines best-in-class DevOps concepts including test automation and test-driven development to enable deployment on-demand. Some instances include:

• Defining standards for testing, environment provisioning, service virtualization and test data management; identifying testing tool stack; and establishing matrix for ownership of test cases
• Increasing automation testing to at least 75% of teams, enabling build-triggered testing, implementing test data management, and increasing adoption of security testing
• Implementing fully automated and build-triggered testing with 90% adoption by teams, and automating test data provisioning

CT/CQ’s goal is to improve code quality accompanied by long-term usefulness and maintainability of code.

Roadblocks to continuous testing and continuous quality

While the benefits and outcomes of CT/CQ are enormous, it doesn’t come without effort. This can present a challenge as teams adopt these best practices for modern DevOps. Agile development results in incremental builds rather than large deployments, allowing scrum teams to rapidly prioritize user stories and determine how the most value can be delivered in the shortest period. This may require teams to constantly create new scripts or update test scripts based on the changes in a feature of functionality.  

Regular script updates due to product backlog or road map change: An agile development is an iterative process in which the product owner owns the direction of the product development. They can change or update user stories during product backlog review meetings, requiring the test teams to constantly update their test scripts.

Test data setup: Running the same load scenario with each subsequent build may require a large amount of test data to be set up early, which can be challenging for the test team. However, this can be easily tackled with efficient test data management.

Missing backend services: When the performance testing starts early, teams risk that an application feature is dependent on a service that is still not built or available. In this case, the test teams should either stub the service or use a virtual service to test the functionality.

Every team’s goal should be to deliver high-quality code that’s useful, maintainable, and scalable. Implementing best practices with adaptable teams that execute continuous testing and continuous quality can make your teams even more effective and your applications resilient to a changing world. 

Unlock the benefits of CT/CQ using multiple-discipline solution levers

To make a real agile transformation successful, organizations must commit to adapting their approach across multiple disciplines through the pre-sprint, sprint, and release portions of the agile workflow. Forming the “learn and iterate” culture and establishing metrics that matter can be the difference between a successful DevOps team and one that can’t keep up with changing demands. To further unlock the benefits of continuous testing and continuous quality, organizations can use these solution levers:

• Transform to a DevOps enablement team collaboration model: Include test management forecast, embed multidisciplinary engineers to support testing, and collaborate between application and operations teams
• Create a dashboard for environment health: Provide visibility into environment information, dependencies, and planning
• Enhance service catalog and enable self-service capabilities: Review current standard test environments for application teams, and automate self-service capabilities for environment provisioning
• Develop metrics that matter: Identify baseline metrics and monitor to identify areas to pivot
• Increase test automation to align with industry standards: Integrate test automation in CI/CD and adopt test-driven development
• Modernize standards for test data management: Resolve any challenges to adoption of test data management, and create a solution for smaller test data environments through synthetic data creation
• Upskill testing talent: Create DevOps and cloud capability training, and share on-the-job coaching to upskill technical and business acumen

What follows is a high-level view of what continuous testing and continuous quality could look like in a typical agile flow.