Real-time DevOps Tech Trends 2014

IT organizations need to better respond to business needs with speed and agility. IT can likely improve the quality of its products and services by standardizing and automating environment, build, release, and configuration management—using tools like deployment managers, virtualization, continuous integration servers, and automated build verification testing. Popular in the agile world, DevOps capabilities are growing in many IT organizations with either waterfall or agile methodologies.

Empowering the business of IT

When it comes to application development (Dev), the business cares about speed and quality. How fast can I get what I want? How close will it be to what I need? Contrast that with IT operations (Ops), held accountable for response times, stability, and efficiency, and focused on how to reduce business disruptions at the lowest cost. These are very different core missions that yield very different behaviors. Dev is looking to compress delivery cycles and adopt “experiment and learn” mentalities. Ops is looking to institute controls and more tightly govern change. The fact that the “build” and “run” shops are typically separate organizations only adds to the divide.

Further complicating matters, both Dev and Ops could each benefit from investments in enabling technology—creating automated capabilities in the business of IT akin to what finance, manufacturing, and the supply chain have invested in over the past decades.¹ But even as requirements management, system maintenance, and other disciplines are upgraded, incremental investments in disconnected activities will go only so far.

The real goal is to bridge the gap between development and operations, supercharging the investments that currently exist in siloed automation by integrating the end-to-end delivery model. Simply stated: real-time DevOps.

Some companies have been using automation to accelerate and improve steps in their development processes for a while. But many more have not. In fact, a recent survey of 1,300 senior IT decision makers revealed that only 39 percent had already invested in pieces of DevOps.² Complacent about their time-tested, over-the-wall approaches to software development, many IT organizations have settled for manual, disjointed processes that have rightly earned a reputation for ineffectiveness and inefficiency.

It’s complicated

Complexity surrounds DevOps. From basic design principles and defect tracking to release management, configuration management, and more, the interdependencies of the software development lifecycle are real and complicated. All the more reason to automate and integrate the process.

The rise of agile is one of the factors driving increased interest in real-time DevOps. What was once seen as an experiment is now mainstream, with more than 70 percent of companies reporting at least partial agile adoption.³ At its core, agile is about “test, experiment, learn, and repeat.” It’s based on short sprints—typically one or two weeks—to get to a potentially releasable product. That’s instead of development cycles that last many months or years.

Accelerated time to market is one possible benefit, but just as important are the side effects of better managing changing priorities and improved alignment with the business. Agile backlogs represent potential work to be completed—stemming from new ideas, unmet requirements, or enhancements and fixes coming back from operations. Each sprint sees the business sponsor (or product owner, in agile parlance) re-establish priorities. Development teams then tackle those items on the top of the list.

Contrary to popular misconceptions, agile development often requires a more disciplined approach than traditional software development methods. It can also provide more transparency in the development process—and real-time visibility into progress. Iterative, rapid development begs for structure and rigor, which are available through the automated, integrated capabilities of real-time DevOps.

Continuous build and continuous integration are big parts of real-time DevOps. Code is constantly being fed back into configuration management and validated via testing automation suites, giving developers affirmation that quality is continuously monitored, with dependencies managed automatically in the background. Companies gain the ability to automate technical compliance and structural correctness, as well as a continuous measure of how requirements are being met.

Environment provisioning and management is another opportunity area. Traditionally, projects had to accept up-front capital spend and ramp-up delays while hardware and software were ordered, installed, and configured. With the rise of virtualization, cloud, and software-defined data centers, this process can be largely automated: not only procurement, but also configuration of servers, networks, storage, operating systems, platforms, applications, transactions, and data—including the automated provisioning of user accounts based on profile types.

Real-time DevOps also includes test automation. When coupled with requirements management, functional, user, and even behavioral scenarios can be scripted for automated quality assurance (QA). This benefits development cycles and also serves as the benchmark for regression scripts—accelerating break/fix and maintenance processes. For areas such as web and mobile development, this might also include build-verification services across devices and operating systems—analyzing across an increasingly fragmented landscape.

Where there’s a waterfall, there’s a way

While agile is helping to showcase the need for real-time DevOps, these concepts are just as relevant for waterfall shops. Handoffs and manual steps typically waste time in the development process. Collapsing the time it takes to develop software, and creating more thoughtful linkages to operations, are universal benefits.

Also, real-time DevOps does not mean that existing Information Technology Infrastructure Library (ITIL) and governance processes need to be scuttled. Indeed, IT service management should have a more explicit link to software development across the lifecycle, with identified issues fed into the backlog for prioritization. Ongoing patching and infrastructure upkeep should still be done, and with real-time DevOps, it will likely be better coordinated with development. Improving outage recovery and minimizing release fail rates are expected outcomes from real-time DevOps—as well as expedited code deployments and improved developer throughput.

Real-time DevOps is not a tool, though tools make it workable. And it’s not only about agile, though agile practices have brought the benefits to light. Instead, real-time DevOps is a process shift that changes the cadence of how much can be done—and in how much time.

Lessons from the front lines

Moving at the speed of commerce

John Lewis PLC, a UK-based retailer with 40 department stores, replaced its customer-facing e-commerce platform in early 2013. This was a complex project involving a team of over 100 employees and consultants working across multiple systems: web storefront, web management, product management, and delivery management. To support this project, multiple development environments were used. Each one was carefully managed to support its respective development stream. Code was then deployed across many additional environments: system testing, integration, performance, training, and ultimately production.

To meet the project’s pace and flexibility demands, John Lewis focused on DevOps and took measures to increase the frequency and richness of communications between the development and project operations teams, resulting in a prioritized list of DevOps-related enhancements. Many of these took the form of process automation in order to improve reliability, repeatability, and speed.

Since the go-live in early 2013, the company has continued to develop and refine its DevOps processes, focusing on efficiency and reliability. Automation has continued to be a main theme: Automated browser-based functional tests have been adapted so that they can be used on larger, fully integrated environments as both smoke and regression test suites. With real-time DevOps practices in place, John Lewis can now deliver one complete (back- and front-end) release per month. Previously, releases were only carried out every six to eight weeks.

With the positive results from using real-time DevOps practices on the e-commerce project, John Lewis is now expanding these practices to other projects, scaling the operation and creating a clear delivery and project operations team for the enterprise. Additionally, the company is looking to orchestrate its automated processes, enabling an end-to-end, “one click” deployment across multiple systems.

Supporting IT’s health and well-being

The state of West Virginia’s Department of Health and Human Resources (DHHR) administers programs that benefit its citizens, such as cash assistance, food stamps, and Medicaid. In support of this mission, the department depends on an integrated solution made up of more than 30 subsystems providing case management, eligibility determination, and benefits issuance functionality.

Given the breadth of business functionality, the size of the solution, and ongoing changes resulting from federal mandates, DHHR wanted an application development and maintenance process that could efficiently support multiple, parallel initiatives. To meet this objective, the organization implemented a schedule of weekly patches and monthly enhancement releases supported by an automated build and deployment process.

DHHR’s DevOps program included multiple integrated, automated components such as defect and change request tracking, build and deployment, smoke testing, and regression testing. Additionally, the organization created an administration dashboard to schedule, manage, and track builds throughout the release cycle. By integrating its defect and change request tracking with automated build and test utilities, DHHR is able to build only those components tied to tested defect fixes or enhancements—preventing untested components from being migrated to higher environments.

As a result of implementing real-time DevOps, DHHR has increased the success rate of software builds by 58 percent. Additionally, the processes have improved the software quality, which means DHHR’s production rollback plan is gathering dust on the shelf.

A healthy dose of collaboration

For a leading health plan, history seemed to repeat itself whenever the company adopted new technologies: The lack of communication across its enterprise resulted in significant delays in delivering new business solutions. Collaboration between operations and testing didn’t occur until the late stages of implementation, and development often proceeded without a holistic view of business requirements. To make matters more challenging, the core implementation teams—operations, requirements, delivery, and testing—were geographically dispersed across multiple locations, both onshore and offshore. As a result, the health plan sought a real-time DevOps and agile/Lean IT approach to streamline communication across the enterprise and accelerate delivery of new business solutions to its users.

The transition to real-time DevOps and agile/Lean IT began by integrating the business functions into a single enterprise environment, creating cohesion from the initial stages of requirements gathering through final deployment. The organization introduced automated processes to build, test, and deploy systems, allowing users to see material progress in four- to six-week release cycles. The process resulted in an efficiency gain of over 50 percent by reducing unproductive wait times throughout the project life cycle, such as by enabling a reduction in the test execution window. There was an increase in overall delivery effectiveness, providing a higher degree of business consistency, flexibility, quality, and satisfaction. When talking about deployment, teams now look at their watches, instead of the calendar.

A new policy for IT

A leading insurance company with a highly distributed IT organization comprising both internal and external resources started a journey to centralize its infrastructure, creating a services company construct. As part of that effort, the company internalized resources, created a more direct reporting structure to the CIO, and implemented a centralized application development group (ADG) of approximately 5,000 people.

One of the goals of implementing the ADG was to jump-start the company’s transition to an agile development organization. As development projects transition into the ADG, they are implemented using an agile methodology. Organization-wide, about 25 percent of development at the company is currently delivered by the ADG using agile. The transition to the ADG has enabled a renewed focus on DevOps and on the goals of standardization, automation, and integration.

Each of the company’s key infrastructure platforms was at varying stages of automation and integration. Real-time DevOps first took hold in two separate virtualized environments that already shared many common elements. As applications were built into those environments, their configurations were standardized. This allowed the company to automate more of its DevOps processes—environment provisioning, release management, system monitoring, and others. Efforts are underway to expand the standardization and automation of the company’s platforms and processes—and to integrate the various capabilities.

One of the ADG’s goals is to continually improve its delivery of what the business wants, which can drive heavy application complexity. With the IT organization moving from a catalog of distinct and separate parts to a service-based model, its approach to building applications has changed accordingly. The ADG is standardizing infrastructure components and applications and is working to automate even more of the work that comes in the door. The company is also looking into which service levels are needed by classifying applications into gold, silver, and bronze categories.

For the company’s leadership, engaging its in-house workforce is a priority. DevOps had a strong, positive impact on relationships within the IT department—in particular, the infrastructure team and the ADG. The intangible benefits of improved collaboration and teaming are valued as much as increased quality of work, cost reduction, or reduced development times. DevOps is also a building block for better partnering between the business and IT.

In this effort, the company realized that the IT organization could not get to a service-based relationship with the business if it didn’t have a service-based model in place. Leadership knew that it was important to map what it did in IT to what the business actually does for its customers. By getting that foundation in place, IT and the business could start having conversations at a higher level.

My take

Ben Gerst, senior vice president of platform and product development, FOX Sports

I’m an unapologetic DevOps geek. As vice president of technology in the PTA at my son’s school last year, I built its website and set up real-time monitoring and automated management processes on a virtual machine the site lived on. It was total overkill for such a small website, but I had fun doing it. With the ease of use and price point of tools, even a kindergarten class can take advantage of real-time DevOps.

At FOX Sports, we are actively building out DevOps and continuous integration. We use Jenkins for the automated build process, which is essentially the beginning of continuous integration for us. We’re also working on quality assurance automation so we can arrive at the point where we are creating an environment that has been tested and deployed—automatically. The next step will be broader testing automation starting with unit and regression testing. We’ll draw a line somewhere; you risk automation overkill if you go too far. But we’re not there yet.

In my role, DevOps is about having the information needed to make informed decisions. A large part of that is the combination of real-time monitoring approaches and mechanisms for teams to communicate. Monitoring can tell you a lot. If you see a spike in website traffic, you can easily pinpoint it and figure out what’s going on. It might be a good thing like a surge in user activity or it could be because of a defect from a recent build or infrastructure maintenance. Early indicators let me take action—hopefully before the business is affected.

Enabling DevOps was driven by both business and technical needs for us. We needed to make process improvements in order to meet our goals for creating leading, timely content. We’re driven by the sports calendar—it makes for lots of hard deadlines and constant activity. This year, the Super Bowl and the Winter Olympics opening ceremony fall within the same week—and we’ll be up for the challenge.

Our strategy for DevOps is ever-changing here at FOX Sports as technologies grow and we learn. Bettering the number of features delivered, turnaround time to get a new environment provisioned, and mean time to repair defects are examples of expected outcomes. Currently we can support a build every 30 minutes, and we’re taking lessons from our current processes to inform our next steps.

My advice for others is to employ DevOps. Look at the data. Get started with continuous integration. Look at your current deployment process—and consider the hours your team is spending to deploy and test code. DevOps requires an investment up front, but pays dividends in the long run. Over time, you can simply make tweaks to the process and keep rolling out code and testing in an automated fashion.

Not only do I recommend it, but I think it has become the rule, not the exception. A little work up front can save significant effort across the board.

Where do you start?

While there are many opportunities to make a shift to real-time DevOps, there are some places where you likely can’t live without it: mobile, social, and big data. In these fast-growing spaces, disjointed, bottleneck-ridden development processes can undermine your efforts. Unless you can find a way to accelerate without sacrificing quality—and real-time DevOps is likely that way—you’ll find yourself out of the loop as the business bypasses IT by going directly to the marketplace.

• Establish the need. Conduct your own benchmarking to identify delays and waste in the software development process. Uncover how much time is spent on manual document capture, build management, build verification, release planning, and test script development. These are opportunities for action.
• Build new skills. Tool configuration and scripting skills are a part of the equation, used to drive version control, configuration management, test harnesses, ticketing, environment provisioning, and system maintenance and monitoring. But soft skills are just as essential for real-time DevOps to take hold. Team members will be collaborating with the business, program and project managers, developers, testers, and the operations teams. Make sure your core team isn’t simply making the new technologies adhere to how things have historically been done.
• Employ services thinking. For real-time DevOps to be viable with legacy ERP and large-scale custom solutions, break down complex systems into components and modular services. This allows for rapid incremental changes within monolithic code bases—and sets up the organization for a broader modernization play.
• Lay down the bases. Once you understand the pain points within your organization, begin automating individual components. Establish a continuous integration build server for your developers, create a small “smoke test” suite of cases to validate builds, and implement a release automation tool. Also, look to add automation within your development, test, and infrastructure tracks in parallel with similar, discrete steps.
• Connect the dots. Once you have some of the automation components in place, look for ways to link them into a single stream that can shorten cycles. Not just integration between requirements, but continuous integration—linking to build, to defect tracking, to configuration management, and to release management. In this model, the handoffs and touch points happen seamlessly in the background.
• Get vendors on board. The opportunity to learn from and build on vendor successes in real-time DevOps is an important way to accelerate your own improvements. You may want to avoid outsourcing agreements with vendors who aren’t using automation as an accelerator.
• Make the leap to test-driven design—or even behavior-driven design. Real-time DevOps enables you to move from build-to-run to build-to-verify. This natural evolution leads to design for end-user engagement. That’s where contractors and vendors that provide application development and maintenance (ADM) and/or application management services (AMS) can help you improve.
• Look beyond cost and speed. Embrace the lower costs and greater speed that come with real-time DevOps, but recognize that more substantial benefits are also possible. If you believe that your technology delivery model can benefit from real-time DevOps, it’s time to get teams delivering against your priorities with a much more compressed cadence.
• Commit. Too many companies dabble in this world—acquiring tools and adopting some of the terminology, but without making hard changes to their operating and delivery models. If there is a case for real-time DevOps, don’t fall for one-off, surface-level investments. Or if you do, don’t be surprised if you get unremarkable results.

Bottom line

Individual tools for automating the software development lifecycle, maintenance, and monitoring have been available for years, and many companies have been taking advantage of them effectively. Yet few have taken the next step to integrate the pieces and commit to a new cadence of development and operations. That’s because the concept of real-time DevOps is only partially understood. In a recent survey, Gartner “found that only one-third of companies surveyed were either in-process or planning to implement DevOps, and close to 44 percent of respondents were still trying to figure out what DevOps means.”⁴

Early adopters have the opportunity to profoundly improve their IT shops—accelerating IT delivery, improving quality, and better aligning with their businesses. Arm IT with the tools to automate and integrate their core disciplines, and the cobbler’s children will finally have new shoes.