Industrializing Machine Learning

Blog

Industrializing Machine Learning

A view on how technology can help you to industrialize your Machine Learning

Machine Learning (ML) can bring tremendous value to organizations, and adoption of ML is growing across multiple industries. Scaling a ML capability comes with complex challenges, such as (re)deploying models at scale and retraining to keep them relevant. Our first article in this series, “If data is the new oil, where is your industrialized refinery?” , advocates a holistic approach to data and machine learning: Machine Learning Operations (MLOps). Using the analogy of an oil refinery, the article illustrates how operational machine learning solutions at scale require seamless combination of technological and organizational components. This article explores the technology behind such a “data refinery”, and gives us a glance at what a continuously operating data refinery could look like.

By Jarvin Mutatiina, Jeroen Monteban & Willem van Hove

Best practices towards an industrialized ML capability

While a fully industrialized data refinery may sound too good to be true, our experience has taught us that significant improvements are feasible. Below, we present five best practices that help organizations to get started and effectively industrialize and scale their ML capability.

Build ML pipelines rather than models
Developing models in a one-off fashion should be a practice of the past by now. Building ML pipelines allows data to flow through the full ML lifecycle: data is extracted, models are (re)trained, evaluated and served to production – all in an automated fashion. This minimizes time spent on operations, allowing your data teams to focus on building and optimizing models, dashboards, and other data products. ML pipelines are easy to set up and available in all popular development environments.

Start monitoring your machine learning services
To ensure model performance remains at the desired level, models need to be retrained once performance degrades. Monitoring enables the tracking of model performance, maximizing the quality and minimizing the risk exposure of services. To create an ‘observable system’, your organization needs to collect the right logs, metrics and traces. While this takes considerable effort, it pays back in increased quality of operational solutions, reliability of your services, and early detection and automated resolution of issues.

Automate ML pipeline deployment
An ML pipeline does not deliver value until it serves models in production. In many organizations, bringing ML pipelines from development to production requires manual effort and lengthy deployment procedures. To standardize and automate this delivery process, MLOps borrows from the best practices in software development, such as Continuous Integration/Continuous Deployment (CI/CD) technology to improve deployment speed from months to minutes. Using CI/CD teams can build, test and deploy ML pipelines at scale at the push of a button, while ensuring quality through automated testing.

Set up the proper tooling and environments
Industrializing machine learning requires the right tools, set up in appropriate environments. Over the past years, a myriad of tools has become available, offering a wide range of capabilities: versioning and lineage, experiment tracking, hyperparameter tuning, model deployment, and monitoring. Certain tools may cover multiple of these capabilities or offer additional features.
An example tool that is key for scaling machine learning is a model registry. It stores and versions models and keeps track of a model’s lineage to ensure your organization stays in control of all models. This allows your organization to smoothly run multiple models side-by-side. Another fundamental tool is the feature store, which maintains all feature data at a central location. This eliminates duplicate work and speeds up the development process of new models bringing them to production faster. Many more useful tools are available: incrementally setting up the right tools catered to your organization’s needs will significantly improve your data refinery.

Start small and scale incrementally
Adopting MLOps requires significant changes at both a technological and organizational level, and experience teaches that these changes do not happen overnight. Starting small and scaling incrementally facilitates gradual adoption of the new paradigm, reducing risk to ongoing operations. Launch a pioneering team that can experiment, and support them to find the right tools for the job. Lessons learned can be applied straight away, and success stories are used to spread the word. From these smaller experiments, teams can incrementally scale their machine learning efforts throughout the entire organization.

A holistic approach to MLOps includes technology and organization

In this article, we discussed the technological side of MLOps. Of course, technology is not a silver bullet. To take full advantage of machine learning, the associated organization needs to operate like a refinery as well. Setting up the proper organization that can build, scale and use a data refinery is one of the key challenges of adopting true MLOps. At Deloitte, we combine our hands-on experience with ML technology with a proven track record of successful organizational transformations to create a holistic approach to MLOps. This equips us to assist our clients in setting up an industrialized ML capability that is ingrained in the whole organization.

The continuous data refinery

An oil refinery functions optimally when it operates continuously, and for machine learning this is no different. Using the five best practices shared in this blog, organizations can work towards a fully industrialized ML capability as illustrated in figure 1.

Figure 1: The continuous data refinery

Multiple environments – such as development and production – hold the code base for a set of ML pipelines, implemented with a variety of tooling like a feature store and model registry. The development environment provides an opportunity for experimentation and CI/CD is integrated to continuously test and deploy any changes to production. Model testing, (re)training and serving can also be automated based on triggers, user input, or monitoring data. Within production, trained models are provisioned as services for consumption. Real-time performance monitoring is defined for metrics like data drift and model staleness. By automating the entire ML pipeline and its deployment using CI/CD, manual effort is limited and the capability can be effectively scaled. Altogether, this setup forms a seamless and industrialized machine learning capability that operates as a continuous data refinery.

More information?

Interested in setting up an industrialized data refinery for your organization? For more information on how Deloitte can help you in your MLOps journey, contact us via the contact details below!

If data is the new oil, where is your industrial refinery?

Read our previous blog here
Did you find this useful?