Cloud platform for data mesh in the automotive industry
Efficient evaluation, assessment, and organization of data
Today, automotive OEMs acquire a large amount of data across the entire value chain and use these data sets to analyze and gain actionable insights on a variety of use cases. Thereby, manufacturers are able to gain new understandings into customer behavior, driving patterns, personalized customer experiences, supply chain, production risk forecasting, asset tracking and tracing. Based on this knowledge, enterprises can make important decisions. But how can the automotive industry use the volume of data to gain efficient and meaningful insights and create new, flexible business models across the value chain?
In most cases, all data accumulated will be converted by the IT system into a centralized ETL-pipeline (extract-transform-load). Thus, the data is stored in monolithic data warehouses or data lakes that have evolved over decades. However, this approach forces OEMs to find a solution regarding the use of the data.
Cloud Uses Cases: Automotive Industry
Challenges for automotive OEMs
In the context of growing data volumes automotive OEMs are facing numerous challenges, such as:
- Complex data sources lead to a difficult and slow integration
- Data sharing problems intensified by a silo mentality
- Lack of expertise regarding data sources in centralized data ingestion and warehousing teams
- Limited ability to address data quality issues at the source or react to changes
- Need for a more economical, effective, and efficient IT architecture as dispersed data is becoming the norm
- Lack of a domain-oriented data catalog to ease data discovery for consumers
As IDC has shown, 64.2ZB (zettabyte) of global data has been created or replicated in 20201. By 2025, IDC forecasts a compound annual growth rate of 23% in terms of global datasphere. But just having more data is not the key. According to IDC's "Global DataSphere Study 2021", less than 3% of the data currently created is analyzed to affect enterprise intelligence. 59% of mid- to upper-level managers responded that they are overwhelmed by the amount of available information. Nevertheless, nearly half of all respondents believe that there is not enough data for appropriate decision-making. Enterprises that overcome these challenges will be successful in the long run.
Data mesh – an architectural paradigm unlocking analytics at scale
Data mesh is a paradigm shift for creating a decentralized data architecture. Under this approach, all data is treated as a product. Specialist teams are responsible for the development and operation of data products in their domain. The concept relies on domain-driven design. Cross-functional data domain teams take on responsibility for specific data, from acquiring the data they need and processing it to making it available to the mesh as a data product. Enterprises generally do not explicitly build a data mesh, but rather create it by putting a federated governance structure in place as well as the means of exchange and discovery. Data discovery and exchange require a set of central tools and policies.
Key benefits of a well-designed and implemented data network include creating domain-centric ownership of data sources, pipelines, and increased data quality. Data assets are offered as products in a "serve & pull", rather than a "push & ingest" model. Furthermore, the quality of the data is guaranteed because teams of specialists are responsible for the various data. And, thanks to data mesh, a faster response of the individual domain teams is possible regarding changes in the source format or quality issues. After all, there is the ability to easily scale the total number of sources and consumers.
Requirements for data mesh in the automotive industry
Regarding the design and implementation, a highly scalable infrastructure is required. In this regard, the following features should be integrated on a public cloud platform:
- Self-service capabilities
- Integration with other data lakes and interfacing applications within the organization
- Security and data protection
- Efficient DevOps tools
- Mesh governance
- Automated data pipelines
- Interoperability among data products
- Modern big data & AI capabilities
This highlights some key features of the cloud platform for data mesh. For example, it is possible for business users to create their own reports/analyses. Thus, users are authorized to include new data sources using ingest/ETL templates. Furthermore, the use of auto-scaling mechanisms helps to easily scale with demand. Consequently, new use cases are more easily accessible by augmenting the DWH with a data lake. Also, the latest BI and Guided Analytics tools help deliver new functionality and a more attractive look and feel.
An easy integration with other applications and data lakes is made possible by standardized APIs. In addition, managed IaaS and PaaS solutions can be deployed to free up development capacity for client-specific value creation. At the same time, reliability is ensured. Finally, the cloud platform for data mesh fulfills the necessary security requirements for identity and access management, encryption, audit logging, archiving/retention, anonymization/deletion, and cyber defense.
Key features of the cloud platform for data mesh
The cloud platform for data mesh offers its users several benefits. On the one hand, the data products are structured in a modularized form. Thereby, defined interfaces support better scalability. On the other hand, the domain team with data ownership has the skills and business knowledge to maintain quality. Data product owners will be able to use federated governance in the data mesh to monitor data usage. In addition, there is an exchange of data within the enterprise at the unit layer. This allows for cross-domain data sharing based on well-defined governance policies.
Additionally, the cloud platform for data mesh offers increased discoverability of data, due to a central and domain-based data catalog. Data products are easier to maintain thanks to a code approach. Thereby, the entry barrier for development of new data products/assets is lowered. Moreover, seamless (zero-downtime) onboarding/migration of individual workloads (datasets, reports, dashboards and KPIs, and custom APIs) is possible.
Last, but not least, the cloud platform for data mesh in the automotive industry enables enterprises to foster innovation and draw resources from a large talent pool. The pay-per-use pricing structure of the public cloud platform helps OEMs reduce the total cost of storage and computing compared to legacy solutions.