Intelligent Data Discovery

The Need

Most modern companies and organizations worldwide have been steadily modernizing their operations, improving productivity through various forms and degrees of digitization, increasingly reliant on inter-connected systems and databases, data warehouses, or data lakes. They have little choice but to do so if they want to remain competitive: those who fail to re-invent eventually find themselves forced onto the sidelines, struggling to catch up.

Data has taken center stage with the emergence of AI as a differentiator. Digitization both within company walls as well as with stakeholders – whether they be customers, partners, suppliers, or regulators – has opened up new possibilities. It is a gargantuan task to organize and combine internal and external data sources in a fashion that lends themselves to analysis and insight. It is also a highly lucrative endeavor: root cause analysis, cross-selling, early warning of potential product issues, supply forecasting, customer brand sentiment, demand forecasting… big data provides a long list of benefits, if you master it.

In addition to profiting from insights, companies, as custodians of often sensitive data, are obligated to comply by a strict set of rules, regulations and laws. They must respect consumer protection laws and general data privacy regulation, often across multiple international jurisdictions. Fulfilling these multiple requirements requires a high standard of data governance.

Both knowing the data (governance) and harvesting it (analytics) pose challenges to any data-rich enterprise, especially so for large and mature organizations that have grown both organically and through acquisition. Established banks are a perfect example: many face a complex patchwork of legacy systems and databases – both home grown and inherited through merger activity – a data management headache which impedes analytics and governance goals. They enviously look at the relatively simple landscape of nimble newcomers, who may lack the rich history of data from the incumbents, but can at least easily analyze what new data they have. In other words, where newcomers may lack “long” datasets (history), they benefit from highly usable “wide” datasets (many variables).

Our Solution: IntelliMap

Deloitte’s data discovery solution “IntelliMap” helps organizations with complex data landscapes regain the upper hand. IntelliMap artfully blends deterministic rules garnered from a wealth of experience in data migration and governance with statistical methods characteristic of unsupervised machine learning. This “hybrid-intelligence” enables IntelliMap to effectively analyze datasets with no prior knowledge of their contents, providing the user with insights into the nature of the data as well as suggested relations between data tables.

The reconnaissance work performed by IntelliMap breaks down barriers to otherwise long and expensive data management projects: system migrations, data warehouses, GDPR-motivated data governance programs… to name a few. By performing the first-pass scan of all available database fields, IntelliMap assists the database architect in quickly isolating variables of interest. Rather than searching through hundreds of potential matches between tables, IntelliMap offers the architect the top few candidates that could link one table to another. This accelerates the arduous, yet critical groundwork behind any data warehouse project. Or when investigating the data of an acquisition target in the context of a due diligence exercise. It can also facilitate the data discovery and validation necessary to obtain and maintain data governance for GDPR compliance.

Advantages/Benefits

• Speed – five- to ten-fold reduction in time over manual mapping of tables
• Cost – reducing the cost of highly specialized resources required for data migrations, API connections, or data warehousing efforts.

Example Use Cases

• System migrations – identifying potential matches between source and target databases.
• Development of APIs – facilitating the development / customization of external tools intended to interact with established systems.
• Data warehouses / data lakes – accelerating the relational linkage between data sources to create a “wide” dataset.
• Data governance / lineage – providing an easily maintainable / updatable level of transparency over the system landscape, for instance identifying where sensitive data may be stored.
• Due diligence – enabling fast insight into the data provided by the acquisition target.

Back to main page