Efficiency and cost savings through application of AI in metadata management has been saved
Efficiency and cost savings through application of AI in metadata management
Leverage AI and text mining to detect, group and predict business rules, enhancing the quality of management decisions
In our previous article we discussed how metadata management and data lineage help financial institutions provide insights into their risk data aggregation processes. In this article we continue where we left off and explore an approach, that showcases the benefit of Natural Language Processing (NLP) and Artificial Intelligence (AI) in resolving challenges financial institutions face in transforming to a more transparent and structured data architecture. The approach accelerates the process of detecting, grouping and predicting business rules in the big and complex “code jungle” often existing within financial institutions.
The “code jungle” challenges of financial institutions
Most approaches financial institutions apply in transforming their data architecture are often inefficient and costly. The approaches, applied currently, demonstrate quite some challenges of which the following three are the most predominant.
Firstly, financial institutions commonly struggle with the lack of consistency in the coding of their business rules and calculations. Historically there has been insufficient focus on metadata management, resulting in data users not knowing of transformations, calculations and derivations already existing within the web of code (e.g. SQL, SAS, etc.). Subsequently, different business units will interpret business rules from their own perspective, leading to variances in reconciliation for the end-user after the data is processed.
Intertwined with the challenge mentioned above is the lack of functional documentation and proper data governance with respect to the implemented code. The absence of such documentation makes it very time consuming, if at all possible, to sort out the functionality of specific pieces of code. Moreover, financial institutions often experience lack of (documented) code semantics and context. Even if the functionality of the code can be traced, it’s still an almost insurmountable challenge to put this functionality into perspective (which business processes are impacted, which source systems are involved, etc.).
These challenges combined give rise to a third challenge. The decentralized implementation of code and lack of functional documentation forces financial institutions to invest significant amounts of (human) capital and manual work in analyses before code can be organized and made transparent.
In order to assist clients, Deloitte envisions an AI-based approach that supports organizations with the challenges discussed above and accelerates the currently labor and time intensive work, while increasing quality at the same time. As we see at our clients, a lot of business logic is captured in the code used to transform, derive and calculate data. This approach turns your code into metadata and by adding context in the form of other metadata (ontology, manuals, data dictionaries, business glossary, regulatory interpretations, etc.) translates it into human understandable information.
The approach is based on five-steps, as seen below:
- A package of code is received and loaded into the solution. Simultaneously various sources of metadata are loaded.
- Pieces of a sub-set of code are manually tagged as being business rules.
- Using Text Mining and Machine Learning classification algorithms, the pieces of code that contain business logic (calculations, aggregations and transformations) are identified.
- Using all the information extracted from the code and combining it with the context provided by the other metadata sources, all pieces of code that contain similar business logic are clustered, at the same time logic that is overlapping and hence partly redundant is tagged.
- With using limited human feedback a recommendation system that predicts and suggests the best version of overlapping business logic based on metadata sources available is trained.
The afore mentioned approach was developed in conjunction with a Dutch retail bank, where similar challenges as described above were encountered, and instantly showcased the most profound benefit: efficiency. By using NLP and AI in the process of structuring the complete set of data transformations and business rules a lot of manual effort, time and hence costs are saved. Compared to the manual analysis this approach creates a more consistent analysis of the preferred end-state of the data architecture, moreover the outcome can also be used as benchmark to compare the manual analysis to. Another benefit is the prediction ability that this approach facilitates, by analyzing the metadata and the variations of the business logic that are tagged as overlapping, the approach enables the prediction of the best variation of the business logic.
This approach also provides a significant acceleration in documenting the metadata and data lineage of the data architecture supporting both the run and change processes of any financial institution, hence easing end-to-end data governance. All in all, this approach will significantly accelerate maturing to a compliant, transparent and structured data architecture.
In our experience automating significant portions of the analysis of code by the deployment of NLP and AI is in the best interest of financial institutions. This approach services best in mitigating the most profound challenges financial institutions currently face in transforming their data architecture. To support with such a journey there needs to be a solid documentation readily available which gives context to your code (e.g. in manuals, position papers, data/business glossaries etc.).
The approach Deloitte has developed enables you to take charge of the code within the data architecture in a cost and time-efficient manner by accelerating the transformation to a transparent and compliant end-to-end data transformation process.
Would you like to know more about Metadata Management? Please contact Yuri Jolly via the details below.