Business Intelligence & Data Science: same, but different
In an IDO, Business Intelligence and Data Science can support each other and improve themselves by leveraging each other’s skills and capabilities.
Stefan van Duin & Bas Schmidt - 8 januari 2018
Journey towards an IDO, a practical example
One of the largest logistics companies in the Netherlands recently initiated their journey to transform into an Insight Driven Organisation. A new Analytics Center of Excellence is created as a center point for all Big Data & Analytics-driven innovation and consists of several data scientists and a leader. The team has two main focuses, supporting the business with high-quality Advanced Analytics solutions and to increase the analytics maturity of the whole organisation. Imagine an Analytics consultancy team within their own organisation with the ability to grow the analytics awareness of all employees. It is kept separate from the existing BICC. Because of a clear, validated service catalog, the BI Competence Center and the Data Science team have a great collaboration in supporting the business. The teams work closely together in data accessibility and in the implementation process of analytical solutions in the business. Started in June 2016 and fully operational from the beginning of 2017, the transition towards an Insight Driven Organisation is at its full speed.
'Data is the new oil', a quote which states that data –when properly used- has proven to be extremely valuable. In the past two decades, many companies have created business intelligence capabilities, focusing on managed reporting, dashboarding and data discovery capabilities. In the past number of years, a new role has emerged: that of the data scientist. Driven by successful companies that have built data analytics in their DNA, (like Amazon, Facebook, Uber), traditional businesses have been keen on transforming themselves into Insight Driven Organisations. This is not an easy journey, shown by the fact that a lot of organisations are “dabbling” rather than “doing”. Is Data Science just the next evolution in BI? Does it make sense to have the BI team build the Data Science capabilities? In this blog, we will explore the two worlds through various lenses and try to come up with some conclusions and lessons learned.
2. The Strategy and ambition of BI versus Data Science
BI and Data Science have one big thing in common: they both take data and transform it into insights. BI, however, is generally focused on creating management information. It is often embedded in planning and control cycles, creating “one version of the truth” as a source for analyzing business performance. Most of the use cases around BI consist of generating periodic, standardized reports with reliable, quality controlled numbers based on structured, internal data. Often it offers some self-service analysis and visualization capabilities, by providing data in multi-dimensional cubes and disclosing the trusted data via “data discovery tools”. This all implies that there is a process that regularly extracts data from operational systems and transforms it into a dataset with harmonized definitions, and someone looks after the quality and reliability of it all. BI is presenting descriptive insights, humans (e.g. managers) are interpreting, drawing conclusions and taking actions based on it.
Data Science tends to have a different focus. It is aiming at generating insights out of data that the human often cannot oversee. This could be complex descriptive analytics, finding relations between multiple variables and events, or the result could be a predictive model. The basis is data, but it can be a combination of internal and external data, and it may very well include unstructured data sources like documents, pictures or videos. The output is often not a report, but rather a Machine Learning model, i.e. a piece of an algorithm that has learned to recognize fraud, predict customer profitability or determine the next best offer to present to customers.
In contrast to BI, Data Science may not be based on predictable requirements, but rather on innovative, experimental ideas of which the feasibility is not completely known upfront.
3. Business Intelligence vs. Data Science within IDO
We will explain the differences and similarities by the five foundational pillars of an Insight Driven Organisation:
Strategy – ‘What does becoming an Insight Driven Organisation mean to our business?’
When thinking about BI and Data Science strategies, Business Intelligence teams aim to provide a high-quality delivery of information for the whole organisation and assuring the principle of ‘one version of the truth’. The teams focus on accessing the most relevant information for decision-makers and other data-driven teams in the organisation to optimize business and decision processes. BI teams tend to focus on the periodic monitoring of the strategic goals, reporting on the key performance indicators and understanding the drivers of over- and underperformance. For BI teams it is essential that reporting and analysis are aligned with the corporate strategy.
Data Science teams are focusing on solutions for specific business problems, applying state of the art, complex algorithms on various data sources, in combination with their business understanding. Data Science teams have a focus on (new) value propositions with an experimental touch to establish competitor advantage in the market. For Data Science teams it is important to align with business priorities, to focus on the right, high-impact problems.
People – ‘Have we got the right people, in the right place, at the right time, ready to perform the right actions?’
Business Intelligence Developers are mostly skilled in Data warehousing capabilities, query languages (for example SQL), data visualization tools (e.g. QlikView, Tableau or PowerBI) and have a mindset for efficient data storage and the performance of queries. The skills of the people are focused on reliability, performance, quality, and user-friendliness.
For Data Scientists, work is normally more experimental, ad hoc and unpredictable. It requires creativity and improvisation to work towards a solution to the problem, even if circumstances (like data availability and quality) are not perfect. Some of the projects they are working on may not result in an actual product since it is often unclear upfront whether a solution will work.
What both have in common is a strong feeling of quality. Both a report and a predictive model should comply with highest quality standards. This requires people that are able to work with sound methodologies and governance, both in their respective domains.
Process – ‘Have we designed an end to end process in which we can accurately identify, correctly prioritize and satisfactorily control the delivery of actionable insights to our business?’
Business Intelligence teams consist mostly of more people than Data Science teams and work in most organisations via gathered business requirements or change requests, potentially via agile working methods. Requests for information such as reports or dashboards are approved, developed, tested and delivered in a structured and preferably in an automated way. Data Scientists, on the other hand, work mostly on ideas from the business with an experimental touch. Think of a project ‘funnel’, where the initiatives have to pass various value gates in order to reach implementation. Initiatives from the business are first evaluated and prioritized based on impact (for example cost reduction or revenue increase) versus complexity (for example resource planning, privacy constraints or data quality). After prioritization, Data Scientists work closely with the business towards a ‘proof of concept’, a ‘prototype’ of the solution. These prototypes are evaluated and possibly piloted into the business and eventually operationalized in the day-to-day decision-making process.
Data – ‘Have we created a clear line of sight from business decisions to data sources, with data management delivered to support and inform this process?’
Data is the key element which connects BI developers and Data Scientists, in different words “it all starts with data”. However, the proportion, structure or format could differ between the two functions Business Intelligence teams aim toward a structured, consistent and high-quality solution to store and present the data, in a repetitive way. It is often stored in large data warehouses or cubes contains much historical data and is mostly structured. Structured data could be described as any data that remains in a fixed field within a record or file. This includes data contained in relational databases from ERP systems. Data Scientists on the other hand, often work with separate data dumps, internal and external data, and sometimes a mix of structured and unstructured data. Several organisations refer to this setup as a ‘sandbox’ environment, a data repository where data scientists have the freedom to store and use all kinds of data formats for experimental algorithm development. Eventually, their (experimental) solutions could end up being part of the structure data flows if this will support the business on a frequent base. The Data Lake is introduced as a way to store all raw company data of all forms, allowing it to be used both by Data Scientists and as a source for structured Business Intelligence,
Technology – ‘Have we constructed an integrated technology infrastructure and architecture which scales to support our long-term vision of becoming an Insight Driven organisation?’
Teradata, Oracle, Amazon, Microsoft, and IBM are all examples of leaders in Data Management Solutions for Analytics (DMSA). Gartner defines the DMSA as a system for storing, accessing, processing and delivering data intended for one of the primary use cases that support analytics. Business Intelligence teams often work with large data warehouse vendors such as previous examples. From a data visualization and dashboarding point of view, BI teams often use the common vendors such as Qlik, Tableau, SAP or Microsoft to support the organisation with reporting, dashboards and self-service visualization. Data Scientists, on the other hand, use a variety of tooling depending which best fits the specific goals of the project. In general, Data Scientists have a preference for open source toolings such as Python or R. These tools have a wide community and are easy to install and use if you have basic programming experience. In some cases, such as GeoSpatial Analytics or Optimization problems, specific tooling like Qgis or AIMMS are a better ‘fit’. Cloud Big Data Platforms, such as Amazon Web Services (AWS) or Microsoft Azure, offer out-of-the-box data science libraries and are able to scale and deploy R and Python code into a productionized environment.
As described, Data Science and Business Intelligence teams, both share a passion for data and the insights from the data but are two significantly different roles. They have a different way of working, serving a different purpose. Both have a significant task in an insight-driven organisation, they can support each other and improve themselves by leveraging each other’s skills and capabilities. We believe that each organisation which makes the transition towards an Insight Driven Organisation needs an operational Business Intelligence team and Data Scientists throughout the organization to fully retrieve value from their data which distinguishes them from their competitors.
Meer weten over Insight Driven Organisation (IDO)?
Wilt u meer weten over IDO? Neem dan contact op met Ron Bergers via +31 (0)88 288 8740