Data migration: why it is often unsuccessful
Successful data migration is built on three pillars
In practice, I have seen many data migration trajectories requiring much manual intervention, facing significant delays, experiencing high level of “fall-outs” or with only sub-optimal results. As a result, data migration often leads to an overrun in delivery budget & time, and decreased productivity due to data quality issues experienced by the end-users in the new system. So why are data migrations often unsuccessful?
Wing Lee - 17 februari 2016
Data migration is in simple terms a process of extracting data from system A (source) and loading it into system B (target). Companies invest large amounts in data integration tools to perform data migration in an almost automated manner. Gartner annually conducts a comparison study (Magic Quadrant) on the market leading data integration tools to help companies in selecting the most appropriate tool. However, that does not guarantee success, because having the right technical tool is only 1 of 3 pillars on which a successful data migration is built.
Before diving into the other 2 pillars, let's zoom into this technology pillar first. Having a good piece of migration software has some advantages:
- It enables the migration team to build the technical migration solution in a shorter amount of time;
- It gives guidance in determining the data migration activities required;
- With an user-friendly GUI, even business people with less technical background could operate the tool and perhaps even develop it;
- The software can be used to explain the technical data migration solution to non-technical people;
- Commercial data integration tools often come with built-in tracking and reconciliation functionalities.
Professional data integration tools unfortunately also come with system requirements and (high) software and/or licensing costs. Note: it is not always a must to invest in expensive data integration tools. Depending on the complexity of the data migration and the experience of the migration team, a good alternative is to build a technical migration solution based on database stored procedures. Custom-built migration tools offer great flexibility, but come with development effort. An experienced database or data warehouse developer would be a good help.
2. Data quality and migration readiness
It surprises me that many companies pay little attention to its data when it comes to data migration. Common complains I come across in practice:
- All data has been migrated into the new system successfully, but the quality is not acceptable;
- The migrated data cannot be used in the new system;
- There are many “fall-outs” during the migration, because records are duplicated or mandatory fields in the new system cannot be populated;
- The actual data migration was painful and dragging, because many data issues had to be manually fixed when loading the data.
The data quality and migration readiness is critical and should deserve appropriate management priority. It not only determines the success of the data migration, but it also affects directly the opinion and the acceptance of the new system by its users.
3. Management of the migration throughout the organization
Another determining factor for a successful migration is how well the migration is managed throughout the organization. I say “throughout the organization”, because there is more to manage than the various delivery teams within the project to ensure the Data migration gets delivered, which are obviously also essential. The following challenges that I have experienced in practice should make this point clear:
- No clear support from Senior Management (i.e. business), leading to de-prioritization of migration activities and budget cuts;
- The migration solution is ready, but Privacy & Security blocks the actual migration to start, due to concerns on unauthorized access to company networks, systems and Production data;
- The development of the Data migration solution is delayed, because necessary IT support has not been timely arranged.
A successful data migration is therefore built on 3 pillars:
- Having the right technical tool to make the actual migration possible;
- Ensure the quality and migration readiness of the to-be-migrated data;
- Continuously manage the migration throughout the organization (i.e. Senior Management, Privacy & Security, and IT).
More information about data migration?
I am interested in your thoughts on and experience with data migration so please contact me if you would like to exchange thoughts. Would you like to know more about data migration? Please do not hesitate to contact me: WingLee@deloitte.nl or +31 (0)88 288 2902.