Data migration: why it is often unsuccessful

Opinie

Data migration: the high level activities for success

5 steps for a successful data migration

Data migration trajectories can use different technologies, depending on the systems and technologies involved in the IT landscape. However, all data migration trajectories generally contain the essential steps of data extraction, validation, transformation, and upload into the target systems. To maximize the data migration success rate though, I would add 3 steps to complete the data migration activities: data cleaning iterations, data reconciliation and error handling.

Wing Lee - september 2016

The high level data migration activities consist in my view of:

  1. Data cleaning iterations; including the data extraction and validation steps.
  2. Data transformation and enrichment
  3. Upload into the target systems
  4. Error handling & reporting
  5. Data reconciliation

 

 

1. Data cleaning iterations

Data migration process starts with extracting the to-be migrated data from the source systems. This data is then validated against technical and functional validation rules, to determine the migration readiness of the data. The technical validation rules are normally set by the target systems, while functional rules originate from requirements of the “new” business processes. This validation step not only ensures that the to-be migrated data can be uploaded successfully in the target systems, but also that it can be processed/used in the to-be business processes.

Data that does not pass the validations, will be fed back to business representatives or preferably data owners for verification and corrective action. The activity of making the data ready for migration and/or improving the quality is also known as data cleaning.

  • This process of data extraction, validation, feedback to business, and data cleaning is best started in parallel to the data migration software development, as it requires quite some time, business support and iterations before the desired data quality level is reached.
  • Data cleaning iterations can also be used by organizations, standalone from a data migration trajectory, to assess and improve its data quality level.


2. Data transformation and enrichment

Data that passes the validation gate, is qualified to be migrated into the target systems. Before the data can be uploaded, there are commonly some transformations and enrichments required. Think about:

  • Transforming data into the target data models
  • Looking up technical references to configuration data and master data
  • Deducing values for new fields based on certain data elements

For this transformation and enrichment step, existing data from the target systems and value mappings are needed, the same as for data validation.


3. Upload into target systems

After the transformation step, the data is ready for upload into the target systems. There are in essence two methods of uploading data into target systems:

  • Make use of existing order processing functionalities
  • Perform inserts (or updates) into target databases


4. Error handling & reporting

Error handling is all about properly following up on errors and fallouts from the data migration, and taking corrective actions to ensure all data entities are migrated completely and correctly into the target systems.


5. Data reconciliation

Data reconciliation is about reporting on the data migration result. Two levels of validation are relevant to the migration success:

  • Migration completeness – has all to-be migrated data been migrated?

    Track migration result per migration entity (e.g. customer, supplier, product, etc.) and log all fallout reasons per migration entity. Fallout reasons can be grouped into data cleaning issues or technical errors.
  • Migration correctness – has all migrated data been migrated correctly?

    Validate data integrity in the target systems after migration, e.g. does every migrated customer have the same contact details and products in the systems.

 

Improved data quality through data migration

Having these high level activities properly in place, not only increases the success rate of the data migration but also ensures a smooth and transparent migration execution. Moreover, through the data cleaning iterations the organization is likely to end up with improved data quality in the target situation.


More information about data migration?

I am interested in your thoughts on and experience with data migration. Would you like to know more about data migration? Please do not hesitate to contact me: WingLee@deloitte.nl or +31882882902.

Vond u dit nuttig?