Did you know that 38% of data migration projects fail?[1]
While this statistic represents a bleak picture of wasted effort leading to a drain on resource and budget, it is important to note that a data migration is a challenging project and the risk involved is high. However, being aware of the common hurdles that could potentially derail your project will increase the likelihood of achieving a smooth transition of data to your new system.
In our discussion paper “Improve the Success of your Data Migration Project” we explored the common problems that are often not addressed and the factors that far too often lead to failure. Here are the “8 data migration hurdles” to steer clear of.
1. Poor Knowledge of Source Data
This knowledge gap includes not being aware of the problems that exist in your data, such as duplicates, missing information, misspellings and erroneous data. It can be all too easy to get complacent and assume that your data can easily be configured into the parameters of a new system however the reality could mean critical failures when it comes to user acceptance. So to ensure success, you need a good understanding of the source data.
2. Underestimating Data Analysis
Due to constraints in computer systems, information can be hidden in obscure places because often there aren’t specific fields to hold all elements of the data or users may not be aware of the purpose of the available fields. This will result in incomplete, inaccurate and outdated information being transferred during the migration, often discovered very late in the day, even after the project has been completed. The outcome can mean not having enough time or the right resources to be able to identify and correct this data. Performing a thorough data analysis at the earliest possible occasion, usually when planning and designing your data migration can help you uncover these hidden errors.
3. Lack of Integrated Processes
Data migrations typically involve a disparate set of people using disparate technologies. The classic example is the use of spreadsheets to document data specifications, which is prone to human errors and cannot be easily translated when analysing data or performing data transformations. Using disparate technologies can lead to failure in the transfer of data and its design between the analysis, development, testing and implementation phases. Things can get lost in translation, resulting in increased costs and wasted time. Organisations must look to utilise a platform that successfully links the critical inputs and outputs from each of the stages to help reduce error and save time and money.
4. Inability to Validate a Specification
While you may well have an understanding of your source data, this will not necessarily result in a strong specification for migrating and modifying data into a target system. As this is early in the stage of the migration, critical misses can have repercussions later in the chain of activities. Validating your data transformation specifications early on with actual data, rather than just documented aspirations can increase the confidence in executing the rest of the steps.
5. Failure to Validate the Implementation
Like before, where your knowledge of source data is evident, you can still hit a brick wall because of a lack of test cases. If you don’t explore various scenarios, you run the risk of developing problems when it is often too late. Testing your migration using full volume data from the real world helps cover a wider range of possibilities and tests for the worst case scenario, which could be missed when using more convenient samples of data.
6. Late Evaluation of the Final Results.
This problem can occur in the testing stage, where users only see the actual data that will be loaded into the new system at the end of the design and development. At this point, one of the worst outcomes can arise – incompatibility of the data in the new system. While an organisation is capable of working without remedying the problem, this is not best practice. Time, money and the embarrassment of a delayed project can be avoided by introducing early and agile testing phases and getting your users involved in evolving the test cases as they see actual prototypes of the data output.
7. Lack of Collaboration
It has already been mentioned that data migrations involve disparate people, using different technologies, and in some cases a mix of internal employees and external contractors. Some of these people may not even be in the same location. Working in silos can reduce efficiency, create more data silos and sometimes lead to misinterpretations. Working together can be difficult and when things start to go wrong most try to avoid blame rather than resolving the issues. Collaborative tools enable all parties invested in a migration to see the same picture of data as it moves through the project stages, leaving little room for assumptions and misunderstandings.
8. Inappropriate use of Expertise
It makes sense to source experts, and usually this is applied to the management and technical aspects of a data migration. However, often the experts on data, usually hidden in the business do not make an appearance until late in the day. All too often those with access to data are unable to decode it, while those that can are unable to obtain access to it, sometimes until the new system is ready. Introducing data experts into your migration projects right from the beginning will ensure they make sense of the disparate data sources, but also guide the data transformation to suit the audience who will use it in the target system.
A data migration project is challenging and high risk but if each of these hurdles are acknowledged during the planning stage and are overcome early on before data is transformed and transferred, you can be sure of success.
[1] Data Migration – 2011, A White Paper by Bloor Research, Author : Philip Howard, Publish date : December 2011