What are the Big Challenges in Data Migration?
Whenever you work with data, you will invariably want to move it somewhere, be it to a different location, application or even just a different format. Generally this is done so that the data can be stored, shared or utilised more effectively. Almost always this will involve some form of extraction, transformation and loading of the data. Sometimes, you may even want to bring together data from multiple sources into one place. With this process there can be a variety of challenges around compatibility, loss of data, security and maintaining quality of data when migrating it.
In this article, we will walk through four of the main challenges when dealing with data migration:
Understanding the data
The first key aspect of data migration is understanding the data that is to be moved. What is stored within the data and how is it stored and formatted? What is the data currently used for and how is it intended to be used in its new location/format?
Understanding what is in the data is crucial for understanding what needs to be captured and moved when the data undergoes migration and to ensure that there is no loss of data when it is migrated.
Understanding how the data is used helps to understand why the data is structured as it is and what crucial elements of the data need to be transferred across to its new location, whilst understanding how the data is to be used in the future helps to understand any improvements or restructuring of the data that can be a applied to better suit its new (or additional) purpose(s). Such understanding can be facilitated by good metadata, and by good recordkeeping on the uses of data both past and present as well as future.
2. What needs to be moved?
Once a good understanding of the data has been achieved, we can begin to consider how to move it. An important consideration is what/how much of the data needs to be moved. In many cases this could be all the data, especially if you are moving from an old, obsolete system that is to be discontinued to a new one, you want to capture everything.
In other scenarios this may not be the case. If you are moving data from a place where it is collected/stored to another place where it is shared/utilised, you may not need or want to migrate all the data. There may be data that will be superfluous to the new requirements and can be left out to save storage space and processing time. It may be easier for users of the data if they are presented only with the handful of data fields that they require, rather than a wide selection that they have to wade through to get what they need.
If the data is being shared, there may be access and security concerns, some parts of the data may contain restricted information that cannot be shared, and thus should not be migrated. Of course, when considering what needs to be moved it is important to consider not only present requirements, but also future ones. This way you can ensure that the new data system meets current needs whilst also endeavouring to future-proof it against new needs that may arise.
3. How is the new data structured?
Now that we know what needs to be moved, we need to consider how the data will be structured in its new location or format. This is especially important if we are migrating several sets of data into one. We need to consider how these data interact. Are all their data fields the same? Do they all have the same potential set of values?
Where there are differences in the data set, we need to consider how these can be overcome in order to provide a clear and comprehensive set of data. Even when we are not combining datasets, it is still important to consider structure. The data could be structured exactly the same way it was before, which may well be the simplest solution. However, we should consider the nuances of the new data format, and any new uses that the data is being put to, which may mean it is more beneficial to restructure the data in a different way. We should also be mindful of any issues with migrating the data, and if they could be resolved by improving the structure in order to make any future migration process easier.
4. Consider all this from the start
The easiest way to address the challenges in data migration is to fully consider them from the beginning when designing the data. ‘Prevention is better than cure’, as the old maxim goes. When structuring new data sets it is important to future proof them against future changes such as migrating to a new system.
Standardisation of data can greatly help with bringing different datasets together and with moving data into a new format. If all data is standardised then it is much easier to understand, merge and migrate it.
Considering the challenges of the future, such as data migration, is an important step of any data architecture process, and should be considered alongside the needs of the present. The best way to resolve these challenges is to solve them from the start!
There we have a walk through of the top key challenges that you are likely to come across when dealing with data migration. We hope that by sharing these insights with you, it will help you through the key steps and stages in the data migration process. For further support with this our team are happy to help, so please do contact us to discuss your particular situation.
Data migration is also one component to data management more generally and we offer a range of support services to help organisations. See our Data Management services to see how we can help and read our article on Why no-one’s using your data management tool.
Author
Tags: