Data Migration: Everything you need to knowReady to migrate your data? Here’s how to get started

Data migration is a large undertaking for any business, but it’s a critical process to ensure better data analytics and security, as well as increased efficiency and compliance.

As well as being time-consuming and cost-heavy, a poorly executed migration could result in a lack of trust in your data and impact decision-making across your company. Detailed planning, a clear strategy and the right data partner are essential components for success.

By consolidating your business data and moving it to a new environment, you can create a complete and holistic view of what your database looks like. This, in turn, allows you to spot any gaps and identify insights and opportunities to use that data in a meaningful way.

A robust data migration project is behind the successful implementation of activities such as:

• Launching new systems
• System upgrades and moving to the Cloud
• Data projects
• Mergers and acquisitions
• De-mergers
• Buyouts
• Regulatory changes

What are the benefits of a successful data migration?

Data migration is a necessary activity to stay abreast of technological, regulatory and business needs. As legacy systems begin to no longer support business needs, it becomes necessary to embark upon new system selection and, inevitably, a well-managed data migration programme.

Despite its complexity and potential challenges, a well-executed data migration brings substantial benefits to a business.

While it’s ultimately your individual business needs and requirements that will determine project success, here are some of the benefits that you can expect:

Risk mitigation

Risk mitigation is a critical benefit from a successful data migration. A data migration can either make or break the implementation of a new system. An unsuccessful data migration can result in substantial costs to rectify and potentially reputational damage to the business.

Better data privacy and security

If you think of data migration as a fresh start for your database, you can implement new security controls that would be difficult to implement retrospectively to old systems. This includes determining who can access and control what data, as well as automatic logging for activity. You can also use the migration as an opportunity to encrypt data and ensure backup and restoration capabilities are in place.

Increased cost savings

Depending on how large your datasets are, you may be paying too much for cloud-storage and usage, when, in fact, you only use a fraction of the information there. By doing a thorough audit and migration, you can see what you have and what’s useful to keep, which will help optimise the support costs.

Reduction in inefficiencies

A large, unruly dataset can mean anyone who uses it has to sift through lots of unnecessary information in order to find and use what they need. And even then there’s no guarantee it’s complete or correct. By migrating your data over to a single storage type, format, or system, you can improve efficiencies in working with the data from the outset, as well as the potential need to amend incorrect work caused by imprecise sources.

Improved compliance

A well-architected migration strategy sets out a project plan for data location, access controls, activity logging, and encryption which in turn allows you to be better equipped when it comes to fulfilling regulatory requirements. Similarly, by implementing privacy by design at the start of your data migration project, you can ensure security and compliance considerations are embedded into the strategy from the get-go.

Better customer service

As well as the obvious benefits of ensuring faster response times and lower error rates, data migration can give you more accurate and detailed access to real-time customer insights. These can then be used to better identify issues, predict trends and needs, and implement actions that can help improve your customers’ experiences.

Increased competitive advantage

From optimising operational activity to informing strategic decision-making, there are plenty of valuable insights that data migration can bring. You can then use this information to gain a competitive advantage over those who have not migrated their data into a more usable system or format.

What are the risks of data migration?

Proper data migration is worlds away from a simple copy-and-paste solution. Issues from too-basic solutions can include moving incomplete records, poorly entered data, data and record duplication, and outdated information from the old system to the new.

While it’s never fun to think about the hurdles and difficulties of data migration, it’s important to understand the things that could go wrong so you can plan for them. After all, you’re moving significant amounts of source data from an old system to something new. With a little forward thinking these things can be avoided for a smooth and drama-free migration.

1. Underestimating the complexity of the project

The assumption that data migration is “just another IT project” or can be delivered by anyone regardless of experience can cause serious headaches down the line, such as data loss or running over budget. A clear yet detailed project plan can outline the resources and time required so everyone is on the same page from the start.

Data migrations can be complex and often involve multiple source systems and multiple target systems. Each source system may have its own schema, data formats, and data quality standards. It’s crucial to thoroughly analyse and understand the structure and content of data from each source system. Mapping the data from multiple source schemas to one or more target schemas requires careful planning.

2. Underestimating the complexity of the data

Data from legacy systems is likely to contain a history of use and abuse over its lifetime.  Sometimes these complexities can be understood through detailed data profiling, but often the data migration team needs to engage with the business to understand how the source systems were used, and how that usage has changed over time. Data fields may have changed use, become redundant, or single text fields could contain a concatenated array of information entered under agreed conventions passed by word of mouth between users.

Similarly, a “system folklore” may have built up around source systems. Users will know which fields are useful for day-to-day operations and reporting, and where they may have fallen out of use. Users will understand where the master data is flawed and how external systems update the source.

Combining all this user knowledge with a full understanding of the data schemas and functionality of the target system, will uncover opportunities to best use the existing source data. This knowledge is also valuable for understanding how the data can be fully utilised in the target systems’ processes, and in addition, identify opportunities where data can be transformed and even augmented to better serve the needs of the business.

3. Poor prioritisation and management of data issues

Those working on data quality or data migration can often fix inaccuracies manually, but this presents challenges of its own. As well as the huge amount of time and resources needed to sift through records, manual administration still leaves room for human error and gaps in understanding which data issues are most prevalent and impactful. These can include:

• Unclear and confusing records thanks to non-unique data descriptors.
• Data appearing ‘missing’ as it has been entered into incorrect or single fields.
• Datasets duplicating information unnecessarily due to too-similar source fields e.g. ‘client name’ and ‘account ID’ presenting the same information.

Similarly, when there’s no coherent and centralised plan it can become easy for a project to go off on a tangent, leading to increasing budget and timelines. Given the business-critical nature of data systems (e.g. trading, payroll, finance, etc.) it’s essential that the correct project controls are in place.

Migration projects often uncover more data issues than anticipated, so contingency time and resources should always be factored in. This in turn makes prioritisation essential as it can help flag which areas are vital to fix before any other. Fixed “go live” dates are rarely a good idea, unless required by an external force.

4. Lack of collaboration and integrated processes

Data silos are easy to come across when you have multiple employees working on various elements of a migration project. This can lead to the use of different processes and technologies and result in a mismatched migration – the thing you’re actively trying to undo. To avoid this, collaboration is key and pinpointing the different project stages will help everyone understand their role and responsibilities, as well as keeping to timelines and leaving little room for assumptions or misunderstanding.

The steps involved in a data migration project

Data migration may appear like a large undertaking because you’re likely going to be working with lots of big, detailed datasets, many of which may have faults and anomalies within them. Here are some key steps you’ll need to take to help your project be a success.

1. Form an all-star data migration team

You’ll need to assemble a team that will manage the data and processes, and ensure quality for the entirety of the project and beyond. This top-tier project team should be cross-functional and draw upon business, technical, and customer-centric expertise. You will likely want to include:

Data quality workflow leader 

A designated leader and champion of data quality will help free up the data migration experts to focus on other areas, safe in the knowledge that data quality is being controlled and coordinated.

Business subject matter experts 

They will know all about the different behaviours and details of your legacy system and can help unravel the meaning of long-forgotten datasets.

Technical subject matter experts 

From software developers and designers to database administrators, data architects, and interface designers, these people will help you understand the various structural architectures and design of your datasets.

Data quality analysts 

You may benefit from team members who can execute data quality assessments, data profiling, and data discovery tasks. They can find issues, present them to the team for resolution, and even design solutions to prepare the data for migration.

2. Determine a project plan and undertake a PMIA

PMIA stands for Pre-Migration Impact Assessment, and it’s essential that all data migrations start with one. A PMIA aims to replace assumptions with evidence and is a quick-fire investigation and analysis that will gather important information to use throughout your project, such as:

• Wide-scale data discovery of legacy systems to uncover the relationships between source data and if there is any probable missing or duplicated data.
• Extensive data profiling to create a definitive catalogue of metadata and data quality rules.
• Discovery and prioritisation of data quality issues so you can adjust your project approach accordingly.

Remember, data migration projects differ from industry to industry; for example, charity-sector data migration can be amongst the most complicated owing to the security and regulations involved. So, be sure to include sector-specific considerations into your PMIA.

Following this, you can set about creating a project plan and checklist, which should include:

• Data profiling
• Data discovery
• Data quality assessment
• Data correction and cleansing
• Data matching
• Data deduplication
• Data enrichment

3. Get the right software in place

Having the right software in place not only makes life easier for those undertaking the practical aspects of a data migration project, but it also puts you in good stead for data reporting, analysis, and management in the future.

4. Assess the data you already have

By assessing the type and quality of data you already have at this stage in the project, you can uncover any defects that could potentially cause issues if they were to be found further down the line.

5. Prioritise each step in the plan

Prioritisation is essential. As with any large-scale project, things can go awry, and this is okay as long as you have the understanding in place of what to focus on. Some issues can be more easily resolved and some are critical to the success of the project so need to take precedence.

6. Undertake a project report

To be sure the data migration project has helped you achieve the mission-critical goals you initially set, there needs to be a reporting structure in place. Think about what you want to measure, how it’ll be monitored, and who will be responsible for it.

7. Put monitoring and assurance processes in place

Without regular management, monitoring and reporting, it can be difficult to spot when defects have reappeared and failures are creeping back in. Creating a data quality process and architecture that you can hand over to your business and IT teams will help ensure data is kept clean, usable, and adequately governed in the long term.