What to expect when doing a data migration: Tips from the Datamine Partners

Undergoing a data migration is no easy feat, particularly if you’re a large organisation with multiple data sources and complex systems you want to integrate into one single source of truth. Matt Wilkins, Partner at analytics consultancy Datamine has helped clients through his fair share of migrations over the years, and there are battle scars to speak of.

Our team recently sat down with Matt to hear his thoughts on both the challenges and opportunities presented by data migrations, as well as his suggestions for organisations looking to undertake this kind of work.

MA: So how many data migrations have you guys done at Datamine?

Matt: I actually counted up recently, and we’ve created 28 marketing datamarts alone - so probably somewhere around 50 data migrations in total? They’re substantial and complex projects, and we’ve definitely felt a lot of pain, both on the client-side and the Datamine side.

MA: What are the biggest pain points, typically?

Matt: I’d say one of the recurring issues we often see is that it’s impossible to fully specify and scope out a migration in advance, which can be frustrating for clients. You need to be super flexible and responsive because there will always be something that comes up and forces you to pivot. You can get a false sense of security by over-specifying ahead of time. For context, I’ve never done a migration where the spec perfectly matched the end output, and you’ll almost always need more budget and time than you anticipate - in fact, it’s usually safe to double what you think it’ll be. This might seem like an over-exaggeration but it’s not!

One of the other major paint points we see is actually in choosing the system you’re migrating to.I’d say more than half the time, the organisation has picked a new system without fully understanding its pros, cons and capabilities. This means they enter the migration process thinking they’ll be able to do certain things, but in the end they have to shell out more time and budget to get those other areas developed. It’s important for companies to fully understand the functionality of the system they’re migrating to, and if there’s stuff missing, how much that extra customisation will cost.

MA: Sounds like a convoluted space. But would you say the benefits of doing a data migration typically outweigh the challenges?

Matt: I think it’s usually worth the struggles. Yes, there’s a lot of pain throughout the process, and the end result might not be exactly what you had initially thought you were going to get, but it’s a ‘long game’ play that enables future analytics and campaign work to be much, much easier. If all goes well, you get improved orchestration, better customer experience and an increase in operational efficiency for the organisation. We’ve seen clients have huge success here.

MA: What kind of team do you need to do a data migration?

Matt: I think one of the most important roles is the project manager who can manage the migration and liaise between all the involved parties. This person needs to have technical knowledge, otherwise, you’ll have to put technical capability around them in order to make sure they’re equipped to champion the project. They also need to understand the interdependencies (from a data point of view) in order to ensure all stakeholders are informed. Internal documentation and specifications need to be shared amongst everyone involved, and people can’t be operating on their own.No one should be making decisions in one isolated area.I’d also say that the person making any decisions about the scope and activities must have a strong connection to the budget holder. I’ve seen in way too many projects decisions being made without prior consultation of the budget holder, and that doesn’t usually go well.

If you’ve never done a migration before, or if you’ve tried and failed in the past, your best bet is probably to work with an agile, flexible partner who can anticipate problems before they occur.Where other types of analytics work might be doable internally, we’ve found that internal teams lack the battle scars and experience needed to grapple with what might seem like an easy process but is actually quite convoluted. Regardless of whether they’re internal or external, data migrations absolutely require subject matter experts who are available and actively working on the project.

MA: Do you have any other advice for organisations looking to undertake a data migration?

Matt: As I touched on earlier, it’s important to focus on provisioning enough budget and understanding the right level of up-front specification. Whenever we talk to people who are looking to do this, they’re concerned about going over budget or having the project blow out of control, and the best way to avoid this is to allow yourself enough flexibility at the outset. Beyond that, I’ve got three other pieces of advice:

• Don’t try to migrate everything in one massive switchover. It’s really risky and hard, particularly if you’re dealing with operational systems and need to switch from one to the other over the weekend with no fall-back period. We suggest building the new datasets and QA-ing as you go, then layering additional data over that over a period of time so that the ultimate ‘switchover’ is just adding the final bits of data to a system that’s already set up and been tested.
• Make sure if you’re combining multiple systems into one that you understand the quality of each system individually, as well as where the data conflicts. You can get 95% of the way pretty quickly, but this step often takes heaps of time.
• 3.If you’re migrating your data to a system that has specific requirements and you’ve got other useful data that doesn’t map into those requirements, don’t get rid of it! You should have a plan for keeping it somewhere accessible so you can still do analysis on it in the future.

Keen to chat with Matt about your data migration questions? Get in touch with him here to set up an initial consultation with Datamine.