Data migration: context
All sectors and markets, including Telecom one, are immersed in a digital disruption. Companies are buying other companies, redefining its strategy, business model, organization, equipment, processes and applications, data and infrastructure that support them to remain competitive in the market.
Nowadays, companies have very clear objectives:
- Engage their customers.
- Innovate through new products and services.
- Adopt new ways of doing business.
Non-native digital companies are involved in Digital Transformation, which is forcing changes within information systems that support business processes. Data migration is a critical element when a new information system is adopted. It doesn’t matter if the new system is bespoke or COTS.
Data migration is, conceptually, a simple task. Data migration consists in transferring data stored from a source system to a destination one without affecting operations. We might question where the complexity of migration lies, but considering the following:
- In source systems we can find the following situations:
- Data could be stored in more than one system.
- Source data could be incomplete.
- In destination systems it is necessary to consider:
- Data usually must be populated in a different format than source and data transformation it is required.
- Destination data model is still being defined.
- It is required to consider some technical constraints, study prerequisites to migrate specific data, balance the volume of data to migrate, data cleansing considering the timing of migration. These aspects show the complexity linked to data migration.
The migration process includes all tasks related to extract from the source system, clean, transform, validate and finally data load in the destination system. Altran organizes all tasks in the following phases:
- Planning is the initial phase of migration. This phase includes the following tasks:
- Identify stakeholders.
- Set KPIs for migration monitoring and tracking.
- Analyze all sources and destinations.
- Define the migration strategy (Big Bang or incremental approach, source shutdown or parallel run…).
- Decide the most appropriate ETL tools to use.
- Identify functional and technical prerequisites.
- Analysis and Design is the stage in which a detailed source profiling is performed and evaluated by key business users to define the needed actions to be carried out. Additionally, a staging area is designed as a working area, mirroring the destination environment, where extractions, transformations, data cleansing and validation processes are designed according to the profiling findings and defined business rules.
- Implementation is the most technical phase within a migration project. The scripts for extracting, transforming, cleaning, validating and loading are developed (by using ETL tools in some cases). All scripts are individually tested in the stating area. Once the scripts are individually tested, a complete set of migration tests are performed to validate results with key business users.
- Migration is the phase where source data is moved to destination systems, also when a parallel operation process, if required, is launched. It is the final phase of a migration project, when results are documented and the project is handed over to key business users.
ALTRAN has selected important challenges faced by any company within a migration process. Depending on how you analyze and address them, migration risks will be higher or lower.
Challenge #1: Parallel run decision
The migration strategy is defined along the planning phase, and parallel run is often one of the topics discussed with key business users. Altran believes that a parallel run migration strategy should be carefully taken, mainly because of the extra effort to maintain two systems in parallel, which can be challenging from both business and technical perspectives.
Parallel run complexity is commonly associated three simple questions:
- Is the master data stable for the duration of the parallel run?
- Is expected a large number of business events to be maintained in both systems?
- Are the business rules different between the source and the destination system? Are they going to remain stable for the duration of the parallel run?
If any of the above questions is answered with yes, it is not advisable a parallel run. For the cases where it is required to verify the results on the destination system, Altran suggests running the process in preproduction environments with a static snapshot (as month-end) to check the results with business users instead.
Challenge #2: Data cleansing
Source data profiling obtains a quality measure of the current data, but migration projects usually require data cleansing to achieve the desired data quality level (data quality KPI) in destination system.
Data quality KPI objective is determined considering the following parameters:
- Data volume affected.
- Time available.
- Effort required.
- Benefits obtained.
In order to identify priorities on data cleansing, Altran suggests classifying all cleansing topics by following concepts: business critical / non-business critical and mandatory data /non-mandatory data.
All topics classified as critical and required must be addressed during migration project and those classified as desirable could be prioritize depending on time available, the volume of data affected, considering the cost and the obtained benefit.
Challenge #3: Different source systems, different coding and a unique data
Quite often the same data is replicated in different systems, with one of the systems working as master and the rest as replica. However, sometimes, special cases may arise where the same data resides in multiple systems with different coding and there is no link between them. (e.g, a client company has contracted something as SMEs and it becomes a large company and it contracts different products / services). Special cases treatment requires analysis and agreement with key business users to make decisions about how to migrate them. Available options are:
- Group data in a single master data.
- Clear data and do not migrate it.
- Keep both master data and create hierarchy group for reporting purposes.
Challenge #4: Select ETL Tool
During the analysis phase, it is also necessary to analyze the destination systems (bespoke and Commercial of the Shelf, COTS). This analysis allows identify reference model and any technical requirements to load data. In general, bespoke systems load data directly in DB, but COTS systems usually have their own data import/ export tools. The function of COTS import/ export tools can be grouped in two different types:
- Type 1: to maintain specific traceability process within the COTS database (such as ERP).
- Type 2: Load tools to perform frequently loading and/ or extraction (such as CRM).
Once identified target systems and technical requirements for performing data loading, this table facilitates ETL tool decision for loading into target systems.
|Bespoke systems||COTS System|
|Load Tool Type 1||Load Tool Type 2|
|ETL tools must have connectors with DB destination.||You have to use the tool of the COTS system. No ETL tool can be used to load data.||ETL tools must have connectors with COTS target system.|
Challenge #5: Validation and Test Migration.
Before performing migration in production, it is required to validate the migration results with key business users and to establish the correct timing for the migration tasks.
Concerning the migration results validation with key business users, Altran recommends to test destination systems (in both integration and User Acceptance Test environments at least) with migrated data. Integration and UAT test cases will identify issues with migrated data within a real business context.
With the aim of establishing the correct sequence and timing of the migration tasks, Altran suggests a full process migration test with all or a sample of the data to be migrated. A test allows you to ensure that the migration tasks are properly identified, sequenced, included in a detailed checklist of the planned steps (where all previous, during and post migration tasks had been identified). Additionally, it is obtained an approximation about the required time for each migration task.
As summary, Altran identifies as key success factor in migration the following topics:
- Manage migration as a separate project to be able to have a picture of the whole migration project and to be able to address migration issues since de very beginning.
- Establish a trust relationship with key business users for defining migration strategy, data cleansing and data validation in all environments.
- Perform tests with migrated data and identify, in advance, any migration issues within business processes.
- Run a complete test of the whole migration process (with all or sample data).