It is data migration time on your SAP business project. Whether your project is implementation, acquisition, or merger, the goal is pretty much the same: the seamless inbound acquisition of master and transactional data from one or more external data sources while ensuring that this activity has minimal impact on the rest of the business. This is where we attempt to move years of neglected master and transactional data from a loosely structured, anything-goes legacy system into a very tightly integrated and highly structured SAP system. You must consider the likelihood that the concept of master data management had not been invented yet when the legacy or source system providing your data was implemented.
How much data to move? How much data to leave behind? What to automate, and what to execute manually? How to gracefully orchestrate and execute a data migration cutover from one system to another? Where and how to fit the data migration plan into the overall business implementation plan? How to continue to run the business during the data migration phase of the business project implementation? These questions are all part of the planning fun!
Data Migration Testing
Processes that we have exercised and continue to exercise on a regular basis are no big deal, do not cause anxiety, or raise any blood pressures. The daily report, the custom transaction, the interface that runs several hundred times a week – they are all second nature now. They happen, and no one notices.
But reflect back to the very first time these processes went live. This is precisely where you are with data migration. For most, it is a once in a lifetime event. It is important, then, to raise confidence levels, and reduce anxiety levels surrounding this activity. This is achieved by practice, practice, practice.
Typically, I like to see at least three data migration test cycles executed in an isolated data migration client. This allows for several attempts at exercising and fine-tuning the data migration plan; collecting nominal run time statistics to have some idea of how long the data migration might take; identifying and fixing any data migration program object defects; identifying and fixing any data mapping and content defects; and identifying and fixing any functional configuration issues. Each of these tasks are done with the goal of significantly improving the fallout rate with each data migration test cycle. These data migration test cycles also give us the opportunity to practice our legacy extract skills, our fallout analysis skills, and our fallout manual cleanup skills.
Source Data Quality
Each data migration test cycle begins with the legacy or source data extract. An important activity between data migration test cycles is the cleanup of the legacy or source data. The data migration process will most likely discover many data problems. Customer and vendor address data alone are enough to bring your bolt-on tax jurisdiction determination software to its knees – that zip code that is too short, too long, or just plain incorrect when combined with the associated city and state; the various abbreviations used in place of the actual city name; the city and state entered in the same field when they should have been entered into separate fields; etc. If the source data is not fixed, your data migration test cycles will begin to look like the movie Groundhog Day.
Source data provisioning and cleanup could very well be problematic, especially in an acquisition or merger scenario where the providers are part of a different organization and have no incentive to participate in your project. If you encounter this scenario, you will most likely need to engage the appropriate management level from your organization to have a serious discussion with the appropriate management level from the providing organization. Recognize this, be in control, and raise that flag early. Do not sit around for several days waiting for data to arrive, as this will only set your project behind schedule.
Data Migration Support for Other Testing
In any business project scenario, the development team will salivate at the thought of testing their customizations against your real data in the conversion test cycle box. Likewise, the functional team will be chomping at the bit to use your real data to test configuration scenarios. And the interface team and the data warehouse team can’t ever get enough data to play with.
While working within your three data migration test cycles, just say NO. You absolutely need a controlled environment for data migration test cycles, and these other teams will not respect that. They have a different focus and purpose which requires changing and manipulating degrees of freedom that you need held constant.
But, we are all on the same team trying to move the project to the finish line within the expected project timeline. So to help your other team members, plan for additional data migration cycles to provide this data to these teams. That’s more data migration practice for you, and it gives everyone else what they need.
The Perfect World of Data Migration
In a perfect world, for each data object to be converted:
- The functional specification and mapping documents are well-written and clear.
- A data load file is built exactly in conformity to the well-written functional specification and mapping documents.
- The data migration ABAP objects are built exactly as specified by the well-written functional specification and data mapping documents.
- No one is insisting that we move a square peg into a round hole.
In a perfect world, for each set of data that is to be migrated:
- The legacy system data has been thoroughly cleansed.
- The providers of the legacy data are genuinely interested in providing accurate data on time.
- Server-resident load files in a Unicode system are correctly encoded to UTF-8.
- A delimiter other than comma has been specified for the load file.
In a perfect world, the data migration test cycle client:
- Is built in a box that is not the development box.
- Is configured exactly like the client where the data migration ABAP objects were built and tested.
- Is not open for configuration, unless by design.
- Is not part of a transport path. to prevent the current cycle of conversion testing from being blindsided by any new configuration changes.
- Is locked down so that only data migration and data validation tasks can be performed.
- Is configured to handle a more background and update processes and fewer dialog processes.
- Is built in a box that has enough disk space to be the repository for the primary load files and any intermediate processing files needed.
In a perfect world, at the project level:
- There is an overall business cutover or implementation plan into which you can assimilate your data migration plan.
- The data migration plan integrates nicely into the overall business cutover plan.
- There is a strategy in place to bring the data current between the time the cutover freeze is enforced and the implementation date.
In the real world, the fun is just beginning. Those perfect world scenarios just never seem to happen. But stay tuned! In my next blog post in this series, I will discuss the details of the Data Migration Plan. After that, in subsequent posts, I will drill down into some real world scenarios that I have encountered and discuss how I dealt with them.