SAP Data Migration – Dealing With Fallout (Part 3)
One of the inevitable aspects of data migration is dealing with fallout from automated data loads. Typically, this process includes identifying the data that will not load, analyzing the error messages to determine the root cause, formatting a readable report that can be used as a tool in the cleanup process, and fixing the root cause of the problem so that it does not happen again.
Why the data will not load correctly.
There is a litany of reasons why some data records will load correctly while others will not. Here is a list of some common root causes:
- Poor quality legacy data. Legacy systems which are not as tightly integrated as SAP, and are not under master data control allow the end user a bit of freedom when entering data. A zip code may contain too little or too many characters; the email address is not properly formatted; numeric fields have transposed digits; various forms of abbreviations (especially in the city field), a quantity of zero (0) permitted by the legacy system and uploaded into a field where SAP will not accept a quantity of 0 and even simple misspellings all can cause stringent validation checks to trigger an error and prevent the record from loading at all. A more sinister type of error occurs when the data is functionally incorrect, but good enough to pass all of the SAP validity checks. In this case, the data record will technically load into SAP, but will not be functionally correct. Duplicate customers, duplicate vendors, and the data entry error for a quantity of 1000 instead of 100, and the wrong pricing condition applied to a sales order line are examples of this scenario.
- Functional configuration and supporting data effects.
Many times I have watched the load statistics for a data object plummet from near 100% in the cycle two test load to near 0% in the cycle three test load. This is very unnerving to the client because the cycle three test load is getting rather close to the go-live date, and “by the way, shouldn’t the statistics be getting better rather than worse?” Functional configuration changes can wreak havoc on any data load. Flipping the switch on a data field from optional to required; turning on batch management or serialization for materials for the first time; changes in the handling of tax, tax codes, and tax jurisdiction codes; that account determination entry that is missing or not set up correctly; a missing unit of measure or unit or measure conversion factor; the storage location in the upload file which does not exist in SAP – any of these can cause a load to drop mostly or completely onto the floor.
While change is inevitable on any project, it is important to control and communicate the change so that the downstream impact can be recognized and understood. Controlled change and communication always works better than total surprise. Perhaps if we all know ahead of time about that data field that is now required, we can impose a requirement on the data extract side to make sure that the data field is populated before it enters the upload file.
- Additional data in the upload file. Inserting a new field in the middle of the upload file data structure might be necessary for the business to close a gap, but if that change is not communicated to the technical team so that appropriate adjustments can be made to the load object’s input structures and processing logic, the new data will surely never load, and may cause misalignment of the data fields which follow it in the upload structure.
SAP PI CTS+: Letting CTS+ Out of the Bag to Get Better Change Management
Anybody familiar with older versions of SAP XI/PI understand that transporting interface development and configuration changes is often a prickly situation. Standard change management in PI relies on the manual packaging and processing of changes into files. These files have many issues:
- No documentation
- A different means of transport than standard SAP transports (need some training people used to ABAP transports)
- Manual audit accountability (what do you do if you lose an exported PI .tpz file)
To help resolve these issues, SAP released CTS+. But, what is CTS+ and how can it help?
SAP Data Migration – The Data Migration Plan (Part 2)
If you are responsible for the success of data migration, you will want to build a detailed plan that will walk you through all of the three phases of data migration: pre-data migration preparation, the data migration itself, and post-data migration cleanup. I like my data migration plan to contain detailed steps that ensure that I don’t forget anything. Each step lists a specific named responsible person along with their area of responsibility and contact information. Unless I am responsible for executing the task myself, I prefer the named person to be a client employee (i.e. the business owner of the process) rather than a project consultant. This is where the responsibility should be, and it requires that the business process owners actually participate in the activity rather than sit on the sidelines and watch.
Coming Soon: SAP .NET Connector (NCo) 3.0
SAP is announcing a new version of SAP Connector for Microsoft .NET 3.0 (now called “NCo 3.0″). A beta program for selected customers and partners is currently underway (Q3, 2010) with the general release of the software coming soon thereafter. I will highlight some of the major differences between the SAP Connector for Microsoft .NET 2.0 and NCo 3.0 (besides the obvious, and much-needed name-shortening).
SAP Data Migration – Answering the Important Questions (Part 1)
It is data migration time on your SAP business project. Whether your project is implementation, acquisition, or merger, the goal is pretty much the same: the seamless inbound acquisition of master and transactional data from one or more external data sources while ensuring that this activity has minimal impact on the rest of the business. This is where we attempt to move years of neglected master and transactional data from a loosely structured, anything-goes legacy system into a very tightly integrated and highly structured SAP system. You must consider the likelihood that the concept of master data management had not been invented yet when the legacy or source system providing your data was implemented.
How much data to move? How much data to leave behind? What to automate, and what to execute manually? How to gracefully orchestrate and execute a data migration cutover from one system to another? Where and how to fit the data migration plan into the overall business implementation plan? How to continue to run the business during the data migration phase of the business project implementation? These questions are all part of the planning fun!
What’s in a Naming Convention? Part II
In my last post, I discussed naming the naming convention that DataXstream recommends for SAP PI Integration Directory (ID) objects. I would like to say that I had a great DataXstream ESR-specific naming convention, however the SAP naming convention guide for PI 7.1 does the job perfectly. Here is the link to the PI 7.1 naming convention guide http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/40a66d0e-fe5e-2c10-8a85-e418b59ab36a?QuickLink=index&overridelayout=true . I would like to point out some things that I feel most people miss, as well as some things that I think are particularly interesting.
Object Name Prefixes
I have seen a lot of places use prefixes or suffixes before objects such as or dt_ or _mt. I have never really been a fan of these prefixes and suffixes, and apparently neither is the above linked naming convention guide. The reason being is that they are unnecessary. It would be difficult to confuse a message type with a data type in a real interface scenario, even if troubleshooting an unknown broken object. Plus it makes message mapping unnecessarily confusing and long since it’s not clear whether to add the prefix, i.e. MT_one_to_MT_two. Good descriptive names are usually all that you need e.g. DEBMAS_to_Customer. The only possible exception to a no-suffix-or-prefix-policy is the service interface, as sometimes it is useful to know which direction and type a service interface is. An argument for a prefix or a suffix to describe a service interface would be to assist in understanding the flow from an SAP perspective in the event that someone who didn’t develop the interface had to come behind and troubleshoot. A argument against would be the fact that it looks silly if you use it for a web service, because you have a name that doesn’t mean anything to a third party user (note operation mappings are told to omit the prefix of direction and mode in the SAP guide).
One interesting thing that I noticed in my investigation of PI 7.1 EHP1 is that it appears that naming conventions on PI can be validated. If the object names do not conform to a naming convention a message will appear. Ter perform this check in the ESR go to menu Tools>Component Check.
Select “Governance” and “Interface Name Checks”:
If the service interface does not end with (In/Out)(SYNC/ASY) the interface check will show as an error (does not impact interface processing). I created 2 interfaces: one good and one bad to show this error.
My suspicion is that SAP will put more in place to force consistent naming standards depending on the service interface pattern in future releases.
Using a software component and Namespace for each “side” of an interface
All objects of an interface should not be grouped in a single namespace. They should to be split among the Software Component Versions of the systems being interfaced. Otherwise when you go to configure, you will not be able to see your operational mapping (OM) in the dropdown box without having to select all in the dropdown menu. A general rule of thumb is: if it’s not easy to configure, odds are you have probably done something wrong. Another reason why an object might not appear in the dropdown menu (for example for an operations mapping on an interface mapping) would be if the installed checkbox is not clicked on the SLD. When done correctly, most interfaces should be able to be configured quickly and intuitively in the integration directory (ID) without the need to select from all SWCV from dropdown menus on the integration builder.
Whatever naming convention you choose to use for the ESR, the important thing to remember is that adhering to the standard makes production support and troubleshooting faster and easier.
SAP Solution Manager (SOLMAN) Template Projects
In my previous post I discussed some of the functionality available in Solution Manager (SOLMAN) to capture and build business process scenarios, processes and steps. Once you captured this information it could be readily transformed into a test plan and SOLMAN functionality used to execute, manage and monitor testing.
This is good functionality and once your business scenarios, processes and steps are in SOLMAN you can use them as a baseline for additional projects.
Consider the “global template” scenario that crops up in companies today: a core set of business processes are designed and rolled out on a global basis – the only deviations allowed are those mandated by local legal requirements (a.k.a. localization). On top of this there are the business scenarios that fall outside of the business template. You don’t want to build a standard implementation project from scratch each time for the core business processes. This is where template projects save you time and effort.
Read more
Using Native SQL in an ABAP Proxy
Recently, I was looking at a requirements document to build an interface to an external system that wants to query customer master data by the customer first name and last name. As I read this, there were a cacophony of thoughts, all demanding equal attention, racing through my head:
- How will I ever match the inbound interface parameter “Tom” with “TOM”, or “tom”?
- How will I ever match the inbound interface parameter “Smith” with “SMITH” or “smith”?
- The ABAP WHERE clause is not case-INsensitive.
- There could be hundreds of customers named Tom Smith.
- KNA1-NAME1 and KNA1-NAME2 are not indexed fields.
- And no, we are not storing any portion of either first or last name in an existing indexed field like SORTL.
- There are well over one million customers in the database.
- We have already decided to use PI for all interfaces.
- I will have to buy the BASIS team a case of beer to get them to agree to create indices on the fields KNA1-NAME1 and KNA1-NAME2 in a table with over one million records.
I arrived at the conclusion that I need a case-insensitive database query, along with database indices created for the fields KNA1-NAME1 and KNA1-NAME2.
But, what is a case-insensitive WHERE clause? A little research and help from colleagues revealed that many had gone before me, and this was nothing new. To implement a case-insensitive WHERE clause in ABAP, you simply needed to use the native SQL UPPER() construct. The database system that is being used is Microsoft SQL Server, but the UPPER() function and its syntax is similar across different database platforms. This seemed like an easy nut to crack. But, as I soon found out, I actually had a lot to learn. Read more
SAP Upgrades, Solution Manager, Test Plans, and Testing
In this entry I’ll pull together a few threads from previous posts (testing, documentation, upgrades and offshore development) and throw in some Solution Manager (SOLMAN) functionality. These pieces can be fitted together to help accelerate a project and testing preparation via use of the SAP Test Workbench.
Nowadays SAP wants you to use SOLMAN to manage your landscape and use it as the main conduit to interact with the mother ship in Walldorf. Lots of SAP installations use SOLMAN as a way to generate developer keys and as a document repository: valid uses, but only a small fraction of the available functionality. Let’s explore some more of that SOLMAN magic. Read more
What’s in a Naming Convention?
It doesn’t take long for a PI implementation to become a complete mess if standards are not put in place before development occurs. Even among seasoned developers, opinions vary as to the best way to name and organize the IR (ESR) or ID (IB), depending on different backgrounds and previous project experience. In the next few blogs I would like to touch upon some of the best practices that we utilize in our implementations at DataXstream, which we have arrived at though our project experiences and discussion, both internally and with other middleware experts. Read more
