Why the data will not load correctly.

There is a litany of reasons why some data records will load correctly while others will not.  Here is a list of some common root causes:

1. Poor quality legacy data. Legacy systems which are not as tightly integrated as SAP, and are not under master data control allow the end user a bit of freedom when entering data.  A zip code may contain too little or too many characters; the email address is not properly formatted; numeric fields have transposed digits; various forms of abbreviations (especially in the city field), a quantity of zero (0) permitted by the legacy system and uploaded into a field where SAP will not accept a quantity of 0 and even simple misspellings  all can cause stringent validation checks to trigger an error and prevent the record from loading at all.  A more sinister type of error occurs when the data is functionally incorrect, but good enough to pass all of the SAP validity checks.  In this case, the data record will technically load into SAP, but will not be functionally correct.  Duplicate customers, duplicate vendors, and the data entry error for a quantity of 1000 instead of 100, and the wrong pricing condition applied to a sales order line are examples of this scenario.
2. Functional configuration and supporting data effects. Many times I have watched the load statistics for a data object plummet from near 100% in the cycle two test load to near 0% in the cycle three test load.  This is very unnerving to the client because the cycle three test load is getting rather close to the go-live date, and “by the way, shouldn’t the statistics be getting better rather than worse?”  Functional configuration changes can wreak havoc on any data load.  Flipping the switch on a data field from optional to required; turning on batch management or serialization for materials for the first time; changes in the handling of tax, tax codes, and tax jurisdiction codes; that account determination entry that is missing or not set up correctly; a missing unit of measure or unit or measure conversion factor; the storage location in the upload file which does not exist in SAP – any of these can cause a load to drop mostly or completely onto the floor.

   While change is inevitable on any project, it is important to control and communicate the change so that the downstream impact can be recognized and understood.   Controlled change and communication always works better than total surprise.  Perhaps if we all know ahead of time about that data field that is now required, we can impose a requirement on the data extract side to make sure that the data field is populated before it enters the upload file.

3. Additional data in the upload file. Inserting a new field in the middle of the upload file data structure might be necessary for the business to close a gap, but if that change is not communicated to the technical team so that appropriate adjustments can be made to the load object’s input structures and processing logic, the new data will surely never load, and may cause misalignment of the data fields which follow it in the upload structure.

The Finger Pointing Game

It’s the load program!  No, it’s the data!  No, it’s the configuration!  No, it’s … (fill in your favorite finger pointing game excuse explaining why data will not load).

If you ever find yourself in the midst of this type of finger-pointing game, immediately stop the madness.  For this and similar situations, I apply a simple litmus test which has never failed me yet – manually enter the EXACT upload data into SAP for the transaction which has failed.  If one can be entered manually, then the program will be able to automatically load thousands with similar upload data and functional configuration.

I have lead many a functional analyst – kicking, screaming, and ranting about how terrible the load program is – to the terminal to play “let’s enter one manually”.  Typically, the result is that one cannot be entered manually due to configuration issues or the lack of supporting values data.  Once these issues are cleaned up, the load program “magically” begins to process thousands of records with no trouble at all.

Sometimes, the load object appears to be the cause, but is not the root cause.  The important data item not being handled by the load object (which was not called out in the functional specification document), the data item which turns blue (because the functional specification document explicitly stated “put this data item into the blue category”), the formula which is not calculating the desired result (but is indeed the exact formula found in the functional specification document) – all are examples of the load object adhering accurately to an incorrect functional specification document.  The root cause, then, is the functional specification document which must first be revised and checked into the controlled document repository before making any code modifications.

It’s the Program

Well, OK, sometimes the load object is at fault.  But it is extremely rare.

Collecting and Reporting the Technical Load Statistics

Load statistics are an important technical metric, indicating what percentage of the upload file has successfully posted a transaction into SAP.  This is a basic and simple record count check.  How many records were presented in the upload file, how many records posted successfully to SAP, and how many records failed to post.  Also, does the number of successful transaction plus the number of failed transactions equal the total number of records presented in the upload file.

Here is how I communicate this technical metric:

This technical metric indicates only that all of the SAP validation rules for posting the transaction have passed.  It does not indicate that the master or transactional data posted to SAP is actually correct in functional terms.

The Functional Review

It is very possible for an entire upload file to technically load at 100%; while at the same time, functionally fail at 100%.  The customer may be missing a partner, the material or article may be categorized incorrectly, the pricing conditions on a sales order may not have the desired validity date range, the GL posting may be to the wrong accounts.  Hence, the need a functional review and validation of the data transacted into SAP.  The data type – master or transactional – determines the type of functional review and metrics to be employed here.

For master data, such as of materials, customers, vendors, etc., it is impossible to individually validate the many thousands of entries in your SAP system.  But statistical methods can be employed which will guide you through a random sampling of the data, while at the same time assuring accuracy at a high degree of confidence levels.

For transactional data, such as inventory, open sales orders, open AR, etc., direct mathematical comparisons can be employed.  On a grand scale, if an inventory value of $15,246,321.44 is being moved from your legacy system to SAP, then that exact amount must arrive in SAP when the migration task is complete.  It may be a bit more difficult to do this mathematical comparison at a more granular level.   If, in the move to SAP you are also redesigning your material/storage location combinations, a direct comparison between the legacy system and SAP may not be possible without a translation factor.  The same scenario exists if you redesign your GL chart of accounts, where one legacy GL account now maps to several SAP GL accounts, or vice versa.

Reporting the Fallout

If an automated data load was not technically 100% successful, a clear set of error messages complete with a link back to the legacy data must be mined, formatted, and presented to the business for analysis.  Such a report really helps to facilitate the fallout cleanup.  The link back to the legacy data must be carefully designed into the load process to make sure that, for example, the legacy customer number, legacy vendor number, legacy sales order number, legacy purchase order number, etc. is included as part of the data being handled.  Without the link back to the legacy data, it becomes very difficult to identify which data record needs to be fixed.

The error message mining technique that I use depends on the load method.  I will describe two here – one for a BDC load method and one for an IDOC load method.

Mining Meaningful Error Messages from a BDC Session Log

At the completion of a batch input session, the batch input session overview screen (SM35) displays the technical load statistics.  In this example, out of a total of 229 transactions, 13 failed and 216 succeeded.

The session log shows the status of all 229 transactions.  This screen snapshot is a fragment of the complete session log for the batch input session.  It shows many successful transactions (Type = S) and one failed transaction (Type = E).  The error message here is clear – the article does not exist or is not activated.

But as you can imagine, the 13 failed transactions with error type = E are sprinkled throughout the many pages of this log file.  With only 229 transactions, this log file is quite easy to pick through to find the 13 errors.  But imagine if the number of transactions were in the thousands or tens of thousands.  How do we extract only the failed transactions and present a concise report of the failed transactions?

To do this, I use SAP transaction SM35P – Batch Input Log Overview.  This transaction has the ability to set a filter on any field in the batch input log file, display the filtered results, and then to export the results to a local file.

To enter the mode where this is possible, first press the PRINT icon.

Next, set the filter.  The appropriate filter field here is SESS. TYPE.

We only want the errors, so set the filter for field SESS. TYPE = E.

The display now shows only the 13 rows containing the error messages.  This can be exported directly to a local spreadsheet for further analysis.

Mining Meaningful Error Messages from IDOCs

Depending on the IDOC and the processing module, mining the error status messages from IDOCs can be very easy or somewhat challenging.  For the more difficult scenarios, you probably will need to hone your EXCEL skills to properly join several extracts together into one complete picture.

When creating IDOCs with a load object, I always note the date, time, and IDOC basic type.  I will use this information as the selection criteria for transaction WE05, which is going to locate the IDOCs and display the results information I need after the IDOCs are processed.  The results that I usually collect are the error status messages, and some data content from a segment or two to illustrate exactly where the problem is in the legacy data.

While the background job is busy processing the IDOCs, I usually take a peek, using transaction WE05, to see how the load is progressing.  If I see that most of the IDOCs are falling onto the floor (IDOC status 51) rather than moving into the database (IDOC status 53), I usually stop the background job and begin an immediate analysis of the fallout.  If the fallout solution does not require a change to the IDOC data content (e.g. a configuration change), then I can replay the fallout using SAP transaction BD87.  If the fallout solution does require a change to the IDOC data content, then the complete set of IDOCs must be regenerated again.

Let’s see what WE05 can tell us about a completed Article Master load.

On the WE05 screen below, we can see that of the total of 14,005 IDOCs, 11,412 have processed successfully and 2,593 have failed to process.  By double clicking the Status 51 folder, the display will show only the Status 51 IDOCs – the fallout.

By pressing the “status list” icon (shown above), the display will show the status messages for the fallout.    Once these messages are displayed, pressing the “export” icon allows me to save the screen contents to a spreadsheet.

Now it would be really nice if I could have the article number in the spreadsheet right next to the error message.  The article number in the ARTMAS IDOC is stored in segment E1BPE1MATHEAD.  The segment content for each IDOC can be displayed by pressing the “list specific segment” icon and entering the segment name in the box.

The segment display will show all fields in the segment, so I usually hide all of the columns that I don’t want to see.  Here is the E1BPE1MATHEAD segment display showing only the article number.  I can use the export icon to save the list of article numbers to another spreadsheet.

Here is a portion of the complete spreadsheet showing the error messages and the article numbers side by side.

A filter applied to the spreadsheet shows that the 2,593 errors are all grouped into one of three error status categories.  By selecting a single category, Excel will also show me the number of records within that failure category.

Sometimes it is easier to mine the status messages directly from the IDOC status table EDIDS.  This is especially true where the processing module is a BAPI which returns an error table rather than a single error message.  In this case, when you press the “status list” icon in WE05, only the first error status message of several is displayed for each IDOC.  I find that the first message is not very helpful (as shown below).  I also find that typically the second or third message in the return status table is usually the important one.  You won’t see it displayed on the WE05 screen, but you can mine it from the EDIDS table.

SAP transactions SE11 or SE16 both support this activity.  For the selection criteria I use the IDOC number range, status 51, and status type E.  On the display screen, choose only the relevant fields for display – the IDOC number (DOCNUM), IDOC status (STATUS), status message (STATXT), the four substitution parameters for the status message (STAPA1, STAPA2, STAPA3, STAPA4) and the message type (STATYP).  All of this can be exported into a spreadsheet.  If you really want to test your Excel skills, you can write code that will move the substitution parameters into their placeholders in the status text.

Preparing for the next data migration cycle – Let the fallout analysis and cleanup begin.

Presenting the fallout report to the business with a set of clear error reasons and links back to the legacy data is key to enabling the legacy data cleanup process to proceed.  In the iterative process of data migration cycles, cleansing the legacy data is a step in the right direction towards an improved next conversion cycle.

I hope you enjoyed this blog series on data migration.  Please feel free to send comments or questions.

Service Desk functionality is delivered to you for use as a ticketing system.  One of the features of it is that it can be used as a ticketing system for both SAP and non-SAP systems as well as in conjunction with other ticketing systems that may be in place already.  In this blog post I’ll briefly touch on some of the scenarios I have encountered and show that there are several ways to deploy Service Desk.

Using Service Desk is beneficial because it can automatically capture a wealth of information about what a user was doing when a problem occurred if the ticket is created directly from SAP.  Also, Service Desk can communicate directly with the SAP mother ship to log issues and manage OSS notes which obviously reduces the risk of transcription errors.  And Service Desk can be extended to include functional components from non-SAP systems which in turn leads to the possibility of one-stop-shopping for ticket management.

Here are some SAP Service Desk scenarios I’ve come across.

New SAP Installation

If you are a new SAP installation and have no ticketing in place Service Desk is a great place to start.  As mentioned the functionality available allows end users to create tickets directly from SAP and have workflow set up to manage ownership and eventual resolution.  Alternatively, a URL can be made available to allow trouble tickets to be logged via the SAP Service Desk web front end.

In this case end users and technical support personnel learn a new tool as part of their go-live experience.

Migrating Non-SAP System Support to SAP Service Desk

Rather than SAP being your sole IT application it is much more likely that you have an existing portfolio of systems and an associated ticketing tool and procedures for ticket management, escalation and resolution.  A conservative approach when bringing up Service Desk is to use it initially only for SAP related tickets and deploy a limited set of functionality.  This allows you to go through the growing pains and adjustments as you learn how to best make Service Desk work for you.

In the longer term migrating ticketing for non-SAP systems to Service Desk brings it all into one system.  Using the Service Desk capability to create custom components and to build an organization structure that fully supports your users and systems facilitates consolidated reporting and the ability to monitor Service Level Agreements (SLA).

This approach works best if you have a fairly simple environment where the level of disruption is limited when decommissioning an existing ticketing.  The impact on people, process and technology is carefully managed and the working environment has some tolerance for the time if and when things aren’t quite as expected at first.

SAP Service Desk Integration with an Existing Ticketing System

In situations where there is a well established ticketing system it is unlikely that that system will be easily displaced by SAP Service Desk and it may not make sense to do so.  In this situation you can choose to not use Service Desk and forgo the system capabilities and instead use the existing ticketing tool to manage SAP issues.  This gives up a lot of delivered SAP Service Desk capability.

Alternatively you can integrate the existing ticketing system and Service Desk and reap the benefits of both worlds.  In this case you need to decide which system is the primary ticketing system and which is the secondary.  DataXstream has worked with CA (formerly known as Computer Associates) to integrate the CA Service Desk offering with SAP Service Desk and now CA delivers a packaged solution for this integration.

SAP Service Desk is the Ticket Initiator

In the scenario where the SAP Service Desk (SAP SD) is primary tickets are initiated in SAP SD and categorized either as SAP related or as non-SAP related.

SAP related tickets are managed to resolution in SAP SD, including any communication with the SAP mother ship, and the secondary ticketing system, e.g. CASD, receives only informational updates on ticket progress, if required.  The complete ticket lifecycle is managed and closed in SAP SD.

Non-SAP related tickets are transferred to the secondary ticketing system, e.g. CASD, where they are worked to their resolution.  In this case ticket status changes and informational updates all occur in CASD and these updates are relayed to SAP SD.

In either case SAP SD and CASD can have a full picture of the ticket status throughout its lifecycle, the main difference being which system “owns” the ticket and has responsibility for working it and closing it out.

Overall the established ticketing system continues its original function and ticket resolution process, the key change being where a ticket is initially created.

SAP Service Desk is NOT the Ticket Initiator

In the scenario where SAP SD is the secondary ticketing system SAP-related incidents originate in the primary ticketing system, e.g. CASD, and are relayed to SAP SD.  Now the SAP related incidents are worked in SAP SD where ticket information can be supplemented with all the SAP unique information that would be useful for analysis and investigation.  Similar to the earlier scenario any significant status or informational updates in SAP SD are relayed to CASD and the ticket is ultimately closed out in SAP SD.

Any non-SAP related tickets are not sent from CASD, for example, to SAP SD.

A Natural Progression: from Island to Integration

SAP Service Desk can be deployed in a variety of ways in an organization: it can be a standalone ticketing and support system; it can be fully integrated with existing support applications.  The role played and the pace of adoption is very flexible and is driven by an organization’s ability to manage the change and adopt new tools and procedures.

The general progression I have seen is to bring up SAP Service Desk as a standalone tool and then after a settling in period one of two things happens:

• Existing ticketing tools and functionality are migrated into Service Desk
• SAP Service Desk is integrated with another ticket management tool

In either case the outcome usually allows SAP related tickets to be created directly from SAP in order to capture as much detail as possible at the time of the incident.

• No documentation
• A different means of transport than standard SAP transports (need some training people used to ABAP transports)
• Manual audit accountability (what do you do if you lose an exported PI .tpz file)

To help resolve these issues, SAP released CTS+.  But, what is CTS+ and how can it help?

What is CTS+?

CTS (with out the plus) is stands for SAP’s Change and Transport System.  This is the transport capability of the SAP NetWeaver AS ABAP.   SAP has enhanced CTS with the capability to transport non-ABAP objects–hence CTS+.  Change management on SAP NetWeaver AS ABAP has been a strength of SAP Netweaver for a very long time. In SAP PI, CTS+ leverages this strength by allowing Enterprise Service Registry (ESR) development objects and Integration Directory (ID) objects on the Java stack to be exported and packaged into standard transports that reside on the ABAP stack.

How Can CTS+ Help?

Documentation

With CTS+ you have the ability to document your transport when you create your request. This is amazingly helpful when it’s time to go to production, as you can see what changes were imported and when they were imported. Documentation is also helpful for the Integration Directory (ID) transports, which require activation and input of values.  To get to transport organizer, export what you want to export (namespace, scenario, etc.) using CTS+ (which should be default once it is set up).  Click on create/modification of request and you will see a screen like this:

Here you can put in details of the transport in the description as well as configure various other parameters about the transport.

Additionally, if your project has transport naming standards that require RICEF ID, defect numbers, change request number, trouble ticket number, etc., you can add that information here.

Standardization of Transports

With CTS+ it is possible to configure a standard way of handling transports that is consistent with your ECC environment. This will improve the overall reliability and robustness of environment change management, and will make life a lot easier for the people responsible for handling transports.  Someone accustomed to ABAP transports in ECC but who hasn’t worked with PI can jump in and change management with CTS+ will make sense. CTS+ also allows SAP PI to be compliant with your standard SAP change management governance.

For More Information

CTS+ is available with AS ABAP and AS Java from SAP NetWeaver SPS 12 onwards, although you will have better results with SPS 14 or higher. The following OSS Notes will be helpful:

• 1003674 Enhancement for non-ABAP systems in CTS
• 1145268 CTS+: Changes from NW 7.0 SP12 -> NW 7.0 SP13
• 1146170 CTS+: Changes from NW 7.0 SP13 -> NW 7.0 SP14

In short, CTS+ is a great tool for transport management. It is totally integrated with SAP PI, it is easy to set up, and has huge advantages when it comes to documenting transports and maintaining consistency with ECC standards.

Before you embark on your project’s first data migration test cycle, your data migration plan will probably start out as a simple list of preparation steps, data load dependencies, and cleanup steps.  As you progress through your several data migration test cycles, your plan should evolve into a rather complex list containing load methods, responsible persons, historical data volumes, historical execution times, steps that need to be modified and additional steps that need to be added.

I always find it rather interesting to compare the very first plan to the plan that is actually used in the cutover process.  Many project factors will cause you to amend your data migration plan.  Newly-discovered gaps, changes in the business requirements, the decision to load all of the legacy customers instead of only those customers which have been active for the past two years, and toggling back and forth between “should this one be a manual or an automated load” are only a few.   Always keep in mind that the goal in amending the plan is to keep it compatible with the changing requirements of the project.

The Data Migration Preparation Phase

Before you actually start the data migration main course, you will probably want to execute a series of preparation steps.  Some of these preparation steps may have a significant lead time; while others may require a system reboot to take effect.  Make sure that you carefully plan accordingly.  Here are some data migration preparation steps that I have found useful:

• Obtain a Data Migration Login Obtain a special data migration login, with all of the authorizations needed to execute all of the required tasks.  This login is only to be used during the data migration process, and is to be deleted upon the completion of the data migration process. Using the data migration login makes identification of migrated data very easy both now and several years down the road.  It also facilitates implementing a general user lockout while the data migration user is performing data migration activities.
• Technical client preparation The BASIS team may want to configure the data migration target client to have less dialog processes and more background and update processes.  This configuration can significantly improve data migration performance, especially if you have the opportunity to parallel process IDOCs.  In a Virtual Machine environment, the infrastructure team can even provision additional processors and memory.  You may also need additional disk space to handle large upload files, and any intermediate files that a load program might need to create.
• Configuration for IDOC processing If a data migration process involves creating and processing IDOCs, you may need to configure the ports, partner profiles, message types, and processing modules.  My preference for IDOC triggering is by using the ABAP program RBDAPP01 rather than immediate triggering.  This allows me to exercise the most scheduling control over the background processing options.
• Change Pointers During a data migration turn off change pointers.  Creating what could potentially be hundreds of thousands of change pointer records that are never going to be processed is an overhead burden on system performance and disk space that you do not need.  If the migrated data needs to be sent to external systems, manually send the data after the data migration is complete.
• Set the Correct Open Posting Period Some of the transactional data will want to post into an open financial or manufacturing posting period.  You will want someone from the business to make sure that the correct posting periods are open for business.
• Configuration Settings Sometimes, special temporary functional configuration settings are made to facilitate the data migration process.  Several functional analysts may be responsible identifying and setting these prior to the start of data migration.  As these are temporary configuration settings, it is expected that they will be set to their production values after the data migration is complete.
• Basic Setup Data Some data migration objects are dependent on reference data – a template from which new data migration objects can obtain some of their default values.  You will want to make sure that those who are responsible for setting up this reference data have completed their task before you start the actual data migration.  Sometimes this data is moved from the Golden Client into the target client via ALE, so you may need to set up RFC destinations, ports, distribution models, etc.
• General user lockout All users except for the data migration user should be locked out of the data migration client during the actual data migration activity.  It is impossible to verify that we moved exactly $15,386,254.23 in inventory from the source system to the target system if users are performing material movements in either system during the conversion.
• Connections to External Systems Depending on what connections to external systems are doing, you may want to make sure that they are disconnected prior to the start of data migration.  This will ensure that inbound interfaces, which may either modify existing migrated data or add new data, are disabled.

The Data Migration Phase

When all of the preparation steps are complete, it is time to start the actual data migration.  Data must be loaded into the target system in a certain sequence that is dictated by their data dependencies.  Customers must exist before sales orders can be migrated, vendors must exist before purchase orders, materials must exist before either sales orders or purchase orders, etc.  This dependency exercise varies a bit from project to project, and should have already been determined well before you start loading the data.

Automated data migration tasks for a single object (e.g. customer master data) usually include some or all of the following:

1. Creation of an upload file from the source system
2. Technical validation of the upload file
3. Functional validation of the upload file
4. Execution of a technical object to load the data into the target system
5. Technical validation of the migrated data
6. Functional validation of the migrated data
7. Collection and reporting of load statistics
8. Analyzing the fallout
9. Correcting the fallout
10. Reprocessing  the fallout

Checkpoints

At a higher level, the data migration plan will string together an end to end list of all of data migration objects.  I strongly suggest that you study the list and judiciously intersperse several checkpoints at appropriate places in the plan.  A checkpoint is an activity where all interested parties meet to review status and metrics, and to determine whether or not to proceed forward.  Keep in mind that what fails to load early in the plan will geometrically cascade into larger fallout later in the plan.  That single vendor which failed to load may cause several hundred open purchase orders to also fail.  And while the business owner of the vendor master data may think the fallout of a single vendor to be quite insignificant, the business owner of open purchase orders may have an entirely different perspective.  It is important here to make sure that all of the right stakeholders are participating, and that all are in agreement about whether or not to proceed.

Backup or Restore Points

Another activity that you should judiciously sprinkle into the plan is several backup or restore points.  It is very painful to have successfully navigated seventy-five percent of the data migration plan, only to have the next load create a huge and unrecoverable mess in the target system.  Without a backup or restore point available, you may have to wipe the slate clean and restart the entire data migration process again.

Depending on the target system and the facilities available, the backup or restore point could be a full system backup, a client copy, or a virtual machine snapshot.  Whatever the means, make sure it is a part of your plan.  It’s really great if you don’t need it.  It’s also really great to have if you do need it!

The Data Migration Cleanup Phase

When the data migration loads are finally complete, it’s time to exercise a series of cleanup steps.  This is where we:

1. Undo the special temporary settings and configurations that were done prior to the start of the data migration activity.
2. Adjust the technical configuration of the system to a more user-oriented system with the right amount of dialog, background, and update processes.
3. Turn change pointers back on.
4. Allow the general user population to access the system.
5. Connect the system to its inbound and outbound communication partners.
6. Disable the data migration user.
7. Archive those massive upload files.
8. Delete any intermediate processing data files.
9. Clean up the IDOC tables if needed.  But first make sure that you do not need to keep the IDOCs  for legal reasons.

An Example Data Migration Plan

Click on the link below to access an example data migration plan.  This plan is not a complete data migration plan, but is intended to demonstrate format and content. Sample Data Migration Plan

Stay tuned!

In part 3 of this series on Data Migration, I will be discussing how to deal with fallout from automated data loads.

NCo 3.0 Logon

One of the new features implemented to combat the malevolent for RFC client attacks is the interface IDestinationConfiguration. This interface contains method IDestinationConfiguration.GetParameters(string destinationName) which allows .NET programmers to provide their own secure method for storing and retrieving logon information, be it encrypted database, encrypted file, or even LDAP.  There is also a corresponding interface and method to secure NCo 3.0 RFC servers.

Additionally, there is updated functionality to support single sign-on (SSO) and X.509 certificates.

Say Goodbye to Data Containers and Generated Code

In SAP Connector 2.0, working with RFCs required the generation of proxy code via the SAP .NET Connector design-time tool.  This tool would convert the IMPORTING, EXPORTING, CHANGING, and TABLES parameters from the RFC you were calling/serving to .NET representations of the same.  With NCo 3.0 there is no longer any kind of generated code.  Instead of one generated proxy method for each function module, there is one single RfcFunction class, whose Invoke() method dynamically executes every given ABAP function module. Additionally, all ABAP parameters will be represented by the class RfcStructure instead of a dedicated generated class for every structure; All tables will be represented by class RfcTable.  So, instead of hard-coding all of the data and variable bindings statically at design time, NCo 3.0 now handles everything dynamically at runtime.

NCo 3.0 uses SAP’s data dictionary to determine the function interface for the called RFCs.  While this data is cached locally for performance reasons, NCo 3.0 is robust enough to detect changes in RFC interface signatures so your code will still execute if you’ve added, deleted, or changed RFC parameters.  If there arises a situation where the dynamic lookup of RFC interface signature is not wanted, there is also an option to hard code the RFC parameter metadata.

The biggest benefit of NCo 3.0′s new dynamic function handling is that NCo 3.0 no longer has a design-time component.  This releases the Visual Studio version restriction that .NET 2.0 had.  Let me rephrase that, NCo 3.0 will work with any version of Visual Studio you like! The only requirement is that at runtime, a .NET Framework version is installed that is compatible with the NCo 3.0 libraries.

Anxiously Waiting

NCo 3.0 figures to be a much-needed update to a vital SAP integration technology.  I, for one, cannot wait for the new functionality to be generally released.  Once NCo 3.0 is released we’ll be posting blogs highlighting the new features.  We’ll also include helpful information regarding a .NET Connector 2.0 to NCo 3.0 upgrade.  In the meantime, if you have any leave any questions or comments you have regarding NCo 3.0 below.  You may also email me directly at cstasila@dataxstream.com if you’d like to set up a detailed conversation.

The Infallible 2-by-2 Matrix and the Magic Quadrant

A long time ago I worked for one of the big consulting firms where every interaction with the client seemed to involve a 2-by-2 matrix.  It got to the point that some of the consultants and most of the client laughed whenever one of these was pulled out.  It became a contest to create 2-by-2 matrices for everything and at one point in my life I aspired to publishing a book of 2-by-2 matrices to enable and empower everyday people to make good life decisions.  I think someone beat me to it at least in spirit.

I think the world has become conditioned by 2-by-2 matrices such that the goal is to always be in the top right hand corner – the so called magic quadrant – whereas being in the bottom left hand corner shows how much progress you still have to make.  As highly paid consultant I am supposed to help you find the path from where you are to where you want to be.  The funny thing about 2-by-2 matrices is that you can make them do whatever you want to serve your needs.  Take the example of where I should go for dinner tonight:  If I plot on one axis how likely the bill will conform to the client expense policy and plot on the other axis how well the restaurant menu prices align with my per diem I get something that looks like the graphic below.

If I follow this chart and adopt the recommendation of the magic quadrant I’ll be eating fast food every night because it’s cheap.  Unfortunately, I’ll feel like Morgan Spurlock in no time.

If I look at this chart with a more skeptical eye the two axes are showing similar information, so the whole thing is pretty bogus.  If you are going to use a 2-by-2 matrix to make a point with a client make sure it serves a useful purpose.  One of my favorite matrices plots employee/consultant willingness against ability as shown below.

In an ideal world all workers fall in the magic quadrant, i.e. are both able and willing to perform a certain task (could this be the nirvana of a low maintenance, competent and talented worker?).  The individuals who fall into other quadrants are potentially much more interesting and I would wager that not every worker falls in the same quadrant all the time.  After all, I’m not a big fan of “busy” work: I am able to do it but not very willing because I don’t value it.  But when it comes to blogging I am both willing and able (obviously!) [Ed. Note: You, sir, are a very able blogger!].  Workers who are able but unwilling present a special set of challenges for project management (you can lead a horse to water but you can’t make him drink), whereas workers who are willing but unable may be lacking skills they want to acquire.

The takeaway here is that there is a place for 2-by-2 matrices in conveying a message: stating an objective; starting a discussion, etc. but use them sparingly and only when it is relevant.

In the meantime, send me your suggestions for 2-by-2 life decision matrices and I’ll see what I can about giving you a credit in my forthcoming publication: Top Right Sure Beats Bottom Left.

It Depends

I don’t know anyone working on SAP projects that doesn’t laugh or roll their eyes when someone says “it depends”.  As much as I mock it, the response does have some merit.  For example, if a client asks you how SAP processes purchase orders it is difficult to give a single definitive reply and “it depends” is a great cop out.  Purchase orders: would you like an account assignment with that?  Do you want to use info records?  Do you have quota arrangements?  Do have to have requisitions for every PO?  And so on.

It is critical to make a rapid transition from the non-value adding “it depends” to a discussion of what matters to your client.  This is where your knowledge of their business and prior project experience come into play.  I would hope that during the project scoping and requirements gathering phases you gained enough of an understanding that you only suggest SAP processing options that are applicable to your current situation.  After all you wouldn’t discuss logistics invoice verification if you are only processing A/P invoices directly through the FI module.

This Happens on Every Project

This phrase shows up in various forms and the two most common variants I’ve heard are “Every project runs into this kind of thing” and “I’ve seen this before on other projects”.  If you find yourself making statements similar to these pause for a moment about how your client will perceive it.

I have only come across these phrases being used when a project has gone astray and has run into a crisis of some sort: testing is revealing a whole new set of requirements that weren’t captured during blueprinting; test scripts don’t have enough details; functional specifications lack detail; too many overlapping yet dependent activities, etc.

The best comeback I heard to “This happens on every project” was from an exasperated client who asked: “Just so I’m not surprised when it happens, what other things happen on every project that I should know about?”.  (Thank you Patrick!)

In short “this happens on every project” can often be traced back to an earlier failure of project management.  Sometimes the situation is unavoidable, for example, despite your guidance about not moving on to the next project phase because of incomplete deliverables your client insists on going forward because the timeline says it is time to move forward.  Sometimes, key milestones are met but the sign-off is far more conditional than it should be.  Regardless of the underlying circumstances your client probably sees you do one of two things:

• Reactively tell them this happens on every project and then describe how every project goes through some level of fire drill and triage to get things back on track
• Proactively manage their expectations and make clear the potential problems the project may run into: for example, designing and developing while integration testing is never a good idea.  In this way you provide leadership, not just management, and set a course to avoid a fire drill.

Typically a client does not like surprises and pro-active behavior is better than reactive.  I like to think that as a project manager I exist to prevent forest fires, not to fight them.

Consulting Catchphrases Conclusions

My underlying thought is that the consulting clichés have an element of truth and validity to them and you should strive to use them as a springboard to bring value to your client.  Think about where the clichés came from – usually a negative experience – and see if you can anticipate where you might feel compelled to use these catchphrases on your project and (ahem, forgive me!) avoid them like the plague.

Please let me know if you have any favorite overworked project phases and I’ll do a follow up posting if I get a good number of responses.

How much data to move? How much data to leave behind? What to automate, and what to execute manually?  How to gracefully orchestrate and execute a data migration cutover from one system to another?  Where and how to fit the data migration plan into the overall business implementation plan?  How to continue to run the business during the data migration phase of the business project implementation? These questions are all part of the planning fun!

Data Migration Testing

Processes that we have exercised and continue to exercise on a regular basis are no big deal, do not cause anxiety, or raise any blood pressures.  The daily report, the custom transaction, the interface that runs several hundred times a week – they are all second nature now.  They happen, and no one notices.

But reflect back to the very first time these processes went live.  This is precisely where you are with data migration.  For most, it is a once in a lifetime event.  It is important, then, to raise confidence levels, and reduce anxiety levels surrounding this activity.  This is achieved by practice, practice, practice.

Typically, I like to see at least three data migration test cycles executed in an isolated data migration client.  This allows for several attempts at exercising and fine-tuning the data migration plan; collecting nominal run time statistics to have some idea of how long the data migration might take; identifying and fixing any data migration program object defects; identifying and fixing any data mapping and content defects; and identifying and fixing any functional configuration issues. Each of these tasks are done with the goal of significantly improving the fallout rate with each data migration test cycle.  These data migration test cycles also give us the opportunity to practice our legacy extract skills, our fallout analysis skills, and our fallout manual cleanup skills.

Source Data Quality

Each data migration test cycle begins with the legacy or source data extract.  An important activity between data migration test cycles is the cleanup of the legacy or source data.  The data migration process will most likely discover many data problems.  Customer and vendor address data alone are enough to bring your bolt-on tax jurisdiction determination software to its knees – that zip code that is too short, too long, or just plain incorrect when combined with the associated city and state; the various abbreviations used in place of the actual city name; the city and state entered in the same field when they should have been entered into separate fields; etc.   If the source data is not fixed, your data migration test cycles will begin to look like the movie Groundhog Day.

Source data provisioning and cleanup could very well be problematic, especially in an acquisition or merger scenario where the providers are part of a different organization and have no incentive to participate in your project.  If you encounter this scenario, you will most likely need to engage the appropriate management level from your organization to have a serious discussion with the appropriate management level from the providing organization.  Recognize this, be in control, and raise that flag early.    Do not sit around for several days waiting for data to arrive, as this will only set your project behind schedule.

Data Migration Support for Other Testing

In any business project scenario, the development team will salivate at the thought of testing their customizations against your real data in the conversion test cycle box.  Likewise, the functional team will be chomping at the bit to use your real data to test configuration scenarios.  And the interface team and the data warehouse team can’t ever get enough data to play with.

While working within your three data migration test cycles, just say NO.  You absolutely need a controlled environment for data migration test cycles, and these other teams will not respect that.  They have a different focus and purpose which requires changing and manipulating degrees of freedom that you need held constant.

But, we are all on the same team trying to move the project to the finish line within the expected project timeline.  So to help your other team members, plan for additional data migration cycles to provide this data to these teams.  That’s more data migration practice for you, and it gives everyone else what they need.

The Perfect World of Data Migration

In a perfect world, for each data object to be converted:

1. The functional specification and mapping documents are well-written and clear.
2. A data load file is built exactly in conformity to the well-written functional specification and mapping documents.
3. The data migration ABAP objects are built exactly as specified by the well-written functional specification and data mapping documents.
4. No one is insisting that we move a square peg into a round hole.

In a perfect world, for each set of data that is to be migrated:

1. The legacy system data has been thoroughly cleansed.
2. The providers of the legacy data are genuinely interested in providing accurate data on time.
3. Server-resident load files in a Unicode system are correctly encoded to UTF-8.
4. A delimiter other than comma has been specified for the load file.

In a perfect world, the data migration test cycle client:

1. Is built in a box that is not the development box.
2. Is configured exactly like the client where the data migration ABAP objects were built and tested.
3. Is not open for configuration, unless by design.
4. Is not part of a transport path. to prevent the current cycle of conversion testing from being blindsided by any new configuration changes.
5. Is locked down so that only data migration and data validation tasks can be performed.
6. Is configured to handle a more background and update processes and fewer dialog processes.
7. Is built in a box that has enough disk space to be the repository for the primary load files and any intermediate processing files needed.

In a perfect world, at the project level:

1. There is an overall business cutover or implementation plan into which you can assimilate your data migration plan.
2. The data migration plan integrates nicely into the overall business cutover plan.
3. There is a strategy in place to bring the data current between the time the cutover freeze is enforced and the implementation date.

Stay Tuned!

In the real world, the fun is just beginning.  Those perfect world scenarios just never seem to happen.   But stay tuned!  In my next blog post in this series, I will discuss the details of the Data Migration Plan.  After that, in subsequent posts, I will drill down into some real world scenarios that I have encountered and discuss how I dealt with them.

Object Name Prefixes

I have seen a lot of places use prefixes or suffixes before objects such as or dt_ or _mt. I have never really been a fan of these prefixes and suffixes, and apparently neither is the above linked naming convention guide. The reason being is that they are unnecessary. It would be difficult to confuse a message type with a data type in a real interface scenario, even if troubleshooting an unknown broken object. Plus it makes message mapping unnecessarily confusing and long since it’s not clear whether to add the prefix, i.e. MT_one_to_MT_two. Good descriptive names are usually all that you need e.g. DEBMAS_to_Customer. The only possible exception to a no-suffix-or-prefix-policy is the service interface, as sometimes it is useful to know which direction and type a service interface is. An argument for a prefix or a suffix to describe a service interface would be to assist in understanding the flow from an SAP perspective in the event that someone who didn’t develop the interface had to come behind and troubleshoot. A argument against would be the fact that it looks silly if you use it for a web service, because you have a name that doesn’t mean anything to a third party user (note operation mappings are told to omit the prefix of direction and mode in the SAP guide).

One interesting thing that I noticed in my investigation of PI 7.1 EHP1 is that it appears that naming conventions on PI can be validated. If the object names do not conform to a naming convention a message will appear. Ter perform this check in the ESR go to menu Tools>Component Check.

Select “Governance” and “Interface Name Checks”:

If the service interface does not end with (In/Out)(SYNC/ASY) the interface check will show as an error (does not impact interface processing). I created 2 interfaces: one good and one bad to show this error.

My suspicion is that SAP will put more in place to force consistent naming standards depending on the service interface pattern in future releases.

Using a software component and Namespace for each “side” of an interface

All objects of an interface should not be grouped in a single namespace. They should to be split among the Software Component Versions of the systems being interfaced. Otherwise when you go to configure, you will not be able to see your operational mapping (OM) in the dropdown box without having to select all in the dropdown menu. A general rule of thumb is: if it’s not easy to configure, odds are you have probably done something wrong. Another reason why an object might not appear in the dropdown menu (for example for an operations mapping on an interface mapping) would be if the installed checkbox is not clicked on the SLD. When done correctly, most interfaces should be able to be configured quickly and intuitively in the integration directory (ID) without the need to select from all SWCV from dropdown menus on the integration builder.

Whatever naming convention you choose to use for the ESR, the important thing to remember is that adhering to the standard makes production support and troubleshooting faster and easier.

This is good functionality and once your business scenarios, processes and steps are in SOLMAN you can use them as a baseline for additional projects.

Consider the “global template” scenario that crops up in companies today: a core set of business processes are designed and rolled out on a global basis – the only deviations allowed are those mandated by local legal requirements (a.k.a. localization).  On top of this there are the business scenarios that fall outside of the business template.  You don’t want to build a standard implementation project from scratch each time for the core business processes.  This is where template projects save you time and effort.

SOLMAN Templates and Business Process Templates

Using SOLMAN and template projects you can capture that global template with its common business scenarios and then extend it for local variations and non-standardized business scenarios.  Let’s work through an example.

Template Project Definition (SOLAR_PROJECT_ADMIN)

Consider a template project (ZTPC300) with some business scenarios defined in the template as shown below.  In this template I initially defined three templates: one for order-to-cash (OTC), one for procure-to-pay (P2P), and one for financial processes (FIN).  From a global business template perspective let’s assume that there are common processes in OTC, P2P and FIN that need to be rolled out in each country (or business unit or division or legal entity… you choose).

Having created these templates in the project administration I go over to project maintenance with transaction SOLAR01.

SOLMAN Project Maintenance (SOLAR01)

Now, in this transaction you can see where business scenarios have been assigned to the templates from the project definition.  In this example the OTC transactional scenarios and the OTC master data scenarios have both been assigned to template ZTPC_TEMP001.

As discussed in my previous post there is further detail at the business process and step level within the SOLMAN project.  This is where I would expect to find detailed descriptions of transaction execution, business rules, etc.

Implementation Projects & Template Projects

At this point you might be wondering what the point is of this set up.  Having invested the time identifying the content of my template project that constitutes my global business template I can use it when I want to build a new implementation project for the next part of my global roll out.

Now when I go to transaction SOLAR_PROJECT_ADMIN and create a new implementation project I see a curious thing under the SCOPE tab in the project definition (see below).

The three template areas from my template project are made available to me and, if I want to, I can further select at the business scenario level to include or exclude the transactional and master data scenarios.  In this case I have selected transactional scenarios only.

If I now save my project and view it with SOLAR01 I see a project that includes the transactional components of the template project along with the lower level details.

This means I don’t have to build out all this content because it was brought in for me from the template.  Pretty neat.

One Implementation Project from Many Templates

In one of the graphics earlier you may have noticed that the templates available to me when I was building my new implementation project included the OTC, P2P and FIN components but there were two other components available: Z_EMS_001 and Z_SDSO_001.  How did that happen?

One of the neat features about building an implementation project is that you can pull in components from multiple templates.  In this example, I have two template projects with available components which means I can select from both template projects to build my implementation project.

This could be useful if, for example, you had templates for historical reasons for common financial processes, common HR processes, common procurement processes and you wanted to roll these out together in one project for a new business unit.  The SOLMAN functionality allows you to pull the required template components from the three source projects and combine them into one implementation project.

Hints & Tips

If you want to start using template projects getting the sequence of events and status changes in the right order will save some frustration, so here’s what I recommend:

• Create the template project with SOLAR_PROJECT_ADMIN
• Build out the project structure with SOLAR01
• Identify the nodes on the project you want to use as template components
• Create the template components with SOLAR_PROJECT_ADMIN and leave them in private status
• Update the project nodes using SOLAR01 with the appropriate template component.  If the template components are in a public status this step is not possible
• Use SOLAR_PROJECT_ADMIN to change the template component status from private to public.  This makes the component available when you build a new project.

Summary

This method in SOLMAN to build implementation projects from template projects is a terrific time saver.  It allows you to re-use and update project artifacts giving you a jump start on value adding activities that should bring value to your business.

• How will I ever match the inbound interface parameter “Tom” with “TOM”, or “tom”?
• How will I ever match the inbound interface parameter “Smith” with “SMITH” or “smith”?
• The ABAP WHERE clause is not case-INsensitive.
• There could be hundreds of customers named Tom Smith.
• KNA1-NAME1 and KNA1-NAME2 are not indexed fields.
• And no, we are not storing any portion of either first or last name in an existing indexed field like SORTL.
• There are well over one million customers in the database.
• We have already decided to use PI for all interfaces.
• I will have to buy the BASIS team a case of beer to get them to agree to create indices on the fields KNA1-NAME1 and KNA1-NAME2 in a table with over one million records.

I arrived at the conclusion that I need a case-insensitive database query, along with database indices created for the fields KNA1-NAME1 and KNA1-NAME2.

But, what is a case-insensitive WHERE clause?  A little research and help from colleagues revealed that many had gone before me, and this was nothing new.  To implement a case-insensitive WHERE clause in ABAP, you simply needed to use the native SQL UPPER() construct. The database system that is being used is Microsoft SQL Server, but the UPPER() function and its syntax is similar across different database platforms. This seemed like an easy nut to crack. But, as I soon found out, I actually had a lot to learn.

Take 1

The examples I found were very explicit, showing actual code snippets.  Using my research examples as a template, I built what was to be my first attempt at using native SQL in a ZTEST ABAP program:

exec sql.
  select kunnr name1 name2
    from kna1
    into table i_cust
    where upper(name1) like :l_name1
      and upper(name2) like :l_name2
endexec.

Well, as a native SQL rookie, I was stopped dead in my tracks.  Native SQL does not allow a select into a table – you can only select into a work area.

Take 2

In my ZTEST ABAP program, I changed the code to SELECT into a work area and append the work area to an internal table in a subroutine:

exec sql performing append_cust.
  select kunnr, name1, name2
    from kna1
    into :wa_cust
    where upper(name1) like :l_name1
      and upper(name2) like :l_name2
endexec.

form append_cust.
  append wa_cust to i_cust.
endform.

I really thought that this would work.  But, I was getting a syntax error from native SQL – the table name was not being recognized.  A little more research revealed that, in some cases, only UPPER CASE is valid for table names and field names.  OK.  I can handle that.

Take 3

I changed the native SQL statement to all UPPER CASE in my ZTEST ABAP program:

EXEC SQL PERFORMING APPEND_CUST.
  SELECT KUNNR, NAME1, NAME2
    FROM KNA1
    INTO :WA_CUST
    WHERE UPPER(NAME1) LIKE :L_NAME1
      AND UPPER(NAME2) LIKE :L_NAME2
ENDEXEC.

form append_cust.
  append wa_cust to i_cust.
endform.

This appearedto work just fine…  Except, I noticed that too many records were being returned.  A little more analysis showed that I was getting records from ALL clients, not just my logon client.

Take 4

Again, in my ZTEST ABAP program, specify the client:

EXEC SQL PERFORMING APPEND_CUST.
  SELECT KUNNR, NAME1, NAME2
    FROM KNA1
    INTO :WA_CUST
    WHERE UPPER(NAME1) LIKE :L_NAME1
      AND UPPER(NAME2) LIKE :L_NAME2
      AND MANDT = :L_MANDT
ENDEXEC.

form append_cust.
  append wa_cust to i_cust.
endform.

Now I was getting the results that I wanted!!!!  So, I pasted this ZTEST ABAP code into the ABAP proxy.

A syntax check of the code immediately rejected the use of subroutines – they are not allowed in OO ABAP. I suppose I should have known that.  The error message indicated that I needed to use a cursor instead of the subroutine.  So, I started cursing – a lot!!!

Take 5 (The Final Version)

Here is the final version of the code which works for OO ABAP, and, hence, the ABAP proxy:

EXEC SQL.
  OPEN CUSTCURSOR FOR
    SELECT KUNNR, NAME1, NAME2
      FROM KNA1
      WHERE UPPER(NAME1) LIKE :L_NAME1
        AND UPPER(NAME2) LIKE :L_NAME2
        AND MANDT = :L_MANDT
ENDEXEC.

DO.
  EXEC SQL.
    FETCH NEXT CUSTCURSOR INTO :WA_CUST
  ENDEXEC.

  IF SY-SUBRC NE 0.
    EXIT.
  ELSE.
    APPEND WA_CUST TO I_CUST.
  ENDIF.
ENDDO.

EXEC SQL.
  CLOSE CUSTCURSOR
ENDEXEC.

Well, I finally got it right after five tries.  As mentioned before, I found many explicit code examples, but none were for OO ABAP with a database  requiring UPPER CASE be used, so that the table and field names would be recognized in the native SQL SELECT statement.

I learned something new about SAP.  I hope that I never stop learning.  I also hope that this blog might save someone a bit of distress in trying to work through a similar situation.  Now, I just need to figure out what kind of beer to buy the BASIS team to get those indexes created…