Discovering Direct Modifications of SAP Standard Objects – One Example

As I discussed in a previous post on this topic, direct modification of SAP standard objects within the SAP namespace carries a high risk in an upgrade project.  It would be very valuable to be able to identify some details about these objects, so that further analysis could be performed to determine their disposition in the upgrade.

Modification of standard SAP objects typically requires that a modification registration key be obtained for that object from the SAP SSCR facility.   The SSCR facility (SAP Software Change Registration) is a procedure which registers all manual changes to SAP sources and SAP Dictionary Objects.  With a valid and authorized OSS ID for an SAP system, you can go to the SAP Support Portal on the web and view the SAP objects that were registered for modification by that OSS ID.  If there are multiple OSS IDs for any SAP system, make sure that you check all of them to capture all objects registered by all of the authorized OSS IDs for an SAP system.  Press the “Objects Registered by Me” button to view a list of the registered objects.

Another place that you can look for this information is directly within SAP.  Table ADIRACCESS stores the registration keys which were obtained from SAP, and which were entered by the developer when the object was to be modified for the very first time.  This entry shown below from table ADIRACCESS shows that function group SCPRPS was registered with SAP for modification.

Registering an object for modification does not mean that the object was actually modified.
What additional evidence can we discover that would show that the object was, in fact, modified?   And, since a function group contains many objects, can we determine exactly which object or objects were modified?

With a little bit of forensic analysis, we can discover the specific object within the function group that was modified, the user name of the programmer who last modified the object, date of the last modification, and the line-by-line source code differences.

If you are familiar with the structure of function groups, you already know that they may contain many function modules, and that the source code for each function module resides within its own include file.  At the moment, we know that the function group name is SCPRPS, but we do not have the list of function modules within that function group.

To obtain that list, we can use the SAP function module FUNCTION_SELECT_TFDIR.
The result list is shown here:

So which, if any, of these has been modified?  To determine this, we need to find all of the rows in table TRDIR where the NAME field is equal to the PNAME entries listed above.

Here is an excerpt from SAP table TRDIR for SAP object names that begin with the characters LSCPRPSU:

Note that for all of these objects, which are not in the customer namespace, the Author Username field (TRDIR_CNAM) is ‘SAP’.  Also note that for most of the entries listed here, the Last Changed By user name field (TRDIR_UNAM) is also ‘SAP’, except for one row, where the Last Changed By username field is not ‘SAP’.  From this row, we can see that include file LSCPRPSU36 was last modified by user DXSDEV on 10/19/2009.

If we look back at the result list of FUNCTION_SELECT_TFDIR and find the row where the field PNAME is equal to LSCPRPSU36, we can see that the name of the modified function module is SCPR_PRSET_CT_IMPORT_INDUSTRY.

Exactly What Changes Were Implemented?

By going to the Version Management Utility for this function module, we can see what the various versions of the code contained.  The Version Management Utility will also show us the transport request numbers, date and time, and username for each version of the code.  The Version Management Utility also has a code comparison tool which will highlight the line-by-line differences between versions of the source code.

Here is an excerpt from the side-by-side source code comparison utility which shows the modified version on the left side and the original version on the right side.  Fortunately, in this instance, the programmer left excellent notes indicating that a correction instruction from an OSS note was being applied.

What Are the Discovery Possibilities?

The manual example described above showed how to discover and analyze customizations that had been applied to a single SAP standard object.  But what if the customizations number in the hundreds or even thousands?  What about all of the customizations that were built entirely within the customer namespace?

It is possible to discover hundreds or even thousands of customizations regardless of which namespace they were developed in.  The key is automation, understanding that this information already resides within SAP waiting to be discovered, and more importantly, understanding the meaning of the discovered information.  I have already developed an ABAP discovery program which will mine, analyze, and report data about the following customized objects:

More Discovery – Usage Statistics and Where-Used

Customization Usage Statistics

The goal of customization usage statistics analysis is to discover the usage frequency of customized objects.  The usage discovery may show usage anywhere from several times daily by key users, to none when analyzed over a long time period.   Since most companies really do not want to incur the cost and the risk of moving an unneeded customization into the upgrade system, objects which show no usage over long periods of usage history become targets.  Care must be used here, as some transactions are exercised only quarterly, semi-annually, or annually.  I always recommend letting the business process owner make the final decision to eliminate any customization object in the upgrade.

SAP keeps track of usage statistics, and this data is available for you to analyze.  You will need to check what facility is available for your particular SAP system, as the transaction codes and programs differ among various versions of SAP.   The amount of data stored within SAP is configurable, and is usually set to a small date range.  Some companies that want to analyze the statistics over longer time periods actually download and store the statistics data in their own external databases.

Customization Where-Used

Like usage statistics, the goal of customization where-used analysis is to discover customized objects that are orphan or not being used.  What about that custom data type that is not used in any table or structure?  What about that function module that shows an empty list when you click the where-used icon in the function builder?

To properly perform a where-used analysis, it is important to understand all of the various possible uses for an ABAP object.  For example, here is the selection screen of possible uses that is presented by the ABAP Dictionary when you click the where-used icon to determine the where-used list for a data table:

And what about that custom function module, which I alluded to earlier, which shows a completely empty where-used list?  Perhaps it is a remote-enabled function module that was installed by, and is currently being called by an external third-party package.   Or, perhaps it is linked to an SAP EDI inbound or outbound process code.  It is really important to be able to properly interpret the data being presented and to not draw any hasty conclusions.

ABAP Programs Where-Used Analysis Example

In the ABAP workbench, SAP provides a where-used facility, which reports the various areas where a single selected object is used.  For a discovery project, you may want to automate this facility to analyze groups of ABAP objects (e.g. all custom ABAP programs, all custom tables, etc.).

Here is an ABAP programs where-used analysis example to help illustrate how a where-used analysis might be utilized.

Your automated where-used analysis program to analyze custom ABAP programs reports fifteen custom programs which show no usage in any of the following areas:

A quick analysis shows that all of these fifteen programs are simple reports.  Since users typically execute their reports via transaction codes, you wonder how these fifteen programs could ever possibly be executed without transaction codes.

You further analyze the SAP job scheduler data as far back as possible and find that only ten of these ABAP programs were ever scheduled in a batch job.  Some of these were scheduled very recently, and some have not been scheduled for a long time (another thread of evidence that we will need to pursue).  This leaves five ABAP programs with no apparent means of being executed.

As further evidence, you perform a cross-check against the usage statistics report and find that two of these five ABAP programs show no usage.

Armed with this analysis, you are now ready to meet with the business process owners to discuss the disposition of the objects on your short list.

A salient point here is that you will need to gather your evidence from several different analysis sources to help paint the most accurate picture about the possible disposition of customized objects.

Summary

Throughout all three parts of this post, I have stressed that customizations, while necessary, add risk and cost to an upgrade project.  And customizations that are not necessary or built poorly without the constraints of best-practice controls can add high levels of risk and cost.  Here are some general guidelines to follow:

Many companies and their auditors consider a well-controlled and secure ERP system to be a fundamental component that is vital to the success of the business.   They recognize that an out-of-control ERP system can easily paralyze a company for extended periods of time.  With this in mind, implement and enforce best-practice controls and standards to make sure that future customizations are reviewed, approved, built, tested, documented, and implemented properly.  This may already be an internal or external audit requirement, so check with your inside and outside auditors to make sure that your controls meet the auditing requirements.

Periodically review, audit, and amend your best-practice controls to make sure that they are still effective in supporting the needs of the business; and that both internal staff and external consultants are in compliance of these controls.

Know your customizations.  Know where they are, why they are, and what they do.  Whenever feasible, it is most efficient to eliminate ABAP object customizations, especially any that were performed within the SAP namespace, and any that are not really needed because they can be implemented within standard SAP.

The degree of risk and cost depends on the specific type of customization that was performed.  Establish a formal review board to own the customization approval process.  Understand that approving a customization also means approving the risk and the cost burdens.  Some companies consider this to be so significant, that they include a high-level business executive as a default member of the review board.

Set and enforce a policy making ABAP object customization the last resort, only after all other options have been thoroughly researched and exhausted.  Here are some examples:

1. Research forums, blogs, become active in user groups, etc. to see how other companies might also be implementing the business function that you require.
2. Find and use customer and user exits instead of directly modifying objects in the SAP namespace.
3. Determine, for example, how to implement and maintain that complex pricing procedure using the pricing condition tables instead of writing custom code and building custom tables.

Share Your Experience.

Once again, I look forward to hearing from our readers to reply to this post with their own experience with customizations, best-practice standard and policies, and success stories illustrating how your company transitioned to best-practice controls.

[Editor's Note] This blog is third of a multi-part series:

1. Part 1
2. Part 2
3. Part 3

In this post, we will look at some of the customization areas which add risk and the cost to an upgrade project.

1. Direct Modifications to SAP Standard Objects

Direct modifications to SAP standard objects bear the highest risk.  During the upgrade, these modifications will be lose–either because they will be overwritten by their respective upgraded SAP standard objects; or because these objects no longer exist and are not used by standard SAP in the upgrade system.

Typically, there are two reasons for performing direct modifications to SAP standard objects:

1. The application of authorized individual OSS corrections to fix problems.
2. Direct customer modification to provide business process enhancements.

1.1. Application of Individual OSS Corrections and Support Packs

Individual OSS corrections are patches supplied by SAP to fix recognized problems with SAP objects.  Sometimes, individual OSS correction instructions are included and delivered within a specific level of support packs.   That means that applying support packs to the specified level will apply the desired correction, and manual intervention is not needed.  Sometimes OSS correction instructions are not yet included in any specific level of support packs, and these will need to be applied either manually or with the SAP Note Assistant.

It is important to carefully analyze the validity range of an OSS note to determine both how it was applied in the legacy system, and how it is to be applied in the upgrade system.  This analysis will help determine how far to patch the upgrade system to make sure that the desired corrections are included in the upgrade system.   Any needed correction that exists at a patch level beyond the highest patch level in the upgrade system will need to be addressed  manually or with the SAP Note Assistant.

The decision of which support pack levels should be applied to the upgrade system must be made carefully.  One approach is to apply the highest level of support packs that are available.    The rationale for applying the latest available support packs is to insure that the most recent problem fixes are applied to the upgrade system.  This is a two-edged sword, as the most recent  support packs may also introduce other problems into the upgrade system.

1.2. Direct Customer Modification of SAP Standard Objects to Provide Business Process Enhancements

Direct modifications of SAP standard objects to provide enhancements are most costly and bear the highest level of risk.  While direct modifications of SAP standard objects may have been technically expedient at the time of implementing the enhancement, they will always require additional work effort and analysis at upgrade time to determine:

1. Whether or not the functionality provided by the enhancement is still required by the business.
2. Whether or not the functionality provided by the enhancement is already included in the upgrade system as standard SAP.  You will need the help of functional subject matter experts here.
3. Whether or not the modified SAP object still exists in the upgrade system.
4. How to move these required enhancements into the customer namespace, using the SAP preferred enhancement procedures.  Moving these types of enhancements to the customer namespace also moves them from the very costly and highest risk category to a lower cost and lower risk category.

2. Custom Interfaces

Custom interfaces are loaded with opportunity for high upgrade risk.  Take, for example, the SAP standard IDOC Basic Type, which was copied as the baseline for an enhanced customized basic type containing additional custom segments and fields.  In the upgrade system, a new version of this SAP standard IDOC Basic Type may now contain additional segments and fields.

Likewise, the associated standard processing module for that standard basic type, which was copied as the baseline for an enhanced processing module to handle the custom segments and fields of the enhanced custom basic type, may also have changed in the upgrade to accommodate the new standard IDOC version.

So, in the upgrade system, especially if you need to implement the new upgrade IDOC version, it may be necessary to start with the new standard basic type and standard processing module, and reapply the customizations.

3. ABAP Code Objects

Of all customized objects discovered in most SAP systems, ABAP Code objects are the most numerous.  These objects include classes, methods, function modules, report programs, dialog programs, and include files which support the all of the other ABAP object types.

At greatest risk are the SAP standard ABAP code objects which were modified directly in the SAP namespace; and those that were copied into the customer namespace and subsequently modified. This is because the standard SAP code objects may have changed in the upgrade.    These changes can range anywhere from simple error corrections to complete process and code redesigns.  In these instances, each modified object must be compared to its counterpart in the upgrade system to determine its existence, and the nature and extent of what might have changed.   If the customization is still needed in the upgrade system, and to stay current with the enhanced standard code, it is probably best to start with the upgrade standard code and reapply the customizations.

ABAP code objects that were developed entirely in the customer namespace (not copied from standard SAP code) are usually low on the risk ladder.  Problems occur when data selected by these objects has moved elsewhere or behaves differently in the upgrade system, when ABAP constructs used are now obsolete in the upgrade system, or when standard called functions are changed in the upgrade system.

Ultimately, the business must decide why SAP standard code was directly modified, why SAP standard code was copied into the customer namespace and modified, why custom code was developed entirely within the customer namespace, whether or not any of these modifications are needed, and how they might be implemented differently in the upgrade system.  This evaluation presents another opportunity to remove some rocks from the customization rock bag.

4. Automated Standard SAP Transactions

Custom ABAP programs which automate standard SAP transactions bear an above-average risk.  This is because the behavior of some SAP transactions may have changed in the upgrade.  These changes can be manifest in the functional performance of the transaction, the technical location and existence of data fields on screens, screen field properties, IMG configuration specific to the transaction, the transaction code associated with the transaction, or even the existence of the transaction or an equivalent in the upgrade system.  The nature and the extent of the changes to the transaction in the upgrade system will determine the amount of effort needed to automate it.

Also, if an “old” SAP transaction has been replaced by a “new” upgraded equivalent, it is usually wise to make the change to the new transaction at upgrade time.  At some point in the future, the “old” version of the transaction will become obsolete and no longer useable or supported.

5. Data Dictionary Customizations

Many customizing projects within SAP require the storage and manipulation of data that is not included in the standard SAP dictionary.   One way in which these requirements can be addressed is by building complete custom transparent tables and data structures.  Another way is by adding append structures and append fields to existing standard SAP tables.  With either approach, it is always an excellent idea to evaluate the need to move these customizations into the upgrade; and another opportunity to lighten the customizing bag of rocks.

6. Customer and User Exits

Customer and user exits are provided by SAP as the SAP-approved mechanism for enhancing standard functionality.  Enhancements performed within these boundaries are guaranteed by SAP to bear low technical risk in an upgrade situation.  While SAP guarantees that the custom code will arrive technically intact in the upgrade system, functional analysis must still be performed to determine the continued need for the enhancement in the upgrade system.

Any enhancement implemented in a customer or user exit that is either functionally covered or determined to be unnecessary in the standard upgrade system should be retired.  This is consistent with the goal of lightening the load in the bag of rocks.  Since a customer or user exits may implement several different enhancements, take extra care to make sure that only the right code or other objects are retired.

7. The Environmental Effects – Operating System Change

OK, changing the operating system during an upgrade project is not customization of SAP objects.  But it does add risk and cost to the project.

Why?

If the upgrade operating system changes from UNIX to Windows or vice versa, then any ABAP program or interface which reads or writes a data file to an operating system folder must be changed.  The changes needed to these ABAP programs are very narrow in scope,  and reflect the manner in which the operating system specifies its file paths.  Additional BASIS work, such as determining the proper file system authorizations, may also be needed.

Using the SAP logical file name facility can help enormously by removing the need to make changes to hard-coded file names and path names in many different ABAP programs.

Share Your Experience

At the top of this post, I stated that I would discuss some of the customization areas which add risk and cost to an upgrade project.  I look forward to hearing from our readers to reply to this post with their own experience with customizations, how they were handled, and how this affected an upgrade project.

In my next post on this topic, I will reveal some of the techniques I use to “discover” customizations, even in the absence of documentation.

[Editor's Note] This blog is the second of a multi-part series:

1. Part 1
2. Part 2
3. Part 3

Why the data will not load correctly.

There is a litany of reasons why some data records will load correctly while others will not.  Here is a list of some common root causes:

1. Poor quality legacy data.
   Legacy systems which are not as tightly integrated as SAP, and are not under master data control allow the end user a bit of freedom when entering data.  A zip code may contain too little or too many characters; the email address is not properly formatted; numeric fields have transposed digits; various forms of abbreviations (especially in the city field), a quantity of zero (0) permitted by the legacy system and uploaded into a field where SAP will not accept a quantity of 0 and even simple misspellings  all can cause stringent validation checks to trigger an error and prevent the record from loading at all.  A more sinister type of error occurs when the data is functionally incorrect, but good enough to pass all of the SAP validity checks.  In this case, the data record will technically load into SAP, but will not be functionally correct.  Duplicate customers, duplicate vendors, and the data entry error for a quantity of 1000 instead of 100, and the wrong pricing condition applied to a sales order line are examples of this scenario.

1. Functional configuration and supporting data effects.
   Many times I have watched the load statistics for a data object plummet from near 100% in the cycle two test load to near 0% in the cycle three test load.  This is very unnerving to the client because the cycle three test load is getting rather close to the go-live date, and “by the way, shouldn’t the statistics be getting better rather than worse?”  Functional configuration changes can wreak havoc on any data load.  Flipping the switch on a data field from optional to required; turning on batch management or serialization for materials for the first time; changes in the handling of tax, tax codes, and tax jurisdiction codes; that account determination entry that is missing or not set up correctly; a missing unit of measure or unit or measure conversion factor; the storage location in the upload file which does not exist in SAP – any of these can cause a load to drop mostly or completely onto the floor.While change is inevitable on any project, it is important to control and communicate the change so that the downstream impact can be recognized and understood.   Controlled change and communication always works better than total surprise.  Perhaps if we all know ahead of time about that data field that is now required, we can impose a requirement on the data extract side to make sure that the data field is populated before it enters the upload file. 
2. Additional data in the upload file.
   Inserting a new field in the middle of the upload file data structure might be necessary for the business to close a gap, but if that change is not communicated to the technical team so that appropriate adjustments can be made to the load object’s input structures and processing logic, the new data will surely never load, and may cause misalignment of the data fields which follow it in the upload structure.

The Finger Pointing Game

It’s the load program!  No, it’s the data!  No, it’s the configuration!  No, it’s … (fill in your favorite finger pointing game excuse explaining why data will not load).

If you ever find yourself in the midst of this type of finger-pointing game, immediately stop the madness.  For this and similar situations, I apply a simple litmus test which has never failed me yet – manually enter the EXACT upload data into SAP for the transaction which has failed.  If one can be entered manually, then the program will be able to automatically load thousands with similar upload data and functional configuration.

I have lead many a functional analyst – kicking, screaming, and ranting about how terrible the load program is – to the terminal to play “let’s enter one manually”.  Typically, the result is that one cannot be entered manually due to configuration issues or the lack of supporting values data.  Once these issues are cleaned up, the load program “magically” begins to process thousands of records with no trouble at all.

Sometimes, the load object appears to be the cause, but is not the root cause.  The important data item not being handled by the load object (which was not called out in the functional specification document), the data item which turns blue (because the functional specification document explicitly stated “put this data item into the blue category”), the formula which is not calculating the desired result (but is indeed the exact formula found in the functional specification document) – all are examples of the load object adhering accurately to an incorrect functional specification document.  The root cause, then, is the functional specification document which must first be revised and checked into the controlled document repository before making any code modifications.

It’s the Program

Well, OK, sometimes the load object is at fault.  But it is extremely rare.

Collecting and Reporting the Technical Load Statistics

Load statistics are an important technical metric, indicating what percentage of the upload file has successfully posted a transaction into SAP.  This is a basic and simple record count check.  How many records were presented in the upload file, how many records posted successfully to SAP, and how many records failed to post.  Also, does the number of successful transaction plus the number of failed transactions equal the total number of records presented in the upload file.

Here is how I communicate this technical metric:

This technical metric indicates only that all of the SAP validation rules for posting the transaction have passed.  It does not indicate that the master or transactional data posted to SAP is actually correct in functional terms.

The Functional Review

It is very possible for an entire upload file to technically load at 100%; while at the same time, functionally fail at 100%.  The customer may be missing a partner, the material or article may be categorized incorrectly, the pricing conditions on a sales order may not have the desired validity date range, the GL posting may be to the wrong accounts.  Hence, the need a functional review and validation of the data transacted into SAP.  The data type – master or transactional – determines the type of functional review and metrics to be employed here.

For master data, such as of materials, customers, vendors, etc., it is impossible to individually validate the many thousands of entries in your SAP system.  But statistical methods can be employed which will guide you through a random sampling of the data, while at the same time assuring accuracy at a high degree of confidence levels.

For transactional data, such as inventory, open sales orders, open AR, etc., direct mathematical comparisons can be employed.  On a grand scale, if an inventory value of $15,246,321.44 is being moved from your legacy system to SAP, then that exact amount must arrive in SAP when the migration task is complete.  It may be a bit more difficult to do this mathematical comparison at a more granular level.   If, in the move to SAP you are also redesigning your material/storage location combinations, a direct comparison between the legacy system and SAP may not be possible without a translation factor.  The same scenario exists if you redesign your GL chart of accounts, where one legacy GL account now maps to several SAP GL accounts, or vice versa.

Reporting the Fallout

If an automated data load was not technically 100% successful, a clear set of error messages complete with a link back to the legacy data must be mined, formatted, and presented to the business for analysis.  Such a report really helps to facilitate the fallout cleanup.  The link back to the legacy data must be carefully designed into the load process to make sure that, for example, the legacy customer number, legacy vendor number, legacy sales order number, legacy purchase order number, etc. is included as part of the data being handled.  Without the link back to the legacy data, it becomes very difficult to identify which data record needs to be fixed.

The error message mining technique that I use depends on the load method.  I will describe two here – one for a BDC load method and one for an IDOC load method.

Mining Meaningful Error Messages from a BDC Session Log

At the completion of a batch input session, the batch input session overview screen (SM35) displays the technical load statistics.  In this example, out of a total of 229 transactions, 13 failed and 216 succeeded.

The session log shows the status of all 229 transactions.  This screen snapshot is a fragment of the complete session log for the batch input session.  It shows many successful transactions (Type = S) and one failed transaction (Type = E).  The error message here is clear – the article does not exist or is not activated.

But as you can imagine, the 13 failed transactions with error type = E are sprinkled throughout the many pages of this log file.  With only 229 transactions, this log file is quite easy to pick through to find the 13 errors.  But imagine if the number of transactions were in the thousands or tens of thousands.  How do we extract only the failed transactions and present a concise report of the failed transactions?

To do this, I use SAP transaction SM35P – Batch Input Log Overview.  This transaction has the ability to set a filter on any field in the batch input log file, display the filtered results, and then to export the results to a local file.

To enter the mode where this is possible, first press the PRINT icon.

Next, set the filter.  The appropriate filter field here is SESS. TYPE.

We only want the errors, so set the filter for field SESS. TYPE = E.

The display now shows only the 13 rows containing the error messages.  This can be exported directly to a local spreadsheet for further analysis.

Mining Meaningful Error Messages from IDOCs

Depending on the IDOC and the processing module, mining the error status messages from IDOCs can be very easy or somewhat challenging.  For the more difficult scenarios, you probably will need to hone your EXCEL skills to properly join several extracts together into one complete picture.

When creating IDOCs with a load object, I always note the date, time, and IDOC basic type.  I will use this information as the selection criteria for transaction WE05, which is going to locate the IDOCs and display the results information I need after the IDOCs are processed.  The results that I usually collect are the error status messages, and some data content from a segment or two to illustrate exactly where the problem is in the legacy data.

While the background job is busy processing the IDOCs, I usually take a peek, using transaction WE05, to see how the load is progressing.  If I see that most of the IDOCs are falling onto the floor (IDOC status 51) rather than moving into the database (IDOC status 53), I usually stop the background job and begin an immediate analysis of the fallout.  If the fallout solution does not require a change to the IDOC data content (e.g. a configuration change), then I can replay the fallout using SAP transaction BD87.  If the fallout solution does require a change to the IDOC data content, then the complete set of IDOCs must be regenerated again.

Let’s see what WE05 can tell us about a completed Article Master load.

On the WE05 screen below, we can see that of the total of 14,005 IDOCs, 11,412 have processed successfully and 2,593 have failed to process.  By double clicking the Status 51 folder, the display will show only the Status 51 IDOCs – the fallout.

By pressing the “status list” icon (shown above), the display will show the status messages for the fallout.    Once these messages are displayed, pressing the “export” icon allows me to save the screen contents to a spreadsheet.

Now it would be really nice if I could have the article number in the spreadsheet right next to the error message.  The article number in the ARTMAS IDOC is stored in segment E1BPE1MATHEAD.  The segment content for each IDOC can be displayed by pressing the “list specific segment” icon and entering the segment name in the box.

The segment display will show all fields in the segment, so I usually hide all of the columns that I don’t want to see.  Here is the E1BPE1MATHEAD segment display showing only the article number.  I can use the export icon to save the list of article numbers to another spreadsheet.

Here is a portion of the complete spreadsheet showing the error messages and the article numbers side by side.

A filter applied to the spreadsheet shows that the 2,593 errors are all grouped into one of three error status categories.  By selecting a single category, Excel will also show me the number of records within that failure category.

Sometimes it is easier to mine the status messages directly from the IDOC status table EDIDS.  This is especially true where the processing module is a BAPI which returns an error table rather than a single error message.  In this case, when you press the “status list” icon in WE05, only the first error status message of several is displayed for each IDOC.  I find that the first message is not very helpful (as shown below).  I also find that typically the second or third message in the return status table is usually the important one.  You won’t see it displayed on the WE05 screen, but you can mine it from the EDIDS table.

SAP transactions SE11 or SE16 both support this activity.  For the selection criteria I use the IDOC number range, status 51, and status type E.  On the display screen, choose only the relevant fields for display – the IDOC number (DOCNUM), IDOC status (STATUS), status message (STATXT), the four substitution parameters for the status message (STAPA1, STAPA2, STAPA3, STAPA4) and the message type (STATYP).  All of this can be exported into a spreadsheet.  If you really want to test your Excel skills, you can write code that will move the substitution parameters into their placeholders in the status text.

Preparing for the next data migration cycle – Let the fallout analysis and cleanup begin.

Presenting the fallout report to the business with a set of clear error reasons and links back to the legacy data is key to enabling the legacy data cleanup process to proceed.  In the iterative process of data migration cycles, cleansing the legacy data is a step in the right direction towards an improved next conversion cycle.

I hope you enjoyed this blog series on data migration.  Please feel free to send comments or questions.

Before you embark on your project’s first data migration test cycle, your data migration plan will probably start out as a simple list of preparation steps, data load dependencies, and cleanup steps.  As you progress through your several data migration test cycles, your plan should evolve into a rather complex list containing load methods, responsible persons, historical data volumes, historical execution times, steps that need to be modified and additional steps that need to be added.

I always find it rather interesting to compare the very first plan to the plan that is actually used in the cutover process.  Many project factors will cause you to amend your data migration plan.  Newly-discovered gaps, changes in the business requirements, the decision to load all of the legacy customers instead of only those customers which have been active for the past two years, and toggling back and forth between “should this one be a manual or an automated load” are only a few.   Always keep in mind that the goal in amending the plan is to keep it compatible with the changing requirements of the project.

The Data Migration Preparation Phase

Before you actually start the data migration main course, you will probably want to execute a series of preparation steps.  Some of these preparation steps may have a significant lead time; while others may require a system reboot to take effect.  Make sure that you carefully plan accordingly.  Here are some data migration preparation steps that I have found useful:

• Obtain a Data Migration Login
  Obtain a special data migration login, with all of the authorizations needed to execute all of the required tasks.  This login is only to be used during the data migration process, and is to be deleted upon the completion of the data migration process. Using the data migration login makes identification of migrated data very easy both now and several years down the road.  It also facilitates implementing a general user lockout while the data migration user is performing data migration activities.
• Technical client preparation
  The BASIS team may want to configure the data migration target client to have less dialog processes and more background and update processes.  This configuration can significantly improve data migration performance, especially if you have the opportunity to parallel process IDOCs.  In a Virtual Machine environment, the infrastructure team can even provision additional processors and memory.  You may also need additional disk space to handle large upload files, and any intermediate files that a load program might need to create.
• Configuration for IDOC processing
  If a data migration process involves creating and processing IDOCs, you may need to configure the ports, partner profiles, message types, and processing modules.  My preference for IDOC triggering is by using the ABAP program RBDAPP01 rather than immediate triggering.  This allows me to exercise the most scheduling control over the background processing options.
• Change Pointers
  During a data migration turn off change pointers.  Creating what could potentially be hundreds of thousands of change pointer records that are never going to be processed is an overhead burden on system performance and disk space that you do not need.  If the migrated data needs to be sent to external systems, manually send the data after the data migration is complete.
• Set the Correct Open Posting Period
  Some of the transactional data will want to post into an open financial or manufacturing posting period.  You will want someone from the business to make sure that the correct posting periods are open for business.
• Configuration Settings
  Sometimes, special temporary functional configuration settings are made to facilitate the data migration process.  Several functional analysts may be responsible identifying and setting these prior to the start of data migration.  As these are temporary configuration settings, it is expected that they will be set to their production values after the data migration is complete.
• Basic Setup Data
  Some data migration objects are dependent on reference data – a template from which new data migration objects can obtain some of their default values.  You will want to make sure that those who are responsible for setting up this reference data have completed their task before you start the actual data migration.  Sometimes this data is moved from the Golden Client into the target client via ALE, so you may need to set up RFC destinations, ports, distribution models, etc.
• General user lockout
  All users except for the data migration user should be locked out of the data migration client during the actual data migration activity.  It is impossible to verify that we moved exactly $15,386,254.23 in inventory from the source system to the target system if users are performing material movements in either system during the conversion.
• Connections to External Systems
  Depending on what connections to external systems are doing, you may want to make sure that they are disconnected prior to the start of data migration.  This will ensure that inbound interfaces, which may either modify existing migrated data or add new data, are disabled.

The Data Migration Phase

When all of the preparation steps are complete, it is time to start the actual data migration.  Data must be loaded into the target system in a certain sequence that is dictated by their data dependencies.  Customers must exist before sales orders can be migrated, vendors must exist before purchase orders, materials must exist before either sales orders or purchase orders, etc.  This dependency exercise varies a bit from project to project, and should have already been determined well before you start loading the data.

Automated data migration tasks for a single object (e.g. customer master data) usually include some or all of the following:

1. Creation of an upload file from the source system
2. Technical validation of the upload file
3. Functional validation of the upload file
4. Execution of a technical object to load the data into the target system
5. Technical validation of the migrated data
6. Functional validation of the migrated data
7. Collection and reporting of load statistics
8. Analyzing the fallout
9. Correcting the fallout
10. Reprocessing  the fallout

Checkpoints

At a higher level, the data migration plan will string together an end to end list of all of data migration objects.  I strongly suggest that you study the list and judiciously intersperse several checkpoints at appropriate places in the plan.  A checkpoint is an activity where all interested parties meet to review status and metrics, and to determine whether or not to proceed forward.  Keep in mind that what fails to load early in the plan will geometrically cascade into larger fallout later in the plan.  That single vendor which failed to load may cause several hundred open purchase orders to also fail.  And while the business owner of the vendor master data may think the fallout of a single vendor to be quite insignificant, the business owner of open purchase orders may have an entirely different perspective.  It is important here to make sure that all of the right stakeholders are participating, and that all are in agreement about whether or not to proceed.

Backup or Restore Points

Another activity that you should judiciously sprinkle into the plan is several backup or restore points.  It is very painful to have successfully navigated seventy-five percent of the data migration plan, only to have the next load create a huge and unrecoverable mess in the target system.  Without a backup or restore point available, you may have to wipe the slate clean and restart the entire data migration process again.

Depending on the target system and the facilities available, the backup or restore point could be a full system backup, a client copy, or a virtual machine snapshot.  Whatever the means, make sure it is a part of your plan.  It’s really great if you don’t need it.  It’s also really great to have if you do need it!

The Data Migration Cleanup Phase

When the data migration loads are finally complete, it’s time to exercise a series of cleanup steps.  This is where we:

1. Undo the special temporary settings and configurations that were done prior to the start of the data migration activity.
2. Adjust the technical configuration of the system to a more user-oriented system with the right amount of dialog, background, and update processes.
3. Turn change pointers back on.
4. Allow the general user population to access the system.
5. Connect the system to its inbound and outbound communication partners.
6. Disable the data migration user.
7. Archive those massive upload files.
8. Delete any intermediate processing data files.
9. Clean up the IDOC tables if needed.  But first make sure that you do not need to keep the IDOCs  for legal reasons.

An Example Data Migration Plan

Click on the link below to access an example data migration plan.  This plan is not a complete data migration plan, but is intended to demonstrate format and content.
Sample Data Migration Plan

Stay tuned!

In part 3 of this series on Data Migration, I will be discussing how to deal with fallout from automated data loads.

How much data to move? How much data to leave behind? What to automate, and what to execute manually?  How to gracefully orchestrate and execute a data migration cutover from one system to another?  Where and how to fit the data migration plan into the overall business implementation plan?  How to continue to run the business during the data migration phase of the business project implementation? These questions are all part of the planning fun!

Data Migration Testing

Processes that we have exercised and continue to exercise on a regular basis are no big deal, do not cause anxiety, or raise any blood pressures.  The daily report, the custom transaction, the interface that runs several hundred times a week – they are all second nature now.  They happen, and no one notices.

But reflect back to the very first time these processes went live.  This is precisely where you are with data migration.  For most, it is a once in a lifetime event.  It is important, then, to raise confidence levels, and reduce anxiety levels surrounding this activity.  This is achieved by practice, practice, practice.

Typically, I like to see at least three data migration test cycles executed in an isolated data migration client.  This allows for several attempts at exercising and fine-tuning the data migration plan; collecting nominal run time statistics to have some idea of how long the data migration might take; identifying and fixing any data migration program object defects; identifying and fixing any data mapping and content defects; and identifying and fixing any functional configuration issues. Each of these tasks are done with the goal of significantly improving the fallout rate with each data migration test cycle.  These data migration test cycles also give us the opportunity to practice our legacy extract skills, our fallout analysis skills, and our fallout manual cleanup skills.

Source Data Quality

Each data migration test cycle begins with the legacy or source data extract.  An important activity between data migration test cycles is the cleanup of the legacy or source data.  The data migration process will most likely discover many data problems.  Customer and vendor address data alone are enough to bring your bolt-on tax jurisdiction determination software to its knees – that zip code that is too short, too long, or just plain incorrect when combined with the associated city and state; the various abbreviations used in place of the actual city name; the city and state entered in the same field when they should have been entered into separate fields; etc.   If the source data is not fixed, your data migration test cycles will begin to look like the movie Groundhog Day.

Source data provisioning and cleanup could very well be problematic, especially in an acquisition or merger scenario where the providers are part of a different organization and have no incentive to participate in your project.  If you encounter this scenario, you will most likely need to engage the appropriate management level from your organization to have a serious discussion with the appropriate management level from the providing organization.  Recognize this, be in control, and raise that flag early.    Do not sit around for several days waiting for data to arrive, as this will only set your project behind schedule.

Data Migration Support for Other Testing

In any business project scenario, the development team will salivate at the thought of testing their customizations against your real data in the conversion test cycle box.  Likewise, the functional team will be chomping at the bit to use your real data to test configuration scenarios.  And the interface team and the data warehouse team can’t ever get enough data to play with.

While working within your three data migration test cycles, just say NO.  You absolutely need a controlled environment for data migration test cycles, and these other teams will not respect that.  They have a different focus and purpose which requires changing and manipulating degrees of freedom that you need held constant.

But, we are all on the same team trying to move the project to the finish line within the expected project timeline.  So to help your other team members, plan for additional data migration cycles to provide this data to these teams.  That’s more data migration practice for you, and it gives everyone else what they need.

The Perfect World of Data Migration

In a perfect world, for each data object to be converted:

1. The functional specification and mapping documents are well-written and clear.
2. A data load file is built exactly in conformity to the well-written functional specification and mapping documents.
3. The data migration ABAP objects are built exactly as specified by the well-written functional specification and data mapping documents.
4. No one is insisting that we move a square peg into a round hole.

In a perfect world, for each set of data that is to be migrated:

1. The legacy system data has been thoroughly cleansed.
2. The providers of the legacy data are genuinely interested in providing accurate data on time.
3. Server-resident load files in a Unicode system are correctly encoded to UTF-8.
4. A delimiter other than comma has been specified for the load file.

In a perfect world, the data migration test cycle client:

1. Is built in a box that is not the development box.
2. Is configured exactly like the client where the data migration ABAP objects were built and tested.
3. Is not open for configuration, unless by design.
4. Is not part of a transport path. to prevent the current cycle of conversion testing from being blindsided by any new configuration changes.
5. Is locked down so that only data migration and data validation tasks can be performed.
6. Is configured to handle a more background and update processes and fewer dialog processes.
7. Is built in a box that has enough disk space to be the repository for the primary load files and any intermediate processing files needed.

In a perfect world, at the project level:

1. There is an overall business cutover or implementation plan into which you can assimilate your data migration plan.
2. The data migration plan integrates nicely into the overall business cutover plan.
3. There is a strategy in place to bring the data current between the time the cutover freeze is enforced and the implementation date.

Stay Tuned!

In the real world, the fun is just beginning.  Those perfect world scenarios just never seem to happen.   But stay tuned!  In my next blog post in this series, I will discuss the details of the Data Migration Plan.  After that, in subsequent posts, I will drill down into some real world scenarios that I have encountered and discuss how I dealt with them.

• How will I ever match the inbound interface parameter “Tom” with “TOM”, or “tom”?
• How will I ever match the inbound interface parameter “Smith” with “SMITH” or “smith”?
• The ABAP WHERE clause is not case-INsensitive.
• There could be hundreds of customers named Tom Smith.
• KNA1-NAME1 and KNA1-NAME2 are not indexed fields.
• And no, we are not storing any portion of either first or last name in an existing indexed field like SORTL.
• There are well over one million customers in the database.
• We have already decided to use PI for all interfaces.
• I will have to buy the BASIS team a case of beer to get them to agree to create indices on the fields KNA1-NAME1 and KNA1-NAME2 in a table with over one million records.

I arrived at the conclusion that I need a case-insensitive database query, along with database indices created for the fields KNA1-NAME1 and KNA1-NAME2.

But, what is a case-insensitive WHERE clause?  A little research and help from colleagues revealed that many had gone before me, and this was nothing new.  To implement a case-insensitive WHERE clause in ABAP, you simply needed to use the native SQL UPPER() construct. The database system that is being used is Microsoft SQL Server, but the UPPER() function and its syntax is similar across different database platforms. This seemed like an easy nut to crack. But, as I soon found out, I actually had a lot to learn.

Take 1

The examples I found were very explicit, showing actual code snippets.  Using my research examples as a template, I built what was to be my first attempt at using native SQL in a ZTEST ABAP program:

exec sql.
  select kunnr name1 name2
    from kna1
    into table i_cust
    where upper(name1) like :l_name1
      and upper(name2) like :l_name2
endexec.

Well, as a native SQL rookie, I was stopped dead in my tracks.  Native SQL does not allow a select into a table – you can only select into a work area.

Take 2

In my ZTEST ABAP program, I changed the code to SELECT into a work area and append the work area to an internal table in a subroutine:

exec sql performing append_cust.
  select kunnr, name1, name2
    from kna1
    into :wa_cust
    where upper(name1) like :l_name1
      and upper(name2) like :l_name2
endexec.

form append_cust.
  append wa_cust to i_cust.
endform.

I really thought that this would work.  But, I was getting a syntax error from native SQL – the table name was not being recognized.  A little more research revealed that, in some cases, only UPPER CASE is valid for table names and field names.  OK.  I can handle that.

Take 3

I changed the native SQL statement to all UPPER CASE in my ZTEST ABAP program:

EXEC SQL PERFORMING APPEND_CUST.
  SELECT KUNNR, NAME1, NAME2
    FROM KNA1
    INTO :WA_CUST
    WHERE UPPER(NAME1) LIKE :L_NAME1
      AND UPPER(NAME2) LIKE :L_NAME2
ENDEXEC.

form append_cust.
  append wa_cust to i_cust.
endform.

This appearedto work just fine…  Except, I noticed that too many records were being returned.  A little more analysis showed that I was getting records from ALL clients, not just my logon client.

Take 4

Again, in my ZTEST ABAP program, specify the client:

EXEC SQL PERFORMING APPEND_CUST.
  SELECT KUNNR, NAME1, NAME2
    FROM KNA1
    INTO :WA_CUST
    WHERE UPPER(NAME1) LIKE :L_NAME1
      AND UPPER(NAME2) LIKE :L_NAME2
      AND MANDT = :L_MANDT
ENDEXEC.

form append_cust.
  append wa_cust to i_cust.
endform.

Now I was getting the results that I wanted!!!!  So, I pasted this ZTEST ABAP code into the ABAP proxy.

A syntax check of the code immediately rejected the use of subroutines – they are not allowed in OO ABAP. I suppose I should have known that.  The error message indicated that I needed to use a cursor instead of the subroutine.  So, I started cursing – a lot!!!

Take 5 (The Final Version)

Here is the final version of the code which works for OO ABAP, and, hence, the ABAP proxy:

EXEC SQL.
  OPEN CUSTCURSOR FOR
    SELECT KUNNR, NAME1, NAME2
      FROM KNA1
      WHERE UPPER(NAME1) LIKE :L_NAME1
        AND UPPER(NAME2) LIKE :L_NAME2
        AND MANDT = :L_MANDT
ENDEXEC.

DO.
  EXEC SQL.
    FETCH NEXT CUSTCURSOR INTO :WA_CUST
  ENDEXEC.

  IF SY-SUBRC NE 0.
    EXIT.
  ELSE.
    APPEND WA_CUST TO I_CUST.
  ENDIF.
ENDDO.

EXEC SQL.
  CLOSE CUSTCURSOR
ENDEXEC.

Well, I finally got it right after five tries.  As mentioned before, I found many explicit code examples, but none were for OO ABAP with a database  requiring UPPER CASE be used, so that the table and field names would be recognized in the native SQL SELECT statement.

I learned something new about SAP.  I hope that I never stop learning.  I also hope that this blog might save someone a bit of distress in trying to work through a similar situation.  Now, I just need to figure out what kind of beer to buy the BASIS team to get those indexes created…

I am currently working on an SAP implementation project that is just starting its realization phase.  One of my first tasks, as a member of the technical implementation team, is to review completed functional specification documents for RICEF objects.  These documents, written by functional subject matter experts, are supposed to detail business requirements that address gaps, and which need to be incorporated into the system being implemented. The purpose of the review is to make sure that the functional specification documents are complete, accurate, and contain the approval signatures required to move on to the technical design phase.

In my career, I have had the pleasure of working with some first-rate functional analysts who know how to draft an excellent functional specification document in a timely manner.  It is this type of performance that helps to move a project along in the right direction, on schedule, and within budget.  Likewise, I have had the not-so-pleasant task of working with not-so-first-rate functional analysts, who draft functional specification documents that are not clear, inaccurate, and incomplete.  The risks here are ultimately manifest as project delays and cost overruns.

The Good…

A really good functional specification document contains enough detailed information about the business process to enable a technical designer to use it as the foundation for drafting a complete and accurate technical design document.  The functional specification document should not only highlight the presence of a gap, but should demonstrate how the business process, accompanied by automation, will close the gap.  This document must also indicate the abnormal processing requirements – what should happen when that report or interface does not run, what are the recovery steps, how are key employees notified of the problem, etc.  The content of a functional specification document must be tuned to the flavor of the RICEF object that it is describing.  Since they perform very different tasks, a report specification document should be very much different from an interface, conversion, enhancement, or form functional specification document.  Using functional specification templates helps to insure the appropriate content for each type of RICEF object.

… and the Not-So-Good.

I am sometimes astonished by the sparse content that is actually offered up for review.  “We need a report” really does not tell me a whole lot about the business process that I am supposed to automate.  Nor does it even hint at the report purpose, content, layout, user interface, execution mode, authorization requirements, or error handling.  And likewise, “Build me an interface” does not even begin to describe the direction, payload content, mapping, frequency, error handling and recovery steps.  It would be so wrong for me to attempt to build a technical design on such meager functional definitions.   One of my favorite cartoons shows a development manager standing in front of rows of programmers saying “You guys start coding.  I’ll go and find out what they want”.

I am further astonished by:

a)      the project managers who apply pressure to accept inaccurate and incomplete functional specification documents, to give the impression that the project is actually moving forward and making meaningful progress.

b)      the functional analysts who whine incessantly when their paltry functional specification document is not accepted.

A functional specification document that does not meet expectations must be upgraded until it does.  But bouncing a functional specification document back and forth like a ping pong ball between the functional team and a technical reviewer is inefficient and wasteful.  I find that the best way to quickly firm up a weak functional specification document is to thoroughly research all of the issues that I found in the document, formulate proposed solutions where possible, and then schedule a face to face collaborative meeting with the functional analyst and the business process owner(s).  This type of collaboration can save hours, days, or even weeks of wasted ping-pong posturing, and that is always best outcome for the project.

Off-Shore Technical Resources

This face-to-face quick resolution scenario typically cannot happen if you have an off-shore technical contingent in play.  In this case, it is absolutely imperative that the functional specification document be most accurate and complete to mitigate the risk of excessive time loss.   Why is that?

Off-shore resources are sometimes time zone shifted eight or more hours ahead of where the project is located.  If a functional specification document is released for review, it will not be analyzed until we have left for the day.  If the off-shore reviewer has questions or raises issues, we will not see these questions or issues until the next day when we arrive at the project site.  When we respond to the questions or issues, the off-shore team will not see our response until we have left for the day.  And so on.

Under these conditions, a poorly written functional specification document with issues takes days instead of hours to resolve.  This leads to unnecessary project delays and cost overruns.

When One is Really Many

That 3PL interface, which was scoped and planned by the business process owners as a single RICEF object named “The 3PL Interface”; and for which only one interface functional specification document is written, is actually many RICEF objects.  We need to move purchase orders, inventory receipts, advance ship notices, inventory picks, and cycle counts between the two interfacing partners.  Each of these represents a different payload, different mapping, is triggered by a different point in the business process, has separate error handling and recovery procedures, and requires a separate RICEF development object.

That single enhancement functional specification document, which addresses all of SD pricing, has the potential to extend into many different user exits.  I just finished coding an ABAP proxy that was functionally specified as one interface.  In fact it was four.  The requirement was to search the database for sales and invoice data starting with either an invoice number, sales order number, customer name, or company name.  Each of these search techniques required the development of a separate method.  The only pieces of code that were shared among the four search techniques were the input parameters and the output return table.

The point here is to make sure that the project planners understand the real complexity and effort required on the development side, and to make sure that the project plan and budget reflects these more realistic metrics.  This really goes a long way to stop everyone from wondering, “It’s only one interface!  What is taking development so long?”

Great Expectations

So what is a reasonable set of expectations for a really good functional specification document?  What is it that we are asking the business analyst to do?

First, let me describe what I do not expect.  I do not expect a business analyst to write code, build tables, design efficient database retrievals, or to decide that one BAPI, function module, class, or IDOC is better than another.

Here is what I do expect:

A clear definition of a business process that is repeatable, and which actually works.  As a pre-automation test for data conversions, I always require the functional analyst to manually enter one of whatever, using the standard SAP transaction for which a conversion program is to be built. Many times, they can’t because the system is not configured correctly, the supporting data is not present, or any number of other reasons which cause the transaction to fail.  An interesting observation is that there is much indignant huffing and puffing during this manual entry “test” process.  But when the manual test fails, I simply remind them that I cannot automate a broken or non-existent business process.

A clear definition of what should happen under abnormal or failure circumstances.  This must include error handling, notification, recovery and reprocessing steps.

A business process that can efficiently be automated.  Requiring a search of sales order header text for the phrase “This is a red order” is a very bad design for automation purposes.  While such a design is technically possible to build, it will certainly be inefficient at run time, and may not always produce all of the red orders.  This is because the key value is a free-form phrase that can and will be misspelled, and abbreviated, along with countless other mutilations of the key phrase “This is a red order”.  There are much better business processes and technical implementations that will more efficiently and more accurately find all of the red orders in your system.

An explanation of the need for development.  Exactly what is the gap, and how will automating the business process close the gap?

Screen shots from SAP transactions depicting data that is to be retrieved or stored.  From the screen shot in the transaction, I can usually determine the exact table and field in the SAP database.  Note that some business analysts are very adept at identifying the actual underlying table and field name.

Clear and concise details with respect to data mappings, formulae, data transformations, conditional processing, etc.  If I come to an intersection and it is unclear whether I should continue to go straight, turn left, or turn right, then the functional specification document needs a bit more detail behind it.

How to insure Consistency in Functional Specification Document Review

Design a separate functional specification document review checklist for each flavor of RICEF object.  Distribute these checklists to the functional specification writers so that they know what the expectations are.  Using a checklist will help to make sure that your review process is consistent and accurate.  Improve these checklists over time.  My Form functional specification checklist document now includes the following check:

Is it physically possible to print the specified content on the specified form using the specified font style and size?  Was an actual printed mock-up provided as proof?

- but only after I had received a functional spec for a form that required four inches of print content on a one inch label.  And somehow, the business analyst who wrote this particular request erroneously thought that it was my problem to solve.   After all, writing code is magic!  Isn’t it?  In this case, I pushed back and insisted that an actual printed mock-up be produced – one that I would then agree to automate.

Summary

A good functional specification document will help tremendously in moving a project forward in the right direction with minimal cost and risk.  A poor functional specification document has serious potential to cause project delays, and  schedule and cost overruns.  The best goal for the project is to achieve a good functional specification document, using whatever means required.

Now, put me behind a keyboard in front of a computer application which I have never seen before. My user experience is all over the map – somewhere in the continuum between most excellent and very poor.  Some application user interfaces are extremely intuitive, well-designed and easy to navigate, logically follow the business process flow, and provide real meaningful help when needed.  Other application user interfaces are extremely difficult to navigate, are not intuitive, do not follow a logical business process flow, and offer little or no meaningful help. And sometimes in these difficult user interfaces, not only has the location of the steering wheel been moved to a totally unsuspecting location, but its appearance has been changed so that, even when I see it, I do not even recognize it as being the application’s steering wheel.

A well-engineered user interface is no accident.  It doesn’t just magically happen.  It must be woven into the fabric of the design and the code; and it should never be shoe-horned into the application as an after-thought.   It takes a lot of up front planning, designing, testing, functional effort and technical effort to produce a really good application user interface.  And yes, designing, building, testing, and implementing a good user interface for your application will extend the delivery time of whatever it is that you are building.

Why is a well-designed and ergonomic user interface so important?  You could have built the best application ever developed.  But if it is unusable, it will never get very far.   Countless hours are lost every day as thousands of frustrated users spend extra time and effort wrestling with poorly designed user interfaces, rather than focusing on their jobs.  And when the frustration levels reach a certain trigger point, the users will seek out and find alternative ways to perform their duties.

Here are a few examples of some very interesting user interface experiences that I have personally encountered.

Let’s Punish the User

I thought we had gotten over this one a long time ago, but I was bitten by it recently while I was completing a web form.  Here is the scenario:

I am completing a screen form containing several fields, and I make an entry error in one of the fields.  When I try to submit the data, the application rejects the submission, and sends me a message indicating where the error field is, and then clears all of the fields on the screen and makes me re-enter all of the data again.

Bad User!  And, because you made a mistake, you get to start the entire form all over again …   and again … and again … and again …

When Help is not Help

I am especially fond of this one.  I need help, so I look for the HELP ICON.  I find the HELP ICON (because it looks like a help icon should look, and it is located where a help icon is typically located … well, sometimes it is) and I press it.  The anticipation of actually getting the help I need is overwhelming.     And the help information finally appears on my screen and reads something like this:

The exact details of what you want to know are located somewhere in our behemoth website.   Now, we could tell you exactly where and actually help you to solve your problem very quickly,    but we’re not going to, because then you won’t have spent hours wandering aimlessly through our website’s colossal labyrinth.  And while you are in there, don’t let the Minotaur get you!   Please go to www.needleinthehay.com.

Well, at least they said “please”.

The Errant Error Message

I can’t tell you how many times I have made a data entry mistake, only to be “helped” by this not-so-helpful error message:

Incorrect Entry

This is irritating.  If I don’t know why it is incorrect, I can’t fix it!  If the application is intelligent enough to determine that the entry is incorrect, it certainly is intelligent enough to tell me which of its check rules were violated (in plain user-oriented language please).

I really like this one:

Program Error 1776

This truly tells me everything I need to know!!!  Maybe the developer knows what error 1776 is, but I don’t.  Maybe I can find the meaning of error 1776 at www.needleinthehay.com. (I hope the answer is not too close to the Minotaur’s lair.)

Hey developer – can I have your phone number so I can call you every time I see one of these messages?  Maybe then you will get the hint and stop the madness!

I’ve fallen in and I can’t get out

Sometimes, I manage to navigate to a screen that has no obvious means of escape.  So I’m trapped until I can figure out the top-secret mouse click or keystroke combination.  Should I place the blue orb on the green pedestal?  I guess I really should have left a breadcrumb trail.  Or maybe the developer was a ZORK player!  Let’s try XYZZY!!!

Just Give me a Good Example

Like pictures, a good example is worth a thousand words.  Good examples are worth their weight in gold.  A really good example can get me to where I want to be – understanding what I need to know – much more quickly than an entire chapter of words.  Good examples are hard to come by.  They require lots of thought and planning.

Here is an example of a bad example:

2  @  2   =  4         where ‘@’ is some arithmetic operation.

OK – did we add or multiply here?  Who knows?  This is totally ambiguous, and a bad example.

Here is an example of a better example:

2  @  3   =  5         where ‘@’ is some arithmetic operation.

It is obvious here that the “arithmetic operation” was addition.

The Cluttered Screen

One of my favorite games as a child was the type of puzzle where a group of objects is hidden somewhere in a very busy picture.  The names of the hidden objects are listed on the side of the page, and the object of this type of puzzle is to “find the hidden objects” that are on the list.

Some user interface designers and builders are very adept at hiding items on their incredibly busy user interface screens.   Perhaps in their youth, like me, they liked to play with “find the hidden objects” puzzles also.  The problem here is that they are designing “find the hidden objects” puzzles into their user interfaces!  Is it really necessary to squeeze in every data field and push button onto a single screen?  Is the perception here that a single screen, no matter how extremely busy, is somehow easier to use?

My end-user work instructions tell me that if I push the “show details” button, I will then know the meaning of life, the universe, and everything.  But where is the “show details” button?  I am really good at playing “Where’s Waldo”!  Just give me an hour or two and I will find it!

The Plethora of Scattered Screens

The opposite of the tightly packing everything onto a single cluttered screen is the scattering of the user interface among way too many screens.  It becomes a memory exercise to remember the user interface elements that I have already touched, and a scavenger hunt to locate the interface elements that I haven’t found yet.  Let’s see … what was that part number that I entered on screen 152?  Or was it on screen 376?  And where do I go from here to get the order number?  Does anyone have a roadmap that I can follow?

Maybe if we could consolidate the enormous number of screens down to a few that represented logical work flow groupings, I could spend less time screen hopping and be more productive.

Let’s Get Colorful and Fontsy

They use several different font styles, sizes, and colors.

I suppose that’s OK, isn’t it?  It really wasn’t too difficult to read the sentence above, was it?

Consistency is Key

Let’s examine the rental car scenario that I posed at the beginning of this blog.  The reason that I am able to quickly enter the car and drive away is because a recognized user interface standard has been established and consistently applied to all automobile makes and models.  Among the various cars that I am faced with, the gas pedal always looks like a gas pedal, and is always located on the driver’s side floor to the right.  Likewise for the steering wheel, the brake pedal, and most other major user interface elements found in a car – they all adhere to standards of look and feel and placement.

It should not be too surprising then, that the not-so-ergonomic user interfaces that I have encountered do not even recognize that a standard might even exist. The really difficult user interfaces mix things up, and change the appearance, behavior, and location of controls from screen to screen.

Summary

The ergonomic user interface – the one that is intuitive, provides real help when needed, follows an organized logical workflow, and is easy to use are important components of a really great application that end users will want to use because it actually will be the best way to do their job.

Here are a few guidelines that I use whenever I build a user interface:

1)      Totally and completely understand what the business requirements are, and what the user needs to accomplish.  This is the first and most important step to building a good user interface.

2)      Design the user interface to follow the workflow.

Organize and logically group functions together.

3)      Determine if standards exist and consistently use them everywhere.

Position controls in expected and predictable locations.

4)      Give feedback to the user in a timely manner.

Don’t wait until screen 3 to inform the user that there is a problem back on screen 1.

5)      Craft meaningful error messages in user-oriented language.

Temper the messages appropriately – if everything is hot, then nothing is hot.

6)      Make sure that the “help” you provide really helps.

Ask the user to confirm before deleting anything.

Use list of values help whenever possible.

7)      Use default values whenever possible.

This avoids potential data entry errors and makes the user interface more efficient.

For example, if I work in warehouse 2, I should not have to enter that information every time I ship an item.

Display default values in pre-populated, grayed-out fields; but add a pushbutton to allow for the rare occasion when the user must override the default value.

8)      Keep training documents current.

As a developer, you may not own the training documents – but someone does.

There is nothing more frustrating than a training document that does not match the actual program behavior screen for screen and mouse click for mouse click.

I hope that this blog can provide some useful hints for building effective and efficient user interfaces.  Please reply to this blog if you have any most interesting user interface stories that you would like to share with our readers.

DataXstream, an SAP Solution Partner, builds, packages, and distributes custom solutions for our clients.  We develop all of our custom add-on products in our own reserved and unique namespace /XSTREAM/.  But, we also need to reserve a separate unique namespace for each add-on product that we package and deliver using the SAP Add-on Assembly Kit.  So, we have a single development namespace /XSTREAM/ and a separate “packaging and delivery” namespace for each add-on product.  Why is that?

Software Component Name and Software Component Version

Let’s begin with the software component name.  A software component name references a set of objects that are always delivered together.  It contains all of the programs, classes, function modules, tables, message classes, BADIs, basic types, transaction codes, … etc. – all of the objects which constitute a custom add-on product.   We try to make our software component names meaningful, so if we are building a product named PRODUCTA, we will try to create its software component name also as PRODUCTA.

This product has a product lifecycle which begins with its initial major release at version 1.0.  As bugs are discovered and fixed or as small enhancements are made, minor updates are released, let’s say at version 1.1, version 1.2, etc.  And after some serious product re-engineering, the product is then distributed as major release 2.0.  So, the software component name represents all of the objects which constitute the product, and the software component version represents the context of the product within its own product lifecycle.  The complete picture is provided by the combination of the software component name and the software component version.  For example, we might refer to PRODUCTA at release 1.3, which is different from PRODUCTA at release 1.2, which is different from PRODUCTB at release 2.5.

At DataXstream, we build and distribute several add-on products, so you can imagine that each product currently exists at a different release level within its own product lifecycle.  PRODUCTA may be at release level 1.3, while PRODUCTB may be at release level 2.5, and PRODUCTC may still be under construction and not yet released.  If we fix bugs in or make minor enhancements to PRODUCTA, then we need to bump the software component version of PRODUCTA (and only PRODUCTA) from release 1.3 to 1.4.

Let me add another degree of complexity to this picture.  We build and deliver the same add-on product to different SAP environments.  For example, we have clients who are running SAP in the 4.7, the ECC 5.0 and the ECC 6.0 environments.  Since these environments are different, it is possible that the same product may exist at different releases in its lifecycle across all of these three SAP environments.

To help us manage this complexity, the actual software component version release names that we actually use are a bit more descriptive than 1.0, 1.1, 2.5, etc.   This will be explained in detail later on in the section Maintaining the Software Component Version.

Maintaining the Software Component Name and its link to a Namespace

The software component name is derived directly from a valid namespace that has already been registered on your SAP system.  It is, in fact, the namespace name without the two enclosing ‘/’ characters.  This establishes a relationship between the software component name and the namespace.  SAP checks and enforces this relationship when maintaining software component names.  We try to make our software component names meaningful, so if we are building an add-on product named PRODUCTA, we will reserve and register namespace /PRODUCTA/.   This allows us to then create the software component name PRODUCTA.

In SAP, we establish our software component name using transaction SSDC.  This is the Software Delivery Composer which is part of the SAP Add-on Assembly Kit.  From the initial screen in SSDC, the menu path is Environment -> Create/Update Add-on Software Component.

On this next screen, you enter the software component name that you wish to maintain.  The software component name must be derived from an already-registered valid namespace.  For example, if you have already registered the namespace /ABCD/, then you can create a software component that is named ABCD.  SAP checks this relationship, and issues an error message if you try to create an invalid software component name that cannot be derived from an already existing namespace.

SAP allows as many as ten characters for the software component version or release name.  This allows us to build meaningful release names, which really helps to sort things out when you are dealing with many products over several SAP environments.  Here are a few demonstrative examples of how we build our release names:

Maintaining the Software Component Version

On this screen, you enter the software component version or release name, and a description, then press the SAVE icon.

Another Piece of the Puzzle – the Package

We want each of our products to have its own unique software component, because each software component or product has its own unique set of objects and needs to exist at its own software component version within its own product lifecycle.  We need to allow all of the development objects for PRODUCTA to be at release 101_470 and all of the development objects for PRODUCTB to be at release 103_500. But, if we perform all of our development for all of our products in the /XSTREAM/ namespace, how do we assign all of the /XSTREAM/ development objects for PRODUCTA to software component PRODUCTA, and all of the /XSTREAM/ development objects for PRODUCTB to software component PRODUCTB?

The answer is in the package or development class.  When creating a package, you must specify the software component that the all of package objects will be assigned to.  The example below shows the creation of a package or development class /XSTREAM/PRODUCTA.  One of the parameters of the package is the software component.  We would assign software component PRODUCTA to this package.

So, when building the product PRODUCTA, we create many development objects (programs, classes, tables, etc.) in the /XSTREAM/ namespace, we assign those objects to the /XSTREAM/PRODUCTA package, which in turn, assigns those objects to the software component PRODUCTA which exists at a particular release level in its lifecycle.

Summary

To summarize, because of the dependencies between namespaces, software components, packages, and development objects, these actions must occur in the following order:

DataXstream, an SAP Solution Partner, builds, packages, and distributes custom solutions for our clients.  We have built and packaged these solutions both in our own SAP landscape and in our client’s SAP landscape.  In doing so, we must be careful about how we manage our namespaces, their associated development and repair license keys, and packages.  It is not surprising, then, that I have received several inquiries asking about our namespace strategy for the development, packaging, and distribution of add-on products.

It all started with two name ranges

If you have ever developed a new object in SAP, you are probably already familiar with the customer name range.  SAP has established the customer name range because within an object type (e.g. report, table, authorization class, etc.), the name of each object must be unique.   It is important, then, to use some organized method of insuring that the names of your custom-developed SAP objects will not conflict with the names of standard SAP objects; and that your custom-developed objects will persist if either a patch is applied or an upgrade is undertaken.

A table showing the Y*, Z*, and other naming prefixes used for naming various SAP objects in the customer name range can be found in SAP documentation.  I have placed this table in the Appendix of this blog.

Then there were naming conventions

For most custom-developed objects that are local in scope, using the customer name range is sufficient to protect your custom development.  This becomes a bit more problematic if you are sharing your custom development with other organizations within your company that also have custom development capabilities.  Here, it may be possible to establish and use a naming convention solution; but this solution can be difficult, as it requires stringent up-front agreements and continuous monitoring.

Protecting custom development from naming conflicts and also from unauthorized modification is also an issue for SAP Solution Partners who distribute their custom-built products and solutions to a wide variety of clients.  In this situation, it is impossible to consider the use of a naming convention.   What is the solution?

Now, the namespace

DataXstream is an SAP Solution Partner.  A service that we offer to our clients is the development of custom solution add-on products.  We can deliver these products to any of our customers, and be assured that our custom development will be protected from unauthorized customer modification, and from SAP patch applications and upgrade procedures.  How are we able to achieve this?

For these situations, and starting with release 4.0, SAP has provided the unique namespace, which is reserved for exclusive use by an SAP customer or partner.  Using the unique namespace not only prevents naming conflicts, but it also prevents unauthorized modification of distributed objects that were built in the unique namespace.

A unique namespace is reserved with SAP against your SAP installation number.  It is not transferrable, and can only be registered and used on that installation of SAP.  (You can find the installation number of your system in the SAP System->Status screen.)  The namespace also comes with a development license key and a repair license key.  Only the owner of the unique namespace development license key can develop objects in their unique namespace; and only after the development license key is registered on the development system.  Likewise, the recipients of products that were built in the unique namespace cannot modify or repair the delivered objects without receiving the repair license key from the namespace owner.

DataXstream has applied for and has reserved the unique namespace /XSTREAM/.    All development at DataXstream that is intended for customer distribution is built within our /XSTREAM/ namespace.  In development, the namespace is used as the prefix for naming all of our objects.  So, for example, if we are creating an ABAP report, we would name it /XSTREAM/myabapreport.  And if we are creating a dictionary table, we would name it /XSTREAM/mytable.    Since the /XSTREAM/ namespace is unique, we never need to be concerned about naming conflicts.  Also, since we hold the development and repair license keys, we never need to be concerned about unauthorized modifications, once the objects are distributed to our customers.

As DataXstream may have several solution products either already completed or under construction, we need a way to organize the objects within each solution project.  We do this by assigning all of the objects that belong to a project to a unique package.  For example, if we had a project named “PROJ1”, we would build a package named /XSTREAM/PROJ1 and we would then assign all of the objects for that project to package /XSTREAM/PROJ1.  This is a most useful way of organizing the objects, since we can then use the SAP Object Navigator to manage all of the objects belonging to a specific package.  It is also useful when packaging a product for delivery, using the SAP Add-on Assembly Kit.  When it comes time to define the list of objects to be included in the delivered package, the Add-on Assembly Kit allows you to indicate that you want to include all of the objects belonging to a specified package.

Obtaining and using a unique namespace

There are four steps needed to acquire and begin using a unique namespace:

1. Request a namespace
2. Reserve an accepted namespace
3. Generate license keys for a reserved namespace
4. Register the namespace on your development system

The first three steps are performed on the SAP Support Portal at this address:

www.service.sap.com/namespaces

This site is very well laid out and the steps are clearly explained.  Here is what it looks like:

Things to have ready before you request a namespace.

Have your namespace name and a brief description ready.  A namespace name can contain no more than ten characters (including the two ‘/’ characters e.g. /ABCDEFGH/).  I would not make this too long, because some object names, like SAP table names, are on the short side.

Select the SAP system where you want to develop in your unique namespace and get its installation number.  The installation number is found on the System -> Status screen.

Step four is performed on your development system using transaction SE03.

In this transaction, under the Administration folder, you want to select Display/Change Namespaces.

It is here, in CHANGE mode, that you will add a new entry to register your namespace, along with the development and repair license keys.  After registering your unique namespace on your development system, you are now ready to begin using it.

Summary

As mentioned before, DataXstream offers custom development and packaging services using either our development landscape or yours.  With respect to strategies for namespaces, packaging using the SAP Add-on Assembly Kit, and product distribution, we have expert consultants who can advise, walk you through the process, or do it entirely for you.

Our contact information can be found at www.dataxstream.com.

Appendix

Customer Name Ranges as shown in SAP Documentation