In this post, we will look at some of the customization areas which add risk and the cost to an upgrade project.

1. Direct Modifications to SAP Standard Objects

Direct modifications to SAP standard objects bear the highest risk.  During the upgrade, these modifications will be lose–either because they will be overwritten by their respective upgraded SAP standard objects; or because these objects no longer exist and are not used by standard SAP in the upgrade system.

Typically, there are two reasons for performing direct modifications to SAP standard objects:

1. The application of authorized individual OSS corrections to fix problems.
2. Direct customer modification to provide business process enhancements.

1.1. Application of Individual OSS Corrections and Support Packs

Individual OSS corrections are patches supplied by SAP to fix recognized problems with SAP objects.  Sometimes, individual OSS correction instructions are included and delivered within a specific level of support packs.   That means that applying support packs to the specified level will apply the desired correction, and manual intervention is not needed.  Sometimes OSS correction instructions are not yet included in any specific level of support packs, and these will need to be applied either manually or with the SAP Note Assistant.

It is important to carefully analyze the validity range of an OSS note to determine both how it was applied in the legacy system, and how it is to be applied in the upgrade system.  This analysis will help determine how far to patch the upgrade system to make sure that the desired corrections are included in the upgrade system.   Any needed correction that exists at a patch level beyond the highest patch level in the upgrade system will need to be addressed  manually or with the SAP Note Assistant.

The decision of which support pack levels should be applied to the upgrade system must be made carefully.  One approach is to apply the highest level of support packs that are available.    The rationale for applying the latest available support packs is to insure that the most recent problem fixes are applied to the upgrade system.  This is a two-edged sword, as the most recent  support packs may also introduce other problems into the upgrade system.

1.2. Direct Customer Modification of SAP Standard Objects to Provide Business Process Enhancements

Direct modifications of SAP standard objects to provide enhancements are most costly and bear the highest level of risk.  While direct modifications of SAP standard objects may have been technically expedient at the time of implementing the enhancement, they will always require additional work effort and analysis at upgrade time to determine:

1. Whether or not the functionality provided by the enhancement is still required by the business.
2. Whether or not the functionality provided by the enhancement is already included in the upgrade system as standard SAP.  You will need the help of functional subject matter experts here.
3. Whether or not the modified SAP object still exists in the upgrade system.
4. How to move these required enhancements into the customer namespace, using the SAP preferred enhancement procedures.  Moving these types of enhancements to the customer namespace also moves them from the very costly and highest risk category to a lower cost and lower risk category.

2. Custom Interfaces

Custom interfaces are loaded with opportunity for high upgrade risk.  Take, for example, the SAP standard IDOC Basic Type, which was copied as the baseline for an enhanced customized basic type containing additional custom segments and fields.  In the upgrade system, a new version of this SAP standard IDOC Basic Type may now contain additional segments and fields.

Likewise, the associated standard processing module for that standard basic type, which was copied as the baseline for an enhanced processing module to handle the custom segments and fields of the enhanced custom basic type, may also have changed in the upgrade to accommodate the new standard IDOC version.

So, in the upgrade system, especially if you need to implement the new upgrade IDOC version, it may be necessary to start with the new standard basic type and standard processing module, and reapply the customizations.

3. ABAP Code Objects

Of all customized objects discovered in most SAP systems, ABAP Code objects are the most numerous.  These objects include classes, methods, function modules, report programs, dialog programs, and include files which support the all of the other ABAP object types.

At greatest risk are the SAP standard ABAP code objects which were modified directly in the SAP namespace; and those that were copied into the customer namespace and subsequently modified. This is because the standard SAP code objects may have changed in the upgrade.    These changes can range anywhere from simple error corrections to complete process and code redesigns.  In these instances, each modified object must be compared to its counterpart in the upgrade system to determine its existence, and the nature and extent of what might have changed.   If the customization is still needed in the upgrade system, and to stay current with the enhanced standard code, it is probably best to start with the upgrade standard code and reapply the customizations.

ABAP code objects that were developed entirely in the customer namespace (not copied from standard SAP code) are usually low on the risk ladder.  Problems occur when data selected by these objects has moved elsewhere or behaves differently in the upgrade system, when ABAP constructs used are now obsolete in the upgrade system, or when standard called functions are changed in the upgrade system.

Ultimately, the business must decide why SAP standard code was directly modified, why SAP standard code was copied into the customer namespace and modified, why custom code was developed entirely within the customer namespace, whether or not any of these modifications are needed, and how they might be implemented differently in the upgrade system.  This evaluation presents another opportunity to remove some rocks from the customization rock bag.

4. Automated Standard SAP Transactions

Custom ABAP programs which automate standard SAP transactions bear an above-average risk.  This is because the behavior of some SAP transactions may have changed in the upgrade.  These changes can be manifest in the functional performance of the transaction, the technical location and existence of data fields on screens, screen field properties, IMG configuration specific to the transaction, the transaction code associated with the transaction, or even the existence of the transaction or an equivalent in the upgrade system.  The nature and the extent of the changes to the transaction in the upgrade system will determine the amount of effort needed to automate it.

Also, if an “old” SAP transaction has been replaced by a “new” upgraded equivalent, it is usually wise to make the change to the new transaction at upgrade time.  At some point in the future, the “old” version of the transaction will become obsolete and no longer useable or supported.

5. Data Dictionary Customizations

Many customizing projects within SAP require the storage and manipulation of data that is not included in the standard SAP dictionary.   One way in which these requirements can be addressed is by building complete custom transparent tables and data structures.  Another way is by adding append structures and append fields to existing standard SAP tables.  With either approach, it is always an excellent idea to evaluate the need to move these customizations into the upgrade; and another opportunity to lighten the customizing bag of rocks.

6. Customer and User Exits

Customer and user exits are provided by SAP as the SAP-approved mechanism for enhancing standard functionality.  Enhancements performed within these boundaries are guaranteed by SAP to bear low technical risk in an upgrade situation.  While SAP guarantees that the custom code will arrive technically intact in the upgrade system, functional analysis must still be performed to determine the continued need for the enhancement in the upgrade system.

Any enhancement implemented in a customer or user exit that is either functionally covered or determined to be unnecessary in the standard upgrade system should be retired.  This is consistent with the goal of lightening the load in the bag of rocks.  Since a customer or user exits may implement several different enhancements, take extra care to make sure that only the right code or other objects are retired.

7. The Environmental Effects – Operating System Change

OK, changing the operating system during an upgrade project is not customization of SAP objects.  But it does add risk and cost to the project.

Why?

If the upgrade operating system changes from UNIX to Windows or vice versa, then any ABAP program or interface which reads or writes a data file to an operating system folder must be changed.  The changes needed to these ABAP programs are very narrow in scope,  and reflect the manner in which the operating system specifies its file paths.  Additional BASIS work, such as determining the proper file system authorizations, may also be needed.

Using the SAP logical file name facility can help enormously by removing the need to make changes to hard-coded file names and path names in many different ABAP programs.

Share Your Experience

At the top of this post, I stated that I would discuss some of the customization areas which add risk and cost to an upgrade project.  I look forward to hearing from our readers to reply to this post with their own experience with customizations, how they were handled, and how this affected an upgrade project.

In my next post on this topic, I will reveal some of the techniques I use to “discover” customizations, even in the absence of documentation.

[Editor's Note] This blog is the second of a multi-part series:

1. Part 1
2. Part 2
3. Part 3

The build of an SAP system during upgrades or cutovers will change during Integration Testing.  Additional kernel patches, support packs, OS or DB patching (Microsoft vulnerability, Oracle security, etc), interface architecture (filesystem mountpoints and system connectivity), and functional transports drastically change the environment between QA and Production Cutover.

Items such as bandwidth issues due to other systems activity or backups, bad install package or kernel file, low disk space in filesystems, transport conflicts or failures due to bug fixes, project plan sequencing issues, and system dependencies are some items that can be discovered prior to production cutover activity.  Many of these are not discovered during the build of QA due to a more relaxed timeline, only working during normal business hours, and changes as a result of formal integration testing, and basis administrator resource changes.

Here are a few guidelines for a successful practice activity.

• Use hardware that is close to your SAP production system (location, networking, CPU speed, OS and DB version)
• Perform a SAP system copy from Production to the practice server to establish baseline
• Run the practice as close to ‘reality’ as possible (day of week, time of day, location of administrators)
• Use same upgrade package that will be used for production
• Use same technical process for applying the patches (terminal server, downtime minimized)
• Follow the production cutover plan for this activity (not capturing timing for backups or file transfers can add hours to a cutover plan)
• Use the practice to monitor and create a ‘watch list’ for long-running support packs or transports
• Practice a SAP system restore activity to understand contingency planning and timing

In this post I’ll do a combined discussion about the use of offshore resources in an upgrade project as well as share some experiences working with remote resources.  My colleague, Mike Salvo, has already discussed ABAP customizations in an upgrade in this post.  Now that you’ve found these customizations, what do you do next?  Actually Mike provides loads of good advice about what to do next in terms of sorting out what is in the overall pile of objects that need to be examined.

What I hope is that you have documentation related to these objects: information that tells you why they were created, what they do, where SAP functionality is deficient in the current release and how you worked around the shortcoming.  This should be helpful in making the evaluation about whether you can remove a particular object or if you need to make sure it works in the new release in a way that satisfies your business and/or technical need.

I going to assume that you have been through the “bag of rocks” analysis described in Mike Salvo’s posts and now have a collection of pebbles, stones and boulders to work through.  This is where you can make good use of offshore resources to help out: there’s a lot of discussion about the use of offshore resources and you can use them really well or really badly.  Let me digress.

SAP Offshore Development Resources

I’ve heard many complaints regarding offshore resources, but the two biggest complaints that are trotted out as a serious offset to the expected cost savings are:

• The time difference makes turnaround time longer than projects plan for or like.  For example, if you work with resources in India you have to synchronize working schedules and often this means team members either work way late or very early in the day.
• The accuracy and completeness of what you ask for determines the quality of what you get back (a.k.a. you get what you ask for, not necessarily what you want): the difference in time zones makes it difficult to get clarifications and address questions quickly.  Consequently, development life cycles can be extended even if the hourly cost is lower.

A straightforward way to address the first issue is to work with an offshore partner in a time zone closer to yours.  In this respect I’ve worked with development partners in South America where the offset with US time zones is small.  This helps address the second complaint: by being in similar time zones it is much easier to either pick up the phone and address questions or send e-mail and get a quick response to a question.  I’ve been on projects where a quick answer to a question had to wait a day because of time zone and work schedule difficulties.  There may be a slight premium for using an offshore team in South America, but the turnaround time and responsiveness usually makes it worthwhile.

How does this digression help your upgrade?

Once you’ve sorted your bag of development rocks you can go through another level of review to rank the objects in terms of complexity and/or difficulty of remediation for the upgrade.  In general, I’d recommend keeping the most difficult work items close at hand and have onshore resources work them.  Any questions about what the object is supposed to do, any business process impacts can be addressed readily with end users and business analysts.  Objects that are less complex can be worked by the offshore team: you stand a good chance of getting what you need by providing clear documentation (all those project artifacts you maintain) and explicit instructions around what you think the remediation entails.  You should also set expectations around what comes back from your offshore partner: code that is free from syntax errors isn’t going to be enough, some level of testing and documented results are more likely what you want.

SAP Upgrade Onshore and Offshore Resource Balancing

I can’t tell you or even estimate how many onshore and offshore resources to engage in an upgrade project without knowing more about your environment and object analysis.  However, I would definitely recommend two key onshore roles:

• Development Object Remediation Lead: an individual who has responsibility for coordinating and prioritizing the objects sent to the offshore team and managing the associated sign out and sign in process.  This person also has the unenviable task of serving as the quality control checkpoint and enforcer of expectations with the offshore team.  There should be a corresponding role in the offshore team, too.
• Development Object Remediation Senior Consultant: one or more individuals with the expertise to address the complex objects

Smart use of offshore resources has the potential to reduce development object remediation cycle times and that should lead to direct cost savings.  Any steps that accelerate your overall upgrade project time line and contain costs should be viewed as a good thing.  Try it – you just might like it!

In short, if you have done a good job with your earlier implementation activities you should have a plethora of materials that can be reused.  Remember that after an upgrade your basic goal is to still be able to do what you could do on the previous release along with anything new you are introducing.  The way you execute business functions may be slightly different, but the underlying objective should be much the same.

For more on SAP Upgrades download the white papers:

SAP Upgrade Checklist

Customization Risk Analysis in an SAP Upgrade Project

SAP Project Artifacts for Upgrades

In short, don’t reinvent the proverbial wheel if you have one available to you already.

As you went through your initial SAP implementation and ongoing support work you probably generated piles of documentation and if you managed this well throughout the ongoing project lifecycle there is a good chance that you can reuse it.  In essence, the better job you have done identifying, capturing and testing the functionality used in your SAP system the easier it should be to reuse those artifacts to confirm continuity of business processes in the new release.  It sounds easy, but it is a rare project that consistently maintains and updates project documentation as functionality is introduced through point releases, production fixes, etc.

Whatever information you have is worth reviewing to see if it can be reused or at least used as a basis for building out the scope of testing for the upgrade.  In addition to the testing scope, you may have test plans, procedures and protocols that can be adapted.  Typical project artifacts for consideration include:

Functional Scope

Any SAP project starts with an initial functional scope which grows over time.  This is a great start point for defining what you need to test and circling back to find where you implemented non-standard functionality.  The standard functionality is likely to stand up to the upgrade and work straight away, whereas the non-standard stuff is where you want to pay special attention.  In this scope you want to make sure you capture any special periodic processing, e.g. daily, weekly, monthly, quarterly and annual processes to be tested

Business Process Procedure Documents

In conjunction with your functional scope, you probably have business process procedure documents that provide detailed step by step instructions on how to execute job functions.  You can use these as a tool to verify whether business process steps have changed.  In the event that the steps have changed it is clear what documentation needs to be updated.

Technical Objects Scope

Similar to the functional scope you should have a catalog of technical objects, e.g. reports, custom programs, user exits, SAP modifications (if you went down that path), etc.  This provides a list of objects to examine and evaluate as part of the upgrade.  This is a topic covered in depth in a separate blog posting.

Test Plans, Scripts, Procedures and Tracking

Now that you have a functional and technical scope defined you can also revisit any test plans you have.  Testing comes in many flavors (click here to discuss some flavors) and you probably have prior test artifacts for functional testing, end-to-end testing, security and authorizations testing, etc. which are worth a review.  On top of this any regression test scripts and automated test tools may be re-deployable in support of upgrade testing

On top of this most projects have established procedures for tracking issues, resolution steps, and progress reporting.

Testing Organization

In conjunction with your test plans you probably have documented roles and responsibilities, test team organization structure, escalation routes and sign-off procedures.  Again, reuse all that you can.

Partner Systems, Interfaces, Batch Processing, Performance Metrics and System Outputs

Very few SAP installations are islands of functionality, instead there are numerous partner systems that interface with SAP as well various outputs in both electronic and hardcopy forms.  An inventory of these items provides another checklist for testing and identifies both internal and external partner systems you need to include in your upgrade test cases.

System performance is often critical for interfaces and batch processing and any historical activity that identified and tested these elements provides a potential guide for ensuring no degradation of these things as a result of the upgrade.

SAP Upgrade Go-Live

Again the experience from other go-live events may form a good basis for building the upgrade production implementation plan.

Supplement these artifacts with additional content for new functionality and modify for anything that has changed because of the upgrade.

Get Organized for Your SAP Upgrade

By no means am I saying that everything you have built and accumulated over the lifetime of your SAP installation is going to lend itself easily to being recycled in support of your upgrade activities.  However, I am saying that you put in a lot of work to get to this point, you already have a good idea of how to test SAP, you know the quirks and nuances of your organization, so don’t feel like you need to start from scratch.  Cast a careful eye over what you have in your project inventory and see what make sense to apply and what to discard.

And if your SAP project inventory is not a robust as it could be, think about how to strengthen it ready for the next time you have go through an exercise like this.

The choice to upgrade your company’s SAP platform is a very important business decision.  Many criteria need to be considered when determining if an SAP upgrade is the right move and, if so, what type of upgrade needs to take place (ECC 6.0, Enhancement Packs, etc.).  A successful SAP upgrade requires the determination of your upgrade requirements, proper planning, and an assessment of technical and functional risk.  Below is a sample of our SAP Upgrade Checklist white paper:

Determine Your Upgrade Requirements

1. What are the business reasons for upgrading? Support from the business for an upgrade project is most important.  If there are no business reasons for upgrading, then you should probably not do it.  Included here are the business risks incurred by not upgrading.
2. What are the technical reasons for upgrading? Included here are the technical risks incurred by not upgrading.  (Posture increasing maintenance fees for old versions or complete support withdrawal as a business risk – not a technical risk.)

For the complete list, download the full white paper:

SAP Upgrade Checklist

Related Links:

1. SAP Upgrade Project Plan
2. SAP Upgrade Project Management Considerations
3. Contact Us

On several occasions, I have been engaged on projects with statements of work containing some or all of:

HELP!  We need to upgrade our SAP system, and we do not know what has been customized, and our original implementation partner has been gone for over several years, our several enhancement partners are also long gone, best-practice controls were never implemented, we have no documentation, and we need all of our customizations, whatever they are and wherever they are, to behave correctly in the upgrade system.”   (And I’m sure that you can add to this list!!!).

At the time of initiating the engagement with a client, how this happens and why this happens is not important.  What is important is that there can be an assessment of the risks, an evaluation of the costs, and an understanding that it may be possible to significantly determine the extent of the customizations, even without documentation.   The customization discovery process, then, becomes a matter of knowing how and where to look.

Early in the process, I remind my clients that customizations are expensive to create, expensive to maintain, and expensive to upgrade.  I like to use the analogy that custom objects are like a bag of rocks which you must drag forward with you into each upgrade.  Of course, if we can empty some of the rocks from the bag (i.e. eliminate unnecessary customizations), then the dragging becomes less labor intensive, the upgrade risk is decreased, and the upgrade cost is lowered.

To assist with emptying the customization rock bag, I start with the concept of custom object usage.  It’s always eye-opening for a client to discover that those “must-have” and “can’t go live without” custom reports, custom data tables, custom transactions, etc., have not been used in a very long time.  The concept of “very long time” varies for each customized object because some business processes are exercised only quarterly, semi-annually, or annually.  Even more disconcerting to the client is the knowledge that they were expensive to design and build, and continue to be expensive to maintain and support.  And now, even if we have to stop and briefly analyze whether or not they should be brought into the upgrade system, here they are again exacting yet another cost.

Another way to empty rocks from the bag is to show that the standard upgrade system is already enhanced to include the desired customized function.  The functional subject matter experts from the business should be adept at doing this.  In fact, they may have already done this exercise as the formulation of the business case for doing the upgrade in the first place.

Yet another way to empty rocks from the bag is to show that the enhancement embodied by the customization was present in the legacy system all the time; and is likewise already included in the upgrade system.  Maybe it was not recognized, maybe it could have been implemented via configuration, and maybe it was just too easy to throw it over the wall at programming, rather than figure out all of those configuration switches and table values.  Whatever the reason, here we are, and there is a better, less risky and less costly way to do it within the standard SAP system.

Custom Object Upgrade Risk Assessment

It is expected that SAP will have enhanced its standard processes in the upgrade system, and that these enhancements may alter the behavior of, or render any customization obsolete.

Custom object upgrade risk assessment is merely a probability that a custom object will or will not perform correctly when migrated to the upgrade system.   It embodies the reality that some customizations performed in the legacy system will fail to work at all in the upgraded system; some will transfer, but will require additional work effort to adapt them to the new upgraded environment; and some will migrate cleanly with no additional work effort needed.

There are no guarantees.  Even a simple report with a seemingly low risk assessment will fail in the upgraded system if the data being reported changes location, format, or function in the upgraded system.  And it may silently fail in the very sinister way of appearing to work correctly while, unbeknownst to the user,  actually reporting incomplete or incorrect results.  It is imperative, then, to exercise all customizations in the upgraded system with staged data, test scripts, and comparison of actual results to expected results, to make sure that they continue to perform as expected.

Read the Full White Paper: SAP Upgrade Customization Analysis

Please fill out the form below to receive access the whitepaper. The whitepaper.pdf document will be sent to the Email address provided below. This paper looks further into some areas of object customization and discuss their relative risk and cost impact on an upgrade project.  In the meantime, please feel free to reply to this post to share your own unique experiences with customization discovery and upgrade risk.

Related Posts

SAP Upgrade Project Management Considerations

SAP Upgrade Project Plan

SAP Upgrade Checklist