In continuing my series of lessons learned with SAP Retail I would like to cover the topic of Client Resources and Planning. Again, in general many of these lesson I have learned could be applied to any project and I can easily draw parallels from other SAP industry solution implementations that I have led in the past. I will do my best to highlight where I think there are specific retail differences.

There Are Never Enough Client Resources

This is the case on almost any large-scale project, not just SAP projects. It impacts smaller organizations and it impacts retail businesses more because they tend to run leaner IT organizations. So you might be saying to yourself, “big deal; what project not funded by the government doesn’t have enough resources?” This one point however might be the most important one I make in this series.

Think about it. In a new SAP Retail implementation you are going to completely replace the core of your IT business systems. This typically impacts all aspects of your business; Finance, Store Operations, Merchandising, and Warehousing. It typically reaches outside your organization and impacts your Business Partners as well. So how do you fix resource problems? As a leader, and I am specifically talking to the CEO (or your organization’s equivalent) here, because he/she is the only one that can make things happen on the resource front. CEO’s, you need to engage the entire business in this process in a meaningful and organized way from the project’s inception and kick-off all the way to go-live and post go-live support.

I know from experience that this kind of ideal C-suite-to-IT alignment does not always occur. My uncle was the president of a division of Honeywell that implemented SAP. I can remember talking with him about his project and being a little frustrated with him. He knew the project was important and impactful but from the language he used I could tell that he did not grasp the significance of the undertaking that an SAP implementation demands and that more overall focus and dedication from the organization as a whole would reduce the time required for the business to successfully adapt to the new system.

Don’t make this mistake in your implementation; engage the best of the best in your organization on the project, and engage the entire organization in meaningful ways throughout the entire project. The more exposure and input people have doing the implementation the easier the transition will be. Always remember, there is no such thing as “cutting corners” in SAP implementations (or any SAP project for that matter). You always end up getting as much from your SAP system as you put into it. Settling for adequate resources at bargain prices will do two things: cut immediate resource overhead costs and produce an adequate SAP system for your business.

Client Management Expectations For Their Internal SAP Resources Are Always Unrealistic

OK, so this is not specific to Retail; it is a recurring theme from all my SAP project exposure.

• First point: unless you have implemented SAP before you and your resources will automatically underestimate what it is going to take to get the job done.
• Second: don’t expect seasoned IT resources to work an SAP implementation and come out of it completely capable of supporting the system on their own after go live. One out of ten will be able to fly solo after go live,  the rest will need assistance for sometime in the future. (By the way, the ones that fly solo need to be rewarded significantly, if not they will be consultants within 12 months)
• Third: look at what you are asking of your resources. If you have them working on business requirements, functional design, test planning, test script creation, test execution, training, etc., do they really have the capacity to do any of these activities simultaneously while maintaining high-quality output? Engage more of the business and spread out the load. After all, implementing SAP is a business commitment, not an IT department commitment. I will make recommendations in other sections on where you can engage more of the business.

There Is Never Enough Time For Knowledge Transfer

Knowledge Transfer (KT) is an activity that is always held high at the beginning of the project but it is the first part of the plan and budget to get absorbed when things don’t go as planned and it is never replaced. In addition, it is never given proper consideration, planning, and execution like any other phase of the project. Failing in KT can potentially squander all the effort and investment you’ve put into an SAP project; new technology has ZERO value if your business users are not proficient at using the system. You should treat knowledge transfer planning and execution just like testing, with specific goals, topics identified and capacity planned. It should also be spread out thoughtfully across the duration of the project. Expectations / level of KT expected also needs to be clearly communicated to both the consulting team and internal team. You need to have realistic expectations here. At the end of KT, your internal resources (especially if new to SAP) will not be able to do what your consulting team can. Knowledge transfer is often approached using the “fire hose” technique. SPOILER: it does not work.

Fire Hose Technique -  Tasking several of your senior SAP consultants to simultaneously knowledge transfer to the same beginner SAP client resource via a two week brain-dump.

There are very few people that can effectively learn this way. Genuine knowledge transfer can only truly be achieved through doing, not watching or listening. Engagement and participation throughout the entire project lifespan will lead to knowledge transfer organically. The most effective way to execute knowledge transfer is to plan a PPS (Post Production Support) phase that keeps the consulting team engaged – some full time, some part time, and some as needed. This will enable the transferring responsibility and knowledge overtime. It is important to understand you will need to push the internal team here, you want the consulting team as a safety net not a crutch.

Client Resources Can Cause Delays Too

OK, now it’s time for me to call out all my clients from the past. This is more of a universal lesson learned but it especially applies to SAP Retail projects. Consultants and consulting teams are not always responsible for project delays. It is our job as consultants in most cases to take the heat and blame whenever deliverable dates are missed; that is part of our job function and it comes with the territory. What sort of things am I referring to? Information needed by the project team needs to flow quickly, and there is often information and tasks that extend outside of the identified SAP project team. It is important that from the top down those tasks are given priority and accomplished with a sense of urgency from Day 1. Oftentimes, teams have to push ahead making assumptions and then have to backtrack to fix wrong assumptions once they receive the information from the business. Good consulting teams never sit around twiddling their thumbs while waiting to receive all the information needed; they assume leadership and use good judgement to help their client be successful.

It is also important for SAP project leadership to monitor for “Analysis Paralysis”. While some consultants can be susceptible to this it is more often the client resources that come down with this serious condition. “Analysis Paralysis” is the most common reason projects get delayed. If you have unlimited budget then go ahead and try and think of everything. Otherwise you need to quickly review the information you do have, pour on some assumptions, and move forward. You will learn more as you progress through the project and can always go back and refine those critical things that you have missed. Prototyping and pilots give you more opportunity to move forward more quickly with an approach that can be refined as you get smarter.

One last point on client resources delaying projects: do not put procrastinators on your SAP Project team. You need to put your resources that hit the ground running and do their best to stay ahead. The average SAP project costs 16 million dollars. This is a pretty significant capital investment for any company. All it takes is one procrastinator in one area of the project to slow down the efficiency of the entire team.

So I got derailed a bit at the beginning of the week so I am behind schedule on my topic list.  I will do my best to cover planning this evening and post sometime tomorrow to keep on track.

Be on the lookout for my next post, which will Lessons Learned for the Planning phase of an SAP Retail project. The remaining schedule is as follows:

Here is an outline of my future lessons learned post topics:

• Week 3
• Design - If It Is Hard or Complex You Most Likely Are Taking The Wrong Approach

• Week 4
• Execution
• Basis - Size Does Matter
• Configuration - Prototype And Prove Out Approach Before You Commit To Process
• Development - RICEFW Can Not Be Developed In Silo’s

•  Week 5
• Testing - Testing Design Is More Important Than Solution Design
• Go-live Planning & Execution - Have An Implementation Methodology That Tests Your Go-live Plan As You Go
• Go-live - Big Bang Not A Good Idea For  An SAP Retail Implementation (Without A Pilot)

• Week 6
• Post Production Support - Figure This Out Prior To Go-live

I have been working with SAP for over 15 years.  This was my first SAP Retail project and once again SAP has proved to me that it can be successfully leveraged and become a competitive advantage for those companies that implement it.  Each time I start a new project in a new industry I think about the vast differences in how the new company will need to leverage SAP and the challenges that unique business will create for the SAP application.  Time and time again a reasonable solution path is achieved and SAP becomes a solid foundation from which the business operates.  The diversity of my own personal experience working with successful SAP customers demonstrates this point.  There are not a lot of similarities in how A Flooring Retailer, Rocket Manufacturer, Pharmaceutical Manufacturer operate, yet they are all very successful at leveraging SAP.

Project Background

A little bit about my background in SAP Retail.  I have played quite a few roles over the last 2 years.  I have been an Infrastructure Architect (Design Of Virtual SAP Landscape), Development Lead (Initial Implementation), Stabilization Lead (Post Go-live), and SAP Program Manager (2 Physical Inventory Projects, Implementation Of Canadian Operations, POS (Point of Sale) Redesign, and Store Receiving Redesign).  Through these various roles within this project I have learned some very important new lessons as well as proved out ones I already new.

Lessons Learned – Too many to fit in one post

I would like to break down my lessons into the various phases of the project, from start to finish I think there are valuable lessons to be learned.  I will start with a lesson learned that applies to all aspects of the project and you will most likely think “thanks captain obvious for pointing this out”.  Rule #1 – Apply Common Sense in all decisions you make and look to see that other decision makers are doing the same.  Yes this is obvious but more often then not common sense is not applied.  I will reference this theme as I point out specific lessons learned.  Many of my lessons learned apply to all SAP projects or any project in general.  I will try and highlight lessons that I feel are specific to a retail.

In order to keep this post manageable I am going to break the lessons learned up into a series of posts.  I will add a new post each week until I have covered all the areas that I think are important to cover.  So lets start with project resources.

Consulting Resources

• Eliminate Under-Performers Quickly – I personally am not very good at this.  It is in my nature to look to try and help develop an individual to be successful.  However when it comes to consulting resources, career development does not apply on a project team.  We had several resources (not on my team) that did not have the skills and/or could not keep up with the pace.  We worked around them instead of eliminating them.  DataXstream did have a contract resource we brought on to my team and within a couple weeks we could see that he had interviewed better than he executed.  It is never easy to say we made a mistake to the client but we did and we replaced him.
• Cheap Resources Are Not Always Cheap – You could also reference the lesson above because it usually applies here.  However decision makers managing budgets tend to try and make the numbers work instead of ensuring they have the right people or are doing the right thing.  If a resource does not have the skills, drive, attitude or lacks all of these then it does not matter if the numbers look good in Excel.  This is a common thread to all SAP projects.
• With Free Resources You Get What You Pay For – There are many free resources that consulting firms provide that add real value.  However it is important to recognize that not all do and some provide negative value.  Free + Negative Value = Expensive.
• A Good (Non-Retail Experienced SAP Resource) is Better Than A Bad (SAP Retail Experienced Resource) – An SAP Retail System is 95% the same as any other SAP system.  Yes, there are differences. A good resource will have worked in a divers set of industries in the past.  They will pick up retail differences.  A weak resource with Retail experience will continue to be a weak resource.

Here is an outline of my future lessons learned post topics:

• Week 2
• Client Resources - There Are Never Enough Client Resources
• Planning - Establish Planning Rules That All Decision Makers Agree To And Then Follow Them

• Week 3
• Design - If It Is Hard or Complex You Most Likely Are Taking The Wrong Approach

• Week 4
• Execution
• Basis - Size Does Matter
• Configuration - Prototype And Prove Out Approach Before You Commit To Process
• Development - RICEFW Can Not Be Developed In Silo’s

•  Week 5
• Testing - Testing Design Is More Important Than Solution Design
• Go-live Planning & Execution - Have An Implementation Methodology That Tests Your Go-live Plan As You Go
• Go-live - Big Bang Not A Good Idea For  An SAP Retail Implementation (Without A Pilot)

• Week 6
• Post Production Support - Figure This Out Prior To Go-live

Hope you have enjoyed this post and I will see you next week.

From time to time it becomes necessary to change the status of SAP IDOCs in SAP. The most common scenario is the requirement to mark SAP IDOCs for deletion. There is no good way to mass mark IDOCs for deletion via the standard IDOC processing transaction BD87. However there is a program that will let you change status.

RC1_IDOC_SET_STATUS

CAUTION: This program should be used with great care and consideration. Improper use of this program can result in data consistency issues. Make sure you know what you are deleting, why you are deleting it, and what is required to correctly update you system after deleting.

Example: Marking IDOCs for Deletion In Mass

It is pretty typical for support users to set the deletion flag on IDOCs that have been incorrectly created and have errored. When there are a small number of IDOCs this is possible via transaction BD87.

An inbound IDOC in error will have the status 51, when it is marked for deletion it has a status of 68.

A view of the IDOCs to be deleted in WE05.

To mass delete IDOCs run the following program via SE38: RC1_IDOC_SET_STATUS via the SAP Transaction: SE38

There are only a few parameters on the selection screen for this program. It is most important that you correctly restrict the IDOCs you select with this program. The program automatically defaults to marking inbound IDOCs in error for deletion.

To mass select IDOCs to be marked for deletion select:

There are many options for selecting and restricting the IDOCs to Mass process. Select by single value or range. Restrict by single value or range.

The allows you to upload a list of IDOCs from a text file.

The allows you to apply a list from your clipboard.

Execute the program

Check the status of the 3 IDOCs in WE05

Example: Changing IDOCs Status To Repost

It is also possible to use this program to reset an IDOC so that it can be reprocessed.

With the following selection we are going to reset the IDOCs with status 68 marked for deletion back to status 64 to try and reprocess them.

Execute the program

As you can see, program RC1_IDOC_SET_STATUS is very helpful, but please be careful when you use it!

SAP Connect can be configured for a number of different message relaying purposes. This blog post will only cover sending E-mail from SAP externally.

Some basic requirements and points:

• SAP Connect configuration is client dependent. It has to be configured in each client where you want to use it.
• SAP Connect requires an external SMTP server to send emails.
• SAP Connect needs to be authorized to use the SMTP server to send emails.
• You need to have applications configured in SAP that send external emails.

High level outline of process:

• SMTP Server Details
• Configuring SAP Connect INT Node
• Configuring Routes For SAP Connect INT Node
• Configuring SAP Connect Default Domain
• Scheduling Send Process
• Setting Up Your User ID with and External Email Address
• Testing Your Configuration
• Monitoring SAP Connect

SMTP Server Details

Let’s start with the most important piece of the puzzle. If you don’t get this part right, your configuration will not work. You need an external SMTP server to relay your outbound email messages. Most companies running SAP will have at least one SMTP server. It is also possible you might have several to choose from. So, you need to identify the appropriate SMTP server.

You will need to know the SMTP server Host Name as well as the Port. The default Port for SMTP is 25, but your administrator has the option to change this. Make sure you confirm the correct port number.

Identifying your SMTP server is the first piece of the puzzle. The next piece is to request permission to use it. Most corporate SMTP servers are protected–they only allow trusted servers / authenticated users to use them. You will most likely need to contact your email system’s administrator to request authorization for your SAP servers. If you will be using SAP Connect in your Development, Test, and Production environments, it is a good idea to request all of them at once. NOTE: If you physically move your SAP server to new hardware in the future or change your servers IP address this authorization will most likely break.

The reason your corporate SMTP server is typically protected is to avoid internal / corporate spammers–both intentional and unintentional.

Configuring SAP Connect INT Node

Now that we have an SMTP server to talk to, we need to configure SAP Connect. The transaction in SAP is SCOT. Remember, SAP Connect is client dependent so you will need to repeat this procedure in all clients that require the ability to send email externally.

When you execute SCOT you should see a similar screen.

You will need to select the INT entry, and from the “View” menu, select “Node”.

Select the INT entry again and click the create paper on the tool bar.

1. Fill out the Description field as desired.
2. Set the “Node in use” flag.
3. Fill out the Mail Host (Host Name or IP Address) – Host Name is recommended.
4. Fill out the Mail Port (Default 25)
5. The default Code Page will work for many implementations.
6. Next, click the SET button next to Internet in the Supported address types

The following screen should appear:

Here we configure allowed routes for external emails. In this example we have configured a “*” which will allow any email address domain to be routed. It is also possible to only allow specific routes, like dataxstream.com or sap.com.

If your system manages HR data or patient data, it is a good idea to check with your compliance office to ensure external email policies are being adhered to in productive systems.

Additionally, you may also want to restrict certain routes in Test and Dev systems so your testing does not generate emails to external business partners.

After you click the green check your screen will look as follows.

Configuring SAP Connect Default Domain

Now you need to configure a default domain for the server. From the settings menu select Default Domain.

Add your company domain name.

Scheduling Send Process

Messages are not relayed synchronously. They are relayed asynchronously via a scheduled background job. You must schedule this job for your messages to be relayed.

On the menu bar click the execute button .

The following screen will pop up.

Now click the the create button pulldown , and select “Schedule Job for INT”.

Name your job. Remember, SAP Connect is client dependent so if you are in a multi client development or test system it is a good idea to include the client number in the name.

It will default address type to INT Internet Mail Address.

If you have a high volume of messages you will want to up the number of work processes as well as consider using a server group if you have them configured.

You will want to schedule with job with a system ID or background ID. In this case we are using BATCH-ID.

Click Continue.

The following screen allows you to customize the setup of the background job. I have schedule this job to run every ten minutes on the two minute of the hour (e.g. xx:02, xx:12, xx:22, xx:32, xx:42, xx:52).

To adjust the period select

In this case we are picking a period more frequent than Hourly so select

Fill in the period that meets your requirement. Hit save (you will need to do this several times to get to the next screen).

Hit save one more time and it should return you to this screen.

You can  review and make changes to your job with the SAP transaction SM36.

Setting Up Your User ID with and External Email Address

If you want SAP users to receive email notifications from SAP applications, you will need to make sure that their email address is correctly maintained on their user account in SAP Transaction SU01.

Please confirm that E-Mail is maintained for users in the Communication Section.

Testing Your Configuration

You can test your configuration using the SAP Business Workplace. The SAP Transaction to access your mailbox is SBWP.

Click the New Message Button .

Create a test message.

To test, you need to select an external email address for the Recipient. Make sure the domain is allowed in the route you setup in the prior step. In this example, any external email address should work.

You can review the status of your message by viewing your outbox.

Your message will be sent at the next execution of the mail-send job we just configured.

You can check the status to the send job with SAP transaction SM36.

Once the send program has executed, check the email inbox for the message recipient. You should see your test email.

Monitoring SAP Connect

To monitor you can use the SAP transaction SOST.

Here you will see all outbound messages and their current send status. You can also process the messages from the menu.

The basic requirement is as follows: SAP needs to be able to write files that JDE can read/delete and visa-versa JDE needs to write files that SAP can read/delete. Both servers are running UNIX.

High level outline of process:

1. Define Example Landscape
2. Create An Integration File System
3. Export File System and Mount
4. Example Issues / Scenarios
5. Creating A New UNIX Group
6. Set Integration Directory Group Ownership and Permissions
7. Adjusting The Umask

Define Examples SAP and JD Edwards Landscape

So let’s create an example landscape. In our example, we have an SAP development system with the SID DEV–we will call this system SAPDEV. We also have a JDE development system that we will call JDEDEV. In this example, we will only discuss the steps to integrate the two development servers. You will need to repeat this process and/or modify this process to configure the rest of your landscape. Please note DO NOT ever mount production systems to QA and DEV systems.

Creating An Integration File System

So, the first step in this process is to create seperate physical file system for your interfaces to use. This file system is to be separate from the OS, SAP kernel, and SAP database. This is a very important step of the process. I have been working with SAP customers for many years and I am always amazed that this step is often skipped. It is important that you create a seperate file system because it is to easy to accidently fill up your file system with interface files. You certainly don’t want your interface files sharing the same file system as the OS, SAP Kernal, or SAP DB. Usually when these componets of the server run out of space bad things happen.

Next you may be wondering how big a file system is required, that answer certainly depends on a lot of project specific variables. In general, a 10 gig file system should be a good starting point for most SAP projects. However, an in-depth analysis is required to determine an appropriate amount of storage.

Export File System and Mount

In our example, we have created a 10 gigabyte file system for the interfaces and have mounted it to the root of the SAPDEV server as the folder /transfer. We have exported this folder from the SAPDEV host and created an NFS mount to the root of the JDEDEV server as the folder /transfer. Our servers now have a shared file system between them.

In AIX you can check your file systems by using the following command:

df -g

Or by reviewing the the filesystems file by using the following command:

more /etc/filesystems

Example Issues / Scenarios

So far in our example this has been a pretty straight forward exercise. Now comes the tricky part; making it so each server has the correct permissions to read and write to the files. Let’s also assume for this example that we are going to open this directory up completely from a permissions perspective.

From the root directory issue the following UNIX command to check the /transfer folders permissions.

ls -l | grep transfer

drwxrwxrwx devadm sapsys 1000 Aug 28 2008   transfer

If your permissions are different from the root director as user <sid>adm, issue the following command:

chmod 777 transfer

This will completely open up the permission on this directory.

In this example the directory was created by the SAP sidadm account (devadm). The sidadm account has a primary group of sapsys. The permissions on this directory are set to allow any user in any group to read, write, and execute from this directory.

When an ABAP program is developed to write a file, it will always write the file with the <sid>adm account (devadm in our example). When a file is written, it will use the default umask for the user to set the file’s permissions. If we were to write an interface file to /transfer it would look something like this in UNIX.

(SAPDEV) /transfer

-rw-r--r--  devadm  sapsys  256  Aug 28 2009  sap_interface_file.txt

Let’s quickly jump over the fence and look at JD Edwards. When a JDE program is developed to write a file, it will always write the file with the JDEXE account. Like SAP, when the file is written it will use the default umask for the user to set the file’s permissions. If we where to write an interface file to /transfer on JDE it would look something like this in UNIX.

(JDEDEV)/transfer

-rw-r--r--  jdexe  jdexe  256  Aug 28 2009  jde_interface_file.txt
-rw-r--r--  devadm  sapsys  256  Aug 28 2009  sap_interface_file.txt

We now have two files in the directory. If you where to list the contents of the directory on either server you will see the two files.

ls -l

/transfer
-rw-r--r--  jdexe  jdexe  256  Aug 28 2009  jde_interface_file.txt
-rw-r--r--  devadm  sapsys  256  Aug 28 2009  sap_interface_file.txt

Simply mounting a file system between the two boxes is not enough to share files. Neither devadm or jdexe can update, move, or delete the file written by the other server. Specifically, sap_interface_file.txt has -rw-r–r– for its permissions JDEXE can open and read this file but can not change it. Most SAP integration scenarios that are file based read the file and then archive/delete it. The same issue is true for jde_interface_file.txt <sid>adm can open and read this file but can not change it.

The Easy Out – Scenario

One simple solution would be to add jdexe to the sapsys group and devadm to the jdexe group. However, this is not a good option from a security perspective. This would give the JDE admin authorization to SAP and SAP admin authorization to JDE.

More Secure – Scenario

Create a new UNIX group on both servers to own the /transfer folder, add both <sid>adm and JDEXE as secondary group members, adjust the users Umasks, and set the GID bit on the folder.

Let’s Create A New UNIX Group

In our example lets call the group “transfer”. We need to add both <sid>adm and jdexe added to this group as a secondary group; althouh they both keep their primary groups, sapsys and jdexe respectively.

To see what groups are configured on your server issue the following UNIX command:

more /etc/group

...
owner:*:10:
everyone:*:12:
group:*:16:
staff:*:20:root
sapsys:*:200:adm
...

The group file contains all the groups configured for a server. In my example, I am going to assume that there is no central user configuration on your UNIX servers. Therefore, when this group is created on both the SAP and JDE server it is important that it is created with the same Group ID. It is important that you pick a Group ID that is not used on either server. On AIX you would use the SMITTY utility to create the group. For this example we will use the group ID 300.

To see the new transfer group created in SMITTY you can check the /etc/group file.

more /etc/group

...
owner:*:10:
everyone:*:12:
group:*:16:
staff:*:20:root
sapsys:*:200:adm
...
transfer:*:300:

Set The Integration Directories Group Ownership and Permissions

Next, we need to change the ownership on the /transfer folder to have the owner group be “transfer” and not “sapsys”.

ls -l | grep transfer

drwxrwxrwx devadm sapsys 1000 Aug 28 2008   transfer

To change the ownership we run the UNIX command:

chown :transfer /transfer

Let’s check our work:

ls -l | grep transfer

drwxrwxrwx devadm transfer 1000 Aug 28 2008   transfer

Notice the group is now “transfer”, not “sapsys”.

Next, we need to change the permissions on the /transfer folder to something a bit more secure, as well as set the GID bit. From UNIX issue the following command:

chmod 2770 /transfer

Let’s check our work:

ls -l | grep transfer

drwxrws--- devadm transfer 1000 Aug 28 2008   transfer

Notice the “s”; that is the GID bit. Setting this bit will force all files to adopt the group ownership of the parent directory not the primary group of the user. We are almost there. Let’s write another file from SAP and JDE and check that file’ permissions. From UNIX issue the following command after writting a file from SAP and JDE to the /transfer directory.

ls -ltr

/transfer
-rw-r--r--  jdexe  jdexe  256  Aug 28 2009 00:00  jde_interface_file.txt
-rw-r--r--  devadm  sapsys  256  Aug 28 2009  00:00 sap_interface_file.txt
-rw-r--r--  devadm  transfer  256  Aug 28 2009  00:00 sap_interface_file._2.txt
-rw-r--r--  jdexe  transfer  256  Aug 28 2009 00:00  jde_interface_file_2.txt

Notice the group on both files is now “transfer”. However, we have still not fixed our problem because both files were created with 644 permissions. We need our permissions to be 664 for every file written to this directory. We will, therefore, need to adjust the UMASK for <sid>adm and jdexe.

Adjusting The UMASK

To determine the UMASK to use, you must take the permission you want and subtract it from 777 to get your mask. In our case we would like the files to have 664 permissions.

777 – 664 = 113

You will need to set the UMASK to 113 for both <sid>adm and jdexe.

Let’s run one more set of tests. Let’s write another file from SAP and JDE and check those files out. From UNIX, issue the following command after writting a file from SAP and JDE to the /transfer directory.

ls -ltr

/transfer
-rw-r--r--  jdexe  jdexe  256  Aug 28 2009 00:00  jde_interface_file.txt
-rw-r--r--  devadm  sapsys  256  Aug 28 2009  00:00 sap_interface_file.txt
-rw-r--r--  devadm  transfer  256  Aug 28 2009  00:00 sap_interface_file._2.txt
-rw-r--r--  jdexe  transfer  256  Aug 28 2009 00:00  jde_interface_file_2.txt
-rw-rw-r--  devadm  transfer  256  Aug 28 2009  00:00 sap_interface_file._3.txt
-rw-rw-r--  jdexe  transfer  256  Aug 28 2009 00:00  jde_interface_file_3.txt

Notice the 3rd set of files have both read and write privleges on the group. This means that both of these files can be read and changed by both user ID’s because they are both part of the transfer group.

You may find it necessary to change the the UID of the sidadm and orasid accounts on your SAP server. The process is not difficult but there are a number of steps that must be followed for it to work correctly. The following is an outline of steps that I have followed in that past on AIX running oracle. Please note you should always test this on a test server to develop a proper procedure for your specific OS and installation. You should also confirm that you have a current backup just in case.
Lets Review The High Level Steps

• Confirm backups are in place and current. Validate the restore procedure.
• Review the current SAP UNIX user setup and develop list of required changes.
• Backup the current passwd file.
• Create a backup list of files owned by the user being changed.
• Stop SAP System and processes owned by <sid>adm.
• Stop Oracle and processes owned by ora<sid>.
• Restart the server.
• Change user UID mapping.
• Change file ownership to new UID mapping.
• Restart the server.
• Restart SAP and Validate server started correctly.

Reviewing UNIX Users and Outlining Changes
First let’s review the setup of the user accounts on the server. You will find the UNIX user accounts and their corresponding UID’s in the passwd file in the /etc/ directory. Let’s review the entries for <sid>adm and ora<sid>

dx1adm:!:207:203:SAP System Administrator:/home/dx1adm:/bin/csh

oradx1:!:208:204:SAP Database Administrator:/oracle/DX1:/bin/csh

So on the DX1 instance of SAP we see the following users are setup as follows dx1adm has a UID of 207 and oradx1 has a UID of 208. In this example let say we need to change dx1adm to 307 and oradx1 to 308.
Backup the Current passwd File

# cd /etc

# cp passwd /home/xd1adm/passwd.bak

Create a Backup List of Files
In this case I am going to create 5 lists just to be safe. These lists are a snapshot of file and directory ownership before we make any change. All of the examples below are run from any directory. The backup lists are written to the /home/xd1adm.

Create a list for: /usr/sap

# ls -ltraR /usr/sap > /home/xd1adm/usr_sap_list.txt

Create a list for: /sapmnt

# ls -ltraR /sapmnt > /home/xd1adm/sapmnt_list.txt

Create a list for all files owned by: xd1adm

# find / -user 207 -exec ls -altr {} \; > /home/xd1adm/xd1adm_list.txt

/oracle/XD1 (you need to su to oraxd1)

# ls -ltraR /oracle/XD1 > /oracle/XD1/oracle_XD1_list.txt

(To put this file in the /home/xd1adm/ directory you will need to copy it there and then delete it from oraxd1 home)

Create a list for all files owned by: oraxd1

# find / -user 208 -exec ls -altr {} \; > /oracle/XD1/xd1adm_list.txt

(To put this file in the /home/xd1adm/ directory you will need to copy it there and then delete it from oraxd1 home)
Stop SAP and Processes
Now we need to stop SAP and Oracle. We need to make sure that we stop all of the processes owned by <sid>adm and ora<sid>.

To stop SAP issue the following command as <sid>adm

# stopsap

To stop the SAP OS collector issue the following command as <sid>adm

# saposcol -k

To stop the Oracle listener issue the following command as ora<sid>

# lsnrctl stop

Restart The Server
This is just to be safe. (Please make sure that you don’t have SAP and Oracle setup to automatically start in your start scripts)
Change The UID for <sid>adm and ora<sid>
In this example we will use smitty to make this change. You will want to be root on the server.

# smitty

In this case we will select the <sid>adm account

Change the User ID field from 207 to 307 and save your change. Repeat this process for the ora<sid> account.
Change File Ownership to New UID’s for <sid>adm and ora<sid>
In order to run this next command you will need to be root on the server.

Change the <sid>adm file ownership to the new UID

The following command will change the owner of all files that are owned by UID 207, the old <sid>adm UID, on a server to the new <sid>adm UID 307.

# find /! (proc | backup) -user 207 -exec chown 307 {} \;

The following command will change the owner of all the symbolic links that are owned by UID 207, the old <sid>adm UID, on a server to the new <sid>adm UID 307.

# find /! (proc | backup) -user 207 -exec chown -h 307 {} \;

You can verify that the ownership on all the files has been changed from 207 to 307 by running the following command. It should not return any results.

# find / -user 207 -exec ls -altr {} \; > /home/xd1adm/xd1adm_list.txt

Change the ora<sid> file ownership to the new UID
The following command will change the owner of all files that are owned by UID 208, the old ora<sid> UID, on a server to the new ora<sid> UID 308.

# find /! (proc | backup) -user 208 -exec chown 308 {} \;

The following command will change the owner of all the symbolic links that are owned by UID 208, the old ora<sid> UID, on a server to the new ora<sid> UID 308.

# find /! (proc | backup) -user 208 -exec chown -h 308 {} \;

You can verify that the ownership on all the files has been changed from 208 to 308 by running the following command. It should not return any results.

# find / -user 208 -exec ls -altr {} \; > /home/xd1adm/xd1adm_list.txt

Restart The Server
This is just to be safe.
Restart SAP Instance
Restart the Oracle Listener issue the following command as ora<sid>

# lsnrctl stop

Restart SAP issue the following command as <sid>adm

# startsap

Confirm you SAP instance has started correctly. I would also check the SAP startup log as well as transaction SM21 to see if there where any issues with starting the instance after making these changes. If you have issues with restarting you instance you can troubleshoot the issue from the startup logs and you also have the backup lists of the directories that show what the system looked like before you made any changes. Again make sure you test this procedure for your environment in a system it like a sandbox, before you implement on any productive instances.

I started off staging a test to look at a performance issue related to an SAP customer load and ended up diagnosing a custom SAP invoice interface problem with Vistex.  We had staged a test to look at why SAP V2 update processes where significantly bottle necking when the customer load program ran.  Just before I pulled the trigger on the test.  I decided to fire up Topas on the AIX to look at how the SAP server was running.  That is when I saw something very interest.  A custom developed SAP invoice interface program had been running for nearly 16 hours and was consuming nearly 1/3 of the CPU capacity of the instance (3 Cores – 4 and change GHz processors).  I did a little checking and we were processing a file with approximately 47,000 invoices.  Now that just ain’t right!  You should be able to process 100’s of thousands of invoices an hour on an SAP instances configured as we are configured.

It was a programming problem and just using Topas, I should have been able to predict the problem and known where to look.  I will explain why I should have known where the problem was and then go into how I actually figured out the issue.  The smoking gun should have been the following clues.  The SAP server we are testing on is a 2 box instance, separate SAP CI and DB.  Topas was reporting one SAP batch process on the CI was consuming 1/3 of the CPU capacity on the box, that process was consuming a pretty normal amount of memory and was stable.  The SAP DB server throughout the process was under no visible load. There was also no real network activity to be measured. The interface program reads a file, formats the data and passes the data to a standard SAP posting function.  We were posting an invoice approximately once every 2 seconds.

Given just the above information what do we know?  Experience tells me that my server should be sized to handle hundreds of thousands of invoices an hour, we are processing 2 per second.  My process is consuming significant CPU. My process is stable from a memory perspective.  My process is not hitting the DB.  What is my process doing?  It is crunching though ABAP. More specifically it is crunch through ABAP and it is not doing DB operations.

So what was the problem?  We had 3 nested loops. The outer loop had 50k rows, the next loop had 150k rows, and the final loop had 800k rows.  That is a lot of looping.  In testing the developer used small files maybe 20 invoices max in the tests.  While the code was still broken during unit testing it would not be visible with small test files. So how did I go about finding the problem?  I kicked off a SQL trace using ST01 and could see that for each invoice we had about 2000 microseconds of operations.  I then reviewed the code, selection screen, file import, and then processing.  It took less than a minute to import the file and start the processing routine.  I commented out the call to the posting function module (AKA the heavy lifting) and replaced it with a write statement.  It was taking approximately 2 seconds per write still.  The problem therefore lied in the routine that formatted the data from the file to present it to the posting function.  As soon as I opened that part of the code it was obvious we had nested loops and looking at the counts on the internal tables it was pretty clear why it was running so long.

This program lets you pick a point in time in one client and synch its configuration changes from that point to another client. The program is RSCC_SCC1_BATCH. You can find more details on the specifics of this program in SAP OSS: Note 645846 – CC-ADMIN: Collective Customizing copy with SCC1.

Program Features:

• Bulk Import Customizing Changes From One Client To Another
• Selections Of SAP Transports By: Transport Number, Owner, Point In Time
• Program Has A Test Mode
• Program Has Ability To Merge Transport Into One New Large Transport
• Program Can Be Schedules To Run As A Periodic Job

The program selection screen is pretty straight forward:

I was most interested in the changes since date.  I had performed a client copy on May 19th and was already missing transports by the 20th.  There was too much testing setup to redo the client copy so we had to get the configuration corrected.

Results:

• In my testing the program typically runs 600 + seconds.
• SAP suggests you use the Merge Requests:  This takes all the request that is selects and merges them into one big request.  (Even in test mode it creates the request it just does not import it.)
• The merged request contains a list of the merged transports in the comments section.
• I was pulling changes from client 400 (Golden Client in QA) to 431 test client.  The transports originated in our development system client 100 so I had to turn off the (Check Request Source Client) Flag
• Configuration was correctly synchronized.

The program creates a spool report:

The report contains summary details of the execution.  You can see the number of the newly created merge transport at the bottom.

The merge transport contains the list of merged transport in the comment section:

Transport is not release unless you set the release flag.