Hershey’s overall goal wasn’t just to update their current technologies, it was also to optimize their current supply chain, reduce costs and tighten their operations through the implementation of SAP.  It was a noble set of goals, and often required of almost any company in today’s corporate landscape.  However, after three years of updating hardware and implementing SAP, their go-live phase was considered a complete and total failure.  $100 million in sales were lost in October 1999; their stock price dropped by 35 percent, and nearly the entire operations of the company was behind schedule (Bill, 2000).  This was not the goal of a three year, $112 million project.

How did this happen?  Could it all have been avoided?  The answer to both of these questions is not as simple as “yes” or “no”.  However, there is no debating the fact that at some point, Hershey was going to have to enter the 21st century and update their legacy systems.

SAP has become a necessary evil for any organization that wishes to survive in today’s cutthroat marketplace.  Simple decisions such as when and how to replenish safety stock or shipping standard quantities to customers on the first of the month need to be made automatically without any room for human intervention or error.  SAP implementation can save a company millions in optimizing their logistics and synergizing individual functional ‘silos’ within a company.  However, in order to get to that point, all companies have to implement the software to begin with, and that is where problems such as Hershey’s typically occur.  Pretty much any type of implementation can have disastrous and unintended consequences if not successful.  Each module of SAP is interconnected with many other facets of a company, such that if one small thing goes wrong in an area most people neglected to consider, then that small mistake could cause far-reaching consequences across the entire company.  This goes for even small “bolt-on” implementations to existing SAP systems.

This is an issue which corporations must face if they want to remain competitive, so it is important that as time goes on, the best methods of implementing are acknowledged and respected as standard business practices.  Even though SAP has been commonplace for many decades now, companies still run afoul by neglecting basic elements that should be expected out of any implementation, no matter how small they are.

As someone who works for an SAP consulting firm, I have witnessed companies that understood the fragility of how their software affects the company.  These organizations often thrive on optimizing their logistics systems and are always looking for a way to get a leg up on the competition.  I have also worked with organizations that have dreaded any type of change to their organization.  These companies were often toxic environments which suffered from very poor top-down leadership.  When all the stakes are considered, the true importance of an SAP implementation can mean the difference between a business staying open and closing their doors for good, potentially putting hundreds of good people out of work.

Management Information Systems have become the heart of any business operation, and as time goes on, will become even more important within the corporate landscape.  SAP implementation is one of the most important, make-or-break steps in a company’s lifespan in the 21st century.  My career has placed me in the center of these situations and there are very clear things that anyone at any level of a company can do to help their company succeed or fail between the moments when it is decided to implement new technologies and when they go-live.  The goal of this blog posting is to analyze what those key actions are and give clear recommendations for any company which implements SAP in the future.

One of the primary mistakes many organizations make is a reluctance to fully understand the importance of their software on the organization as a whole.  I have worked for managers and executives who feel their company’s software is something which “stands in the way” between them and “making money”.  Yes, these were words once given to me by a high-ranking official of a global manufacturer of carbide parts.  By devaluing a tool which can enable a company to meet and exceed their financial goals only inhibits the broader mission of the organization.

FoxMeyer Drug was a $5 billion dollar subsidiary of pharmaceutical company FoxMeyer Health, which had very complex supply chains due to the variable nature of their field and product requirements.  A decision was made in the early 90’s to implement SAP as a solution to their logistic woes.  On paper, it seemed like a dream come true.  Inventory fluctuations would be kept to a minimum, transactions would be monitored in real time, and their order process would be automated and simple.    FoxMeyer has an obligation to iron out their supply chain issues and trim the fat in order to remain competitive, so the decision to utilize SAP was a very easy one to make.

FoxMeyer made the choice of adopting a “Big Bang” approach to implementing the complex software (Bulkeley, 1996).  A “Big Bang” approach is when a company has all segments of their operations move into their new system at the same time.  It’s described as an “instant changeover” and is often compared to rushing an entire football team through a single small door in a matter of seconds.  For smaller companies, this can be a useful strategy.  For larger ones, it’s more like fitting an entire parade through a small door in a matter of milliseconds, it’s ill-advised and simply not that great of an idea, despite the need to go-live as quickly as possible.

FoxMeyer faced many more problems than just a their choice of implementation strategy.  Warehouse workers who were not properly educated on the benefits of the project felt that their jobs were in danger of being replaced by machines.  A poor initial transition to the SAP software in the first warehouse did not help boost employee morale either.  This led to a massive influx of disgruntled employees and an eventual loss of $34 million worth of inventory (Scott, 2000).  This could have easily been prevented by greater communication between the warehouse workers and the executives of the company as to what the ultimate goal of the implementation was, as well as how SAP could greatly benefit the company and the workers.  An overall negative view of the SAP implementation project can easily derail the process due to an end-user’s unwillingness to accept and learn how the software can benefit their jobs and ultimately make their lives easier.

Eventually, FoxMeyer Drug’s implementation had gone well over-time and budget, resulting in over $100 million dollars sunk into a project which was beginning to cost the company hundreds of millions annually.  Eventually, despite annual revenue of over $5.5 billion, the division of FoxMeyer Health filed for chapter 11 bankruptcy in 1996, only 3 years after it began its implementation of SAP (Unknown, 1996).  While there were many reasons for the failure of the implementation, in hindsight it remains very clear the root cause of the bankruptcy was the implementation itself, and the financial hurdles that were left for the company in its wake.

One of the key elements of SAP is its ability to interconnect all segments of a company.  This means that each individual transaction, accounting document and shipment can affect the financial position as they happen, keeping the system up-to-date and becoming a one-stop-shop for anything that needs to be known in order to make a decision.  Mexican-based oil company Pemex has a “control room” deep in their corporate headquarters which monitors their entire operations in real time (Chavez, 2008).  What is essentially a room filled with charts and graphs connected directly to SAP allows executives to see the effect millions of individual transactions have on the company as they happen.  So when someone fills their tank with Pemex at a gas station somewhere in the world, you can actually see the effect in real time with their custom-made gauges on display within the “control room”.  It is this type of real-time information which allows decision-makers at the top of a company to make the most accurate and informed decisions possible in a marketplace where everyone wants to be at several steps ahead of everyone else.
This is a double-edged sword however; because having all components of an organization interconnected through SAP also means that any small change within that software can dramatically shift all of the information which is flowing through a company at any given time.  If a configuration is not setup correctly, or a default setting was not considered during implementation, a small adjustment in quantities or pricing could result in an error multiplied by the number of transactions in the system, which could manifest itself in the form of a high dollar value error.  The larger the company, the more severe a minor error in a sensitive spot can be.

In Part 2: I’ll analyze individual SAP implementation failures and their subsequent fallout’s.

Can’t wait to read the rest of the Mind the Gap series? Submit your information below to receive the entire series today!

I have been working with SAP for over 15 years.  This was my first SAP Retail project and once again SAP has proved to me that it can be successfully leveraged and become a competitive advantage for those companies that implement it.  Each time I start a new project in a new industry I think about the vast differences in how the new company will need to leverage SAP and the challenges that unique business will create for the SAP application.  Time and time again a reasonable solution path is achieved and SAP becomes a solid foundation from which the business operates.  The diversity of my own personal experience working with successful SAP customers demonstrates this point.  There are not a lot of similarities in how A Flooring Retailer, Rocket Manufacturer, Pharmaceutical Manufacturer operate, yet they are all very successful at leveraging SAP.

Project Background

A little bit about my background in SAP Retail.  I have played quite a few roles over the last 2 years.  I have been an Infrastructure Architect (Design Of Virtual SAP Landscape), Development Lead (Initial Implementation), Stabilization Lead (Post Go-live), and SAP Program Manager (2 Physical Inventory Projects, Implementation Of Canadian Operations, POS (Point of Sale) Redesign, and Store Receiving Redesign).  Through these various roles within this project I have learned some very important new lessons as well as proved out ones I already new.

Lessons Learned – Too many to fit in one post

I would like to break down my lessons into the various phases of the project, from start to finish I think there are valuable lessons to be learned.  I will start with a lesson learned that applies to all aspects of the project and you will most likely think “thanks captain obvious for pointing this out”.  Rule #1 – Apply Common Sense in all decisions you make and look to see that other decision makers are doing the same.  Yes this is obvious but more often then not common sense is not applied.  I will reference this theme as I point out specific lessons learned.  Many of my lessons learned apply to all SAP projects or any project in general.  I will try and highlight lessons that I feel are specific to a retail.

In order to keep this post manageable I am going to break the lessons learned up into a series of posts.  I will add a new post each week until I have covered all the areas that I think are important to cover.  So lets start with project resources.

Consulting Resources

• Eliminate Under-Performers Quickly – I personally am not very good at this.  It is in my nature to look to try and help develop an individual to be successful.  However when it comes to consulting resources, career development does not apply on a project team.  We had several resources (not on my team) that did not have the skills and/or could not keep up with the pace.  We worked around them instead of eliminating them.  DataXstream did have a contract resource we brought on to my team and within a couple weeks we could see that he had interviewed better than he executed.  It is never easy to say we made a mistake to the client but we did and we replaced him.
• Cheap Resources Are Not Always Cheap – You could also reference the lesson above because it usually applies here.  However decision makers managing budgets tend to try and make the numbers work instead of ensuring they have the right people or are doing the right thing.  If a resource does not have the skills, drive, attitude or lacks all of these then it does not matter if the numbers look good in Excel.  This is a common thread to all SAP projects.
• With Free Resources You Get What You Pay For – There are many free resources that consulting firms provide that add real value.  However it is important to recognize that not all do and some provide negative value.  Free + Negative Value = Expensive.
• A Good (Non-Retail Experienced SAP Resource) is Better Than A Bad (SAP Retail Experienced Resource) – An SAP Retail System is 95% the same as any other SAP system.  Yes, there are differences. A good resource will have worked in a divers set of industries in the past.  They will pick up retail differences.  A weak resource with Retail experience will continue to be a weak resource.

Here is an outline of my future lessons learned post topics:

• Week 2
• Client Resources - There Are Never Enough Client Resources
• Planning - Establish Planning Rules That All Decision Makers Agree To And Then Follow Them

• Week 3
• Design - If It Is Hard or Complex You Most Likely Are Taking The Wrong Approach

• Week 4
• Execution
• Basis - Size Does Matter
• Configuration - Prototype And Prove Out Approach Before You Commit To Process
• Development - RICEFW Can Not Be Developed In Silo’s

•  Week 5
• Testing - Testing Design Is More Important Than Solution Design
• Go-live Planning & Execution - Have An Implementation Methodology That Tests Your Go-live Plan As You Go
• Go-live - Big Bang Not A Good Idea For  An SAP Retail Implementation (Without A Pilot)

• Week 6
• Post Production Support - Figure This Out Prior To Go-live

Hope you have enjoyed this post and I will see you next week.

Discovering Direct Modifications of SAP Standard Objects – One Example

As I discussed in a previous post on this topic, direct modification of SAP standard objects within the SAP namespace carries a high risk in an upgrade project.  It would be very valuable to be able to identify some details about these objects, so that further analysis could be performed to determine their disposition in the upgrade.

Modification of standard SAP objects typically requires that a modification registration key be obtained for that object from the SAP SSCR facility.   The SSCR facility (SAP Software Change Registration) is a procedure which registers all manual changes to SAP sources and SAP Dictionary Objects.  With a valid and authorized OSS ID for an SAP system, you can go to the SAP Support Portal on the web and view the SAP objects that were registered for modification by that OSS ID.  If there are multiple OSS IDs for any SAP system, make sure that you check all of them to capture all objects registered by all of the authorized OSS IDs for an SAP system.  Press the “Objects Registered by Me” button to view a list of the registered objects.

Another place that you can look for this information is directly within SAP.  Table ADIRACCESS stores the registration keys which were obtained from SAP, and which were entered by the developer when the object was to be modified for the very first time.  This entry shown below from table ADIRACCESS shows that function group SCPRPS was registered with SAP for modification.

Registering an object for modification does not mean that the object was actually modified.
What additional evidence can we discover that would show that the object was, in fact, modified?   And, since a function group contains many objects, can we determine exactly which object or objects were modified?

With a little bit of forensic analysis, we can discover the specific object within the function group that was modified, the user name of the programmer who last modified the object, date of the last modification, and the line-by-line source code differences.

If you are familiar with the structure of function groups, you already know that they may contain many function modules, and that the source code for each function module resides within its own include file.  At the moment, we know that the function group name is SCPRPS, but we do not have the list of function modules within that function group.

To obtain that list, we can use the SAP function module FUNCTION_SELECT_TFDIR.
The result list is shown here:

So which, if any, of these has been modified?  To determine this, we need to find all of the rows in table TRDIR where the NAME field is equal to the PNAME entries listed above.

Here is an excerpt from SAP table TRDIR for SAP object names that begin with the characters LSCPRPSU:

Note that for all of these objects, which are not in the customer namespace, the Author Username field (TRDIR_CNAM) is ‘SAP’.  Also note that for most of the entries listed here, the Last Changed By user name field (TRDIR_UNAM) is also ‘SAP’, except for one row, where the Last Changed By username field is not ‘SAP’.  From this row, we can see that include file LSCPRPSU36 was last modified by user DXSDEV on 10/19/2009.

If we look back at the result list of FUNCTION_SELECT_TFDIR and find the row where the field PNAME is equal to LSCPRPSU36, we can see that the name of the modified function module is SCPR_PRSET_CT_IMPORT_INDUSTRY.

Exactly What Changes Were Implemented?

By going to the Version Management Utility for this function module, we can see what the various versions of the code contained.  The Version Management Utility will also show us the transport request numbers, date and time, and username for each version of the code.  The Version Management Utility also has a code comparison tool which will highlight the line-by-line differences between versions of the source code.

Here is an excerpt from the side-by-side source code comparison utility which shows the modified version on the left side and the original version on the right side.  Fortunately, in this instance, the programmer left excellent notes indicating that a correction instruction from an OSS note was being applied.

What Are the Discovery Possibilities?

The manual example described above showed how to discover and analyze customizations that had been applied to a single SAP standard object.  But what if the customizations number in the hundreds or even thousands?  What about all of the customizations that were built entirely within the customer namespace?

It is possible to discover hundreds or even thousands of customizations regardless of which namespace they were developed in.  The key is automation, understanding that this information already resides within SAP waiting to be discovered, and more importantly, understanding the meaning of the discovered information.  I have already developed an ABAP discovery program which will mine, analyze, and report data about the following customized objects:

More Discovery – Usage Statistics and Where-Used

Customization Usage Statistics

The goal of customization usage statistics analysis is to discover the usage frequency of customized objects.  The usage discovery may show usage anywhere from several times daily by key users, to none when analyzed over a long time period.   Since most companies really do not want to incur the cost and the risk of moving an unneeded customization into the upgrade system, objects which show no usage over long periods of usage history become targets.  Care must be used here, as some transactions are exercised only quarterly, semi-annually, or annually.  I always recommend letting the business process owner make the final decision to eliminate any customization object in the upgrade.

SAP keeps track of usage statistics, and this data is available for you to analyze.  You will need to check what facility is available for your particular SAP system, as the transaction codes and programs differ among various versions of SAP.   The amount of data stored within SAP is configurable, and is usually set to a small date range.  Some companies that want to analyze the statistics over longer time periods actually download and store the statistics data in their own external databases.

Customization Where-Used

Like usage statistics, the goal of customization where-used analysis is to discover customized objects that are orphan or not being used.  What about that custom data type that is not used in any table or structure?  What about that function module that shows an empty list when you click the where-used icon in the function builder?

To properly perform a where-used analysis, it is important to understand all of the various possible uses for an ABAP object.  For example, here is the selection screen of possible uses that is presented by the ABAP Dictionary when you click the where-used icon to determine the where-used list for a data table:

And what about that custom function module, which I alluded to earlier, which shows a completely empty where-used list?  Perhaps it is a remote-enabled function module that was installed by, and is currently being called by an external third-party package.   Or, perhaps it is linked to an SAP EDI inbound or outbound process code.  It is really important to be able to properly interpret the data being presented and to not draw any hasty conclusions.

ABAP Programs Where-Used Analysis Example

In the ABAP workbench, SAP provides a where-used facility, which reports the various areas where a single selected object is used.  For a discovery project, you may want to automate this facility to analyze groups of ABAP objects (e.g. all custom ABAP programs, all custom tables, etc.).

Here is an ABAP programs where-used analysis example to help illustrate how a where-used analysis might be utilized.

Your automated where-used analysis program to analyze custom ABAP programs reports fifteen custom programs which show no usage in any of the following areas:

A quick analysis shows that all of these fifteen programs are simple reports.  Since users typically execute their reports via transaction codes, you wonder how these fifteen programs could ever possibly be executed without transaction codes.

You further analyze the SAP job scheduler data as far back as possible and find that only ten of these ABAP programs were ever scheduled in a batch job.  Some of these were scheduled very recently, and some have not been scheduled for a long time (another thread of evidence that we will need to pursue).  This leaves five ABAP programs with no apparent means of being executed.

As further evidence, you perform a cross-check against the usage statistics report and find that two of these five ABAP programs show no usage.

Armed with this analysis, you are now ready to meet with the business process owners to discuss the disposition of the objects on your short list.

A salient point here is that you will need to gather your evidence from several different analysis sources to help paint the most accurate picture about the possible disposition of customized objects.

Summary

Throughout all three parts of this post, I have stressed that customizations, while necessary, add risk and cost to an upgrade project.  And customizations that are not necessary or built poorly without the constraints of best-practice controls can add high levels of risk and cost.  Here are some general guidelines to follow:

Many companies and their auditors consider a well-controlled and secure ERP system to be a fundamental component that is vital to the success of the business.   They recognize that an out-of-control ERP system can easily paralyze a company for extended periods of time.  With this in mind, implement and enforce best-practice controls and standards to make sure that future customizations are reviewed, approved, built, tested, documented, and implemented properly.  This may already be an internal or external audit requirement, so check with your inside and outside auditors to make sure that your controls meet the auditing requirements.

Periodically review, audit, and amend your best-practice controls to make sure that they are still effective in supporting the needs of the business; and that both internal staff and external consultants are in compliance of these controls.

Know your customizations.  Know where they are, why they are, and what they do.  Whenever feasible, it is most efficient to eliminate ABAP object customizations, especially any that were performed within the SAP namespace, and any that are not really needed because they can be implemented within standard SAP.

The degree of risk and cost depends on the specific type of customization that was performed.  Establish a formal review board to own the customization approval process.  Understand that approving a customization also means approving the risk and the cost burdens.  Some companies consider this to be so significant, that they include a high-level business executive as a default member of the review board.

Set and enforce a policy making ABAP object customization the last resort, only after all other options have been thoroughly researched and exhausted.  Here are some examples:

1. Research forums, blogs, become active in user groups, etc. to see how other companies might also be implementing the business function that you require.
2. Find and use customer and user exits instead of directly modifying objects in the SAP namespace.
3. Determine, for example, how to implement and maintain that complex pricing procedure using the pricing condition tables instead of writing custom code and building custom tables.

Share Your Experience.

Once again, I look forward to hearing from our readers to reply to this post with their own experience with customizations, best-practice standard and policies, and success stories illustrating how your company transitioned to best-practice controls.

[Editor's Note] This blog is third of a multi-part series:

1. Part 1
2. Part 2
3. Part 3

A Short Selective Retrospective on Key Constituencies

All project events and project success stem from testing and testing well.  I’ve written about various types of testing before and how that can lead to some confusion because of issues with definitions.  Here I want to discuss some areas where testing really can make or break a project and ideas for how to minimize the chances of things turning out badly.

SAP Testing with Project Team Members

Unsung heroes abound on SAP projects and your client’s key business analysts can be fabulous participants or sacks of rocks you have to carry everywhere for no apparent benefit.  Always choose the former or trade up if you discover you got the latter. Engaged business analysts are great when they bring business knowledge, a willingness to learn SAP, insight to the hot-button, day-to-day issues and the understanding that the consulting team may know a lot, but not necessarily everything.  Partnership between consultants and business analysts can be very fruitful at teasing out mainstream, what happens 90% of the time, business scenarios and building coherent test cases.

These key project team members can provide a sanity check when you try to bring prior experiences and findings from other clients to the project.  A crazy design you built on the last project may not be relevant here and your BA can save you from any tendency to over engineer a solution.  Conversely, your BA might be the one who sets you down the path to an unusual design.  Either way, it has to be tested and between you and the BA the bases should be covered.

I’ve found that the BA’s on a project are usually there for one of two reasons: either a motivated high performer with knowledge and credibility; or someone found the SAP project was a place to dump a personnel problem.  Fortunately, the latter is rare these days as more and more people turn into high performers out of necessity.  Consequently, your BA is going to know key areas of process and functionality that must work for the project to be a success and he/she drives you to deliver.  Never be afraid to let a BA talk about the business, what it needs to be able to do and what needs to be tested.

However, the BA probably isn’t going to be using the system as an end user on a day-to-day basis and can only get you so far with the completeness of testing.  This is where you need to branch out from the immediate project team and into the realm of the end user.

SAP Testing with End Users

In an ideal world, the systems we build would be foolproof, but there are some smart fools out there and they don’t always behave the way we want them to.  As you probably know, SAP is not the most intuitive system to navigate and use (would an overhaul of the GUI make system adoption quicker and easier?) and people don’t always do what they are told.  I know you are shocked to hear this!

If I had a lot more time on my hands I’d love to watch people using systems and seeing all the various ways people try to do things.  It’s a fairly dopey example, but I’m one of those mouse-and-menu people when I use MS Office.  The ALT, CTL, and SHIFT keys are a tiny part of my repertoire because I learned how to use these products a certain way, but I know some people who do almost everything with keyboard shortcuts.  I can scarcely follow how they do things, although I can see what they are doing.  The fun is seeing different ways to achieve the same end.  SAP is a bit like this in that there are different ways to approach a task: using menu paths; transaction codes; single session; multiple sessions; favorites list entries; and different entry points to get to data.  Unless you have really nailed down tightly how to execute tasks users will eventually find all possible ways, but more importantly, without nailing it down users will find ways to do it wrong and not realize it.  This is a potential disaster in the making.

Despite the best intentions of project team members (business analysts, consultants, trainers, supervisors) end users will use the system in ways you did not anticipate.  After all, the way you use the system works brilliantly, so no one would think to do it any differently, would they?  Except, they do use it differently.

Including a round or two of real end user testing on the QAS system is probably going to be quite revealing.  Ideally, after a dry run conversion and before going live you want to let loose on the system and do some serious day-in-the-life and periodic processing.  And you want to back off on your desire to hand hold.  This way you can see how a user actually uses the system, the steps they take when they get into trouble, and what they try to do as corrective actions.  The intuitive and natural screen flow you came up with?  Perhaps its not as intuitive as you thought.

Here’s the point: carving out time in a project to involve end users in testing is invaluable.  Initially they will stick to the script but after a short time they will extemporize and that is when you find out if you built a robust system that guides people down the desired path or a mess that allows mistakes and the attendant downstream problems.

It is better to take the time before you go live to ensure you don’t get downstream problems than to discover the chaos in production.

SAP Testing to Make Executives Happy

A final group of people with a vested interest in SAP project success is the executives.  These are rarely the folks who execute transactions on a daily basis; at best they might run a financial statement or a flash report of some description.  Nonetheless, these people along with departmental heads need to feel confident that the system works in an abstract way: a little like the way you know the engine in your car is working although you don’t really understand how all the various components fit and work together.  You’re pretty upset when the car won’t start, or the heater blows hot and cold intermittently, or you can’t tell if you have an almost empty gas tank.  Similarly executives get upset if they can’t get information that is important to them or they are unable to complete key tasks: for example, flash reports that don’t run or give “funny” numbers, not being able to close an accounting period.

These executives can be especially challenging to a project because they have so many things to worry about that they may only truly pay attention to an SAP project during times of crisis or around go-live.  At earlier times in the project life cycle they may have deferred to the PMO and existing management structure, but at crunch time things change.  Consequently, a conscious effort to find out from them what they need in order to feel comfortable with the system is critical: discovering what they need their organizations to do is critical.  Once you know what they want you can build it into project test plans and activities and show success.

This is one constituency that might be late coming to the party but you can’t ignore them or chastise them for tardiness – they are probably paying the bill!

Key Ingredients for SAP Success: Testing, Testing & (You Guessed It) Testing

It is stating the obvious (a core competence of mine) to say that an SAP project is a complicated endeavor.  I’d say there are no easy projects, only less difficult ones and testing is one of the most difficult areas to do well.

There are groups who judge testing success in different ways and with varying degrees of thoroughness.  A corner cut here and there might be acceptable, but too many cut corners and it all unravels.  By identifying key constituencies, identifying their needs, building test plans and strategies to address those needs, and executing against them greatly increases your chances of overall success.  And, if you do it right, you ensure their involvement and make them accountable for the results.

Going Virtual: How Lumber Liquidators Optimized Its IT Investments and Lightened the Demand on Its IT Organization

The April 2011 issue of SAP insiderPROFILES magazine will feature an editorial on the virtualization of Lumber Liquidators’ SAP implementation. The piece takes you inside the thought process of Lumber Liquidators’ IT team as they recount the factors that led them towards virtualization, the selection of DataXstream as their implementation partner, and how they feel today about about the decisions they made a year ago during the virtualization process.

Initially the project team had the expected you-can’t-do-that reaction when the idea of a mid-month go-live was suggested.  We took three main steps to determine whether or not we were crazy or had a viable go-live option:

1. We asked SAP.  As one of the main participants on the project we got them to do an internal review with some platinum consultants with the objective of telling us why we could not go-live mid-month.
2. We asked our project team, both client and consulting resources.  Again, the goal was to tell us why we couldn’t do it.
3. We Googled like maniacs to find something to support and justify the conventional wisdom.  We failed to find anything substantial that would deter us.

Armed with the conviction that there was no reason we couldn’t go-live mid-month we set about defining the details of how we would pull it off.

Conventional Wisdom: Why You Don’t Go-live Mid-Month

My big disclaimer is that each SAP project has unique characteristics and what worked for us may not work for you, nonetheless I encourage you to keep an open mind and push until you find your insurmountable barrier, if it exists.  I would even contend that one of the biggest barriers may be your most experienced project team members and their gut reaction.  I admit my first reaction to the suggestion was “NO,” but as we drilled into the specifics of our project my inchoate objections crumbled.  The conventional wisdom is typically grounded in truth and experience, but conventional wisdom only applies to conventional situations.  To use a consulting cliché: challenge your assumptions.

In our research we found a consensus that there are three main areas that make a mid-month cut-over a bad idea.

Assets: I’m no expert on assets but the general word was that depreciation gets “messed up” if you cut over mid-month.  Owing to my limited experience in this area I’ll take that on faith, but this was not considered an issue as assets was out of scope for our project and asset postings will be done by journal entries.

Payroll: Again, I’m not an expert on payroll, but this was another situation where payroll was out of scope as the process is outsourced to a third party provider and journal entries are posted instead.  It was not considered a show stopping concern.

Year-to-Year Period comparisons: by cutting over mid-month some folks on the interwebs expressed concern that the month-over-month comparison for the go-live year would be no use until the third year.  Let’s say you go-live in the middle of June 2010 with partial balances in SAP (the remainder being in the legacy system), when June 2011 comes around you can’t compare the two Junes because one month is partial and one is complete.  You have to wait until June 2012 for a meaningful comparison.  This was not an issue for us because we converted three prior fiscal years of account balances with the go-live month balances being the balances as of the go-live date along with the transactional activity for the rest of the month.

Factors Leading to an SAP Mid-Month Go-Live

The ugly truth is that we went live mid-month out of necessity, not by choice.  Our original plan was to go-live on the first of the month, but we weren’t ready and a focused re-planning exercise identified that we could go live three weeks later.  Changing the go-live date to the first of the next month was not acceptable for a variety of reasons, including political considerations – it’s never a purely technical decision, so no surprise there.

Adapting the SAP Go-Live Strategy for Mid-Month Cut-Over

Once we knew we were going live mid-month we had to work out how to handle key areas of functionality: particular areas of interest were open A/P items, inventory balances, customer deposits, and the GR/IR account.  On top of this historical account balances needed to be loaded and we needed to ensure that FI conversions reconciled with logistics conversions.

Finance and Logistics Reconciliation

We came up with an approach that allowed us to readily identify differences between account balances converted by FI compared to the conversion by MM.  Consider the example of inventory conversion.

Standard SAP postings for the initial inventory balance with movement type 561 post to a pair of accounts similar to these:

• DR Inventory (135010)
• CR Conversion account (399175)

Instead of making this posting we changed the account determination for MM conversion to post as follows:

• DR Inventory (135010)
• CR Inventory conversion (135011)

Now when we converted historical balances into FI the inventory posting was:

• DR Inventory conversion (135011)
• CR Conversion account (399999)

The benefit of this approach was that any balance on the 135011 account meant there was a difference between what was converted via MM and via FI.  This became the basis of the work list that had to be reconciled.  The actual operational G/L account for inventory (135010) could be used immediately after the go-live without worrying about it becoming part of the ongoing reconciliation process.

Also, standard SAP does not allow direct posting to the inventory account (135010) and this approach allowed us to leave it as delivered.

We used a similar process, i.e. a conversion account instead of operational account, to support the cut over for accounts payable, customer deposits and the GR/IR account.

GR/IR Conversion

In our environment we are fortunate to have a situation where invoices are not received before the goods.  Consequently at the time of conversion there were purchase orders where goods have been received and not invoiced.  The typical GR/IR process is as follows:

Goods Receipt:

• DR Inventory (135010)
• CR GR/IR (211200)

Invoice Receipt:

• DR GR/IR (211200)
• CR Accounts Payable

At the time of conversion the good receipt has already happened and we did not want to reconstruct the PO processing so that we could use MIRO processing when the invoice is received.  Instead we converted historical balances for the GR/IR account by posting to a new account, 211201 as follows:

• DR Conversion account (399999)
• CR GR/IR conversion (211201)

Now when the invoices are received they will be posted with an FB60 transaction as follows:

• DR GR/IR conversion (211201)
• CR Accounts Payable

We know we are giving up some integrated capability for a period of time until these purchase orders and invoices wash through the system, but it means that once the GR/IR conversion account gets to zero we are done.  The typical life cycle for PO is 6-8 weeks so this is a temporary situation.  The true GR/IR account, 211200, will be used as intended and the benefits of MIGO and MIRO processing will be realized.

Month End Processing in the Old System

Going live mid-month also meant that the prior period was closed in the old legacy system and the month end balances were converted over in the same way as any other period.  It was a straightforward extension of our process to pull the mid-month account balances from the legacy system during go-live weekend and load them into SAP, too.

Financial Statement Impacts

The approach that we took meant we introduced several accounts to the chart of accounts for conversion only.  Consequently we had to update our financial statement versions to include these accounts and assign them to the correct positions in the report structure.  Clearly this is not a difficult task, but one that is needed.

SAP Mid-Month Go-Live Conclusions

I am skipping over some of the detailed considerations of how we made a mid-month go-live work for our project.  The details of how to make it work, the nuts and bolts of reconciliation, how we tested it (iteratively!), how we worked the differences identified between FI and MM/SD conversions, etc. aren’t here.  It wasn’t always easy and we had to do it in a short amount of time, but my key message is that we found a way to do it: the conventional wisdom wasn’t and isn’t necessarily wrong, just we found another path.

One area in which businesses can make improvements to reduce waste is in Virtualization.  Eddie White, Sr. VP with Sentilla, noted that data centers account for 2% of the power usage in the United States.  In addition to actual hardware costs, the cost of power to the data center, power drawn by the servers and power used in cooling, can be reduced by 25% through efficient use of hardware and through virtualization.

In addition, virtualization allows a business to “sweat the assets it already has instead of adding new ones.”  Through smart virtualization a CIO can delay the need for a new data center, which can cost in the area of 10 million dollars.  White noted that IBM’s data center accounts for 6% of the company but utilizes 30% of the company’s power usage.

Stezler stated it simply.  “Virtualization has a huge impact.”  SAP was 60% virtualized servers in 2009 and the goal for 2010 is 80% virtualization.   In more detail Stezler notes that Energy Management is a new challenge for IT.  In recent studies 50% of clients and investors state energy efficiency and carbon footprint are among deciding factors when making consumer or investor decisions.   The data center is typically 30% to 40% of energy cost associated with any business.

With this in mind DataXstream offers virtualization services as part of our SAP integration focus.  DataXstream is a certified partner with Dell and has also partnered with VMware the global leader in Business Infrastructure Virtualization.  These partnerships allow DataXstream to streamline the virtualization process as well as allow excellent pricing options.

In each session Timo Stezler advocated, “Become a strategic Sustainability advisor to your business.”  Work to streamline efforts within your business.

For more information on Virtualization:

SAP Virtualization Solutions:  http://www.dataxstream.com/products-services/sap-virtualization-solutions/

SAP AIO Virtual Infrastructure: http://www.dataxstream.com/sap-aio-virtual-infrastructure/

For more information on Green IT within SAP:

Green IT blogs on SDN https://go.sap.corp?Green-IT-blogs

SAP Green IT Community: https://go.sap.corp/GIC

SAP Sustainability Report: http://www.sapsustainabilityreport.com

Before you embark on your project’s first data migration test cycle, your data migration plan will probably start out as a simple list of preparation steps, data load dependencies, and cleanup steps.  As you progress through your several data migration test cycles, your plan should evolve into a rather complex list containing load methods, responsible persons, historical data volumes, historical execution times, steps that need to be modified and additional steps that need to be added.

I always find it rather interesting to compare the very first plan to the plan that is actually used in the cutover process.  Many project factors will cause you to amend your data migration plan.  Newly-discovered gaps, changes in the business requirements, the decision to load all of the legacy customers instead of only those customers which have been active for the past two years, and toggling back and forth between “should this one be a manual or an automated load” are only a few.   Always keep in mind that the goal in amending the plan is to keep it compatible with the changing requirements of the project.

The Data Migration Preparation Phase

Before you actually start the data migration main course, you will probably want to execute a series of preparation steps.  Some of these preparation steps may have a significant lead time; while others may require a system reboot to take effect.  Make sure that you carefully plan accordingly.  Here are some data migration preparation steps that I have found useful:

• Obtain a Data Migration Login
  Obtain a special data migration login, with all of the authorizations needed to execute all of the required tasks.  This login is only to be used during the data migration process, and is to be deleted upon the completion of the data migration process. Using the data migration login makes identification of migrated data very easy both now and several years down the road.  It also facilitates implementing a general user lockout while the data migration user is performing data migration activities.
• Technical client preparation
  The BASIS team may want to configure the data migration target client to have less dialog processes and more background and update processes.  This configuration can significantly improve data migration performance, especially if you have the opportunity to parallel process IDOCs.  In a Virtual Machine environment, the infrastructure team can even provision additional processors and memory.  You may also need additional disk space to handle large upload files, and any intermediate files that a load program might need to create.
• Configuration for IDOC processing
  If a data migration process involves creating and processing IDOCs, you may need to configure the ports, partner profiles, message types, and processing modules.  My preference for IDOC triggering is by using the ABAP program RBDAPP01 rather than immediate triggering.  This allows me to exercise the most scheduling control over the background processing options.
• Change Pointers
  During a data migration turn off change pointers.  Creating what could potentially be hundreds of thousands of change pointer records that are never going to be processed is an overhead burden on system performance and disk space that you do not need.  If the migrated data needs to be sent to external systems, manually send the data after the data migration is complete.
• Set the Correct Open Posting Period
  Some of the transactional data will want to post into an open financial or manufacturing posting period.  You will want someone from the business to make sure that the correct posting periods are open for business.
• Configuration Settings
  Sometimes, special temporary functional configuration settings are made to facilitate the data migration process.  Several functional analysts may be responsible identifying and setting these prior to the start of data migration.  As these are temporary configuration settings, it is expected that they will be set to their production values after the data migration is complete.
• Basic Setup Data
  Some data migration objects are dependent on reference data – a template from which new data migration objects can obtain some of their default values.  You will want to make sure that those who are responsible for setting up this reference data have completed their task before you start the actual data migration.  Sometimes this data is moved from the Golden Client into the target client via ALE, so you may need to set up RFC destinations, ports, distribution models, etc.
• General user lockout
  All users except for the data migration user should be locked out of the data migration client during the actual data migration activity.  It is impossible to verify that we moved exactly $15,386,254.23 in inventory from the source system to the target system if users are performing material movements in either system during the conversion.
• Connections to External Systems
  Depending on what connections to external systems are doing, you may want to make sure that they are disconnected prior to the start of data migration.  This will ensure that inbound interfaces, which may either modify existing migrated data or add new data, are disabled.

The Data Migration Phase

When all of the preparation steps are complete, it is time to start the actual data migration.  Data must be loaded into the target system in a certain sequence that is dictated by their data dependencies.  Customers must exist before sales orders can be migrated, vendors must exist before purchase orders, materials must exist before either sales orders or purchase orders, etc.  This dependency exercise varies a bit from project to project, and should have already been determined well before you start loading the data.

Automated data migration tasks for a single object (e.g. customer master data) usually include some or all of the following:

1. Creation of an upload file from the source system
2. Technical validation of the upload file
3. Functional validation of the upload file
4. Execution of a technical object to load the data into the target system
5. Technical validation of the migrated data
6. Functional validation of the migrated data
7. Collection and reporting of load statistics
8. Analyzing the fallout
9. Correcting the fallout
10. Reprocessing  the fallout

Checkpoints

At a higher level, the data migration plan will string together an end to end list of all of data migration objects.  I strongly suggest that you study the list and judiciously intersperse several checkpoints at appropriate places in the plan.  A checkpoint is an activity where all interested parties meet to review status and metrics, and to determine whether or not to proceed forward.  Keep in mind that what fails to load early in the plan will geometrically cascade into larger fallout later in the plan.  That single vendor which failed to load may cause several hundred open purchase orders to also fail.  And while the business owner of the vendor master data may think the fallout of a single vendor to be quite insignificant, the business owner of open purchase orders may have an entirely different perspective.  It is important here to make sure that all of the right stakeholders are participating, and that all are in agreement about whether or not to proceed.

Backup or Restore Points

Another activity that you should judiciously sprinkle into the plan is several backup or restore points.  It is very painful to have successfully navigated seventy-five percent of the data migration plan, only to have the next load create a huge and unrecoverable mess in the target system.  Without a backup or restore point available, you may have to wipe the slate clean and restart the entire data migration process again.

Depending on the target system and the facilities available, the backup or restore point could be a full system backup, a client copy, or a virtual machine snapshot.  Whatever the means, make sure it is a part of your plan.  It’s really great if you don’t need it.  It’s also really great to have if you do need it!

The Data Migration Cleanup Phase

When the data migration loads are finally complete, it’s time to exercise a series of cleanup steps.  This is where we:

1. Undo the special temporary settings and configurations that were done prior to the start of the data migration activity.
2. Adjust the technical configuration of the system to a more user-oriented system with the right amount of dialog, background, and update processes.
3. Turn change pointers back on.
4. Allow the general user population to access the system.
5. Connect the system to its inbound and outbound communication partners.
6. Disable the data migration user.
7. Archive those massive upload files.
8. Delete any intermediate processing data files.
9. Clean up the IDOC tables if needed.  But first make sure that you do not need to keep the IDOCs  for legal reasons.

An Example Data Migration Plan

Click on the link below to access an example data migration plan.  This plan is not a complete data migration plan, but is intended to demonstrate format and content.
Sample Data Migration Plan

Stay tuned!

In part 3 of this series on Data Migration, I will be discussing how to deal with fallout from automated data loads.

The Infallible 2-by-2 Matrix and the Magic Quadrant

A long time ago I worked for one of the big consulting firms where every interaction with the client seemed to involve a 2-by-2 matrix.  It got to the point that some of the consultants and most of the client laughed whenever one of these was pulled out.  It became a contest to create 2-by-2 matrices for everything and at one point in my life I aspired to publishing a book of 2-by-2 matrices to enable and empower everyday people to make good life decisions.  I think someone beat me to it at least in spirit.

I think the world has become conditioned by 2-by-2 matrices such that the goal is to always be in the top right hand corner – the so called magic quadrant – whereas being in the bottom left hand corner shows how much progress you still have to make.  As highly paid consultant I am supposed to help you find the path from where you are to where you want to be.  The funny thing about 2-by-2 matrices is that you can make them do whatever you want to serve your needs.  Take the example of where I should go for dinner tonight:  If I plot on one axis how likely the bill will conform to the client expense policy and plot on the other axis how well the restaurant menu prices align with my per diem I get something that looks like the graphic below.

If I follow this chart and adopt the recommendation of the magic quadrant I’ll be eating fast food every night because it’s cheap.  Unfortunately, I’ll feel like Morgan Spurlock in no time.

If I look at this chart with a more skeptical eye the two axes are showing similar information, so the whole thing is pretty bogus.  If you are going to use a 2-by-2 matrix to make a point with a client make sure it serves a useful purpose.  One of my favorite matrices plots employee/consultant willingness against ability as shown below.

In an ideal world all workers fall in the magic quadrant, i.e. are both able and willing to perform a certain task (could this be the nirvana of a low maintenance, competent and talented worker?).  The individuals who fall into other quadrants are potentially much more interesting and I would wager that not every worker falls in the same quadrant all the time.  After all, I’m not a big fan of “busy” work: I am able to do it but not very willing because I don’t value it.  But when it comes to blogging I am both willing and able (obviously!) [Ed. Note: You, sir, are a very able blogger!].  Workers who are able but unwilling present a special set of challenges for project management (you can lead a horse to water but you can’t make him drink), whereas workers who are willing but unable may be lacking skills they want to acquire.

The takeaway here is that there is a place for 2-by-2 matrices in conveying a message: stating an objective; starting a discussion, etc. but use them sparingly and only when it is relevant.

In the meantime, send me your suggestions for 2-by-2 life decision matrices and I’ll see what I can about giving you a credit in my forthcoming publication: Top Right Sure Beats Bottom Left.

It Depends

I don’t know anyone working on SAP projects that doesn’t laugh or roll their eyes when someone says “it depends”.  As much as I mock it, the response does have some merit.  For example, if a client asks you how SAP processes purchase orders it is difficult to give a single definitive reply and “it depends” is a great cop out.  Purchase orders: would you like an account assignment with that?  Do you want to use info records?  Do you have quota arrangements?  Do have to have requisitions for every PO?  And so on.

It is critical to make a rapid transition from the non-value adding “it depends” to a discussion of what matters to your client.  This is where your knowledge of their business and prior project experience come into play.  I would hope that during the project scoping and requirements gathering phases you gained enough of an understanding that you only suggest SAP processing options that are applicable to your current situation.  After all you wouldn’t discuss logistics invoice verification if you are only processing A/P invoices directly through the FI module.

This Happens on Every Project

This phrase shows up in various forms and the two most common variants I’ve heard are “Every project runs into this kind of thing” and “I’ve seen this before on other projects”.  If you find yourself making statements similar to these pause for a moment about how your client will perceive it.

I have only come across these phrases being used when a project has gone astray and has run into a crisis of some sort: testing is revealing a whole new set of requirements that weren’t captured during blueprinting; test scripts don’t have enough details; functional specifications lack detail; too many overlapping yet dependent activities, etc.

The best comeback I heard to “This happens on every project” was from an exasperated client who asked: “Just so I’m not surprised when it happens, what other things happen on every project that I should know about?”.  (Thank you Patrick!)

In short “this happens on every project” can often be traced back to an earlier failure of project management.  Sometimes the situation is unavoidable, for example, despite your guidance about not moving on to the next project phase because of incomplete deliverables your client insists on going forward because the timeline says it is time to move forward.  Sometimes, key milestones are met but the sign-off is far more conditional than it should be.  Regardless of the underlying circumstances your client probably sees you do one of two things:

• Reactively tell them this happens on every project and then describe how every project goes through some level of fire drill and triage to get things back on track
• Proactively manage their expectations and make clear the potential problems the project may run into: for example, designing and developing while integration testing is never a good idea.  In this way you provide leadership, not just management, and set a course to avoid a fire drill.

Typically a client does not like surprises and pro-active behavior is better than reactive.  I like to think that as a project manager I exist to prevent forest fires, not to fight them.

Consulting Catchphrases Conclusions

My underlying thought is that the consulting clichés have an element of truth and validity to them and you should strive to use them as a springboard to bring value to your client.  Think about where the clichés came from – usually a negative experience – and see if you can anticipate where you might feel compelled to use these catchphrases on your project and (ahem, forgive me!) avoid them like the plague.

Please let me know if you have any favorite overworked project phases and I’ll do a follow up posting if I get a good number of responses.

How much data to move? How much data to leave behind? What to automate, and what to execute manually?  How to gracefully orchestrate and execute a data migration cutover from one system to another?  Where and how to fit the data migration plan into the overall business implementation plan?  How to continue to run the business during the data migration phase of the business project implementation? These questions are all part of the planning fun!

Data Migration Testing

Processes that we have exercised and continue to exercise on a regular basis are no big deal, do not cause anxiety, or raise any blood pressures.  The daily report, the custom transaction, the interface that runs several hundred times a week – they are all second nature now.  They happen, and no one notices.

But reflect back to the very first time these processes went live.  This is precisely where you are with data migration.  For most, it is a once in a lifetime event.  It is important, then, to raise confidence levels, and reduce anxiety levels surrounding this activity.  This is achieved by practice, practice, practice.

Typically, I like to see at least three data migration test cycles executed in an isolated data migration client.  This allows for several attempts at exercising and fine-tuning the data migration plan; collecting nominal run time statistics to have some idea of how long the data migration might take; identifying and fixing any data migration program object defects; identifying and fixing any data mapping and content defects; and identifying and fixing any functional configuration issues. Each of these tasks are done with the goal of significantly improving the fallout rate with each data migration test cycle.  These data migration test cycles also give us the opportunity to practice our legacy extract skills, our fallout analysis skills, and our fallout manual cleanup skills.

Source Data Quality

Each data migration test cycle begins with the legacy or source data extract.  An important activity between data migration test cycles is the cleanup of the legacy or source data.  The data migration process will most likely discover many data problems.  Customer and vendor address data alone are enough to bring your bolt-on tax jurisdiction determination software to its knees – that zip code that is too short, too long, or just plain incorrect when combined with the associated city and state; the various abbreviations used in place of the actual city name; the city and state entered in the same field when they should have been entered into separate fields; etc.   If the source data is not fixed, your data migration test cycles will begin to look like the movie Groundhog Day.

Source data provisioning and cleanup could very well be problematic, especially in an acquisition or merger scenario where the providers are part of a different organization and have no incentive to participate in your project.  If you encounter this scenario, you will most likely need to engage the appropriate management level from your organization to have a serious discussion with the appropriate management level from the providing organization.  Recognize this, be in control, and raise that flag early.    Do not sit around for several days waiting for data to arrive, as this will only set your project behind schedule.

Data Migration Support for Other Testing

In any business project scenario, the development team will salivate at the thought of testing their customizations against your real data in the conversion test cycle box.  Likewise, the functional team will be chomping at the bit to use your real data to test configuration scenarios.  And the interface team and the data warehouse team can’t ever get enough data to play with.

While working within your three data migration test cycles, just say NO.  You absolutely need a controlled environment for data migration test cycles, and these other teams will not respect that.  They have a different focus and purpose which requires changing and manipulating degrees of freedom that you need held constant.

But, we are all on the same team trying to move the project to the finish line within the expected project timeline.  So to help your other team members, plan for additional data migration cycles to provide this data to these teams.  That’s more data migration practice for you, and it gives everyone else what they need.

The Perfect World of Data Migration

In a perfect world, for each data object to be converted:

1. The functional specification and mapping documents are well-written and clear.
2. A data load file is built exactly in conformity to the well-written functional specification and mapping documents.
3. The data migration ABAP objects are built exactly as specified by the well-written functional specification and data mapping documents.
4. No one is insisting that we move a square peg into a round hole.

In a perfect world, for each set of data that is to be migrated:

1. The legacy system data has been thoroughly cleansed.
2. The providers of the legacy data are genuinely interested in providing accurate data on time.
3. Server-resident load files in a Unicode system are correctly encoded to UTF-8.
4. A delimiter other than comma has been specified for the load file.

In a perfect world, the data migration test cycle client:

1. Is built in a box that is not the development box.
2. Is configured exactly like the client where the data migration ABAP objects were built and tested.
3. Is not open for configuration, unless by design.
4. Is not part of a transport path. to prevent the current cycle of conversion testing from being blindsided by any new configuration changes.
5. Is locked down so that only data migration and data validation tasks can be performed.
6. Is configured to handle a more background and update processes and fewer dialog processes.
7. Is built in a box that has enough disk space to be the repository for the primary load files and any intermediate processing files needed.

In a perfect world, at the project level:

1. There is an overall business cutover or implementation plan into which you can assimilate your data migration plan.
2. The data migration plan integrates nicely into the overall business cutover plan.
3. There is a strategy in place to bring the data current between the time the cutover freeze is enforced and the implementation date.

Stay Tuned!

In the real world, the fun is just beginning.  Those perfect world scenarios just never seem to happen.   But stay tuned!  In my next blog post in this series, I will discuss the details of the Data Migration Plan.  After that, in subsequent posts, I will drill down into some real world scenarios that I have encountered and discuss how I dealt with them.