Let’s say that Customer 15 in your SAP system buys products from your company.  It sends inbound EDI Orders to you using three different partner IDs because they have 3 internal divisions and they want all transactions to be separate.  You want to keep all sales data for this customer under one customer number in your SAP system and just separate them by a different division.  You are required to send out EDI invoices to this customer, but they must go to the correct EDI Partner ID.  Let’s say you would normally use the Standard SAP Output Type RD00 and  Access Sequence 0003 (Sales Org, Distribution Channel, Division, Customer Number) for producing your INVOIC IDOCs.

Your first step would be to make three copies of the RD00 Output Type (ZRD0,  ZRD1, ZRD2) using transaction V/40.  We will need to make three copies because the Standard RD00 Output type uses Access Sequence 0004.   Select Change mode for this transaction and find the RD00 entry.  Select the RD00 entry and click on the ‘Copy As’  icon on the Menu bar or press the F6 key.

Change the Output Type to ZRD0 and the Access Sequence to 0003 and hit your Enter key.

A prompt will appear with three options.  Copy all, only copy entry, or cancel.  Select the copy all option.

An information message will appear displaying the number of dependent entries have been added.

Hit the enter key and the main Output Type screen will appear with your new output type.  Select the entry and double click the Processing Routines option in the left hand column.  Make sure that the EDI Medium is listed under the Processing Routines.  If not, add it as shown below.

Double click on the Partner Functions option in the left hand column.  Make sure the Partner Function you will be using is listed for the EDI Medium.  In this case we will be using the BP (Bill-to Party) Partner Function.

Hit the save button to save the new output type.  Repeat this process for the other two new output types.  If the Access Sequence you require is not available a new one can be created.  Creating a new Access Sequence is not covered in this blog.

Now we can set up the Partner Profiles for this customer using transaction WE20.  On WE20, expand the Partner Type KU folder and select customer number 15.  Under the Outbound parmtrs section click on the ‘Create Outbound Parameters’ icon.

Fill in the proceeding screen with the as follows.  If you are using an IDOC Extension enter that also.  For this example we are just using the Standard INVOIC02 IDOC.  As you can see I have entered 10 in the Message Code field.  This represents one of the Divisions for this customer.  You can use this field and the Message Function field to differentiate between Divisions or other criteria you wish to use.

Select the Message Control tab and click on the Insert Row icon.

Your Application will be V3 (Billing), your Message Type will ZRD0 (the new output type), and your Process Code will be SD09 (Invoice).  Enter these three values and save the entry.

You will create two more Outbound Parameter entries for the other Divisions.  In these two new entries you will place the other divisions in the Message Code field and the other output type field in the Message Control Message Type field.  Everything else will be the same.

When you are done, the Partner Profile for this customer will look like this.

Finally we will need to create the output determination so the output is created automatically once the invoice is completed.  For invoicing, the transaction for output determination is VV31 for create and VV32 for changes.  We will use VV31 since this is the first time this output type will be used.  Enter ZRD0 as the Output Type and hit Enter or click the Key Combination button.  If more then one Key Combination is available then a pop up window will appear asking which one you want to use.  Once you select one and click the Green check the key fields will appear.  If there is only one, then the key field will appear immediately.  Select the SOrg/Distrib Ch/Division/Customer option.

Enter the Sales Org, Distribution Channel, and Division data.  Then for each customer you will be using this Output Determination for you would enter the Customer Number, Partner Function, Medium (6 = EDI), Date/Time (This represents when/how the output will be processed), and Language (EN = English).  Then hit the save button.  Repeat this process for the other two output types.

Once this is completed every time an invoice is created for this combination of Sales Org, Distribution Channel, Division, and Customer number an Output will be created using the specified Output Type.  Once the output is processed an IDOC will be created with the Division in the Control Record of the IDOC which can then be used in the EDI Software to determine which EDI Partner the data should be sent to.

How do you convert these external customer numbers into your internal SAP customer numbers?

Some may hard code these conversions into their EDI maps.  This approach can be very high maintenance as customers can add new ship-to locations or reorganize their internal numbers which would require changes to your maps.

Others may set up a cross reference table within their EDI translation table to perform the conversion.  This works well at times, but then you are at the mercy of your EDI group to update the table with any new additions or changes to existing entries.

I find the best way to perform this conversion is within your SAP system.  Mainly because SAP has given you a standard utility to perform the conversion.  All you need to do is populate a few fields of the incoming IDOC and populate the EDPAR table using transaction VOE4 with the internal and external customer numbers.  The Function Module used to process the IDOC takes care of the rest.  SAP uses the Sender Partner Number of the IDOC Control record along with the PARVW and LIFNR elements from the E1EDKA1 segment to look up the correct entry in the EDPAR table.  In your EDI maps you would populate the PARVW element with the partner function code for the external customer number (AG = Sold-To, WE = Ship-To) and populate the LIFNR field with the external customer number received in the EDI file from that partner.  All other fields would remain blank in the E1EDKA1 segment.

Sender Information Partner Number from the Control Record:

E1EDKA1 Segment for the Sold-To Partner (AG):

E1EDKA1 Segment for the Ship-To Partner (WE):

Use transaction VOE4 to populate the EDPAR table.  You enter the value from the Sender Partner Number in the IDOC control record in the Customer field, the function code for the partner in the Ext. Function field.  Note: The Ext. Function field will not be the same as the Partner Function (PARVW) element of the IDOC.  (AG in IDOC = SP on VOE4, WE = SH).  The External Partner is the number received in the EDI data from the customer which is also the number in the LIFNR element of the IDOC.  Int. no is the SAP customer number associated with the External number.

VOE4 transaction with required entries for the Sold-To and Ship-To for IDOC information above:

Using the EDPAR table for customer conversion allows this function to remain in control of those responsible for customer accounts.  Users can be trained to use the VOE4 transaction to update the EDPAR table so when new ship-to or sold-to numbers need to be added they can be done in a timely matter and not rely on other departments.  It also takes this responsibility away from the EDI group so they don’t have to maintain maps or tables for this function.

This function can be used for other incoming IDOCs as well.  You will need to check the Function Modules used to process incoming IDOCs to determine if EDPAR can be used for customer number conversion and on which IDOC segment you would send the external number.

Outbound IDOCs can use either the EDPAR table or the PUMA table to cross reference external customer numbers.  Again, it depends on the IDOC type.  I will cover this topic in a separate blog.

Let’s look at how this process works.  Let’s say we have created an invoice document in SAP.  In this case, the Sold-to, Ship-to, and Bill-to partners are all SAP customer number 15.  If we want to create an INVOIC02 IDOC on which the external customer numbers are populated for all three of these partners we would have to set up three EDPAR entries as displayed on the below screen shot.  The Customer field will contain the SAP partner number (Sold-to, Ship-to, Bill-to).  The Ext. Function field will contain the Partner Function (SP = Sold-to, SH = Ship-to, BP = Bill-to).  The External Partner field will contain the external partner number that the customer is expecting on the EDI file.  And the Int. no. field will contain the SAP partner number (Same as the Customer field).

After we have set up the Partner Profile (WE20) for the INVOIC for this customer we can produce the output through transaction VF02 to create the IDOC.  The INVOIC02 IDOC will display the Sold-to and the Bill-to partners at the Header level in the E1EDKA1 segments, but because we could have multiple Ship-tos on an invoice the Ship-to partner(s) are located at the Item level in the E1EDPA1 segments.  In both segments, the external partner numbers will be populated in the LIFNR element.  The segments for the three partners are displayed below:

As you have probably noticed, the Partner Functions on the IDOCs do not match the Partner Functions in the EDPAR table.  Those would translate as follows: SP in EDPAR = AG in the IDOC, SH = WE, BP = RE.  As you can see the LIFNR field is populated with the values in the External Partner field of the EDPAR table.  These values can now be used in the EDI mapping tool to send the value to the customer.

This is a useful tool as new partners can be added easily using the VOE4 table by a customer service representative.  It also keeps the cross reference within SAP so that the EDI group does not have to build their own functionality to convert the partner numbers and maintain it as new customers are added or locations are changed.

This table can be used for other partner types and other outbound IDOCs.  You would need to check the function module that processes the output to see if it uses the EDPAR table and for which partners.  Another internal SAP table that is used for partner cross reference is the PUMA table.  This is used when creating DELVRY and SHPMNT IDOCs whose function modules do not use the EDPAR utility.  The use of the PUMA table will be discussed in a separate blog.

Why the data will not load correctly.

There is a litany of reasons why some data records will load correctly while others will not.  Here is a list of some common root causes:

1. Poor quality legacy data.
   Legacy systems which are not as tightly integrated as SAP, and are not under master data control allow the end user a bit of freedom when entering data.  A zip code may contain too little or too many characters; the email address is not properly formatted; numeric fields have transposed digits; various forms of abbreviations (especially in the city field), a quantity of zero (0) permitted by the legacy system and uploaded into a field where SAP will not accept a quantity of 0 and even simple misspellings  all can cause stringent validation checks to trigger an error and prevent the record from loading at all.  A more sinister type of error occurs when the data is functionally incorrect, but good enough to pass all of the SAP validity checks.  In this case, the data record will technically load into SAP, but will not be functionally correct.  Duplicate customers, duplicate vendors, and the data entry error for a quantity of 1000 instead of 100, and the wrong pricing condition applied to a sales order line are examples of this scenario.

1. Functional configuration and supporting data effects.
   Many times I have watched the load statistics for a data object plummet from near 100% in the cycle two test load to near 0% in the cycle three test load.  This is very unnerving to the client because the cycle three test load is getting rather close to the go-live date, and “by the way, shouldn’t the statistics be getting better rather than worse?”  Functional configuration changes can wreak havoc on any data load.  Flipping the switch on a data field from optional to required; turning on batch management or serialization for materials for the first time; changes in the handling of tax, tax codes, and tax jurisdiction codes; that account determination entry that is missing or not set up correctly; a missing unit of measure or unit or measure conversion factor; the storage location in the upload file which does not exist in SAP – any of these can cause a load to drop mostly or completely onto the floor.While change is inevitable on any project, it is important to control and communicate the change so that the downstream impact can be recognized and understood.   Controlled change and communication always works better than total surprise.  Perhaps if we all know ahead of time about that data field that is now required, we can impose a requirement on the data extract side to make sure that the data field is populated before it enters the upload file. 
2. Additional data in the upload file.
   Inserting a new field in the middle of the upload file data structure might be necessary for the business to close a gap, but if that change is not communicated to the technical team so that appropriate adjustments can be made to the load object’s input structures and processing logic, the new data will surely never load, and may cause misalignment of the data fields which follow it in the upload structure.

The Finger Pointing Game

It’s the load program!  No, it’s the data!  No, it’s the configuration!  No, it’s … (fill in your favorite finger pointing game excuse explaining why data will not load).

If you ever find yourself in the midst of this type of finger-pointing game, immediately stop the madness.  For this and similar situations, I apply a simple litmus test which has never failed me yet – manually enter the EXACT upload data into SAP for the transaction which has failed.  If one can be entered manually, then the program will be able to automatically load thousands with similar upload data and functional configuration.

I have lead many a functional analyst – kicking, screaming, and ranting about how terrible the load program is – to the terminal to play “let’s enter one manually”.  Typically, the result is that one cannot be entered manually due to configuration issues or the lack of supporting values data.  Once these issues are cleaned up, the load program “magically” begins to process thousands of records with no trouble at all.

Sometimes, the load object appears to be the cause, but is not the root cause.  The important data item not being handled by the load object (which was not called out in the functional specification document), the data item which turns blue (because the functional specification document explicitly stated “put this data item into the blue category”), the formula which is not calculating the desired result (but is indeed the exact formula found in the functional specification document) – all are examples of the load object adhering accurately to an incorrect functional specification document.  The root cause, then, is the functional specification document which must first be revised and checked into the controlled document repository before making any code modifications.

It’s the Program

Well, OK, sometimes the load object is at fault.  But it is extremely rare.

Collecting and Reporting the Technical Load Statistics

Load statistics are an important technical metric, indicating what percentage of the upload file has successfully posted a transaction into SAP.  This is a basic and simple record count check.  How many records were presented in the upload file, how many records posted successfully to SAP, and how many records failed to post.  Also, does the number of successful transaction plus the number of failed transactions equal the total number of records presented in the upload file.

Here is how I communicate this technical metric:

This technical metric indicates only that all of the SAP validation rules for posting the transaction have passed.  It does not indicate that the master or transactional data posted to SAP is actually correct in functional terms.

The Functional Review

It is very possible for an entire upload file to technically load at 100%; while at the same time, functionally fail at 100%.  The customer may be missing a partner, the material or article may be categorized incorrectly, the pricing conditions on a sales order may not have the desired validity date range, the GL posting may be to the wrong accounts.  Hence, the need a functional review and validation of the data transacted into SAP.  The data type – master or transactional – determines the type of functional review and metrics to be employed here.

For master data, such as of materials, customers, vendors, etc., it is impossible to individually validate the many thousands of entries in your SAP system.  But statistical methods can be employed which will guide you through a random sampling of the data, while at the same time assuring accuracy at a high degree of confidence levels.

For transactional data, such as inventory, open sales orders, open AR, etc., direct mathematical comparisons can be employed.  On a grand scale, if an inventory value of $15,246,321.44 is being moved from your legacy system to SAP, then that exact amount must arrive in SAP when the migration task is complete.  It may be a bit more difficult to do this mathematical comparison at a more granular level.   If, in the move to SAP you are also redesigning your material/storage location combinations, a direct comparison between the legacy system and SAP may not be possible without a translation factor.  The same scenario exists if you redesign your GL chart of accounts, where one legacy GL account now maps to several SAP GL accounts, or vice versa.

Reporting the Fallout

If an automated data load was not technically 100% successful, a clear set of error messages complete with a link back to the legacy data must be mined, formatted, and presented to the business for analysis.  Such a report really helps to facilitate the fallout cleanup.  The link back to the legacy data must be carefully designed into the load process to make sure that, for example, the legacy customer number, legacy vendor number, legacy sales order number, legacy purchase order number, etc. is included as part of the data being handled.  Without the link back to the legacy data, it becomes very difficult to identify which data record needs to be fixed.

The error message mining technique that I use depends on the load method.  I will describe two here – one for a BDC load method and one for an IDOC load method.

Mining Meaningful Error Messages from a BDC Session Log

At the completion of a batch input session, the batch input session overview screen (SM35) displays the technical load statistics.  In this example, out of a total of 229 transactions, 13 failed and 216 succeeded.

The session log shows the status of all 229 transactions.  This screen snapshot is a fragment of the complete session log for the batch input session.  It shows many successful transactions (Type = S) and one failed transaction (Type = E).  The error message here is clear – the article does not exist or is not activated.

But as you can imagine, the 13 failed transactions with error type = E are sprinkled throughout the many pages of this log file.  With only 229 transactions, this log file is quite easy to pick through to find the 13 errors.  But imagine if the number of transactions were in the thousands or tens of thousands.  How do we extract only the failed transactions and present a concise report of the failed transactions?

To do this, I use SAP transaction SM35P – Batch Input Log Overview.  This transaction has the ability to set a filter on any field in the batch input log file, display the filtered results, and then to export the results to a local file.

To enter the mode where this is possible, first press the PRINT icon.

Next, set the filter.  The appropriate filter field here is SESS. TYPE.

We only want the errors, so set the filter for field SESS. TYPE = E.

The display now shows only the 13 rows containing the error messages.  This can be exported directly to a local spreadsheet for further analysis.

Mining Meaningful Error Messages from IDOCs

Depending on the IDOC and the processing module, mining the error status messages from IDOCs can be very easy or somewhat challenging.  For the more difficult scenarios, you probably will need to hone your EXCEL skills to properly join several extracts together into one complete picture.

When creating IDOCs with a load object, I always note the date, time, and IDOC basic type.  I will use this information as the selection criteria for transaction WE05, which is going to locate the IDOCs and display the results information I need after the IDOCs are processed.  The results that I usually collect are the error status messages, and some data content from a segment or two to illustrate exactly where the problem is in the legacy data.

While the background job is busy processing the IDOCs, I usually take a peek, using transaction WE05, to see how the load is progressing.  If I see that most of the IDOCs are falling onto the floor (IDOC status 51) rather than moving into the database (IDOC status 53), I usually stop the background job and begin an immediate analysis of the fallout.  If the fallout solution does not require a change to the IDOC data content (e.g. a configuration change), then I can replay the fallout using SAP transaction BD87.  If the fallout solution does require a change to the IDOC data content, then the complete set of IDOCs must be regenerated again.

Let’s see what WE05 can tell us about a completed Article Master load.

On the WE05 screen below, we can see that of the total of 14,005 IDOCs, 11,412 have processed successfully and 2,593 have failed to process.  By double clicking the Status 51 folder, the display will show only the Status 51 IDOCs – the fallout.

By pressing the “status list” icon (shown above), the display will show the status messages for the fallout.    Once these messages are displayed, pressing the “export” icon allows me to save the screen contents to a spreadsheet.

Now it would be really nice if I could have the article number in the spreadsheet right next to the error message.  The article number in the ARTMAS IDOC is stored in segment E1BPE1MATHEAD.  The segment content for each IDOC can be displayed by pressing the “list specific segment” icon and entering the segment name in the box.

The segment display will show all fields in the segment, so I usually hide all of the columns that I don’t want to see.  Here is the E1BPE1MATHEAD segment display showing only the article number.  I can use the export icon to save the list of article numbers to another spreadsheet.

Here is a portion of the complete spreadsheet showing the error messages and the article numbers side by side.

A filter applied to the spreadsheet shows that the 2,593 errors are all grouped into one of three error status categories.  By selecting a single category, Excel will also show me the number of records within that failure category.

Sometimes it is easier to mine the status messages directly from the IDOC status table EDIDS.  This is especially true where the processing module is a BAPI which returns an error table rather than a single error message.  In this case, when you press the “status list” icon in WE05, only the first error status message of several is displayed for each IDOC.  I find that the first message is not very helpful (as shown below).  I also find that typically the second or third message in the return status table is usually the important one.  You won’t see it displayed on the WE05 screen, but you can mine it from the EDIDS table.

SAP transactions SE11 or SE16 both support this activity.  For the selection criteria I use the IDOC number range, status 51, and status type E.  On the display screen, choose only the relevant fields for display – the IDOC number (DOCNUM), IDOC status (STATUS), status message (STATXT), the four substitution parameters for the status message (STAPA1, STAPA2, STAPA3, STAPA4) and the message type (STATYP).  All of this can be exported into a spreadsheet.  If you really want to test your Excel skills, you can write code that will move the substitution parameters into their placeholders in the status text.

Preparing for the next data migration cycle – Let the fallout analysis and cleanup begin.

Presenting the fallout report to the business with a set of clear error reasons and links back to the legacy data is key to enabling the legacy data cleanup process to proceed.  In the iterative process of data migration cycles, cleansing the legacy data is a step in the right direction towards an improved next conversion cycle.

I hope you enjoyed this blog series on data migration.  Please feel free to send comments or questions.

Before you embark on your project’s first data migration test cycle, your data migration plan will probably start out as a simple list of preparation steps, data load dependencies, and cleanup steps.  As you progress through your several data migration test cycles, your plan should evolve into a rather complex list containing load methods, responsible persons, historical data volumes, historical execution times, steps that need to be modified and additional steps that need to be added.

I always find it rather interesting to compare the very first plan to the plan that is actually used in the cutover process.  Many project factors will cause you to amend your data migration plan.  Newly-discovered gaps, changes in the business requirements, the decision to load all of the legacy customers instead of only those customers which have been active for the past two years, and toggling back and forth between “should this one be a manual or an automated load” are only a few.   Always keep in mind that the goal in amending the plan is to keep it compatible with the changing requirements of the project.

The Data Migration Preparation Phase

Before you actually start the data migration main course, you will probably want to execute a series of preparation steps.  Some of these preparation steps may have a significant lead time; while others may require a system reboot to take effect.  Make sure that you carefully plan accordingly.  Here are some data migration preparation steps that I have found useful:

• Obtain a Data Migration Login
  Obtain a special data migration login, with all of the authorizations needed to execute all of the required tasks.  This login is only to be used during the data migration process, and is to be deleted upon the completion of the data migration process. Using the data migration login makes identification of migrated data very easy both now and several years down the road.  It also facilitates implementing a general user lockout while the data migration user is performing data migration activities.
• Technical client preparation
  The BASIS team may want to configure the data migration target client to have less dialog processes and more background and update processes.  This configuration can significantly improve data migration performance, especially if you have the opportunity to parallel process IDOCs.  In a Virtual Machine environment, the infrastructure team can even provision additional processors and memory.  You may also need additional disk space to handle large upload files, and any intermediate files that a load program might need to create.
• Configuration for IDOC processing
  If a data migration process involves creating and processing IDOCs, you may need to configure the ports, partner profiles, message types, and processing modules.  My preference for IDOC triggering is by using the ABAP program RBDAPP01 rather than immediate triggering.  This allows me to exercise the most scheduling control over the background processing options.
• Change Pointers
  During a data migration turn off change pointers.  Creating what could potentially be hundreds of thousands of change pointer records that are never going to be processed is an overhead burden on system performance and disk space that you do not need.  If the migrated data needs to be sent to external systems, manually send the data after the data migration is complete.
• Set the Correct Open Posting Period
  Some of the transactional data will want to post into an open financial or manufacturing posting period.  You will want someone from the business to make sure that the correct posting periods are open for business.
• Configuration Settings
  Sometimes, special temporary functional configuration settings are made to facilitate the data migration process.  Several functional analysts may be responsible identifying and setting these prior to the start of data migration.  As these are temporary configuration settings, it is expected that they will be set to their production values after the data migration is complete.
• Basic Setup Data
  Some data migration objects are dependent on reference data – a template from which new data migration objects can obtain some of their default values.  You will want to make sure that those who are responsible for setting up this reference data have completed their task before you start the actual data migration.  Sometimes this data is moved from the Golden Client into the target client via ALE, so you may need to set up RFC destinations, ports, distribution models, etc.
• General user lockout
  All users except for the data migration user should be locked out of the data migration client during the actual data migration activity.  It is impossible to verify that we moved exactly $15,386,254.23 in inventory from the source system to the target system if users are performing material movements in either system during the conversion.
• Connections to External Systems
  Depending on what connections to external systems are doing, you may want to make sure that they are disconnected prior to the start of data migration.  This will ensure that inbound interfaces, which may either modify existing migrated data or add new data, are disabled.

The Data Migration Phase

When all of the preparation steps are complete, it is time to start the actual data migration.  Data must be loaded into the target system in a certain sequence that is dictated by their data dependencies.  Customers must exist before sales orders can be migrated, vendors must exist before purchase orders, materials must exist before either sales orders or purchase orders, etc.  This dependency exercise varies a bit from project to project, and should have already been determined well before you start loading the data.

Automated data migration tasks for a single object (e.g. customer master data) usually include some or all of the following:

1. Creation of an upload file from the source system
2. Technical validation of the upload file
3. Functional validation of the upload file
4. Execution of a technical object to load the data into the target system
5. Technical validation of the migrated data
6. Functional validation of the migrated data
7. Collection and reporting of load statistics
8. Analyzing the fallout
9. Correcting the fallout
10. Reprocessing  the fallout

Checkpoints

At a higher level, the data migration plan will string together an end to end list of all of data migration objects.  I strongly suggest that you study the list and judiciously intersperse several checkpoints at appropriate places in the plan.  A checkpoint is an activity where all interested parties meet to review status and metrics, and to determine whether or not to proceed forward.  Keep in mind that what fails to load early in the plan will geometrically cascade into larger fallout later in the plan.  That single vendor which failed to load may cause several hundred open purchase orders to also fail.  And while the business owner of the vendor master data may think the fallout of a single vendor to be quite insignificant, the business owner of open purchase orders may have an entirely different perspective.  It is important here to make sure that all of the right stakeholders are participating, and that all are in agreement about whether or not to proceed.

Backup or Restore Points

Another activity that you should judiciously sprinkle into the plan is several backup or restore points.  It is very painful to have successfully navigated seventy-five percent of the data migration plan, only to have the next load create a huge and unrecoverable mess in the target system.  Without a backup or restore point available, you may have to wipe the slate clean and restart the entire data migration process again.

Depending on the target system and the facilities available, the backup or restore point could be a full system backup, a client copy, or a virtual machine snapshot.  Whatever the means, make sure it is a part of your plan.  It’s really great if you don’t need it.  It’s also really great to have if you do need it!

The Data Migration Cleanup Phase

When the data migration loads are finally complete, it’s time to exercise a series of cleanup steps.  This is where we:

1. Undo the special temporary settings and configurations that were done prior to the start of the data migration activity.
2. Adjust the technical configuration of the system to a more user-oriented system with the right amount of dialog, background, and update processes.
3. Turn change pointers back on.
4. Allow the general user population to access the system.
5. Connect the system to its inbound and outbound communication partners.
6. Disable the data migration user.
7. Archive those massive upload files.
8. Delete any intermediate processing data files.
9. Clean up the IDOC tables if needed.  But first make sure that you do not need to keep the IDOCs  for legal reasons.

An Example Data Migration Plan

Click on the link below to access an example data migration plan.  This plan is not a complete data migration plan, but is intended to demonstrate format and content.
Sample Data Migration Plan

Stay tuned!

In part 3 of this series on Data Migration, I will be discussing how to deal with fallout from automated data loads.

What a great idea!  Too bad I hate this suggestion… and it’s all SAP’s fault!!

You see, SAP transaction BD53 just does not play well with others.  My full HRMD_A solution used a custom filter object (transactions BD95 and BD59) and a custom outbound IDOC formatting function module to convert change pointers into IDOCs (transaction BD60).  The dirty little secret of transaction BD53 is that every time you save and update the IDOC reduction in transaction BD53, SAP goes and screws up a bunch of configuration  displaying a message that says “Interface changed for message type <your_message_type>:  Settings reset”

Wait! What!?!

What does that message even mean?  Well, that message is telling you in very cryptic language that SAP has gone and reset the settings in BD59 and BD60 to the SAP standard! So, every time the field reductions were updated, kiss your custom filter assignments and custom IDOC processing function goodbye!

Why couldn’t the message say:

Hey Buddy! You know all that custom configuration you did to get your interface working just so? Well, its gone and you’re going to have to redo it all in the following transactions:
BD59
BD60
etc.

I guess SAP is just far too polite.

So, be very cautious when you pair IDOC reduction with custom filter assignments and custom IDOC formatting function modules and be aware that every time the IDOC reduction is updated, you will need to go back in to reconfigure transactions BD59 and BD60.

For more information, see SAP OSS Note 655321

The reduction at segment level (see also Note 301223) was facilitated in order to be able to copy message type HRMD_A. This involved setting the reduction at field level, which was not taken into account in the code because it is not possible to centrally determine which fields are mandatory and which are dependent for all country versions. In addition, for one info type there may several records for different periods of time, which may also cause a problem when you are reducing fields.

So, in essence, field-level reduction isn’t supported for HRMD_A.  The solution supplied in the OSS note suggests creating user-exit code to filter out the unwanted fields.  I already have implemented BADI HRALE00OUTBOUND_IDOC to add some other processing logic (if you’re lucky I’ll cover that logic in this blog, too), so I have a module in which I can place the code. And while the code to loop through an IDOC and clear out all instances of E1P0002-PERID, E1P0002-GBDAT, and E1P0002-GESCH is very easy, I can’t help but think I can make the code more useful.  You see, I hate unitasking code.  I can see it now.  If I take the shortcut path and wrote the unitasking code to clear those 3 fields, the code won’t be in production for more than a month before the requirements change and now there is another field that needs to be filtered out as well.  With unitasking code, I would have to crack open the code, create a transport, unit test, transport to Q, integration test, wait for a transport window, blah, blah, blah.  Ugh!  What a waste of time for a minor change.  We can do better.

I decided to create a multi-tasking module that will work for HRMD_A message types.  The fields to be cleared will be stored in TVARV in the format <SEGMENT>-<FIELD> (e.g. E1P0002-PERID).  That way, whenever the HR functional team wanted to filter out a new field, all they had to do was to update a TVARV variable.

To implement the solution, I created a new method called CLEAR_IDOC_FIELDS in my class implementation for BADI HRALE00OUTBOUND_IDOC.  CLEAR_IDOC_FIELDS has the same method signature as IF_EX_HRALE00OUTBOUND_IDOC~IDOC_DATA_FOR_RECEIVER_MODIFY:

The code is pretty straight forward.  To allow multiple interfaces to coexist in method CLEAR_IDOC_FIELDS, the TVARV variable that stores the fields to clear is in the format ‘ZHR_CLR_’ concatenated with the IDOC message type.

METHOD CLEAR_IDOC_FIELDS .
*----------------------------------------------------------------------*
* Class:     ZCL_IM_HRALE00OUTBOUNDIDOC                                *
* Method:    CLEAR_IDOC_FIELDS                                         *
* Author:    Craig Stasila                                             *
*----------------------------------------------------------------------*

  TYPES: BEGIN OF TY_FILTER,
           TABNAME   TYPE DFIES-TABNAME,
           FIELDNAME TYPE DFIES-FIELDNAME,
           POSITION  TYPE DFIES-POSITION,
           OFFSET    TYPE DFIES-OFFSET,
           LENG      TYPE DFIES-LENG,
        END OF TY_FILTER.

  DATA: WA_FILTER      TYPE TY_FILTER,
        LT_FILTER      TYPE TABLE OF TY_FILTER,
        LS_TABNAME     TYPE DDOBJNAME,
        LS_SAV_TABNAME TYPE DDOBJNAME.

  DATA: LS_NAME   TYPE TVARV-NAME,
        LT_TVARVC TYPE TABLE OF TVARVC,
        WA_TVARVC TYPE TVARVC.

  DATA: LT_DFIES TYPE TABLE OF DFIES,
        WA_DFIES TYPE DFIES.

  DATA: LR_SEGNAM TYPE RANGE OF EDIDD-SEGNAM,
        WR_SEGNAM LIKE LINE OF LR_SEGNAM.

  FIELD-SYMBOLS: <FS> TYPE LINE OF EDIDD_TT.

* Fields to clear are in variable
  CONCATENATE 'ZHR_CLR_' MESTYP INTO LS_NAME.

* Get list of fields to clear from TVARV
  SELECT * FROM TVARVC
    INTO TABLE LT_TVARVC
   WHERE NAME = LS_NAME
     AND TYPE = 'S'.

  CHECK SY-SUBRC = 0.

* Build list of fields to clear
  LOOP AT LT_TVARVC INTO WA_TVARVC.
    CLEAR WA_FILTER.
    SPLIT WA_TVARVC-LOW AT '-' INTO WA_FILTER-TABNAME WA_FILTER-FIELDNAME.
    CHECK SY-SUBRC = 0.

    COLLECT WA_FILTER INTO LT_FILTER.
  ENDLOOP.

  SORT LT_FILTER BY TABNAME.
  CLASS CL_ABAP_CHAR_UTILITIES DEFINITION LOAD.

* Get field offsets
  LOOP AT LT_FILTER INTO WA_FILTER.
    LS_TABNAME = WA_FILTER-TABNAME.

    IF LS_TABNAME NE LS_SAV_TABNAME.

      CALL FUNCTION 'DDIF_NAMETAB_GET'
        EXPORTING
          TABNAME   = LS_TABNAME
        TABLES
          DFIES_TAB = LT_DFIES
        EXCEPTIONS
          NOT_FOUND = 1
          OTHERS    = 2.

      IF SY-SUBRC <> 0.
        CONTINUE.
      ENDIF.

      SORT LT_DFIES BY TABNAME FIELDNAME.

    ENDIF.

    LS_SAV_TABNAME = LS_TABNAME.

    READ TABLE LT_DFIES INTO WA_DFIES
         WITH KEY TABNAME   = WA_FILTER-TABNAME
                  FIELDNAME = WA_FILTER-FIELDNAME
         BINARY SEARCH.

    CHECK SY-SUBRC = 0.

    WA_FILTER-POSITION = WA_DFIES-POSITION.
    WA_FILTER-OFFSET   = WA_DFIES-OFFSET / CL_ABAP_CHAR_UTILITIES=>CHARSIZE.
    WA_FILTER-LENG     = WA_DFIES-LENG.

    MODIFY LT_FILTER FROM WA_FILTER.

* Build segment name range
    IF SY-SUBRC = 0.
      WR_SEGNAM-SIGN   = 'I'.
      WR_SEGNAM-OPTION = 'EQ'.
      WR_SEGNAM-LOW    = WA_FILTER-TABNAME.
      COLLECT WR_SEGNAM INTO LR_SEGNAM.
    ENDIF.
  ENDLOOP.

* Loop at IDOC to clear fields
  LOOP AT IDOC_DATA ASSIGNING <FS>.
    IF <FS>-SEGNAM IN LR_SEGNAM.
      LOOP AT LT_FILTER INTO WA_FILTER WHERE TABNAME = <FS>-SEGNAM.
        CLEAR <FS>-SDATA+WA_FILTER-OFFSET(WA_FILTER-LENG).
      ENDLOOP.
    ENDIF.
  ENDLOOP.
ENDMETHOD.

I also added the following code to IF_EX_HRALE00OUTBOUND_IDOC~IDOC_DATA_FOR_RECEIVER_MODIFY:

* Clear IDOC fields
CALL METHOD ME->CLEAR_IDOC_FIELDS
  EXPORTING
     IDOC_CONTROL = IDOC_CONTROL
     RECEIVER     = RECEIVER
     MESTYP       = IDOC_CONTROL-MESTYP
  CHANGING
     IDOC_DATA    = IDOC_DATA.

Finally, I maintained the following TVARV variable:

That’s all it takes!  Now individual field values can be easily be filtered from outbound HRMD_A IDOCs!.

UPDATE: My colleague informed me that instead of using TVARV, I should continue to use BD53 to manage the IDOC reduction and change my code to look up the field reductions in TBD24.  This would be a great idea, but my full HRMD_A solution uses custom filter objects and a custom IDOC formatting function module.  And BD53 does not play well with these customizations.

From time to time it becomes necessary to change the status of SAP IDOCs in SAP. The most common scenario is the requirement to mark SAP IDOCs for deletion. There is no good way to mass mark IDOCs for deletion via the standard IDOC processing transaction BD87. However there is a program that will let you change status.

RC1_IDOC_SET_STATUS

CAUTION: This program should be used with great care and consideration. Improper use of this program can result in data consistency issues. Make sure you know what you are deleting, why you are deleting it, and what is required to correctly update you system after deleting.

Example: Marking IDOCs for Deletion In Mass

It is pretty typical for support users to set the deletion flag on IDOCs that have been incorrectly created and have errored. When there are a small number of IDOCs this is possible via transaction BD87.

An inbound IDOC in error will have the status 51, when it is marked for deletion it has a status of 68.

A view of the IDOCs to be deleted in WE05.

To mass delete IDOCs run the following program via SE38: RC1_IDOC_SET_STATUS via the SAP Transaction: SE38

There are only a few parameters on the selection screen for this program. It is most important that you correctly restrict the IDOCs you select with this program. The program automatically defaults to marking inbound IDOCs in error for deletion.

To mass select IDOCs to be marked for deletion select:

There are many options for selecting and restricting the IDOCs to Mass process. Select by single value or range. Restrict by single value or range.

The allows you to upload a list of IDOCs from a text file.

The allows you to apply a list from your clipboard.

Execute the program

Check the status of the 3 IDOCs in WE05

Example: Changing IDOCs Status To Repost

It is also possible to use this program to reset an IDOC so that it can be reprocessed.

With the following selection we are going to reset the IDOCs with status 68 marked for deletion back to status 64 to try and reprocess them.

Execute the program

As you can see, program RC1_IDOC_SET_STATUS is very helpful, but please be careful when you use it!

So now that you have achieved control over your change pointer data, how do you make sure that it does not go out of control again?

1. Continuously purge PROCESSED and UNPROCESSED change pointer records based on established ageing parameters.

Once a change pointer record is processed, it has no further use – there is no reason to keep it for several weeks or longer.  Someone once tried to convince me that they needed to change pointer records as proof that an IDOC was created.  I suggested that they use SAP transaction WE05 instead.

Determine and agree upon a reasonable time period for which processed and unprocessed change pointer records should be permitted to persist in the change pointer tables.  Once these values are determined, you should set up an SAP job using ABAP program RBDCPCLR with an appropriate variant that will insure that change pointer records cannot persist beyond the specified length of time.

Unprocessed change pointer records that persist beyond the specified date indicate a problem.  These should be allowed to persist long enough for analysis purpose to determine and fix the root cause of why these records still remain.  This should be a matter of days and not weeks.

2. Establish a process to periodically review whether or not the current change pointer
configuration settings are supporting the business needs.

Make sure that you include your functional business owners in this process.

3.Consider the impact of a database restore.

While this is a rare occurrence, a major catastrophe may trigger the need to restore the SAP database from a backup that was created some time in the past.  For example, restoring the database from yesterday’s backup would resurrect some “processed” change pointers because yesterday, they were “unprocessed”.  Now consider that the external consumers of your IDOCs are not resetting their databases to yesterday, and have already received yesterday’s IDOC distributions.

4. Continuous monitoring.

Continue to monitor the size of tables BDCP and BDCPS.  Unless there is an unusually large amount of master data maintenance occurring at a given time, the size of these tables should remain relatively static.

Ideally, the monitoring process should be automated.  Possible automation alert triggers are:
a)      The change pointer tables increase in size by a pre-determined percentage or count above the
nominal value.
b)      Records exist in the change pointer tables that are aged beyond a specified number of days.

In this blog, I tried to cover the most common scenarios that I have encountered.  Because I see and learn new and wonderful things all the time, I realize that you may be facing a scenario that is entirely different from what was discussed here.  But I hope that this blog provides enough information to help you solve your particular issues.

Of course, if you have any questions or suggestions, I sincerely look forward to hearing from you.

Discussion points and procedures for the cleanup process.

1. Unless there is custom ABAP code depending on the continued existence of processed change pointers, they serve no further purpose and should be deleted.  ABAP Program RBDCPCLR should be scheduled on a regular basis so that recent “processed” change pointers are purged.

If there is a large quantity of very old “processed” change pointer records, then consider special executions of RBDCPCLR where the selected date range is old enough to collect and purge these records.

To cleanup processed change pointers, use the Processed Change Pointers section on the select option screen of program RBDCPCLR.  Make sure that the “Processed Change Pointers” check box is checked, and enter the proper date range.  You can further limit the data selection by entering a message type in the message type box.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.

2. “Unprocessed” change pointer records that are very old are most likely not useable.  If the IDOCs that they represent were triggered now, the data distributed might no longer be valid.  The important discussion point here is the definition of “very old”.  Is it one year, six months, two weeks?  This decision definitely needs input from the functional business owners.

Once the definition of “very old” has been established, schedule executions of RBDCPCLR with the “Obsolete Change Pointers” box checked, and the agreed-to “very old” date entered in the date box.  This execution will remove both “processed” and “unprocessed” change pointer records up to the specified date.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.

3. “Unprocessed” change pointers that are deemed recent must be analyzed to determine why these records are configured for creation if they are never actually processed.  Perhaps, they were needed at one time, and are no longer needed now.  What I need from the functional business owners here is an agreement to turn off, in configuration, the creation of these change pointers.  Not turning them off will only continue to add unnecessary records to the change pointer tables.

After the creation of these change pointers is turned off in configuration, then schedule an execution of RBDCPCLR using the obsolete section on the selection screen.  Enter the message type in the message type box and the current date.  This will purge all change pointers of the specified message type, regardless of their status, up to the specified date.  Special care must be taken here to make sure that the selection options are entered correctly.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.

4. Compare the list of message types that are configured to create change pointers (use SAP transaction BD50), to the list of message types in RBDMIDOC executions and the list of message types RBDCPCLR executions.  Make sure that all three lists agree.  Review these lists with the functional business owners.  Make sure that what is configured is meeting the needs of the current business processes.

I also ask the functional business owners to review the selected table-field values that, when changed or created new, will trigger the creation of change pointer records (use SAP transaction BD52).  The goal, again, is to make sure that what is configured is meeting the needs of the current business processes.

Now that we have cleaned up, how do we keep the size of the change pointer tables in check?
In my next post, I will discuss items that I recommend monitoring.