Before you embark on your project’s first data migration test cycle, your data migration plan will probably start out as a simple list of preparation steps, data load dependencies, and cleanup steps.  As you progress through your several data migration test cycles, your plan should evolve into a rather complex list containing load methods, responsible persons, historical data volumes, historical execution times, steps that need to be modified and additional steps that need to be added.

I always find it rather interesting to compare the very first plan to the plan that is actually used in the cutover process.  Many project factors will cause you to amend your data migration plan.  Newly-discovered gaps, changes in the business requirements, the decision to load all of the legacy customers instead of only those customers which have been active for the past two years, and toggling back and forth between “should this one be a manual or an automated load” are only a few.   Always keep in mind that the goal in amending the plan is to keep it compatible with the changing requirements of the project.

The Data Migration Preparation Phase

Before you actually start the data migration main course, you will probably want to execute a series of preparation steps.  Some of these preparation steps may have a significant lead time; while others may require a system reboot to take effect.  Make sure that you carefully plan accordingly.  Here are some data migration preparation steps that I have found useful:

• Obtain a Data Migration Login Obtain a special data migration login, with all of the authorizations needed to execute all of the required tasks.  This login is only to be used during the data migration process, and is to be deleted upon the completion of the data migration process. Using the data migration login makes identification of migrated data very easy both now and several years down the road.  It also facilitates implementing a general user lockout while the data migration user is performing data migration activities.
• Technical client preparation The BASIS team may want to configure the data migration target client to have less dialog processes and more background and update processes.  This configuration can significantly improve data migration performance, especially if you have the opportunity to parallel process IDOCs.  In a Virtual Machine environment, the infrastructure team can even provision additional processors and memory.  You may also need additional disk space to handle large upload files, and any intermediate files that a load program might need to create.
• Configuration for IDOC processing If a data migration process involves creating and processing IDOCs, you may need to configure the ports, partner profiles, message types, and processing modules.  My preference for IDOC triggering is by using the ABAP program RBDAPP01 rather than immediate triggering.  This allows me to exercise the most scheduling control over the background processing options.
• Change Pointers During a data migration turn off change pointers.  Creating what could potentially be hundreds of thousands of change pointer records that are never going to be processed is an overhead burden on system performance and disk space that you do not need.  If the migrated data needs to be sent to external systems, manually send the data after the data migration is complete.
• Set the Correct Open Posting Period Some of the transactional data will want to post into an open financial or manufacturing posting period.  You will want someone from the business to make sure that the correct posting periods are open for business.
• Configuration Settings Sometimes, special temporary functional configuration settings are made to facilitate the data migration process.  Several functional analysts may be responsible identifying and setting these prior to the start of data migration.  As these are temporary configuration settings, it is expected that they will be set to their production values after the data migration is complete.
• Basic Setup Data Some data migration objects are dependent on reference data – a template from which new data migration objects can obtain some of their default values.  You will want to make sure that those who are responsible for setting up this reference data have completed their task before you start the actual data migration.  Sometimes this data is moved from the Golden Client into the target client via ALE, so you may need to set up RFC destinations, ports, distribution models, etc.
• General user lockout All users except for the data migration user should be locked out of the data migration client during the actual data migration activity.  It is impossible to verify that we moved exactly $15,386,254.23 in inventory from the source system to the target system if users are performing material movements in either system during the conversion.
• Connections to External Systems Depending on what connections to external systems are doing, you may want to make sure that they are disconnected prior to the start of data migration.  This will ensure that inbound interfaces, which may either modify existing migrated data or add new data, are disabled.

The Data Migration Phase

When all of the preparation steps are complete, it is time to start the actual data migration.  Data must be loaded into the target system in a certain sequence that is dictated by their data dependencies.  Customers must exist before sales orders can be migrated, vendors must exist before purchase orders, materials must exist before either sales orders or purchase orders, etc.  This dependency exercise varies a bit from project to project, and should have already been determined well before you start loading the data.

Automated data migration tasks for a single object (e.g. customer master data) usually include some or all of the following:

1. Creation of an upload file from the source system
2. Technical validation of the upload file
3. Functional validation of the upload file
4. Execution of a technical object to load the data into the target system
5. Technical validation of the migrated data
6. Functional validation of the migrated data
7. Collection and reporting of load statistics
8. Analyzing the fallout
9. Correcting the fallout
10. Reprocessing  the fallout

Checkpoints

At a higher level, the data migration plan will string together an end to end list of all of data migration objects.  I strongly suggest that you study the list and judiciously intersperse several checkpoints at appropriate places in the plan.  A checkpoint is an activity where all interested parties meet to review status and metrics, and to determine whether or not to proceed forward.  Keep in mind that what fails to load early in the plan will geometrically cascade into larger fallout later in the plan.  That single vendor which failed to load may cause several hundred open purchase orders to also fail.  And while the business owner of the vendor master data may think the fallout of a single vendor to be quite insignificant, the business owner of open purchase orders may have an entirely different perspective.  It is important here to make sure that all of the right stakeholders are participating, and that all are in agreement about whether or not to proceed.

Backup or Restore Points

Another activity that you should judiciously sprinkle into the plan is several backup or restore points.  It is very painful to have successfully navigated seventy-five percent of the data migration plan, only to have the next load create a huge and unrecoverable mess in the target system.  Without a backup or restore point available, you may have to wipe the slate clean and restart the entire data migration process again.

Depending on the target system and the facilities available, the backup or restore point could be a full system backup, a client copy, or a virtual machine snapshot.  Whatever the means, make sure it is a part of your plan.  It’s really great if you don’t need it.  It’s also really great to have if you do need it!

The Data Migration Cleanup Phase

When the data migration loads are finally complete, it’s time to exercise a series of cleanup steps.  This is where we:

1. Undo the special temporary settings and configurations that were done prior to the start of the data migration activity.
2. Adjust the technical configuration of the system to a more user-oriented system with the right amount of dialog, background, and update processes.
3. Turn change pointers back on.
4. Allow the general user population to access the system.
5. Connect the system to its inbound and outbound communication partners.
6. Disable the data migration user.
7. Archive those massive upload files.
8. Delete any intermediate processing data files.
9. Clean up the IDOC tables if needed.  But first make sure that you do not need to keep the IDOCs  for legal reasons.

An Example Data Migration Plan

Click on the link below to access an example data migration plan.  This plan is not a complete data migration plan, but is intended to demonstrate format and content. Sample Data Migration Plan

Stay tuned!

In part 3 of this series on Data Migration, I will be discussing how to deal with fallout from automated data loads.

How do you know when an Outbound IDoc has physically left SAP?

It’s a simple fact that if your subsystem never receives an IDoc from SAP, the subsystem will not be able to route or process the IDoc.   I have always gone by the notion that once the subsystem has successfully received the IDoc, the subsystem now has all the responsibility to process and route the IDoc, but before that time, SAP has the responsibility to ensure that the IDoc is fully dispatched.  The subsystem should have processes in place to determine if the IDoc has been received at the target system successfully or not.  In this blog I will focus on the handshake between SAP and the subsystem.  This will entail Outbound IDoc status and a couple of batch jobs that should be setup to ensure a better chance of a successful handoff between SAP and the subsystem.

First, let’s cover the normal, expected progression of statuses for outbound IDocs.  The following graphic shows the collection of status records for a random outbound IDoc.

The status records at the bottom of the list depict the IDoc status before the status records at the top of the list.  So, in this example, the IDoc started in status “01 – IDoc generated”, then progressed to status “30 – IDoc ready for dispatch (ALE service).”  These two statuses usually are created in rapid succession for outbound IDocs and mean that the IDoc has been created in the SAP system and is ready to be sent via tRFC to the subsystem.  The next two IDoc status, 03 and 12, can be a little confusing.  They both seem to mean that the IDoc has been dispatched from SAP.  What is the difference?  I have found that sometimes there is some confusion on weather the IDoc has left SAP or not.  After I explain the difference, I will show 2 batch jobs that should be setup in order to update the status if the Outbound IDoc has physically left SAP.

So what do statuses ‘03’ and ‘12’ really mean?

The status ‘03’ actually means “Data passed to port OK” which basically means that the IDoc has been dispatched to the tRFC queue and an attempt was made to send the IDoc to the subsystem.  However, that does not necessarily mean that the IDoc has left SAP.   There could be many reasons why SAP would fail to send the IDoc.  One example could be that the subsystem was down during the transmission of the IDoc.  Errors during data transmission would cause the IDoc to remain in the tRFC queue.  Thus, the IDoc would still remain in SAP and remain with a status of ‘03’.

When the Status changes to ‘12’; a successful connection was made to the subsystem and the IDoc was sent.  This status represents that the Outbound IDoc is no longer in the tRFC queue  and, therefore, has physically left SAP.  The IDoc is in the hands of the subsystem which is now responsible for processing or routing the IDoc.

But what if I never see a status of ‘12’ on IDocs that were successfully sent?

For one reason or another, sometimes the batch job to update the status to ‘12’ is not scheduled.  In Production this batch job is usually set up, however in the QA or Dev environment this may or may not be setup depending on the client.

1)  The following job should be setup in order to process a status ‘12’.  I typically set this to every 10 minutes depending on the client.

Program:  RBDMOIND – Schedule this program in order to check whether the IDoc have been successfully sent out of the tRFC queue.  If the IDoc has been sent successfully to the subsystem, the program RBDMOIND will change the status of the IDoc from status ‘03’ to status ‘12’.  This means that the handshake with subsystem was successfully and the IDoc has physically left SAP.

2)  There may be times when the subsystem is unavailable at the time the SAP is trying to send the Outbound IDoc.  By default, SAP will only make one attempt to transmit the IDoc.  By scheduling the following program in a batch job, SAP will try to resend the Outbound IDoc again at a specified time interval.  I also usually set this at job every 10 min, depending on the client.

Program:  RSARFCEX – Schedule this program in a batch job in order to resend any IDocs that have are stuck on the tRFC queue.

Ok I have set up these jobs, but the status does not change.  Now what?

Sometimes an Outbound IDoc could remain stuck on the tRFC queue and never get sent out successfully.  Some issues could be that the RFC Destination for the subsystem is configured incorrectly or there could be an authorization issue.  These types of errors need to be fixed before the batch job for RSARFCEX will send the IDoc out successfully.  You may need to get some additional detail of the problem.  Execute the following transaction below.

Transaction SM58 – This will check what is currently on the tRFC queue.   Once you display the list, you also have the ability to drill down and get some additional detail of what has caused the failed transmission.

Conclusion

Typically you won’t see many issues with Outbound IDocs.  From time to time, IDocs will get stuck on the tRFC queue and if not managed properly, could be stuck there a long time.  This blog should give you more visibility in respect to IDoc status to ensure that the Outbound IDoc has successfully left SAP for processing.

The reduction at segment level (see also Note 301223) was facilitated in order to be able to copy message type HRMD_A. This involved setting the reduction at field level, which was not taken into account in the code because it is not possible to centrally determine which fields are mandatory and which are dependent for all country versions. In addition, for one info type there may several records for different periods of time, which may also cause a problem when you are reducing fields.

So, in essence, field-level reduction isn’t supported for HRMD_A.  The solution supplied in the OSS note suggests creating user-exit code to filter out the unwanted fields.  I already have implemented BADI HRALE00OUTBOUND_IDOC to add some other processing logic (if you’re lucky I’ll cover that logic in this blog, too), so I have a module in which I can place the code. And while the code to loop through an IDOC and clear out all instances of E1P0002-PERID, E1P0002-GBDAT, and E1P0002-GESCH is very easy, I can’t help but think I can make the code more useful.  You see, I hate unitasking code.  I can see it now.  If I take the shortcut path and wrote the unitasking code to clear those 3 fields, the code won’t be in production for more than a month before the requirements change and now there is another field that needs to be filtered out as well.  With unitasking code, I would have to crack open the code, create a transport, unit test, transport to Q, integration test, wait for a transport window, blah, blah, blah.  Ugh!  What a waste of time for a minor change.  We can do better.

I decided to create a multi-tasking module that will work for HRMD_A message types.  The fields to be cleared will be stored in TVARV in the format <SEGMENT>-<FIELD> (e.g. E1P0002-PERID).  That way, whenever the HR functional team wanted to filter out a new field, all they had to do was to update a TVARV variable.

To implement the solution, I created a new method called CLEAR_IDOC_FIELDS in my class implementation for BADI HRALE00OUTBOUND_IDOC.  CLEAR_IDOC_FIELDS has the same method signature as IF_EX_HRALE00OUTBOUND_IDOC~IDOC_DATA_FOR_RECEIVER_MODIFY:

The code is pretty straight forward.  To allow multiple interfaces to coexist in method CLEAR_IDOC_FIELDS, the TVARV variable that stores the fields to clear is in the format ‘ZHR_CLR_’ concatenated with the IDOC message type.

METHOD CLEAR_IDOC_FIELDS .
*----------------------------------------------------------------------*
* Class:     ZCL_IM_HRALE00OUTBOUNDIDOC                                *
* Method:    CLEAR_IDOC_FIELDS                                         *
* Author:    Craig Stasila                                             *
*----------------------------------------------------------------------*

  TYPES: BEGIN OF TY_FILTER,
           TABNAME   TYPE DFIES-TABNAME,
           FIELDNAME TYPE DFIES-FIELDNAME,
           POSITION  TYPE DFIES-POSITION,
           OFFSET    TYPE DFIES-OFFSET,
           LENG      TYPE DFIES-LENG,
        END OF TY_FILTER.

  DATA: WA_FILTER      TYPE TY_FILTER,
        LT_FILTER      TYPE TABLE OF TY_FILTER,
        LS_TABNAME     TYPE DDOBJNAME,
        LS_SAV_TABNAME TYPE DDOBJNAME.

  DATA: LS_NAME   TYPE TVARV-NAME,
        LT_TVARVC TYPE TABLE OF TVARVC,
        WA_TVARVC TYPE TVARVC.

  DATA: LT_DFIES TYPE TABLE OF DFIES,
        WA_DFIES TYPE DFIES.

  DATA: LR_SEGNAM TYPE RANGE OF EDIDD-SEGNAM,
        WR_SEGNAM LIKE LINE OF LR_SEGNAM.

  FIELD-SYMBOLS: <FS> TYPE LINE OF EDIDD_TT.

* Fields to clear are in variable
  CONCATENATE 'ZHR_CLR_' MESTYP INTO LS_NAME.

* Get list of fields to clear from TVARV
  SELECT * FROM TVARVC
    INTO TABLE LT_TVARVC
   WHERE NAME = LS_NAME
     AND TYPE = 'S'.

  CHECK SY-SUBRC = 0.

* Build list of fields to clear
  LOOP AT LT_TVARVC INTO WA_TVARVC.
    CLEAR WA_FILTER.
    SPLIT WA_TVARVC-LOW AT '-' INTO WA_FILTER-TABNAME WA_FILTER-FIELDNAME.
    CHECK SY-SUBRC = 0.

    COLLECT WA_FILTER INTO LT_FILTER.
  ENDLOOP.

  SORT LT_FILTER BY TABNAME.
  CLASS CL_ABAP_CHAR_UTILITIES DEFINITION LOAD.

* Get field offsets
  LOOP AT LT_FILTER INTO WA_FILTER.
    LS_TABNAME = WA_FILTER-TABNAME.

    IF LS_TABNAME NE LS_SAV_TABNAME.

      CALL FUNCTION 'DDIF_NAMETAB_GET'
        EXPORTING
          TABNAME   = LS_TABNAME
        TABLES
          DFIES_TAB = LT_DFIES
        EXCEPTIONS
          NOT_FOUND = 1
          OTHERS    = 2.

      IF SY-SUBRC <> 0.
        CONTINUE.
      ENDIF.

      SORT LT_DFIES BY TABNAME FIELDNAME.

    ENDIF.

    LS_SAV_TABNAME = LS_TABNAME.

    READ TABLE LT_DFIES INTO WA_DFIES
         WITH KEY TABNAME   = WA_FILTER-TABNAME
                  FIELDNAME = WA_FILTER-FIELDNAME
         BINARY SEARCH.

    CHECK SY-SUBRC = 0.

    WA_FILTER-POSITION = WA_DFIES-POSITION.
    WA_FILTER-OFFSET   = WA_DFIES-OFFSET / CL_ABAP_CHAR_UTILITIES=>CHARSIZE.
    WA_FILTER-LENG     = WA_DFIES-LENG.

    MODIFY LT_FILTER FROM WA_FILTER.

* Build segment name range
    IF SY-SUBRC = 0.
      WR_SEGNAM-SIGN   = 'I'.
      WR_SEGNAM-OPTION = 'EQ'.
      WR_SEGNAM-LOW    = WA_FILTER-TABNAME.
      COLLECT WR_SEGNAM INTO LR_SEGNAM.
    ENDIF.
  ENDLOOP.

* Loop at IDOC to clear fields
  LOOP AT IDOC_DATA ASSIGNING <FS>.
    IF <FS>-SEGNAM IN LR_SEGNAM.
      LOOP AT LT_FILTER INTO WA_FILTER WHERE TABNAME = <FS>-SEGNAM.
        CLEAR <FS>-SDATA+WA_FILTER-OFFSET(WA_FILTER-LENG).
      ENDLOOP.
    ENDIF.
  ENDLOOP.
ENDMETHOD.

I also added the following code to IF_EX_HRALE00OUTBOUND_IDOC~IDOC_DATA_FOR_RECEIVER_MODIFY:

* Clear IDOC fields
CALL METHOD ME->CLEAR_IDOC_FIELDS
  EXPORTING
     IDOC_CONTROL = IDOC_CONTROL
     RECEIVER     = RECEIVER
     MESTYP       = IDOC_CONTROL-MESTYP
  CHANGING
     IDOC_DATA    = IDOC_DATA.

Finally, I maintained the following TVARV variable:

That’s all it takes!  Now individual field values can be easily be filtered from outbound HRMD_A IDOCs!.

UPDATE: My colleague informed me that instead of using TVARV, I should continue to use BD53 to manage the IDOC reduction and change my code to look up the field reductions in TBD24.  This would be a great idea, but my full HRMD_A solution uses custom filter objects and a custom IDOC formatting function module.  And BD53 does not play well with these customizations.

Cross Reference To SAP Sales Area From External Customer Number – SAP EDSDC Table

If you have ever converted an external file into a SAP Sales Document IDOC (ex: ORDERS05) you may have hard coded the E1EDK14 segments of the IDOC which specify Sales Area and order type for the SAP Sales Document.  However, these segments are not required if you populate the E1EDKA1-LIFNR segment of the IDOC with the External Customer Number and populate the EDSDC table with the Sales Area and Order Type for the customer.  Your SAP system contains a standard utility that converts the External Customer Number to the Sales Area values and Order Type value contained in this table.

Sales Area Cross Reference From IDOCs ORDERS05 and DELFOR02

This utility is available in the standard SAP package to convert External Customer Number to SAP Sales Area when processing inbound IDOCs (ex: ORDERS05, DELFOR02) from non-SAP systems.  It eliminates the need to add E1EDK14 segments to the IDOC as it populates the Sales Organization, Distribution Channel, Division, and Sales Document Type using the entries in the EDSDC table which is populated using transaction VOE2.  In order for this process to work, you must have the Customers External number populated in the E1EDKA1-LIFNR field of the IDOC where E1EDKA1-PARVW is ‘AG’ and the EDPAR Customer Conversion table set up to convert the Customer External number to the SAP Customer number.  See my blog ‘SAP EDPAR Table – External Customer Number cross reference to SAP Customer Number’ for instructions on the External Customer Conversion.  By properly doing this, the standard SAP function module will use the EDSDC table to convert the External Customer Number to the SAP Sales Area and Document Type.  The following instructions will explain how to populate the inbound IDOC and the EDSDC table.

On the Inbound IDOC do not create any E1EDK14 segments.  The use of the EDSDC table will populate the Sales Area and Document Type of the Sales document and therefore no E1EDK14 segments are required.  Populate the E1EDKA1 segment for the Sold-To party where the PARVW field is ‘AG’ and the LIFNR field is the External Customer Number (Note: the EDPAR table must be populated to convert this External Customer Number to the SAP Sold-To number).

Use transaction VOE2 to populate the EDSDC table.  Enter the SAP Sold-To Customer number in the Customer field and your Customer’s External Sold-To number in the Vendor number field.  You will then populate the SOrg, DChl, and Dv fields on VOE2 with the Sales Area information(Sales Organization, Distribution Channel, and Division) related to your SAP Customer number.  Finally, you will enter the Sales Document Type(ex: OR = Standard Order) in the SaTy field.  These values will be used as the Sales Area and Document Type on the SAP Sales Document once the IDOC is processed.

So eliminate hard coding multiple E1EDK14 segments in the inbound IDOC, just use the Standard SAP Utility designed to populated the Sales Area and Order Type of your Sales Documents.  Populate the EDSDC table using the VOE2 transaction and send the External Customer Number in the E1EDKA1-LIFNR field of the IDOC.

Have you ever had a project where you are converting a customer file into an SAP Sales Document IDOC (ex: ORDERS05) and the customer file contains their Internal Customer Number?  So you need to determine what the SAP Customer Number will be on the IDOC.  If you are not familiar with the SAP interface process you may hardcode the SAP Customer Number or create external tables in your third party mapping software to accomplish this conversion process.  However, this involves customer specific logic or updating the tables with your SAP Customer Numbers and having a process to maintain this table as new SAP Customer Numbers are created.  There is a much easier way to perform this conversion with no additional coding as it is a standard SAP utility.

Customer Conversion For IDOCS ORDERS05 and DELFOR02

This utility is available in the standard SAP package to convert External Customer Numbers to SAP Customer numbers (Sold-To, Ship-To, etc.) when processing inbound IDOCs (ex: ORDERS05, DELFOR02) from non-SAP systems.  When working with data that will be imported into SAP via IDOC format the data usually  contain your customers external sold-to and ship-to numbers and not the associated SAP customer numbers.    Therefore, you would need to convert this External Customer Number to  your internal SAP Customer Number.  Instead of creating cross-reference tables or conversion routines in your third-party translation software you can populate the E1EDKA1-LIFNR field of the inbound IDOC with the External Customer Number and use transaction VOE4 to populate the EDPAR table.  By properly doing this, the standard SAP function module will use the EDPAR table values to convert the External Customer Number to the SAP Customer number.  The following instructions will explain how to populate the inbound IDOC and the EDPAR table.

Configuration Of IDOC Segment E1EDKA1

On the inbound IDOC, the only fields you will need to populate on the E1EDKA1 segment is the PARVW(Partner Function) field and the LIFNR(Vendor Number at Customer Location) field.  In most cases you will have an E1EDKA1 segment for the Sold-To party and another E1EDKA1 segment for the Ship-To party.  For the Sold-To party you would enter ‘AG’ in the PARVW field and the External Sold-To Number in the LIFNR field.

For the Ship-To party you would enter ‘WE’ in the PARVW field and the External Ship-To Number in the LIFNR field.

Populating SAP EDPAR Table Via Transaction VOE4

Prior to processing an inbound IDOC you would use transaction VOE4 to populate the EDPAR table.  The SAP Sold-To number will always be entered in the Customer number field.  The Ext Function field will be populated depending on the External number.  If you are converting the Sold-To number you would enter ‘SP’ in the Ext Function field.  If you are converting the Ship-To number you would enter ‘SH’ in the Ext Function field.  The External Sold-To/Ship-To number would be entered in the External partner field.    Finally, you would enter the SAP Sold-To number or Ship-To number in the Int no. field.  This value will be used as the Sold-To/Ship-To on the SAP Sales Document once the IDOC is processed.

So if you ever need to convert External Customer Number to SAP Customer Numbers take advantage of the utilities that SAP supplies.  You can even incorporate the EDPAR table entry updates  into the SD Business Configuration processes so when a new customer number is created and it is designated as an EDI Customer, the personnel creating the new customer can also update the EDPAR table entries.

From time to time it becomes necessary to change the status of SAP IDOCs in SAP. The most common scenario is the requirement to mark SAP IDOCs for deletion. There is no good way to mass mark IDOCs for deletion via the standard IDOC processing transaction BD87. However there is a program that will let you change status.

RC1_IDOC_SET_STATUS

CAUTION: This program should be used with great care and consideration. Improper use of this program can result in data consistency issues. Make sure you know what you are deleting, why you are deleting it, and what is required to correctly update you system after deleting.

Example: Marking IDOCs for Deletion In Mass

It is pretty typical for support users to set the deletion flag on IDOCs that have been incorrectly created and have errored. When there are a small number of IDOCs this is possible via transaction BD87.

An inbound IDOC in error will have the status 51, when it is marked for deletion it has a status of 68.

A view of the IDOCs to be deleted in WE05.

To mass delete IDOCs run the following program via SE38: RC1_IDOC_SET_STATUS via the SAP Transaction: SE38

There are only a few parameters on the selection screen for this program. It is most important that you correctly restrict the IDOCs you select with this program. The program automatically defaults to marking inbound IDOCs in error for deletion.

To mass select IDOCs to be marked for deletion select:

There are many options for selecting and restricting the IDOCs to Mass process. Select by single value or range. Restrict by single value or range.

The allows you to upload a list of IDOCs from a text file.

The allows you to apply a list from your clipboard.

Execute the program

Check the status of the 3 IDOCs in WE05

Example: Changing IDOCs Status To Repost

It is also possible to use this program to reset an IDOC so that it can be reprocessed.

With the following selection we are going to reset the IDOCs with status 68 marked for deletion back to status 64 to try and reprocess them.

Execute the program

As you can see, program RC1_IDOC_SET_STATUS is very helpful, but please be careful when you use it!

So now that you have achieved control over your change pointer data, how do you make sure that it does not go out of control again?

1. Continuously purge PROCESSED and UNPROCESSED change pointer records based on established ageing parameters.

Once a change pointer record is processed, it has no further use – there is no reason to keep it for several weeks or longer.  Someone once tried to convince me that they needed to change pointer records as proof that an IDOC was created.  I suggested that they use SAP transaction WE05 instead.

Determine and agree upon a reasonable time period for which processed and unprocessed change pointer records should be permitted to persist in the change pointer tables.  Once these values are determined, you should set up an SAP job using ABAP program RBDCPCLR with an appropriate variant that will insure that change pointer records cannot persist beyond the specified length of time.

Unprocessed change pointer records that persist beyond the specified date indicate a problem.  These should be allowed to persist long enough for analysis purpose to determine and fix the root cause of why these records still remain.  This should be a matter of days and not weeks.

2. Establish a process to periodically review whether or not the current change pointer configuration settings are supporting the business needs.

Make sure that you include your functional business owners in this process.

3.Consider the impact of a database restore.

While this is a rare occurrence, a major catastrophe may trigger the need to restore the SAP database from a backup that was created some time in the past.  For example, restoring the database from yesterday’s backup would resurrect some “processed” change pointers because yesterday, they were “unprocessed”.  Now consider that the external consumers of your IDOCs are not resetting their databases to yesterday, and have already received yesterday’s IDOC distributions.

4. Continuous monitoring.

Continue to monitor the size of tables BDCP and BDCPS.  Unless there is an unusually large amount of master data maintenance occurring at a given time, the size of these tables should remain relatively static.

Ideally, the monitoring process should be automated.  Possible automation alert triggers are: a)      The change pointer tables increase in size by a pre-determined percentage or count above the nominal value. b)      Records exist in the change pointer tables that are aged beyond a specified number of days.

In this blog, I tried to cover the most common scenarios that I have encountered.  Because I see and learn new and wonderful things all the time, I realize that you may be facing a scenario that is entirely different from what was discussed here.  But I hope that this blog provides enough information to help you solve your particular issues.

Of course, if you have any questions or suggestions, I sincerely look forward to hearing from you.

Discussion points and procedures for the cleanup process.

1. Unless there is custom ABAP code depending on the continued existence of processed change pointers, they serve no further purpose and should be deleted.  ABAP Program RBDCPCLR should be scheduled on a regular basis so that recent “processed” change pointers are purged.

If there is a large quantity of very old “processed” change pointer records, then consider special executions of RBDCPCLR where the selected date range is old enough to collect and purge these records.

To cleanup processed change pointers, use the Processed Change Pointers section on the select option screen of program RBDCPCLR.  Make sure that the “Processed Change Pointers” check box is checked, and enter the proper date range.  You can further limit the data selection by entering a message type in the message type box.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.

2. “Unprocessed” change pointer records that are very old are most likely not useable.  If the IDOCs that they represent were triggered now, the data distributed might no longer be valid.  The important discussion point here is the definition of “very old”.  Is it one year, six months, two weeks?  This decision definitely needs input from the functional business owners.

Once the definition of “very old” has been established, schedule executions of RBDCPCLR with the “Obsolete Change Pointers” box checked, and the agreed-to “very old” date entered in the date box.  This execution will remove both “processed” and “unprocessed” change pointer records up to the specified date.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.

3. “Unprocessed” change pointers that are deemed recent must be analyzed to determine why these records are configured for creation if they are never actually processed.  Perhaps, they were needed at one time, and are no longer needed now.  What I need from the functional business owners here is an agreement to turn off, in configuration, the creation of these change pointers.  Not turning them off will only continue to add unnecessary records to the change pointer tables.

After the creation of these change pointers is turned off in configuration, then schedule an execution of RBDCPCLR using the obsolete section on the selection screen.  Enter the message type in the message type box and the current date.  This will purge all change pointers of the specified message type, regardless of their status, up to the specified date.  Special care must be taken here to make sure that the selection options are entered correctly.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.

4. Compare the list of message types that are configured to create change pointers (use SAP transaction BD50), to the list of message types in RBDMIDOC executions and the list of message types RBDCPCLR executions.  Make sure that all three lists agree.  Review these lists with the functional business owners.  Make sure that what is configured is meeting the needs of the current business processes.

I also ask the functional business owners to review the selected table-field values that, when changed or created new, will trigger the creation of change pointer records (use SAP transaction BD52).  The goal, again, is to make sure that what is configured is meeting the needs of the current business processes.

Now that we have cleaned up, how do we keep the size of the change pointer tables in check? In my next post, I will discuss items that I recommend monitoring.

Now, it is time to analyze the change pointer data directly.  Here are some targets that I pursue:

1.  Count. How many change pointer records are in your tables – thousands, hundreds of thousands, millions, billions? This simple statistic alone reveals an interesting story.

While I typically try to use transaction SE11 to count the number of records in table BDCP, I have found, on occasion, that this method would time out due to an excessively large number of records in this table. In this case, you can write a simple ABAP which can be scheduled to run in background.

2.  Age and status. Using the view BDCPV, determine the age and the status of your change pointer records.  The age of a change pointer record is indicated by the field BDCPV-CRETIME, and the status of a change pointer record is indicated by the field BDCPV-PROCESS.

In general, are the change pointer records days, weeks, months, or years old? Are different ancient records found to be in both the ‘processed’ and ‘unprocessed’ status? Are different recent records found to be in both the ‘processed’ and ‘unprocessed’ status?

3.  Message type and status. Using the BDCPV view, determine the message type and status of your change pointer records. The message type of a change pointer record is indicated by the field BDCPV-MESTYPE.

Are there different change pointer records for some message types that are never found in the ‘processed’ state? Are there different change pointer records for a particular message type that are in both the ‘processed’ and the ‘unprocessed’ status?

Custom Reporting For a more sohpisticated data analysis, you might consider writing an ABAP program to categorize your change pointer records into ageing buckets, message types, and by status.  The cross-tab report shown here is one that I use to analyze change pointer records.   It contains three sections - Unprocessed Change Pointers by Message Type Within Ageing Category, Processed Change Pointers by Message Type Within Ageing Category, and a Grand Totals Summary.

Example Scenario Leading to Runaway Change Pointer Records Of course, there are many reasons why you might observe some of the scenarios described above.  Here is one example:

Many years ago, your company’s business processes required the daily distribution of material master data additions and changes to a remote plant.  Your functional analysts correctly configured the change pointers and the ALE process for message type MATMAS to send this data to your remote plant. RBDMIDOC and RBDCPCLR were both running daily for the material master data message type MATMAS.   Change pointer records for material master data were created whenever users added or changed certain data fields, IDOCs were created and sent, the change pointer records were marked as ‘processed’, and were then purged – all on a daily basis.

Last year, business process changed, and your remote plant no longer required the distribution of material master data changes.  The batch jobs running RBDMIDOC and RBDCPCLR for the material master data message type MATMAS were cancelled, so that IDOCs would no longer be sent to the plant, and because there was no need to clean up after these IDOCs for this message type.

But…

No one remembered to turn off the creation of change pointers in configuration for the material master data message type MATMAS.  So, every time a user added or changed a material, new change pointer records were still being created, causing the change pointer tables to needlessly increase in size. And the status of these change pointer records will forever remain ‘unprocessed’.

Do you have custom code that is creating change pointer records? There is another source of runaway change pointer table size that I would like to discuss here.    In some SAP systems that I have worked on, I have found that change pointer records were being created via custom ABAP programs.  This was done to support a custom process, having nothing to do with the SAP intended use of the change pointer records – the support of ALE distribution of master data.  Unfortunately, the system designers also forgot to write a custom program to either clean up their “custom-purposed” change pointer records, or to allow these records to be recognized by the SAP cleanup program.

Next Steps So, now that we have discovered unnecessary change pointer records cluttering up our SAP database, what do we do next? What is the recovery cleanup process? How do we stop the creation of change pointer records that are no longer needed?

This will be discussed in my next post.

In their SAP environment they produce a single DESADV IDoc per order/purchase order.  This could amount to several hundred DESADV IDocs being generated.  The initial development was a 1 to 1 translation in that each DESADV IDoc that was produced would be translated into a single EDI-856 document.

Because of a single data error, due to department procedures not being followed in the warehouse in the closing of orders, the bad data element resulted and was repeated throughout all of the DESADV IDocs that were generated.  Translation of the IDocs was successful because this error was not conceived at the point of map development.  The hundreds of translated documents were sent to the trading partner and subsequently resulted in failed documents.

Because the trading partner generated a charge back for each document sent, my client was billed thousands of dollars in charge back fines.  This simple example shows how a single error repeated in each document can turn out to be a very costly problem.

I hope to show you in the next few postings how to bundle the IDocs together to:

• Bundle and reduce the number of IDocs being translated
• Reduce the number of EDI documents being sent to the trading partner
• Minimize the charge back cost, should and error occur