SAP Data Migration – The Data Migration Plan (Part 2)

If you are responsible for the success of data migration, you will want to build a detailed plan that will walk you through all of the three phases of data migration: pre-data migration preparation, the data migration itself, and post-data migration cleanup.  I like my data migration plan to contain detailed steps that ensure that I don’t forget anything.  Each step lists a specific named responsible person along with their area of responsibility and contact information.  Unless I am responsible for executing the task myself, I prefer the named person to be a client employee (i.e. the business owner of the process) rather than a project consultant.    This is where the responsibility should be, and it requires that the business process owners actually participate in the activity rather than sit on the sidelines and watch.

[Read more...]

Are You Managing Your Change Pointers Properly Part 5 – Going Forward

In my last post, I discussed a collaborative effort with functional business owners to devise and execute a proper cleanup plan.  I also discussed a functional review of the configuration to make sure that change pointers are being created only when needed.

So now that you have achieved control over your change pointer data, how do you make sure that it does not go out of control again?

[Read more...]

Are You Properly Managing Your SAP Change Pointers Part 4 – The Cleanup



In my last post on this topic, I discussed several change pointer data forensic methods that I use to point me in the right cleanup direction.
The change pointer data forensics will determine how to design the cleanup process.  Armed with meaningful statistics, I usually approach the functional business owners to collaborate on a solution.  Typical of the many questions that I ask are “Why are change pointers configured for this message type if we are never distributing the IDOCs?”

Discussion points and procedures for the cleanup process.

1. Unless there is custom ABAP code depending on the continued existence of processed change pointers, they serve no further purpose and should be deleted.  ABAP Program RBDCPCLR should be scheduled on a regular basis so that recent “processed” change pointers are purged.

If there is a large quantity of very old “processed” change pointer records, then consider special executions of RBDCPCLR where the selected date range is old enough to collect and purge these records.

To cleanup processed change pointers, use the Processed Change Pointers section on the select option screen of program RBDCPCLR.  Make sure that the “Processed Change Pointers” check box is checked, and enter the proper date range.  You can further limit the data selection by entering a message type in the message type box.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.
delete change pointers 1

2. “Unprocessed” change pointer records that are very old are most likely not useable.  If the IDOCs that they represent were triggered now, the data distributed might no longer be valid.  The important discussion point here is the definition of “very old”.  Is it one year, six months, two weeks?  This decision definitely needs input from the functional business owners.

Once the definition of “very old” has been established, schedule executions of RBDCPCLR with the “Obsolete Change Pointers” box checked, and the agreed-to “very old” date entered in the date box.  This execution will remove both “processed” and “unprocessed” change pointer records up to the specified date.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.
delete change pointers 2

3. “Unprocessed” change pointers that are deemed recent must be analyzed to determine why these records are configured for creation if they are never actually processed.  Perhaps, they were needed at one time, and are no longer needed now.  What I need from the functional business owners here is an agreement to turn off, in configuration, the creation of these change pointers.  Not turning them off will only continue to add unnecessary records to the change pointer tables.

After the creation of these change pointers is turned off in configuration, then schedule an execution of RBDCPCLR using the obsolete section on the selection screen.  Enter the message type in the message type box and the current date.  This will purge all change pointers of the specified message type, regardless of their status, up to the specified date.  Special care must be taken here to make sure that the selection options are entered correctly.

Note that on the selection screen there is also a “Test Run” check box that will list which records would be selected for purging, without actually purging them.  I find this feature very useful, and I strongly recommend using it prior to actual purging.
delete change pointers 3

4. Compare the list of message types that are configured to create change pointers (use SAP transaction BD50), to the list of message types in RBDMIDOC executions and the list of message types RBDCPCLR executions.  Make sure that all three lists agree.  Review these lists with the functional business owners.  Make sure that what is configured is meeting the needs of the current business processes.

I also ask the functional business owners to review the selected table-field values that, when changed or created new, will trigger the creation of change pointer records (use SAP transaction BD52).  The goal, again, is to make sure that what is configured is meeting the needs of the current business processes.

Now that we have cleaned up, how do we keep the size of the change pointer tables in check?
In my next post, I will discuss items that I recommend monitoring.

Are You Properly Managing Your SAP Change Pointers Part 3 – Using Change Pointer Data Forensics



Your batch job analysis shows that you are currently running the change pointer processing program RBDMIDOC,
and the change pointer cleanup program RBDCPCLR; but you notice that the size of your change pointer tables
continues to increase.

Now, it is time to analyze the change pointer data directly.  Here are some targets that I pursue:

1.  Count.
How many change pointer records are in your tables – thousands, hundreds of thousands, millions, billions?
This simple statistic alone reveals an interesting story.

While I typically try to use transaction SE11 to count the number of records in table BDCP, I have found,
on occasion, that this method would time out due to an excessively large number of records in this table.
In this case, you can write a simple ABAP which can be scheduled to run in background.

2.  Age and status.
Using the view BDCPV, determine the age and the status of your change pointer records.  The age of a change
pointer record is indicated by the field BDCPV-CRETIME, and the status of a change pointer record is indicated
by the field BDCPV-PROCESS.

In general, are the change pointer records days, weeks, months, or years old?
Are different ancient records found to be in both the ‘processed’ and ‘unprocessed’ status?
Are different recent records found to be in both the ‘processed’ and ‘unprocessed’ status?

3.  Message type and status.
Using the BDCPV view, determine the message type and status of your change pointer records.
The message type of a change pointer record is indicated by the field BDCPV-MESTYPE.

Are there different change pointer records for some message types that are never found in the ‘processed’ state?
Are there different change pointer records for a particular message type that are in both the ‘processed’ and the ‘unprocessed’ status?

Custom Reporting
For a more sohpisticated data analysis, you might consider writing an ABAP program to categorize your
change pointer records into ageing buckets, message types, and by status.  The cross-tab report shown here
is one that I use to analyze change pointer records.   It contains three sections -
Unprocessed Change Pointers by Message Type Within Ageing Category,
Processed Change Pointers by Message Type Within Ageing Category,
and a Grand Totals Summary.

crosstab report

Example Scenario Leading to Runaway Change Pointer Records
Of course, there are many reasons why you might observe some of the scenarios described above.  Here is one example:

Many years ago, your company’s business processes required the daily distribution of material master
data additions and changes to a remote plant.  Your functional analysts correctly configured the
change pointers and the ALE process for message type MATMAS to send this data to your remote plant.
RBDMIDOC and RBDCPCLR were both running daily for the material master data message type
MATMAS.   Change pointer records for material master data were created whenever users added
or changed certain data fields, IDOCs were created and sent, the change pointer records were marked
as ‘processed’, and were then purged – all on a daily basis.

Last year, business process changed, and your remote plant no longer required the distribution of
material master data changes.  The batch jobs running RBDMIDOC and RBDCPCLR for the material
master data message type MATMAS were cancelled, so that IDOCs would no longer be sent to the plant,
and because there was no need to clean up after these IDOCs for this message type.

But…

No one remembered to turn off the creation of change pointers in configuration for the material master
data message type MATMAS.  So, every time a user added or changed a material, new change pointer
records were still being created, causing the change pointer tables to needlessly increase in size.
And the status of these change pointer records will forever remain ‘unprocessed’.

 

Do you have custom code that is creating change pointer records?
There is another source of runaway change pointer table size that I would like to discuss here.    In some SAP systems that I have worked on, I have found that change pointer records were being created via custom ABAP programs.  This was done to support a custom process, having nothing to do with the SAP intended use of the change pointer records – the support of ALE distribution of master data.  Unfortunately, the system designers also forgot to write a custom program to either clean up their “custom-purposed” change pointer records, or to allow these records to be recognized by the SAP cleanup program.

Next Steps
So, now that we have discovered unnecessary change pointer records cluttering up our SAP database,
what do we do next?
What is the recovery cleanup process?
How do we stop the creation of change pointer records that are no longer needed?

This will be discussed in my next post.

Are You Properly Managing Your SAP Change Pointers Part 2 – A Simple Source of Runaway Growth

Last time, in my 50,000 foot view post, I presented an overview of the purpose, the location, and the
expected lifecycle of change pointer records.    I also briefly discussed two programs:

RBDMIDOC -  triggers IDOC distribution for ‘unprocessed’ change pointer records, and marks
those change pointer records as ‘processed’.
RBDCPCLR -   deletes the ‘processed’ change pointer records from the change pointer tables.

One obvious source of runaway change pointer table growth is never running the cleanup program RBDCPCLR.
Using transaction SM37 (batch jobs), you can determine if RBDCPCLR is running in a batch job, and how frequently it is running.
As shown in the screen shot below, enter RBDCPCLR in the ABAP Program Name box, enter a relevant date range,
and check the various job status boxes that you might be looking for.  I recommend setting a significantly wide date
range, as I have found situations where RBDCPCLR was running in a scheduled batch job a while ago, but for some
unknown reason is no longer currently running.

Pressing the EXECUTE button will produce a list of batch jobs containing RBDCPCLR.

SM37 Simple Job Selecion Screen

Analyze the scheduled batch job list.

If the list is empty, then RBDCPCLR is not running in a regularly scheduled batch job.  Unless RBDCPCLR
is executing via some other means, ‘processed’ change pointer records are not being purged.
This is one simple source of runaway change pointer table growth.

I have encountered many reasons why RBDCPCLR is not currently running when change pointers are turned on.
In some instances, my clients did not know that active management of the change pointer tables was necessary.
In others, they knew about managing the change pointer tables, had at one time, run periodic batch jobs executing
the program RBDCPCLR, but for some reason, were not currently running these jobs.

While you are here in transaction SM37, you might also want to check out the frequency of your RBDMIDOC
executions.  If RBDMIDOC is not running, it is possible that new change pointers are created as master data
is changed, never triggering the creation of their outbound IDOCs, and never marking change pointer records
as “processed” so that RBDCPCLR can clean them up.

If the scheduled job list or other analysis shows that RBDMIDOC and RBDCPCLR are currently running periodically,
and your change pointer tables are still increasing in size, then you need to analyze the change pointer data itself
to determine the cause.

I like data forensics.  It can reveal some rather interesting facts if you know how to analyze your data.
In my next post, I will discuss several change pointer data forensic methods that I use to reveal other
sources of runaway change pointer table growth.

Are You Properly Managing Your SAP Change Pointer Tables Part 1 – The 50,000 Foot View

As a consultant, I have the opportunity to work on many different SAP installations.  I continue to be surprised by the discovery that many of these companies are not properly managing their change pointer tables, allowing these tables to continuously increase in size over long periods of time.  Using data forensics, I have seen change pointer record counts as large as two billion (and still growing), and change pointer record creation dates that have aged many years with status of both processed and unprocessed.
What are change pointers, where are they located, and what is their purpose in an SAP system?
When turned on in configuration, change pointer records are written to SAP change pointer data tables when master data is maintained, as part of the shared master data facility.  Their primary purpose is to facilitate the distribution of master data additions and changes from the SAP host system to remote systems, via ALE using IDOCs.  Configuration customizing is used to determine whether or not change pointer record creation is turned on or off, and if turned on, which message types and data table-field changes will trigger the creation of change pointer records.
Change pointer records reside in data tables BDCP and BDCPS.  There is also a view BDCPV which combines fields from these two tables.  Table BDCP is the primary change pointer data table, and table BDCPS contains the status (processed or unprocessed) of the change pointer record.
A new change pointer record is created in the ‘unprocessed’ state, and creates a “demand” in the SAP system to create an outbound IDOC.  This “demand” is recognized by the SAP program RBDMIDOC (typically scheduled in a periodic batch job) which triggers the creation of the outbound IDOC, and then alters the state of the change pointer record to ‘processed’.  But while the status of change pointer record is marked as ‘processed’, the record still resides in the BDCP and BDCPS tables.
So, how are these consumed change pointer records deleted from the change pointer tables?  This is accomplished with the SAP Program RBDCPCLR (typically scheduled in a periodic batch job).  Depending on your business processes, the life cycle of a change pointer record is typically not that long.  If you are distributing your master data change IDOCs to remote systems daily, then the related change pointer records can be purged daily.  If your distribution cycle is weekly, then your cleanup cycle should also occur weekly.
If everything is working properly, the creation of change pointers is turned on, the creation of change pointer records is conditioned by message type and data table-fields, RBDMIDOC is running in a scheduled periodic batch job to distribute the master data changes via IDOCs and mark the change pointer records as ‘processed’, and RBDCPCLR is running in a scheduled periodic batch job to clean up the ‘processed’ change pointer records.
What, then, would cause runaway growth in the change pointer tables?
In my next two posts, I will review several methods I use to identify sources of runaway growth in the change pointer record tables.