Twas the Friday before Christmas, and all through the house
Not a creature was stirring, not even my spouse;
(Truth be told she was out earlier than most,
With finals to give and semester grades to post);
I was back home from consulting on the road,
with dread of expenses and timesheets to upload;
Doing the last-minute search for the right gift,
That might delight the special person on my list;
When from my work email there arose such a clatter,
I turned away from Twitter to see what was the matter.
The most urgent email was filled with despair,
“The upgrade, I fear, is damaged beyond repair”.
With many apologies to Clement Moore and poetry fans everywhere, I could not resist having a bit of fun with this latest adventure in SAP upgrades. I could have continued with the poetry, however, I could not find anything to rhyme with MAIN_PROFEND/KX_CHANGE_PUPG!
BACKGROUND: The upgrade was from NW 7.01, SP16 BW to NW 7.31, SP 04 BI — ABAP standalone. The upgrade of the sandbox system had been relatively uneventful, so I was definitely not expecting an issue with the BI development box. The MAIN_PROFEND phase is near the end of the Execution stage for the Software Update Manager — very close to the completion of the entire upgrade.
The screenshot provides one clue that provides some hope for a solution: “see F:\******\SUM\abap\log\SAPCPE.OUT for details”. As most Basis admins know, SAPCPE is the process that runs during the SAP startup process and is responsible for copying the executable files from the shared executable directory to the local executable directory. My first guess was a file permission problem or an executable running (like sapstartsrv). However, since this is a Windows 2008 server, it seemed somewhat unlikely. And of course, it had nothing to do with permissions. In the details of the log file was the message that 33 files were not found in the source directory. Of course all of the files were there. Comparisons with the sandbox server, the QAS and PRD servers could find not a single file missing from the directories.
All of this consumed around 4 hours of time searching SCN, ServiceMarketplace, Google and several rocks in the landscaping around my house without success. I finally searched ServiceMarketPlace for sapxxcpe.c (presumably a reference to the error line in the source code). This yielded SAP note 1785829 titled “sapcpe failed to copy files”. The issue is the continuing incompatibility of Windows and Unix text files — specifically the Windows CR+LF character mis-interpretation by Unix. sapcpe uses the files are the *.lst files to determine the expected file inventory in the directory. As described by the note, I opened all 7 of these files in SAPpad and saved them with Unix-style LF character.
Fortunately for this SAP Basis consultant, SAP note 1785829 saved his Christmas shopping time (with a few hours to spare!). The upgrade was completed without further problems and the system turned over for testing — after Christmas.
There are a few points that come to mind after this situation:
1) There is not a great volume of knowledge regarding SUM from the SCN community. There are 100s of notes and PDF documents without question. What is not out there are discussions — with resolution or blogs providing insight into some of the problems that can be encountered.
2) Because of point 1, if you hit a problem with your upgrade do NOT limit your focus to only upgrades or installations. As this case proves, an issue that might occur during normal operation can also occur during an upgrade.
3) Windows and Unix have coexisted for over 30 years. Why do we still have this stupid new line incompatibility problem? And for SAP development: why can’t an additional bit of code be written to check for this? Can I make this my first wish for the new year?
So accept this modest addition to the Software Update Manager knowledge store on SCN. And may you 2013 be filled with happiness, prosperity and NO upgrade problems.
On The Road to EHP6
Implementing an SAP Enhancement pack across a typical ECC6 landscape compares better to a journey than an event. Some trips are like a pleasant drive in the country, other trips become a replay of the movie Duel… 1
Day 1 Upgrade of ECC6 sandbox to EHP6 + nZDM starts now. #sapadmin
— Kevin Grove (@kgrovetx) June 12, 2012
Looking back, that tweet looks more appropriate for a young “fresher”. It was instead from a somewhat grizzled veteran who should have known better.
If you expect this blog to be a step-wise how-to guide – you are in the wrong place. The Service Marketplace: https://service.sap.com/erp-ehp-inst has all the documentation you could ever want.
Tip #1 Plan Ahead
- Grab the EHP6 Master Guide as your guide-book or GPS.
- Next read the Upgrade and Update Guide for EHP6. In this document are the latest SAP notes for EHP6. Download all notes that pertain to your landscape for the upgrade.
- Read and re-read all the downloaded PDFs and SAPnotes.
- Make a checklist of steps based on these documents. For example, I grab the upgrade key from the master note and place at the beginning of my checklist. For me this saves having to dig back through the notes while SUM is running. (for some reason the authors of SAP notes seem to delight in burying these keys deep within the note)
- Read the Software Update Manager (SUM) guide, particularly if this is your first experience with this upgrade tool.
Tip #2 Expect the Unexpected
- Make sure that ALL of your systems are up-to-date in your SLD and SolMan. The sandbox system I planned to upgrade was the only ECC 6 system that had not been updated since last support pack installation.
- When LVM was introduced last year, I quickly got it downloaded and installed. The tool is not quite as good as I hoped. REALITY check for ABAP systems it is quite possible to have many “correct” (ie. status green) solutions for the relevant products. (lost time: about 3 days due to a messy SMSY in Solution Manager).
- Add about 50% to the disk space usage for the SUM install directory and the directory where the downloaded support packs are stored. All of the support packs are copied to the SUM/data directory for unknown reasons.
- Download the latest version of the tp executable from SAPService marketplace. Keep the file handy so that you can copy to the correct directory locations during the SUM process.
- Open a document and screenshot EVERY dialog screen and add notes immediately. This can be a huge timesaver later.
- In stage 5.2 a dialog listing 20 or so SAP notes to review and implement will stop the installation process. Relax- on our system only 2 notes were needed. Add these to the preparation check list so that the next system is prepped before starting SUM.
Tip #3 –DETOUR AHEAD– Third-Party Add-ins
There are many vendors providing add-on products for SAP ECC systems. Many of these products are completely unaffected by an Enhancement Pack upgrade. If an add-on appears as a system component (check table CVERS to be sure), issues can effect your upgrade. SAPnotes 1262124, 1117309 and 1226284 list many vendors who provide keys to allow you to “ignore” the add-on during the upgrade. If one of your add-ins is not there contact your vendor BEFORE you begin your installation. If you encounter issues with SUM stopping due to issues with your add-in:
The answer is 1324910
One wrong turn and you are on the Highway to Hell
For SAP enhancement packs and compatibility with third-party add-ons, the answer is always 1324910. The short translation: SAP will not provide any support. You must contact the vendor for resolution. If your maintenance contract with your vendor is up-to-date, the proper workaround for the upgrade conflict should be readily available.
In our situation, the add-on has not been used in around 4 years and maintenance was dropped 2 years ago (after our EHP4 upgrade). The message with SAP support was opened on June 14. To date SAP support has been insistent that it is not an SAP problem. However after my incessant questions regarding options and clarification of error messages, SAP support responded with this information:
07-12-2012 – 09:19:39 CST – Reply by SAP
It’s required the vendors to provide you the “Deletion CD”. However, currently it’s not possible for the vendors to provide an “Deletion CD” to delete the add-on. Again please contact the vendor to get the appropriate key for you to continue the upgrade.
So after 1 month of back and forth with support, is it actually an SAP problem? How can a vendor provide a deletion CD if SAP has not enabled it?
The vendor has been less than helpful as well. The best offer provided was for us to purchase a new license + maintenance + configuration consulting. The total was over $100K US – for a product we do not use. We have chosen not to accept this as a solution either.
SHOUT-OUT — I would like to recognize some SAP support folks. First via Twitter: @SAPSupportCE – particularly Patricia, was helpful in getting the message unstuck after 5 days in the status “Message Sent to SAP”. Also, the SAP CIC phone support team has also been helpful on several occasions in getting the support people to respond to my questions.
BETWEEN A ROCK AND A HARD PLACE
We are now examining various “manual” options for getting around this problem – without compromising the future supportability of our systems.
A few final thoughts:
- This situation is probably unique in all the SAP landscapes in the world. There is certainly no information on SCN or through Google that comes close to this situation.
- I do not believe that any of the third-party add-in vendors, including the particular vendor in our situation are acting with bad intent.
- At the same time, I will never accept a new third-party add-in that does not come with a documented removal process. That process will be tested repeatedly during the initial set up and configuration. And the process will be reviewed every year during DR review time to ensure the procedures are still correct.
SAP is on a roll currently. Record breaking Q2 financial results, lots of shiny new products: HANA, Cloud deliveries, SuccessFactors, Ariba, mobility and more. All of this new technology is amazing, transformative and may lead major IT transformations to come. But back here on earth, the vast majority of SAP customers are still using the boring old stuff. We are running systems that bear the scars of numerous upgrades, support packs and customizations. These boring old systems are the ones that are running the business of the companies that are SAP’s customers. At times, one wonders if SAP has not become a little too focused on the shiny new stuff and forgotten those of us still working on with the legacy, tried and true, systems.
UPDATE: To restart the upgrade process, a Kobayashi Maru solution was implemented in our sandbox system.
1 Duel is a classic thriller movie: http://www.youtube.com/watch?v=5MtAMc4i8OA starring Dennis Weaver, directed by some kid named Steven Spielberg.
A few days ago while I was working on a new monitoring script for our SAP systems, one of the SAP team members came by and said: “I just got the strangest error message on my screen, did I break something?”
Above is the error message in a screenshot I took to memorialize the event. One thing I know is that a joke that requires explaining is likely not very effective. For any non-Americans: in the US a pool table is a pocket billiard table. This caused the first bit of confusion. Secondly, for the majority of us for whom SQL is not a second or – God forbid – first language “LIKE with ESCAPE” has absolutely no meaning.
So to the person who alerted me to this, it appeared that somehow their actions had caused SAP to collapse into a state of returning random meaningless phrases — SAP’S BROKE! The short story, of course is that SAP was not broken. Instead a diligent programmer provided an informational message to warn against using a search string with the “_” character in it. Of course the search returned valid results, did not run for 30 minutes or any other bad thing.
I began reflecting on how difficult great programming really is. Creating the code to work as designed, returning valid information is a great accomplishment. Building in the facility to react reasonably to the input from the human interface adds further complication and skill. If I had one recommendation for the SAP style guide it would be that when an entry field contains input that is not expected (ie non-numeric) PLEASE, PLEASE do not generate an ABAP dump, instead return a message pointing out the field requiring correction.
Alternatively, I offer my standard error return code in the spirit of Open Source:
echo “Dear %username% that is incorrect. In order to prevent further computational error/abuse the hard drive on this computer will self-destruct in
5 …. 4…..3…..2…..1………………………………………………………………….”
terminate 2>&1 >/dev/null
Share your favorite confusing messages in the comments. I’d love to hear or see them!