Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt3)

Welcome back to the R12 Upgrade House of Terror.

All I can promise you is pain, heart-ache (that Oracle Support girl for the AP Trial Balance broke mine…..ahhhhhhhhh) and impending doom. Abandon all hope on your R12 for those who read on……..

So you’ve been mad enough to start an R12 project (they’ll lock you up in the asylum they will !!!) and you’ve made it through Implementation (designing any new stuff, build, test, UAT, etc) and you are sitting there smugly thinking you’re smarter than your average 5th grader and that this column was just scare-mongering……..Well lets go to the Transition Phase and see how confident you are………

 Cutover is complex

Deployment involves high number of objects

Deployment involves significant application setup

Deployment carries high risk areas

Users are not familiar with the new software

Large number of users require training

Users are distributed across countries

Help Desk Staff are not trained to take calls

Transition is poorly documented

Transition is not rehearsed

Change in Business Procedures for R12

Before you start the cutover, just wondering if you actually trained all your users? Yes R12 is mostly the same, but there are significant variations between R11 and R12 in Payments, TCA, SLA, etc. Now if those are spread across countries, it’s going to be even worse. A new system with new features suddenly appearing will lead to a lot of angry users, all wanting to talk to you on Day 1 of R12 as you are inundated with other more serious issues……..Now I went through four audits on my R12 and got a clean bill of health on all, except for training (even though we did some). The auditor did a survey and the users said they would have liked MORE TRAINING. But the products almost the same I said bar a few key areas as we were doing a Technical Upgrade !!!!!!!! I got burnt at the Halloween stake on this – OK ever so slightly singed. Don’t make this mistake. At least offer some training on delta differences and some educational classes with no commitment to implementing (get your lawyer to write the disclaimer) new functionality.

Has anyone told the Helpdesk of your R12 transition? Has anyone told the business they will lose their systems for 4 days (weekend, plus two working days). Has anyone told the external users or put out appropriate announcements? Are there key business activities that expect to run during your upgrade? Have you scheduled your upgrade right in the middle of these? People are going to be seriously upset if payroll is delayed for your upgrade. Or month end closing is delayed and you have serious problems with your new R12 that goes live right before the preparation of those financial results. Wondering if you told your corporate website team that pulls out summary information from ERP to display to your company website that ERP won’t be available over the transition? Or how about the legacy team that are still sending you files to interface into ERP? I know you are feeling pretty insecure now. I guess you are wondering “Have I missed anyone in the announcements……”.

Cutover is a nightmare. Get this wrong and you see your entire project collapse in a pile of dust, on day 1 of Production, with the Board asking why your business has ceased to function and lost 5M US. Pulling the transition on R12 at the last minute is really bad, but even worse is going live and then finding out you have serious issues……..Either has a good chance of ending your career.

Has everything been tested? Has every bit of code your about to deploy to Production actually really been through UAT (or did the developer slip in that one last easy fix with no testing?). Does your Application Setup really reflect the setup you tested? Did the Consultant complete the BR.100 before he left for a contract paying 30% more? Can anyone actually follow that BR.100……?

So you have a thousand customization files (actually isn’t a lot when you count Java files, setup, etc). Have you ever taken that as a single software release and had your DBA’s (not your developers that know to run script 2 if script 1 falls over and hack script 3 to make it work, without anyone finding out) and applied to an exact clone of your Production system and had a 100% success rate? Have you repeated this multiple times as you roll towards the go-live date, just to make sure that any changes between the time you last ran it and any updates to Production didn’t cause any adverse impact? Are you sure that the DBA’s can run all your customization deployments in a very short-time frame. (Your typical go-live window is a maximum of 4 days with two of those the weekend, after that your business will scream if you go one hour over). Is your deployment manual and if so, will the DBA do the same steps each time? If your deployment is automated will it actually work?

We had 600 application changes. Have you ever checked how long it would take to deploy all of those changes? Have you ever done an entire setup and checked all the setup was valid and tested properly? Have you thought of the issues of letting knowledgeable people do the setup in Production in terms of access, SOX, etc? If new people need to do the setup, as the project team cannot have access to Production, are they familiar enough not to make mistakes? Have they ever practiced the setup on an R12? Interesting how long it takes to carefully setup and check 600 application setups. Just wondering if you ever thought how long it actually takes to setup 600 application setups for a single person. Do you know it’s impossible to do that setup with one person? Have you ever bothered to do a careful review of all setup, access needed and then balance setup streams across different people to make it achievable in the very short transition time the business has granted you whilst still ensuring segregation of duties, SOX compliance, etc?

Have you ever considered setting up some functions, menus, etc in R11 BEFORE you go-live. Or some new program definitions? Or perhaps having these as Loader files, so you can load them instead of manually setting up? Of course if you load them, have you checked that the developers have generated these LDT files correctly?

How many minutes will the R12 Upgrade run on Production hardware? (I ask for minutes because you should know very precisely the estimated time of this – if you don’t you are in serious trouble. For info an R12 will run for a very, very long time indeed, even on a small ERP database.). Have your DBA group ever run an R12 Upgrade with no failures? How long will it take to upgrade the Technology components or database? Have you ever seen a detailed step by step upgrade document from the DBA Team? Are there any hot wiring the DBA team apply during the upgrade due to known failures? Are you aware of these known failures and the implications? Do you know that an R12 Upgrade will require some very serious shift working? Do you know that your DBA’s will be seriously burned out as they have the very first task in the actual upgrade of running the hundreds of thousands of scripts that the standard R12 upgrade runs. And they need to actually be awake and watch this constantly, because if it falls over, you don’t have the time to lose 8 hours on an upgrade weekend…….

Wondering if you’ve actually even got a transition plan? Have you got an hour by hour schedule, with milestones and an effective method to monitor setup, customizations, actual upgrade, etc? On an R12 upgrade transition, you have no slippage time for anything. If you slip, the team will get stressed, you’ll get stressed and mistakes will be made.

Bottom line is have you REALLY rehearsed an R12 cutover BEFORE you actually do it? Do you really feel confident that first time you do it for real will be on your Production system……….if so you must be a clown……by the way do you think the McDonald’s clown is intrinsically evil????? (My Attorney has advised me at this point to clearly state that is a question and in no way implies anything on McDonalds whose food I have been legally advised to state I love and the picture below is in no way related to said company).

If you take a look at the Oracle Upgrade manuals, it’s interesting how many pre-req steps potentially have to be done. Ironically if you miss some of these you can have major support issues going live. I wouldn’t leave any Payments in the middle of process. I’d be carefully checking the effect of an upgrade half way through each of your workflows. Is there any effect on transactions that are left half completed? Do you even have these in your cutover procedures and does everyone in the Business know from your carefully documented plan exactly what their responsibilities are prior to bringing down their R11 ERP. If you (or the Business) makes mistakes here, you are in real trouble. Do you have scripts to check everything has been accounted, no payments are sitting halfway or are you just relying on your business to do the right thing with no checking whatsoever…….Just to add a little time pressure, your DBA’s will be screaming at YOU if you go one minute over the agreed time to shut down R11 and hand it to the DBA team……..of course you’ll have to herd dozens of business units across 27 countries to complete all activities before that time…….but then they want to do their business not your upgrade……which leaves the DBA’s screaming at you…….(The DBA’s will be working all through the night, so every hour you delay, means an hour extra in the middle of the night so a little empathy for them here folks……..Show them you care for their welfare….at least till your upgrade is over. Alternatively take the selfish approach that the DBA that is working all night might be the same one running your customization scripts after two hours of sleep because of your poor planning that could wreck your upgrade when he makes a serious mistake because he’s so tired……….)

Have you made any plans to backup the original system? What are your fallback plans if the upgrade fails? When will you be prepared to pull the plug and under what criteria, over the transition cutover to R12? Who is going to take responsibility to give the green light to go live? Will the senior management even be there at the weekend to make that decision?

Just wondering if you told your team and others (Unix, Email, Legacy, etc) that they would need to work some very stressful and long hours over the cutover weekend. Wondering if any of those key staff have their holidays planned…….?

Have you made plans to clone and test the actual R12 on the cutover weekend? If not how do you know all your setup, customizations, etc actually were promoted cleanly and your R12 actually works? You will need a large team to test in such a short period of time. Do you even have a cutdown test plan to make sure you hit the key functionality? Have you got all this planned? Probably not…….And if you find stuff missed, do you have a proper controlled mechanism to track it, test it and release it to Production in a horrifically short and highly pressured weekend in a safe manner?

What’s amazing is that everyone plans a transition, but no-one actually thinks about turning on the R12……….when you enable the workflow and the concurrent managers are you just going to open the tap, or are you carefully going to release production jobs and monitor. I’ll tell you that your single most stressful time is when you commit to switching on these elements over the weekend. I bet you that your heart will be pounding and you will be sweating at this point. The entire company ERP and Business hangs on your Go or No-Go signal. Once you’ve done that, there is no turning back, there is no backing out and you are on R12. At that point if you’ve got it wrong, you’ll be looking for another job. (Doesn’t that picture below remind you of your boss when it all goes wrong……?????). I wonder if that Oracle Support girl I dealt with for the AP Trial Balance wore red lipstick…….ahhhhhhh……..

So your live, but it’s not over……..

Serious Outages can occur

Lack of Staff to address serious support issues post go-live

Patches are required after go-live

Are you ready for serious outages? Did your transition plan have proper contingency plans for these events? If not exactly what will you tell the Board and the Business?

Do you have the appropriate staffing plan to handle what could be a very rough few days (or weeks) or months? An R12 will almost always lead to a spike in calls. Are you ready? Did you bother to give your people proper notification that they may be required to work late nights and weekends for the first few weeks after go-live? Or is it just a line item on your plan that you didn’t bother to communicate?

Did you re-synchronize all of your ERP and Legacy systems in terms of interfaces? Some of these integrations may require manually running interfaces to catch-up with transactions for the days you were down.

With major business events such as payroll and closing, do you have any plans to clone and rehearse these events, prior to these happening? Now if you can clone and simulate an R12 closing after you are live, but before the closing happens, it’s better to find out serious problems and have a few weeks to fix them right? (Of course this has you wondering if you should close in R11 and then immediately move to R12, but of course, then your Financial reporting would be straight after go-live, so you can’t win right?)

Have you got any plans for patches after go-live? What happens if you really need to apply patches? Are your project team still there? Did you set business expectations just in case? Always better to make business aware of the real risks inherent in an R12 Upgrade. That way, they’ll be incredibly happy if you manage to get it right.

I hope you remembered to get before and after reports from key areas, such as your AP Trial Balance (that Oracle Support girl was just so nice, ahhhhhhhhh), GL Trial Balance, etc. And keep a copy of your Production database on R11, whether on tape or disk. When the auditors come and take an interest in it, as they will it would be unfortunate if you’ve not done this and it causes issues with your auditor’s signing off your company accounts.

Also before you go live just have a quick review of JVM sizing and parameters. If you fail to tune and size this properly you could see your Applications crashing, intermittent crashes, etc. Getting your JVM sizing wrong leads to absolute nightmares……..Ask your DBA’s “Have you size the JVM’s and if they say what’s that, start to worry……”. If you have too many users and not enough JVM’s and OACORES then BOOM !!!!!!!!

Finally is there monitoring of key components in place? Are you touching base with all the areas of Business (and Legacy and Web and email and and DBA and Unix and other teams). Are you even aware of major production issues or is your Production setup so poor in terms of support that you’ll be the last to know……..

And lets break for another movie recommendation. The single scariest movie I have EVER seen. This was the 1970’s television mini series called Salem’s Lot. It is absolutely terrifying. Get it on DVD….it’s one creepy movie from the Master of Horror Mr. Stephen King. I saw this when I was ten, and I didn’t go out for weeks after dark……It’s a classic ten out of ten movie. But REALLY TERRIFYING.

This isn’t an exhaustive list of what can go wrong in an R12. Now note this article has been written in a strong manner to scare the crap out of you because if I do that then you’ll start to take every last point out of this article and make sure you’ve covered it. Make no mistake – R12 Upgrades are a serious risk to your career. (However don’t let some consulting company come in and scare the crap out of you. They are in it to make as much money as possible from you for an R12 by scaring you as much as possible. I write these articles for FREE and hope people can avoid the mistakes that I and others made in our R12 Upgrades). If you read this article and take everything on board, you’ll be strongly versed in R12 risks and be able to evaluate what these consulting companies are telling you and separate the fact from the fiction…….

And a final note. Premier Support for 11.5.2 ends in November 2010…..that gives you about a month left……Interestingly you also need to have minimum baselines in place for R11 if you want proper support from November 2010……….And will you get hit by fees if your still on R11 in November 2011??????

Have you read Metalink Note 1178133.1…….

“Starting on Dec 01, 2010, Oracle E-Business Suite customers on Release 11i10 will only receive extended support for new bugfixes as described in My Oracle Support, Minimum Baseline Patch Requirements for Extended Support on Oracle E-Business Suite 11.5.10.”

That’s a lovely little bit of legal small print from Oracle with major implications for you………Getting to those minimum levels will be enough to “persuade” most customers to jump to R12 instead.

It’s funny also that so many people don’t think they can start doing preparatory work in R11 that will greatly ease R12. Have a look at the references below. Work spread over R11 and R12 will greatly ease your stress in the R12 Upgrade itself, as you’ll have less to do. (Did you know Solution Beacon even has Vision instances you can play with for free, even before you install an R12 in your company’s test environment…..)

Let me tell you briefly our R12 story. We had some very serious problems with quality (12.0.4 RUP5) – an early R12 release in many respects. However despite that we got through a Finance, HRMS and Procurement Suite R12 Upgrade across 20+ countries with strong planning. Our transition finished 30 minutes ahead of the planned 96 hour time window (I wasn’t kidding when I said you should know the transition in minutes…..). Yes, I was terrified when I made the call to go-live. But due to all the planning and a great Project team, we went live with no major issues on 3rd August 2009 and I went home at 5PM two days after go-live (And for the cynics, no I didn’t spend the first two days constantly in the office 48 hours straight…..I got home at a reasonable time the first two days as well). We cleared Payroll, breezed through month end and went live on two Payroll rollouts to two new countries one month after go-live.

Of course now I’ve told you all the risks, well you’d be a fool to actually fall into those holes right? Take this article, make a list, make sure you cover each and every one. Then you’ll make your R12 a huge success.

An R12 is certainly a very achievable project, with significantly reduced stress and risk, if planned correctly.

Hope I haven’t scared you too much……..

Enjoy your Halloween.

References

As always, my appreciation goes to the kind authors and contributors of the following articles and resources. Some of these are especially note-worthy, because those people are putting out their failings (as well as successes) on R12 in a very public and open way, for the benefit of those about to travel the R12 Upgrade road.

General Blogs and Websites

OAUG Website

Chris Warticki’s Blog – EBS R12 Support Resources – Consolidated

Oracle Applications Upgrade Guide: Release 11i to Release 12 (B31566-01)

Oracle Financials Applications Blog – R12 Lessons Learned

Analyzing, Planning and Executing an R12 EBS Upgrade

Steven Chan Blog

Risk Management: Tricks of the Trade for Project Managers (Rita Mulcahy)

My example of an R12 Risk Spreadsheet (Excel 2007) – Available on Request

 

Metalink

Oracle E-Business Suite Release 12.1.3 Release Update Pack (Patch 9239090)

 Oracle E-Business Suite Release 12.1.3 Readme (Note 1080973.1)    

Upgrade Manual Script (TUMS) – Patch 5120936

Metalink Note: 580299.1 – Best Practices for Adopting Oracle E-Business Suite, Release 12

Metalink Note: 394692.1 Oracle E-Business Suite Upgrade Resources

Metalink Note: 562887.1 R12: Helpful Tips for a Successful R12 Oracle Payables Implementation

Metalink Note: 437422.1 R12 Troubleshooting Period Close in Payables

Metalink Note: 73128.1 R12 Troubleshooting Accounting Issues

Metalink Note: Oracle Support Upgrade Advisors (250.1)

                             Tech Stack                     Note 253.1

                             Financials                     Note 256.1

                             HRMS HCM                    Note 257.1

                             Manufacturing              Note 258.1

Metalink Note: Extended Support Patch Level Verification in Oracle E-Business Suite Release 11.5.10 (Note 1178133.1)

Metalink Note: R12 Upgrade Considerations by Product (Note: 889733.1)

Oracle E-Business Suite R12.1 Financials Pre-Upgrade, Setup and Operational Tips (Note: 1104163.1)

Oracle E-Business Suite Release 12.1 Information Center (806593.1)

Database Preparation Guidelines for an Oracle E-Business Suite Release 12.1.1 Upgrade (Note: 761570.1)

Oracle Applications Installation and Upgrade Notes Release 12 (12.1.1)

R12: Period-End Procedures for Oracle Financials E-Business Suite (Note: 961285.1)

Oracle Open World 2010 – On Demand

For those with access to Oracle Open World 2010 On Demand there are good references at http://www.oracle.com/us/openworld/index.htm

Mission Impossible: Oracle E-Business Suite 12 Upgrade on 3 Continents in 6 Months

Success with Oracle E-Business Suite Release 12.1.2 Upgrade Drivers

Algar Telecom Upgrades to Oracle E-Business Suite R12   

Get Ready for Oracle E-Business Suite Release 12.1: Tasks to Complete Now

Oracle E-Business Suite 12 Upgrade : Best Practices to Obtain Business Value

Planning your Oracle E-Business Suite Upgrade from Release 11i to Release 12.1 (Superb presentation – highly recommended)

Oracle E-Business Suite 12 Upgrade: An Easier Ride on Nine Miles of Bad Road

Oracle E-Business Suite R12 Upgrades: Have you Thought about the Details?

Real-Time System Assessment of Oracle E-Business Suite 12 Upgrade: Case Study

And finally talk to other companies, use forums, other areas (such as http://www.linkedin.com) and make contacts with people that have done R12. This again will help you considerably on your R12.

Footnote:

Some of the great graphics were the work of:

WWW.MYSPACEGRAPHICSANDANIMATIONS.COM

Advertisements

Tags: , , , , , , , , , ,

3 Responses to “Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt3)”

  1. Gareth Says:

    Classic 😉

  2. Michael Walsh Says:

    Hi,

    I would love to get a copy of your R12 Risk Register please.

    We are debating whether an upgrade or re-implementation is appropriate for us as we are heavily customized.

    Thanks (and great arcticle, by the way)
    Mike.

    • irroberts Says:

      Thanks for your kind comments.

      I’d still recommend for almost all companies an upgrade, rather than re-implementation. A lot of your customs may have very valid business reasons for being done and you may find yourself simply redoing many. R12 is a very large and difficult upgrade, so re-implementation just makes it even more so. Hopefully with 12.1.3 Oracle has reached stability which should make it easier than those who were early adoptors of 12.0.1 to 12.04

      I’ve attached a simple risk register we used.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: