Archive for April, 2009

R12 Upgrade Plan

April 9, 2009

1. The most important part prior to starting an R12 Upgrade is to build support for the project. If the Business isn’t on board, it’s going to hit real problems when you want their time – and it’ll take a lot of their time……Look for the key drivers of why your business needs an upgrade; for us we had many new Oracle modules and we felt it was better to do those projects on R12. rather than implement in 11.5.10 and upgrade in R12 later. A move to Linux and the age of 11.5.10 were other critical factors. With R12 being out for 2.5 years by the time we implement we figured it would be relatively stable overall – a lot of companies got badly burned by jumping way too early into R12.

For those companies not in R12 now is a good time to start thinking – the product is stable in most areas (even AP is getting there) and the market is depressed, so there are a lot of available resources (albeit mainly in 11.5.10, but most modules remain pretty similar overall)

2. A study approach of building a Business Case to assess the Upgrade may be a good idea. It won’t cost a lot, but will allow you to make a very informed choice ; the new functionality, the customizations you have, the estimate of effort, time and cash it going to take, pitfalls and other issues.

This approach lets you build concensus and support. It minimizes risk and allows you to build a strong business case to get necessary funding.

For us, we identified several major issues upfront; Payables was a complete re-implementation for us. Payments was a complete re-implementation. We had a LOT of customizations. General Ledger customizations require re-work; small but 60% of our customizations were hit…..Self Service screens must be migrated. All Payment formats require new build. This was going to be the biggest upgrade I have seen in a very long time…..Understand also the architecture – Oracle ERP isn’t an island. It usually talks to an awful lot of other systems and if you don’t understand this at the outset it may hit you just before go-live or worse……after.

We also used a lot of good documentation on R12. Look on Metalink, google, OAUG. People have made plenty of mistakes and published where they went wrong – with the documents available now you should avoid the pitfalls for most.

Remember this is not just an upgrade – Oracle has brand new modules so this is also a major implementation project; Payments is new and completely different, EBiz Tax is new, Subledger Accounting is new and if you are running AP , you have no choice but to implement the Payments. The other two are now core modules that must be implemented, even if your not using Tax………

Be careful of your plans for current Production during the Upgrade. Most of your resources will sink into R12, so try and reduce big new commitments.

3. Starting the project. Make sure you have enough hardware……you’ll need to run a lot of instances AND still support your existing Production on 11.5.10…….If you don’t have the hardware your project will suffer badly. Resourcing wise there are lots of options, but it will take a lot more than an upgrade from 11.5.5 to 11.5.10. I can’t stress enough how important it is to get top tier consultants – a few really good people (albeit expensive) actually saves you money. Depending on your customizations, you’ll need a fair chunk of development resources. But by doing the study you’ll be able to make good estimates. (Don’t underestimate Payables and Payments – these will take a VERY LONG TIME).

Remember that resourcing won’t just be Oracle people; you will need people from the feeder systems involved (Legacy Teams, Data Warehouse Teams – the stuff you change may create work for them), the DBA Team, the Unix Team, the Email Team, the ERP System Admins, potentially people from external (such as Citibank). Most of all , your going to need your users.

4. So your ready to start. Once your over the ubiquitous kickoff meetings (which shouldn’t have any surprises as you did all the planning, lobbying, studies in advance right :-)? ), its time to get the team going.

Obviously an R12 ERP would be helpful. Make sure that you have the hardware, know how to upgrade a copy of Production and have it available BEFORE you have your Upgrade army. It’s a lot of cash to burn through otherwise….If you’ve got a good DBA Team they should be able to get you an instance in 2-4 Week timeframe.

If you have good internal resources, then this really helps. Otherwise external consultant will need to take a bit of time to understand how you use Oracle ERP – sure most companies work in roughly the same way, but every company has its difference; some are very different indeed.

At this point I normally split off into streams; I split into modules, team leads across each module (we have 31 standard and custom modules, spanning HRMS, Procurement and Finance).

The Functional Team should be preparing an initial high level Test Plan, working closely with Business. In parallel they should use their knowledge (and the R12 Upgrade Guides) to identify the relevant changes to the company. The Test Plan should be comprehensive and signed off once complete. This should cover all Test Scenarios across all Business Areas, including external systems. The Functional Team should also try to identify, in conjunction with the Business and Development Team, customizatons that can be removed with new functionality provided as standard. (We removed 10 check formats in a single morning – imagine the saving when you need to rewrite these, do Technical MD.070, Functional MD.050, Testing, etc. A Little time doing this could bring significant savings).

The Development Team should again be split across Modules. If you have a customization register (and we have all 1,600 customizations registered so we don’t miss any) you can use this as a very good monitoring tool. Working closely with Functional Team, identify the core stuff – there is no point in working on a report first, if you can’t enter the invoice to test the report. Once you prioritize the customizations (and possibly modules if you have limited resources), start working to upgrade these, document and test.

Now based on R12 experience, from a Technical Perspective, AP and Payments are very difficult. GL customizations had 60% small updates required. Self Service needed a re-implementation to move from MOD*PLSQL to standard Java pages. Custom modules were largely unaffacted and OAF upgrades with minimal overall issues.

If the Developers update the Customization Register with flags saying Unit Tested, Fixed, Require Fix, it becomes an excellent monitoring tool providing exact progress on your project. Ensure that you use source control. There are cheap, cheerful and excellent solutions including Subversion. This is a great tool.

Once the Test Plan is signed off, Consultants and QA’s should start ensuring that comprehensive Test Scripts are available. As they are being written, the application should be tested initially to throw up the bugs. The earlier you find them the earlier you get them to Oracle Support to get fixes. R12 does have quite a chunk of bugs, especially in Payables. Keeping a close eye on Metalink is important – when new RUP’s are released, you have a very important decision to make on whether to apply. If your well into your project, the advice I would give is not to apply….they’ll cause a lot of issues, it’s newly released and you’ll be the poor guys finding all the new bugs. Leave that to someone else. Obviously if you have critical errors, pile the pressure on to Oracle to provide single patches. If you are early in the project stage, then apply the RUP’s, but do it on a Patch System and check it first…..they can cause real damage. At some point though your going to have to freeze patches, otherwise you will never finish.

Whilst all this is going on your DBA team should be working on practise upgrades and optimization. It will take many upgrades to get it right.

5. So you have your Test Plan, Test Scripts and Customizations. We run what we call an Internal UAT. Basically its a UAT but it is carried out by the R12 Upgrade Team, not the user. Try and get an exact copy of Production, upgrade fully, deploy the objects and do the setup. (Be careful to carry out pre and post upgrade steps)

Given that some modules are going to be delivered before others, you may want to do this staged. It’s not ideal but it does compress the overall project time. You will have problems on Payables, and should that hold up your HRMS track? I would argue no.

The point of the Internal UAT is to run through the entire test, as the users would do. This includes external systems such as Legacy, your planning systems, Citibank, or whatever. All bugs should be recorded. (We use a tool called JIRA which is fantastic)

Be particularly careful to look at the upgrade process itself. You should simulate a close on your R11.5.10, reconcile on 11.5.10, then do the upgrade then reconcile again on R12 and make sure the R11.5.10 and R12 add up. This is critical and we found what we think is a major issue on accruals when we did this.

6. Another clone of Production should be taken. All objects, customizations, setup should be done on the UAT environment (If you have not been keeping deployment registers, BR.100 would strongly suggest you start doing so……) Usual UAT kickoff meetings should be held and contact lists given to users. Your team will need to work hand in hand with the Business Users to co-ordinate the UAT. It’s going to be a lot of work, a lot of new and unfamiliar screens and quite a few complaints, no matter how well you’ve done till now. The Test Plans and Test Scripts should have been agreed months ago, with full review and signoff.

One trick here is to put the Test Plan on a shared drive and ask the user to mark Pass/Fail/To Test as they progress. As an Upgrade Manager you can then review and summarize easily progress across the entire ERP Suite. Now that is sweet…….Again delivering some modules into UAT before others may be an option if you want to progress quicker. Others prefer to deliver the entire UAT. What I have found is that by progressing some modules quicker, it gives the project momentum and when you put out weekly progress tables to users, they don’t want to be last to finish.

Make sure all Test Scripts are tested. Make sure Month End and Year End are similuted, with proper reconciliations. Make sure all the systems to and from Oracle ERP are tested and continue to work with R12. Make sure integration between modules is smooth and working – organizing this across departments is tricky. A UAT room for this can be handy…..providing cookies and drinks (non alcoholic please…..don’t want people testing when they are drunk….) creates a good ambience.

Record all issues and bugs as you go – again a tool like JIRA is great. Monitor closely and ensure that when your users are doing UAT, your R12 Team are there to assist. Some areas may need twice weekly meetings on certain modules. This will help to manage difficult UAT areas, stop user frustration and make the UAT a little less painful for all. We did this on Payables in particular.

On completion, ask the head of each area and other users to sign off a standard AIM Acceptance certificate.

7. Whilst the UAT was going on I hope you were also planning to do something else right? As UAT goes on, you should be planning what will be a very compex and difficult cutover. The team should also be doing rehearsals by taking copies of production, upgrading, deploying, setting up and testing. All bugs should be recorded.

Don’t underestimate the deployment.

You should have deployment registers for each module, including tasks for DBA, System Admin, etc. You should also have a big thick document on the steps the DBA’s do in the actual upgrade. Separate deployment registers with hyperlinks to relevant setup docs, BR.100, etc.

Review the deployment registers and figure out what can be done in advance of R12. We transferred new programs, menus, value sets, etc to R11.5.10 without switching on certain components. All of this saves a little bit of time and will ease the stress of the upgrade weekend.

Ensure the BR.100 (or other setup instructions) are clear, concise and complete.

Call meetings with the users and make sure that the cutover (and possibly closing processes) are clear.  We are doing the closing in R11.5.10 just before cutover, closing down all transactions, etc. Why? Couple of reasons. If data is left open in R11 (say unaccounted) it will give real problems in R12. (We worked for months to get the script from Oracle and luckily we found this prior to go-live). Secondly it allows you a whole month before your first closing in R12. You can do a couple of simulations of Production and sort out issues if needed. The users will lose Production for a few business days, unless you do it on a holiday……..its a big upgrade. Our’s runs for 42 hours.

Get a register of who needs what responsibility on the day and how many setups they need to do. Set these up in advance so you know who needs what access and are not scrambling on the day. It also lets you balance the workload. We have almost 600 setups. A lot in Payments.

Deployment of code. If you have used Subversion (or another Source control system) then you can grab all changes and give to the DBA’s. Make sure your directories match Oracle’s and Subversion as that will make deployment easier. A lot can be automated and this will save a lot of time on the cutover. Now with all the prep work the DBA’s started over the previous months maybe this is something they could also have been working on…..There are a lot of standard Loaders that can save lots of time.

Talk to Oracle and make them aware of your key dates, cutover, month end, payroll, etc. They can get you critical support management over that time.

Try and do as many rehearsals as you can whilst users are doing UAT. The closer the rehearsal is to an exact recent copy of Production the better.

Make sure you do patch reconciliations between your UAT and Rehearsal databases.

Hold a few internal meetings to ensure that everyone in IT knows exactly their role. Watch out for people booking holidays in advance. One key person out could derail your go-live.

As PM, you should be writing a PM.030 Transition and Continegency Plan that covers all the angles.

If the cutover fails all your hardwork will have gone to waste.  Rehearse, Rehearse, Rehearse……

8. Cutover

Generally over a four day window. Holidays are ideal but make sure you tell Business well in advance if their key system is going to be down during business hours. We plan to be down by 5:00PM on a Wednesday and up by 7:00AM Monday. We also have a migration to Linux thrown in. Wednesday, Thursday are days for the actual Upgrade, Friday the Upgrade Team roll in to do setup, customizations, etc. Saturday and Sunday are for testing with a go-live decision on Sunday evening. As we are moving server, our contingency is simple; if the cutover is a no-go we simply turn our old server back on……Don’t repeat the errors you made in rehearsals, keep a log of everything that failed and make sure it doesn’t happen on your go-live, as you simply won’t have time to troubleshoot.

Be very careful that all transactions are accounted, workflows complete (as much as possible), payments complete. Review the Best Practises from Oracle for more detailed advice. If you don’t do this you will hit serious problems.

9. Post Go-Live. There are going to be problems on anything of this scale. Thats a given. Have all teams on standby so that when issues arise, they can be addressed. With Oracle Critical Support you’ll hopefully be in better shape than with standard support. As soon as possible clone production to a test system and simulate the payroll, closing and other key events in advance.

10. Party? If you do an R12 Upgrade, you deserve a big party. R12 is the most challenging upgrade you will ever have done in Oracle ERP. Good Luck !!!