Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt 2)

So it’s heading for Halloween. Only a few days left……..And looks like you’ve come back for the 2nd part of the R12 Halloween Upgrade. So I guess you don’t scare too easy? Prepare to be afraid. Prepare to be very, very afraid.

So after a whopping amount of things that can go wrong just in the initial stages of an R12, we can finally talk about the actual R12 Implementation in terms of analyzing, designing, building, testing, new functionality, etc. It’s amazing so much can go wrong BEFORE you even start……..here’s a previous Project Manager that was running an R12…….(he’s the guy 3rd from the right, the one 2nd from the right did the R11 Upgrade and the one on the right did the initial ERP implementation….isn’t Oracle ERP pretty unforgiving project wise……)

Here’s a few things on Implementation to help you lose sleep.

Applications do not provide a good fit to Business Requirements

Subsequent Analysis during the Project drives Significant Change

Not possible to achieve a consistent Business Process across all units

Some modules are largely custom built

Large number of customizations

Undocumented customizations

Lack of development standards

Information is confidential

System Integration is complex

System integration is high risk

System integrates with non Oracle technology

Complex Oracle Application Configuration Management

Complex custom development configuration management

Poor Technical Designs

Customizations are delayed or lacks resources in Development Team

Functional Designs are late or incomplete

Technical Designs are late or incomplete

Patch Management is poor

Complex Setup

Testing misses requirements or scenarios

Testing is incomplete by Project Team

Testing is incomplete by Business Team

Testing is delayed or lacks resources for Project Team

Testing is delayed or lacks resource for Business Team

Testing integration is delayed by dependencies

Legacy data is of poor quality

Legacy Data requires data cleansing

Legacy Data has complex structure

Legacy data involved high data volumes

Changes in Legacy System required

Oracle does not have interfaces to support conversion

Oracle uses complex API’s for conversion

Ignore the Legacy Conversion stuff. I use this spreadsheet for more than just R12 Upgrades…..

Now if you’ve ignored the advice to aim for a R12 Technical upgrade, then you may find yourself knee deep in very long, difficult and time consuming business requirements, business change, etc. On an R12 Upgrade, trust me just don’t go there. Keep it clearly 100% out of the scope of the project. It can wipe out your project unless absolutely forced to. Which brings us nicely on to R12 Oracle Payments.

Payments module is a re-implementation. This will take some adjusting as it looks nothing like the R11. With this and other new features, you run the risk of not fitting to business requirements, or wanting new processes or wanting changes. The problem is if you show the new stuff (and with Payments you have to….) you could have significant change in your project scope and therefore significant risk. If you don’t show the new stuff you are going to be accused of not showing it, not training users, not taking advantage of the “amazing” new functionality, etc. So if you show it, you are screwed. If you don’t you are screwed. For me R12 is complex enough to pull off without adding anything other than like for like. New functionality for us could wait (and it did in 99% of cases, despite some fairly tough fights).

Of course if you are rolling out R12 across multiple countries, then it’s even worse to agree all the new functionality, legislative stuff, etc.

Then you get into your biggest nightmare of all. Customizations. Imagine someone having a decade to write customizations on R11. Imagine no discipline, documentation or standards. And you’ve got to upgrade something that you have no clue what it is you actually have to upgrade. Talk about a Pandora’s box……Or worse does anyone remember the Hellraiser horror 80’s movie flick with the box that brought out the demons…….Do you really want to open that R12 box??????

When you open an R12 Upgrade project, you are opening a Pandora’s box unless you did an advance study. Or worse you could be the guy hearing the doorbell, going to his door and seeing a burning brown paper bag with R12 written on it on his doorstep. You go to stamp it out and find something VERY unpleasant on your feet that your company that implemented your ERP left you with but didn’t tell you…….Is your current R11 a brown paper bag with a pile of crap inside waiting for you to stamp on it with your R12 Upgrade?

One single customization missed can cause a catastrophic failure. Funnily enough I saw one Upgrade where they had account generators in the hooks of a standard package for Costing. Now yes, this was supported, but totally undocumented. Funny that generated a pile of entries to General Ledger that took weeks to figure out and calculate the reversals to, after the code was fixed post go-live. So the message is, it isn’t even enough to look at the obvious custom code. What about the standard packages that support hooks? What about overwriting standard code in packages, forms, reports and other customs? What about the change of seeded menus? None of this should happen, but don’t count that it hasn’t.

If you don’t have a customization register before you start you are in very serious trouble. You are going to miss stuff, and that one thing you miss could wipe you out in Production.

Also do you have any idea what R12 will do to your customizations. We saw our Payment customizations wiped out and need completely rewritten. (BI Publisher). Any workflow changes? They’ll need to be redone. Any Self Service in the old MOD PL*SQL? They’ll need to be redone. Any interfaces to GL or reports or anything else? They’ll need to be fixed as 80% of our stuff stopped working. Small changes on a lot of customizations add up to a lot of work, time and risk that can seriously delay your schedules. Did you know there have been very significant changes to the data models between R11 and R12 in certain areas……all that will cause your existing R11 customizations to crash and burn……

If you are doing an R12 Upgrade, you’ll have frequent clones. If you don’t encrypt data you could be incurring serious risk as all those outside resources can see vast amounts of confidential information. (See R12 Security and the Auditors from Mars article to scare you some more).

From a Systems Integration viewpoint, watch out that your R12 Upgrade doesn’t just focus on Oracle ERP. When we did the upgrade, we had Legacy, email , Web, external 3rd parties (Citibank and others), Health Insurers, etc. Now us Oracle guys like to think ERP is the center of the universe. Sometimes it is, sometimes it isn’t. But your typical R12 (or R11) can have interfaces, data extractions, data imports, links to spreadsheets, etc that can number many, many hundreds of integration points. Are you aware of what they are and if not how the hell are you going to test and make sure everything works day 1? Think about missing the testing on say Payroll interface to the Bank. That would be a nice one to find out doesn’t work on say, Payroll day for 50,000 employees. Or perhaps you forgot to test the interface to your planning system that orders the parts to build the products you make, which would stop your production line for 5 days, cost your copy millions of dollars and definitely get you on the “skull decoration” shown at the start of the article in the Boardroom or hanging on your managers wall….. . Hope you sleep well thinking of your integration points to ERP and the catastrophic damage it does if you get it wrong. I bet you that your one of these people that DOESN”T EVEN HAVE an architecture diagram of your ERP integration points. And your company spent like what 50 Million on that system with your integrator and you didn’t even get a coffee mat Integration and Interface Architecture chart????? You were ripped off and it may come back to haunt you…….

By the way, speaking of integration, does all the tech stack count? Just think, adding Forms 10G, Reports 10G, JDeveloper 10G, downloading the extensions for OA Framework, your Discoverer upgraded, BI Publisher. I guess not so much integration as getting the absolute basic tools to even start your R12, as all of these change between R11…….so you’d better sort that before you hire all those developers to upgrade your customizations…..

There will be a need to update or add new functional designs unless you are really a vanilla site. We did a lot of work again around Payments. Will those designs be decent and cover the requirements? Will they be reviewed in a timely manner? Will they be signed off? Keep this to a minimum (no new functionality, that’s gold plating an R12 Project) and you’ll avoid this risk.

A lot of companies also do MD.070 Technical Specifications. I’m not a fan of these to be honest as they have too much info, take too much time, are subject to not getting updated so become quickly outdated, and are largely done by many to tick the box. If you are doing these, make sure the quality and approach of customizations is correct for R12. (And don’t build what R12 provides – that’s one area I would add to an R12 to replace customizations with standard functionality where possible. We took out a raft of custom modifications which are the worst types of customs and vastly reduced our risk both to our upgrade and future upgrades/patches).

So I saw a movie with my kid. It was called The Hole (and it was in 3D so the ticket price is like double !!!!) Now this was a PG13 which meant that it shouldn’t have been that scarey, except it scared the crap out of my kid. More to the point it scared the crap out of me even more. So what’s it about? The hole generates your darkest fears and then these monsters come out. And what’s my darkest R12 fear? It’s about managing all of the configurations on an R12 Upgrade Project – that is my worst nightmare. The Hole gets a 7 out of 10 on the movie review. Well worth seeing, but if your kids get scared easy, approach with caution.

Now managing all your environments, patches, setup and customizations is a truly epic and very complex task. Really someone could write a book on this by itself, but the point is, you lose track of this and you lose track of your project. The lack of source control, the lack of BR.100’s, the lack of configuration management spreadsheets (or software) will cause your project to descend into chaos, especially as your ERP clones spiral from a development instance, to a sandbox, to a project team test instance, to a UAT instance, to a patch instance, to a DBA instance, to an instance with a DMZ, to an instance just for one module as it’s getting piles of patches and needs isolated……..your configuration management isn’t about a single instance. We had 10 instances at one point during the upgrade…….Or to put it another way our R12 had over 600 Functional Setups between R11 and R12, dozens of patches, thousands of test scripts all requiring data/setup to be in exact scenarios and thousands of customization files, loaders, packages, etc……Just curious on how exactly you are going to manage that………..? Are you one of these people that manage this type of thing on the backs of bits of paper or a cigarette packet? If so you’ll need to become a chain smoker as you’ll need a mountain of cigarette packets to keep track of all your configurations items on an R12 Upgrade project……

What I always find most worrying on an upgrade is playing the Oracle Patch Lottery. This is cool. You’ve got your R12 say Test environment all setup. You’ve got your thousand customization files, 600 setups and all the test data ready and you apply a patch, having no idea what that patch changes. And it trashes your entire environment, wipes out your 80% test completion and trashes the modules down-line of your testing. Every patch you apply is a roll of the dice unless you very carefully manage each and every patch. So when you apply a patch during your upgrade and have no clue/plan/analysis of that patch, the question is, do you feel lucky punk?

Testing is full of pitfalls on any upgrade. Have you got all the scenarios covered? Are your test scripts complete? Is your test data valid? Have you covered month end testing? Have you covered year end testing? Have you covered integration testing? Did you try the Web ADI? What about tools like Discoverer? Have you covered testing to and from Oracle and 3rd party companies that you barely even talk to (and met in 2007 when you put in your interface to their products which run offsite – think bank interfaces). Does your new patch invalidate Test Scenario 220, 221, 222 all the way through to 230 for Payables and therefore invalidate your testing across 80% of Payments? Or what if you apply a patch on TCA? Does it hit PO, AP and Payments? Or if you apply your customization, what have you invalidated? Will the auditors pick you up for claiming a sign off from the user, when they tested on a completely different configuration of custom, patch, etc? Are you sure your environments are all the same? Or do some differ in terms of setup/customizations/etc? If so, then how valid is your testing?

Here’s another great one to keep you awake. So all your systems around ERP. I bet you one joker that owns for instance your Legacy planning system or say your Payment middleware is planning to upgrade at exactly the most inconvenient time for your R12. Now you’ve always hated that guy and he’s always hated you. I mean is he doing this at the most inconvenient time in your R12 Upgrade without telling you just to screw you over yet again to get that next promotion over you ? If your not aware of everything you need in your R12 Upgrade and everything in your company’s universe (planned upgrades, planned business initiatives, politics, etc) that can screw you over, I need to ask, why did they even give you the R12 upgrade to do?????

Even if you get all this right, does your team have enough time to thoroughly test? Did you properly track all the bugs? Are you monitoring the bugs? Are you monitoring the Service Requests? What happens if Oracle decides to give a RUP (Rollup Patch) for you in your UAT and refuses to provide a single patch? Do you also take that latest security patch? Did you know that security patches have been known to screw up the Applications…….

It’s also interesting that an R12 DOES have data migration. Think about all those scripts running for the upgrade from Oracle Corporation. They are re-arranging your ERP data pretty substantially in some areas. What if the standard upgrade scripts for changing the structure of the Bank data (to TCA) goes wrong or cause problems? Think that won’t happen? Have a look at our Risk Register for R12………Or did you know you can run a script to convert your previous data to Subledger Accounting? Did you know R12 actually does some fairly heavy data conversion for SLA as standard? Are you confident it will all work with no need to reconcile? Are you confident that the new way that R12 handles accruals will work, as you actually have to run programs to build all the Accrual data………If so have a look at our Risk Register…..we didn’t spend 4 months reconciling and talking to the guy that had written this (in ironically Redwood shores…..those conference calls time-wise between Asia and California, weren’t easy) and apply multiple patches because it was working first time or second time or third time or fourth time……..Again have a look at the risk register from us. Or how about your AP data being converted? Again R12 does some very serious data conversion in the background. Try running the AP Trial Balance and see if you can reconcile…I didn’t spend two months just trying to talk to the lovely girl at Oracle Support to get a date from her….there honestly was a problem with the AP Trial Balance although I did close the SR immediately once it was transferred to a guy……..And did you know all your supplier data is converted by Oracle’s Upgrade process to TCA……..

The good news is that newer versions of R12 are a lot better, largely thanks to “suckers” like me that spent months with Oracle making all of the above data conversions that the standard R12 does in the background work. Of course until you check and reconcile all this for yourself, how confident do you really feel trusting some guy on a blog that claims to be a prophet yet never picks decents stocks………?

On top of all that resource management across your R12 will be the usual nightmare. Competing projects, production issues and users screaming for enhancements of R11. Key Project People walking off to lucrative new contracts and positions. Lack of R12 experience in the market. The time delays finding and hiring people to replace those that walk off. Dealing with unexpected issues that you didn’t plan for that have a catastrophic effect on your resourcing across the project (try cleaning up 3,000 banks when you find out what Oracle does to these during an R12 Upgrade….)

We’ve pretty much covered the testing phase above but just a quick recap to really scare you…..

Testing misses requirements or scenarios

Testing is incomplete by Project Team

Testing is incomplete by Business Team

Testing is delayed or lacks resources for Project Team

Testing is delayed or lacks resource for Business Team

Testing integration is delayed by dependencies

Test Plans are poor

Test Scripts are poor

Reconciliation of Ledgers and Subledgers is not carried out

Test Coverage is poor

Numerous Patches are required disrupting Test Cycle

All of the above can wreck an R12 Upgrade. It’s easy to miss key scenarios. Test scripts can run into thousands. Integration testing can have hundreds of points including external parties making it very complex indeed. User testing is beyond your control in many respects, especially for those that see no need for an R12 Upgrade. Reconciliation of ledgers and data that R12 has converted in the background is a significant task. Patches create havoc on internal test cycles and formal UAT. Remember you can’t just get each group of users to test each module. Each module talks to the other modules. Data from one module flows to the other for the next scenarios. PO needs Inventory. PO feeds the receipts. AP matches on PO receipts which then feeds to assets. IRecruitment flows to HRMS which then trigger hiring which then triggers Payroll which then relies on Advanced Benefits which then drives Self Service. The beauty and wonder of a fully integrated system makes testing co-ordination far from a stand alone list of test plans for each module………Managing an R12 in terms of testing is an extremely complex and challenging endeavor. Try doing that across 45 business areas and 27 countries, each speaking a different language……What’s Halloween in Spanish……??????

With an R12 Upgrade, vendor management is unavoidable…..

Software has significant patches

Software purchased is unstable

Hardware Vendor provides poor support

Software Vendor provides poor support

Poor quality in Product (bugs)

Software purchased is recently released

You are going to spend a significant amount of time dealing with Oracle Support. Depending on who you get on your Service Request can really screw you up big-time, as quality varies significantly. Get the right person (such as the lovely Indian girl we got on our Trial Balance problem, ahhhhhhhh…….) and you could get an immediate response. Get the wrong person and you could see weeks or months wasted, with the “apply this next rollup patch, we’re sure one of the thousand+ fixes will fix your minor problem and retest your entire R12 again” attitude.

R12 was highly unstable in 2008. 2009 saw it got better. By now reports are it is solid……….but I wouldn’t make that assumption, until I had actually implemented it and got my UAT signed off 100%.

When you implement R12 you may need to upgrade your database. (most will). When you upgrade your database you may need to upgrade your Operating System. When you upgrade to R12 you may upgrade or replace your servers, depending on where they are in their lifecycle. Now companies don’t generally have massive dealings with their hardware vendors (as Unix/Linux is incredibly reliable). But during an upgrade that relaxed relationship is going to be potentially very heavily tested. Question is, can the vendor provide strong support, resolve issues quickly and get you safely onto a server that is the very foundations of your business? If you fail to install the correct O/S plus interoperability patches you could see your business crippled…….It’s funny that the Infrastructure Manager that you were fighting with two weeks ago, kind of holds your entire career in the balance and you don’t really know anything about Servers, questions to ask, etc so are 100% dependent on him for your project to succeed.

Just wondering if your R12 needs more resources in terms of CPU, disk, memory, etc? Have you checked? It would be a shame if the server collapsed on day 1 of go-live.

Lastly the million dollar question. Oracle is on R12.1.3 at time of writing. Do you implement this version (and be the one to find the bugs as no one else is using it) or do you go for say 12.1.1. At this stage of the R12 lifecycle I’d go for R12.1.3 but then again you will get some additional pain of being one of the first. Obviously by the time you get to UAT, 12.1.4 will be out……….then do you jump to that or perhaps apply that new ATG patch or a RUP Patch that has come out as it has some really needed functionality……..All of these create massive risks not just for the stability of the project but also your budget, timescales and resourcing. Larry Ellison is a little devil….always releasing new ATG, Security or RUP Patches just at key points in your project, just to tempt you and cause major indecision in your mind. Question is, is that patch a poisoned apple that will drop your project at a key time??????

All hail Larry Ellison. The True and Only Oracle Prophet !!!!!

Happy Halloween. The clock is ticking down to the midnight hour. Before Halloween is over, I will scare you. The witching hour is coming with Transitioning from your R11 to R12, the actual cutover and post go-live, with all the reference material and a prebuilt R12 Risk Spreadsheet as a treat for all the kids in the next article……

Delivered on Halloween !!!!! Enjoy………..


See the other Prophecies including Business Intelligence and the Kung Fu Dragons of Wudang at



Tags: , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: