Archive for the ‘R12 Upgrade’ Category

The Quest for the Holy Grail of R12 Patch Management

April 26, 2011

With new technological innovations, please wear 3D Glasses and Headphones for optimal experience whilst reading this blog. This represents the first ever blog to cover ERP Columns and Movie Columns, together with hot off the press exposes that Oracle Technology came from Alien UFO’s captured at Roswell, all in 3D and Dolby Sound. Popcorn and a coke are recommended.

Ensure your PC sound is turned down if in the office with no headphones.

In the time of King Arthur, legend had it there was a magical cup, a holy grail.

Whoever drank from the cup would have immortality. During the time of King Arthur, many knights embarked on dangerous quests to find this magical grail, always to fail. The question is, were those pursuing their quest searching for something, that perhaps did not actually exist?

Sometimes it feels like Oracle Patch Management is perhaps a search for the holy grail. To be able to stay current on patches, to be able to apply patches and still ensure that no adverse effects are inflicted on the Business seems an almost impossible quest. This column looks at some of the strategies that can be used, to perhaps not achieve the holy grail, but certainly move towards the holy grail of patching ERP.

Now previously some crazy oracle prophet had written about the Art of Zen Patching . Let’s be clear on this – the quest for the Holy Grail of R12 Patching is not such a zen-like endeavor. It is an arduous, painful and often disappointing journey, frought with failure, despair and danger.

And at this point I feel a movie review is appropriate.

One of the most famous movies (and one of my all time favorites EVER) covering the times of the Holy Grail has to be Excalibur. Highly recommended on DVD. A superb and very haunting movie. It is one of the most atmospheric movies I have ever seen. You just have to see it.

So the movie is about a search for something that people dream of but quite possibly doesn’t exist. It’s about human dreams that become all consuming leading people to eventual madness and despair.

For many years I have been searching for what can only be described as the Holy Grail of ERP Patching. I started off pretty sane, but as the years went on my sanity deteriorated until I started writing columns combining movie reviews and Oracle ERP tips. I tried to turn to Zen (The Art of Zen Patching), but lately, just maybe I found the Holy Grail of Patching. I uncovered this during an R12 Upgrade Quest. In the midst of battle, with patches coming out daily from Oracle (on 12.0.4) we perhaps found not a perfect grail, but certainly something that made the quest worthwhile.

As I explained in a previous column there are a few patch scenarios:

1. Security Patches, Database Patches and ATG Patches on a Production System. This column could be used for that, but I’d also recommend the Art of Zen Patching for these types of patches. These patches are generally stable (if left to others to apply first) and cause less problems from experience. Note however we HAVE had problems on each of these types of patches, so don’t underestimate the fact that the quality is better that you can be sloppy on your testing.

2. Patches that come out during an R12 Upgrade Project or need to be applied. I guess this column kind of applies, although depending on how far your upgrade is you can take a more aggressive patch approach. If you are early in an upgrade, you generally throw in the Rollup Patches or others as needed. The later in the project you are the more careful you need to be.

3. The Holy Grail of Patching refers to the approach taken for Patches on a Production System. It is like searching for the grail, a very slow and dangerous process, with little in the way of rewards.

As everyone knows Oracle has patches to fix the patches you applied. This isn’t good, because like Forest Gump, Oracle gives you a box of chocolates, and your never sure if there is something unpleasant in one of them…….Or as I say, our office coffee machine (I live in the Far East) is like an Oracle Patch. Occassionally you get a cockroach dispensed……..(and no I am not joking on this one – cockroaches like Nescafe too…….)

Our approach to the Holy Grail of Patching can be described as a highly conservative and disciplined approach, based largely on the principles of ITIL. Our primary aim is not one of speed, but of keeping our Production Systems safe and stable, to avoid major business disruption. It is a very slow and very painful process.

So let me walk you through the process (Apologies, it’s going to get a bit serious from here on in. Have a look at the Alien Technology blog if you need something lighter……Oracle Fusion finally went live in one company. Poor buggers………Although it is based on captured Alien Technology from Roswell……………so I am sure as a first release of an Oracle product quality is great as it always is……. )

The bug process starts either with the EBS Support Team (I assume you are proactively look for bugs/problems before they happen, by keeping an eye on Oracle Support Updates) or the business user reporting an incident. The Incident Management process should record the details and check if it’s a known problem or there is a workaround.

This should then be passed to your Problem Management Process, where you investigate the problem reported (DS10B-001).

You should then try to replicate the problem (DS10B-002).

If you can replicate, then it’s time to log on to My Oracle Support (DS10B-006). (I’ll slowly introduce some ITIL Terminology into this column. ITIL is a framework to manage IT Systems and it’s very, very good indeed, applicable not only to ERP but to any IT activities. If it was a movie I’d give it a 10 out of 10, but it’s a bunch of great books that can be found here. Introducing these ITIL concepts can seriously improve your service to your ERP users.

And onto another movie review. It has to be Monty Python and the Holy Grail. This is about as British as humor gets. A great movie and very, very original. It’s a 9. Get it on DVD.

On My Oracle Support you should check for similar bugs. It’s a great database and no doubt someone else has hit your bug before you (unless you are one of the suckers to take an early release).

You’ll get to one of two points here. You’ll either identify a patch (which may have pre-reqs) or you’ll come up blank. Lets deal with the second point.

If there is no patch (and don’t be lazy not bothering to look) then raise a Service Request (DS10B-012). Now Oracle’s game isn’t necessarily to help you fix the problem………..Typically responses are as follows, but we suggest you counter-act these by using phrases from the movie Monty Python and the Holy Grail. Assuming Oracle Support doesn’t hang up, this will certainly be a tactic they have not seen from other companies………..

Oracle Support – Good Morning. How can I help?

You – “Go and tell your master that we have been charged by God with a sacred quest.”

Oracle Support – We’re very sorry to hear that.

You – “Everytime I try to talk to someone it’s ‘sorry’ this and ‘forgive me’ that and ‘I’m not worthy’!”

Oracle Support – Sorry Sir, could you explain the problem with your system?

You – “Come and see the violence inherent in the system!”

(We recommend you explain the problem, maintaining the Monty Python style and English accent – pretend you are an old English queen)

Oracle Support – That’s standard functionality Sir. The accounting on the Payables isn’t supposed to do proper exchange rate variance calculations, otherwise your General Ledger would be correct for the 30M Dollars you spent on Oracle. Now that sounds perfectly plausible, doesn’t it Sir?

You – “Your mother was a hamster and your father smelled of elderberries.”

Oracle Support – Sorry Sir, let me just check (By this point Oracle Support is a little concerned)

Oracle Support – Ah, it’s a known bug. It’s in development………..should be out in six months…….

You – “I’ll bite your legs off!”

Oracle Support – Wait Sir, Wait Sir, let me check. I’ll check again. You need to go to R12.0.6 (By this time the support person is probably thinking “nutcase on the phone, better help him quick”……)

You – “Jesus Christ!”

Oracle Support – This small patch will definitely fix your problem !!!! (Now Oracle Support knows this has nothing to do with your problem but ask you to apply it anyway just so you get off the phone and they can close the call………)

You – “Pull the other one!”

Oracle Support – OK we suggest you apply the latest Rollup Patch

You – “I fart in your general direction.”

Oracle Support – OK how about this patch, which has 300M of Pre-Requisites. That will do right?

You – “Aaarrrggghhh!”

Oracle Support – What about a one off patch that no-one else has that will trash your system and is cobbled together by our developer?

You – “You tiny-brained wipers of other people’s bottoms!”

Oracle Support – But we’ll need to spend 4 weeks of burueacracy giving you a hard time to justify our development team doing this for you, even though it’s a serious bug we should fix……..

You – “You can’t expect to wield supreme executive power just ’cause some watery tart threw a sword at you.”

Oracle Support – Wait I’ve actually done my job and found a correct patch………

You – “I feel happy!”

A very Monty Python approach, but every scenario above represents some actual conversations with Oracle Support……(bar the Monty Python quotes – I normally pretend to be Harry Potter when talking to Oracle Support).

They may also pretend it’s a problem for another team (e.g. AP blamed the SLA Team and vice versa) or point to the good old technology stack components or database components. Common tactics by Oracle Support.

There’s a very serious point to the above. It shows the strategies that Oracle Support uses against you when it comes to patches. Your end game with Oracle Support is to get you to the minimum patches required to fix your problem. You and Oracle have very different aims sometimes (Oracle’s current aim is to move everyone onto minimum of 12.0.6…..), so you have to be very careful in how you manage the Oracle Support Analyst because that seriously affects your patch outcome and therefore testing work and eventual risk to your system.

And it’s got to be time for a movie review. This is another favourite Indiana Jones and the Last Crusade. Great movie for all the family. It’s a 9 out of 10 and keeps our Grail movie theme moving along nicely. This makes me feel old, because when the first Indiana Jones movie came out I was 10. Now Indiana Jones (Harrisson Ford) is a pensioner…….

So you’ve got that patch sitting on your server waiting to be applied. Do you feel lucky punk? You’ve reached DS10B-016 in the Patch Management Process.

A quick excerpt from a previous conversation I had with a person in one company:

Techie Guy -We need to apply the payment patches this weekend.

Oracle Prophet – Why?

Techie Guy – My manager told me to

Oracle Prophet – What did he say?

Techie Guy –“If it’s not done by sunrise, I’ll cut your balls off.”

Techie Guy – So we had real problems with the application of patches in a Test System that knocked out all Payment functionality.

Oracle Prophet – So you’ve never had a clean run of applying this patch?

Techie Guy – No, that’s right. Never been applied correctly in any test system. (Stated not with concern, but an unbelievable stupidness that is rarely found). So can we go ahead and apply it in Production this weekend?

Oracle Prophet – “I’ve got no option but to sell you all for scientific experiments.”

This was a true conversation of one person considering applying patches that he had never gotten to work during a testing phase… our Global Payments Systems……….truly a new way of testing patches. “Yep it didn’t work in any test environments, but maybe it will behave differently in Production when we apply……….”.

Our first step for Patch Application is normally to look at the size of the patch. Now a lot of people say size does not matter – many woman beg to differ. In this case, the smaller the better – the patch I mean. It gives an indication of just how many objects it’s going to hit potentially. Not scientific but a quick check gives a reasonable impact assessment. However……….

Applying patches is about a horrible and painful quest with little reward. But going back to King Arthur the Oracle Prophet recommends you seek a mystical wizard. Yep, your thinking I’ve lost the plot again on this one………..Just trust me. Go into your Manager, now obviously your credibility is shot to pieces after suggesting using the Art of Zen on corporate systems. Obviously he thinks you are a bit mental after suggesting using captured alien technology for deployments of ERP customizations.

But trust me, when you tell him you should seek out a mystical wizard to improve patching, you will truly have made those other suggestions look almost coherent………

So King Arthur had Merlin as his wizard. On your quest, we suggest that you take a look at the Patch Wizard from Oracle. (I surpass myself in corny literally prose this time……..)

Oracle has had a Patch Wizard for a considerable amount of time. I am not here to teach you all there is in this (see the excellent authors at the end of this column). I am here to raise awareness of these tools, what they do and how they fit into the Holy Grail of R12 Patching. The Patch Wizard can be found under the System Administrator Responsibility.

Once we have assessed the patch size, read me, etc, we normally try to apply the patch on a temporary database (we usually have one we can apply on). (We’re now on DS10B-007 in the Process Chart previously).

The Patch Screens below are a very useful impact assessment tool. Now size isn’t always an indication of how satisfying the patch will be for you………After applying the patch, bring up the Applied Patch screen and query the patch applied.

So a patch may be 300M. But it may not apply many files, if your patch levels are already recent.

Looking at the files copied and Action Summary will give you a very good idea of just how satisfying that big patch has been for you.

If a patch shows that it is replacing database objects, forms, etc then it certainly has had potentially an impact on your system.

We’ve faced large patches, but when looking at the files applied, we may only end up with a handful of files. This allows us to then use our technical guys to put a reasonable assessment on roughly what we have to test.

With an idea of what we have to test, we can now draw up a reasonable test plan for the patch. Without the patch screens we would be looking at guessing the best approach to testing. Now it is not easy to work out from the files exactly what to test – you need to be careful in that respect.

Now a message to Oracle. Why don’t you enhance the Patch Wizard to do a mapping between code files and Functionality. This would allow Functional Users/Support Teams to far more easily assess the patches, then draw up a far more accurate test plan. This would be an excellent enhancement for Patch Management for companies globally.

One of the problems with patches is where they overwrite your customizations (customizaton by modification). I suggest that where you modify shipped Oracle code to make new customizations, you record these in the appcust.txt file. To quote the manual “Oracle Applications uses this file during patch processes to generate warning messages that customizations are being overwritten or may need to be replaced after the patch”. It’s a handy tip. A missed customization that is overwritten by a patch can cause immense damage later……….

Our approach to testing patches is comprehensive.

We apply the patch onto a DBA Instance. (Step DS10B-008). This lets the DBA’s check the pre-reqs, make sure the patch applies cleanly, makes sure objects are not invalidated, etc. We then do a very quick sanity test.

We apply the patch onto a PTH Instance. (Step DS10B-009). Our test here is not thorough, but gives us confidence that the patch has fixed the problem and does not break any other key functionality.

We apply the patch onto a DEV Instance. (Step DS10B-010). Testing becomes more thorough at this point from our internal team. (Make sure it is a clean DEV instance, else use another clean environment). Once complete by our IT ERP Team, we release the Patch to our users.

We also test critical functionality when applying a patch. The drop dead functionality such as entering an invoice and paying. We also do an end to end around the patch when necessary (e.g. a quick cycle through procure to pay). Testing patches well in the IT area of ERP ensures quality patches are delivered to the Business. The messy dealings with Oracle and cycle of patches to fix patches is kept out of the Business Users view, as sometimes this could undermine confidence in the patches themselves otherwise.

We apply the patch onto a TST Instance. (Step DS10B-011). Our users then independently test this patch, using their own test plan and test cases. Keeping the testing independent (separate IT and Business Testing) gives more chance of any issues with the patch being caught.

For testing patches, we have suites of comprehensive test scripts/test plans across our application. We built these up as part of our previous R11 and then R12 Upgrade. This allows us to test our scenarios easily and quickly. It also covers integration to other systems. With freehand testing, formal test script usage, business user testing and IT Testing, we avoid mistakes.

With sign-off from the Business Users (DS10B-018), the patch is passed to our Release Management process for scheduling into Production.

With the use of the Patch Wizard and Patch Screens in R12, our testing becomes much more targeted, minimizing testing effort but re-utilizing our test script library selectively.

By having multiple databases with multiple testing cycles from both business and IT, our testing becomes much safer, reducing risk of a bad patch reaching production. It also protects other environments including your Development and Test Instances. You wouldn’t want to wreck these with a bad patch usually.

There are a few options we are looking at to further improve our Patch Management:

We are looking at Oracle Configuration Manager as a potential to alert us to patches that need to be applied, moving to a more proactive patch management policy.

We are looking at Testing Automation. Our preference is the HP Tools with pre-built testing scripts for ERP, but we know there is a huge amount of work before we could get to proper automation. Oracle also has a Testing Tool, but I still prefer HP’s to be honest. Automated Testing does seem to be the holy grail eventually – to be able to press a button at 5:00PM, have all the test scripts executed and come in at 8:00AM and see the results really is something that I think companies should aim for. However it’s expensive, difficult and often a huge failure when companies try to do this.

Finally we just bought Oracle RUEI. I have written about this in the past. It is a support revolution. We can now see the errors that users are encountering, the system performance that they are experiencing and so much more. Again it’s a move to a much more proactive incident/problem management. It even has an option to playback user errors. As said no more having a helpdesk asking “And what step did you do next. What data did you enter, blah, blah, blah and failing to take the right information for the Support Analysts”. It has pre-built integration into ERP and Oracle had this up and running for us in a matter of days. It even works with our custom modules……… I’d recommend this product to anyone running ERP. It’s a super product, quite revolutionary in terms of support.

And the final movie recommendation. Dan Brown’s Da Vinci Code. I read the book (also highly recommended) on a long haul flight to the Far East. Then I watched the movie on the TV a few years later. The movie is pretty interesting and I’d say it’s an 8.

Personally I always thought Mona Lisa looked absolutely miserable……

But Dan Brown’s book caused a huge amount of controversy. It’s a full circle from where I started the column. Many looked for the holy grail yet never found it. Dan Brown’s book was based on the premise that the Grail wasn’t really an object, but the grail was actually a person.

I’d say that to find a Holy Grail of ERP patching isn’t perhaps looking for an object or a magic bullet. I’d say that the Holy Grail of ERP Patching is also, actually a person. A person that puts in place the strong processes, controls and procedures, whilst thinking smart to reduce the work to enable a safe and reasonable patching strategy that supports the business goals. That is the true Holy Grail of ERP patching.

One last question – why are you wearing those 3D Glasses? You look pretty stupid in those in the office, given this is just an ordinary web page……..

And to end on an equally silly Monty Python note:

Woman -“He’s the Messiah !!!!”

Oracle Prophet – “I am not the Messiah!”

Man – “He’s not the Messiah. He’s a very naughty boy.”

Until next time when we look at R12 Patching – Friday 13th – A Horror Movie Special.

Further secret blogs and prophecies can be found at:


This column was based on some very hard-won experience. However other parts of this column came from research from generous authors in the Oracle Community. Below are some excellent articles that further shine light on the difficult process of Patching ERP safely.

Oracle E-Business Suite Patch Wizard – Path to Less Errors

Release 12 eBusiness Suite Codelines, Codelevels and PatchWizard

The 11i Patch Wizard

Managing R12 EBS

Tips and Tricks for Patching R12 EBS

Best Practises for Patching and Maintaining EBS R12

Real User Experience Insight

Global Thermonuclear War – The New Oracle R12 Feature

January 18, 2011

This column will be short and sweet, explaining how someone can launch Global Thermonuclear War on you, completely wiping you out. Nice topic and not any jokes this month I’m afraid for such a serious subject. Of course the movie reviews will still be included, otherwise I’d lose my rating of the only combined Oracle ERP and Movie Rating Column on the Web……….

Now Oracle ERP has grown immensely over the years, adding module after module. Perhaps this column is about a new module that controls nuclear missiles? Computers (and Oracle ERP these days) seems to control everything else but thankfully Oracle hasn’t quite got to the point of having a module to do this. Worrying if they ever did, given the number of bugs in the early R12’s.

Oracle Support: Good morning. Can I help you?

User: Yes, we implemented Oracle’s Global Nuclear Missile Control module in Fusion Apps and it’s launched a nuclear missile accidentally against a large city that will kill millions in two minutes from now.”

Oracle Support: Yes, that’s a known bug. We will work on a patch and get back to you in a few days.

User: The missile will hit in two minutes. We need to escalate.

Oracle Support: We’ll have the duty manager phone you within 3 hours. Goodbye.

Ironically we’ve had a few Severity One Service Requests that make this conversation horribly familiar………

But what would happen if a hacker managed to get into your ERP system? Access to your Payroll. Access to your Financial results. Access to your HRMS System. Access to Payments. Imagine a hacker being INSIDE your system. Have a look at the column R12 and the Auditors from Mars. That will give you an idea of the horrible consequences of someone being inside your systems………..

Which brings us nicely to a movie recommendation. A hacker inside the system? It has to be the original Tron which I would give 9 out of 10 for it’s vision, way ahead of it’s time. The more recent version was OK, but lacked something I felt. But still worth seeing.

Now to get down to the serious business.  R12 did have a rather nasty payload, of thermonuclear proportions. I don’t normally write (or disclose) hacking vulnerabilities, but given this is already out on the web and represents a serious threat to you, I thought it now appropriate to warn everyone about what is a real global thermonuclear device, just waiting to go off in your ERP System with potentially catastrophic results.

In R12 a JSP file was shipped – jtfwcpnt.jsp. This JSP takes a query that executes against your database opening you up to SQL Injection based attacks………..Now let me see if I had access to an ERP Database as a hacker where would I want to start…….??????

I am not going to go into the details of how this is exploited, but you should strongly check if this file is used and then remove it if not. This warning is applicable for anyone who is using products such as iRecruitment, iSupplier or other DMZ based products in R12. (although an internal attack could equally be done).

This vulnerability seems to be across all R12 releases judging by other reports on the web. (We’re currently upgrading R12.1.3 and will be checking this also shortly).

This file represents a very serious risk to your entire ERP and therefore to your company

And to end this rather serious column, we need a movie recommendation. Well that has to be the original War Games movie. A great story from 1983 and decades ahead of it’s time. It’s all about how computers controlling everything internally are accessed from external sources to almost start a nuclear war. For me given it’s relevant 30 years later, it’s a 9 out of 10.

There’s quite a number of very serious points to this column.

We have to wonder what Oracle was doing shipping stuff like this, whilst busily shipping security patches quarterly. I am just utterly stunned this ever got out as part of the shipped R12 product.

I’d also suggest that companies start looking very seriously at security of their ERP, especially those running products in the DMZ.

Review the papers on Metalink on the best practises for DMZ. Review Steven Chan’s column as there is always great information, but most of all, out of this learning experience, google regularly for security vulnerabilities on the web about R12 or R11 – I know most people don’t do this, which is why I published this vulnerability in this column. Solution Beacon also provides some good security information. Also make sure you have decent firewalls (Oracle has released a new product just recently) and software to protect against SQL Injection and other similar attacks.

Also do keep your ATG and Quarterly Security patches up to date. I know how difficult that is, but it is critical. (A previous security patch closed a hole in iRecruitment that could be exploited from outside). See R12 Patching and the Art of Zen for an approach that makes this less painful.

Security is very much a multi-layered approach and your ERP needs heavy protection like any other corporate system. (and arguably even heavier than most).

The hacking days of Windows and Internet trojans will continue as they have done for many years, but there’s a new age of hacking dawning and there is a real awareness from hackers on other areas, and that now includes ERP Systems such as SAP and Oracle.

This is a real wake-up call in terms of security with ERP and I hope that everyone really starts looking at ERP security as a priority in their companies, over and above anything else.

The dawn of the ERP Hacking Wars is beginning……

Further Prophecies can be found at

The Oracle Grinch Who Stole the R12 Christmas

December 30, 2010

Every Who Down in Whoville Liked Christmas a lot…
But the Grinch, Who lived just north of Whoville, Did NOT!
The Grinch hated Christmas! The whole Christmas season!
Now, please don’t ask why. No one quite knows the reason.
It could be his head wasn’t screwed on just right.
It could be, perhaps, that his shoes were too tight.
But I think that the most likely reason of all,
May have been that his heart was two sizes too small.
Whatever the reason, His heart or his shoes,
He stood there on Christmas Eve, hating the Whos,
Staring down from his cave with a sour, Grinchy frown,
At the warm lighted windows below in their town.

The Grinch was a particularly mean spirited creature, who’s prime aim was to make sure that everyone had a very miserable holiday season.

This will be a short but important column for the “Heroes of R12”, the companies that adopted R12 early, took all the pain and whom Oracle even honored as the best R12 Upgrade sites. To all those companies that took the encouragement from Oracle to upgrade early to R12.  Or to the 95% of companies that didn’t, the suckers that found out all the bugs.

Now every knows the story of the Grinch that hated all things Christmas.

Well as much as I love Steven Chan’s columns, Oracle gave a nasty Christmas surprise for all those R12 early adopters.

From 2012, February 1st, Oracle R12 will move from Premier to Extended Support. Great no problem, as Oracle has always kept extended support for all those on the relatively new versions such as 12.03, 12.04, 12.05 (OK not totally recent but still on recent releases than most companies)

Unfortunately as with R11, Oracle has now added a particularly Grinch like clause onto R12 Extended Support…….

“To be eligible for Extended Support, [EBS 12.0] customers will need to have applied at minimum the 12.0.6 Release Update Pack (Note ID 743368.1) and the Financials CPC July 2009 (Note ID 557869.1).   Additional minimum requirements for Oracle E-Business Suite Release 12.0 have not been finalized yet. “

So your options?

Upgrade to 12.0.6, but that’s going to be a significant upgrade to do, so may as well go to 12.1.3.

Upgrade to 12.1.3 (which is the most sensible, or 12.1.4 if it comes out in Q1/Q2 2011)

Or take your chances of not getting support (sure I fancy that one as I still have accounting issues and AP transactions getting stuck so no support will leave me in real trouble, but applying patches is massively risky and my users will scream at another upgrade in a 2-3 year time span).

Anyway the full story is here (and don’t blame Steven, he’s just the messenger), but I can’t help but feeling that Oracle has really screwed those early customers that made R12 the success and stable product that it now is.

Head’s Up – Preparing for E-Business Suite 12.0 Extended Support

If there’s a Grinch this Christmas, it’s definitely sitting in Oracle HQ, Redwood Shores and it’s delivered a very nasty Christmas present indeed to R12 early adoptors.

That gives you 13 months to get off R12.01-12.05 and move to R12.1.3. That’s not long to get the funding, get the support, get the project running, deliver it to users, through UAT and into Production.

And for the movie recommendation? I’m afraid I can’t recommend the Grinch (movie poster below). Sorry, but it sucks. On the plus side there were some good movies this holiday season including Harry Potter and the Deathly Hallows Part 1, Narnia – The Voyage of the Dawn Treader and Tron was kind of average. The Little Fockers is coming out and the same words come to mind to describe my thoughts on R12 Support and Oracle……….

And on that note, I’d just like to wish everyone all the very best in 2011. I hope it is a successful and prosperous year, for all those doing R12 Upgrades (and perhaps a few Fusion Apps implementations).



R12 Deployment using Captured Alien Technology from Roswell

December 26, 2010

On June 13th 1947, residents of Roswell reported seeing bright lights in the sky, moving at remarkable speed and then a huge explosion. 1947. The next day Mac Brazel found the crash site, and the biggest coverup in human history began. Today I am pleased to reveal a WORLD EXCLUSIVE of what really happened and how that technology worked it’s way into Oracle technology you use today. I promise with all the thousands of Oracle blogs on the internet, you will read this nowhere else……..

The crashed  UFO was taken to Area 51 (the photo above was smuggled out and is 100% genuine we believe)  and housed in a large top secret highly restricted warehouse. For over two decades scientists tried to break into the UFO’s computers with no success. From this 200 million dollar research program funded by the US Taxpayers, the only success was the discovery from the UFO of how to make non-stick frying pans. It was at this point during the famous alien autopsy that scientists also found out that the famous grey aliens keep their brains in their ass, explaining their relatively low intelligence compared with the sentinel aliens. Further there is strong evidence that a recent US President was indeed an alien, as many stated “he had his brains in his ass”.

At the same time, a bright but underachieving student named Larry Ellison, was flunking university with some style. In 1964 the government inadvertently invited Chicago students (including Larry) to the highly secure, non-existent area 51 for a field trip (Normally intruders are shot on sight but a clerical error led to 100 students, including those with far-left tendencies to be invited in). During this field trip, as the scientists were showing the non-stick frying pans Larry lost interest and wandered off, stumbling on the UFO (shown above in the 100% genuine photo). He started to mess around with the UFO computer and stumbled upon tape drives, which the 200 Million dollar research program scientists did not understand just what these were for almost twenty years. Luckily Larry had 20 large 10 inch tapes with him and he downloaded the entire UFO software. As he walked out Larry thought the game was up as security stopped him. “Do you need a hand with those tapes young man?” said the security guards. The kind guards got these dropped off at Larry’s house from the highly classified, non-existent area 51 site.

Larry spent the next ten years working out exactly how aliens were using advanced software technology using these tapes, before formally starting Oracle in 1977.

The truth is that Oracle software is based on alien technology

Now using alien technology was not all plain sailing (Larry can you stop spending so much on this sailing stuff – great pun and literally transition eh? – and start paying more in dividends….???). When Larry used alien technology to create Oracle Apps 10.7SC he did not realize that the aliens had tried this exact same strategy on their home planet of Ursula Minor Quadrant 5 with an ERP product called Grey Apps SC (note the naming similiarity) and everyone was laughing at how crap it was………The same thing happened here on earth to Larry’s home grown 10.7SC version. Larry eventually apologized for this ERP Release.

However the Oracle Database, ERP and now Fusion (even sounds alien) can now be revealed as being based on the software from the captured UFO in Roswell. And of course with such software, other vendors such as IBM and SAP (that 2 Billion dollar fine must sure have hurt), offering vastly inferior software are doomed….you may as well sell their shares now. It’s interesting to note it took a team of Oracle scientists a further 20 years to break the alien tapes for Fusion software. Oracle claims to have thousands of people working on this but this is just disinformation. Think about how little information is out there. Why is Oracle so secretive on Fusion Apps? When it does come out look for the alien code, that has not been taken out amongst the 50 million lines of source code.

Fusion Apps was delayed because Oracle couldn’t break the alien code. Fusion Apps is actually being developed not by a team of thousands as Oracle claim, but by a single programmer named Steve in his bedroom using the captured alien technology from Roswell.

And I feel a movie recommendation coming on. So captured alien technology that humans then re-engineer for their own ends? Well this one absolutely fits the bill. Terminator was a classic (as were all the Terminator movies) and they score a 10 out of 10. All are great movies to rent or buy.

Interestingly the new Exadata is similiary based on blueprints from those tapes. When you turn one of these Exadata servers on it uses less power than a lightbulb (it has alien fusion power sources and chips and a special plug from Home Depot). When you turn on the IBM servers with the same performance, all the lights in New York go out as it drains the power grid (allegedly according to a guy I talked to in a bar at 2:00AM. The lights of the bar went out and that’s when he stated someone turned on the IBM server again……..). It’s pretty conclusive proof of the use of captured alien technology from Roswell.

At this point I’d just like to say I don’t believe in conspiracy theories, or alien abductions, etc. So if you’re some UFO nut that stumbled on this column and think this is all true, please don’t stalk me for photos and email me for further stories, as I’ve got good lawyers and I’ll take out a restraining order on you……..Now onto the serious business of the column – Teleportation. (Your thinking I’ve really lost the plot today right? Stay with me on this one for one more paragraph)

Look at the definition of teleportation. It breaks down physical structures and moves them from one place to another. I’d like to claim the Nobel Science prize for showing we already have this today and finally get to the point of today’s column (some are happy because they get hopefully useful information after reading like 10 pages of crap, others probably think here come the boring stuff and it was amazing Oracle based all their products on captured Alien Technology….we suggest those people get a life…….). Don’t worry, I’ll try to keep it interesting as I’ve absolutely got to do some sci-fi movie recommendations given the title of this column right?

So the topic for today is migration of all your R12 setup, customiztions, etc. (Well think about it. You take physical information, sitting on a disk, transfer it over a network which might even be some fibre optics channel, so actually converting physical info to light… that is super advanced ….. and then reconstruct on the other disk. Now that’s the very definition of teleportation).

Now there are two approaches. Either you manually do this (and risk all the screw ups) or you use teleportation. I’d love to see your bosses face as two weeks ago you were telling him of Zen Patching (See the article on the Art of Zen Patching) and today in your management meeting when you are asked how to solve deployment issues you suggest TELEPORTATION with a straight face, which you got from some guy called the Oracle Prophet on the web…….

When you do an Oracle implementation, or an Upgrade, or a new module or a single project or even a simple customization, there is potentially a lot of information that needs teleported. I wonder how many people have been fired as a result of my columns so far……

So today is a high level overview of the options you have to safely move the various components around your R12 instances with the minimum of fuss and the minimum risk. This is going to be a handy column for all those doing an R12 Upgrade in 2011 (given Premium Support has ended and R11 Support is dependent on some nasty small print of applying minimum patch levels…….)


Lets start with one which I hope just about everyone knows. FNDLOAD. Now this has been around since the first aliens came down and built the Pyramids……..So to quote Oracle Oracle Applications System Administrator’s Guide – Configuration Manual:

“The Generic Loader can download data from an application entity into a portable, editable text file”. How cool is that? You can download lots of your setup and edit it, giving you the ideal opportunity to screw things up bigtime.

But FNDLOAD is an amazing utility. With a single command line you can download an incredible array of setup including:


Attachment Setup

Concurrent Program Executables

Concurrent Program Definitions

Descriptive Flexfields




Key Flexfields

Lookup Types

Lookup Values




Printer Styles Definitions

Profile Options

Profile Values

Request Groups



Value Sets

(and a few more not documented………)

Using FNDLOAD is a breeze. Simply type in a one line command to download and a one line command to upload. Now if your doing 1,000 Menus across 12 Project Instances during Implementation, the time savings are vast….

FNDLOAD apps/apps_psswd 0 Y Mode configfile datafile entity [param]

It really is as simple as a single one line command.

A great blog entry on the details can be found at Anil Passi Blog. This gives a much more detailed overview than this column can and is a great blog with lots of other info too.

So which movie made teleportation famous? Well that has to be Star Trek, which started way back in 1966. Now I’m not a Star Trek fan, but the the 2009 Star Trek Movie was pretty good and I’d give that an 8 out of 10. Definitely worth a rental on DVD in 2011.

Generic File Manager Access Utility

Well here’s one I’ve never even heard off – Generic File Manager Access Utility. Now in a manner befitting Star Trek and during research for this article I came across this and we should boldly go where no man has gone before (Note Star Trek finally became gender aware and change this to a more acceptable politically correct sentence…..)

As said can’t say much on the Generic File Manager Access Utility as never used it, but its covered again in the Oracle Applications System Administrator’s Guide – Configuration Manual. From what I can see it’s used to transfer Help Files between Systems, but as with everything in an Oracle manual, a lot is left unstated………

Oracle Alerts

Now Oracle has the ability to define Alerts across pretty much any business area. These are very useful and used by a lot of companies. But they can take a fair chunk of time to define and migrate. Not many people know this, but alerts can also be transferred between databases. Having a quick look at Oracle Alert User’s Guide it’s a pretty straightforward procedure:

Firstly go to the Alert Screen.

Simply choose from the Tools Menu the Transfer Alert option and fill out the screen above. Very straightforward and saves a lot of time.

Now apparently there is also an option to transfer alerts using FNDLOAD

FNDLOAD apps_user_name/apps_password O Y DOWNLOAD $ALR_TOP/patch/115/import/alr.lct my_file.ldt ALR_ALERTS APPLICATION_SHORT_NAME=”INV” ALERT_NAME=”Alert_to_download”

See Meta Link Note: 400295.1 for more details.

Now the interesting point is that you can download all your configuration files and once your happy, store them in a Source Control System such as Subversion or PVCS. Once you are ready to deploy to a new instance all the files can be automatically loaded using a Shell script (or some other newer options I’ll discuss) in a matter of minutes. Interestingly if you combine using a database flashback option and an automated shell script you can practise deployments many times in quick succession until you get it right. That keeps your Production very safe, as you will have removed all the bugs from deployment by repeateded practise runs. The Database Flashback option lets you go back to a previous version of the database very quickly indeed.

So time for another movie recommendation. Back in the 80’s, when I was a kid, I saw one of the most magical and memorable movies of my life. E.T. This movie is timeless and one your kids will definitely love. It’s funny how ET used a Texas Instruments Speak and Spell and a few bits of wire to make an intergalactic telephone to call across the Universe for help. Ironically putting a bunch of very simple tools together (FNDLOAD, ISetup, etc) can equally help you in either your R12 Upgrades or R12 Implementations………..

Data Load

So how do you spot a Functional Consultant? One way is that generally Functional Consultants have weird hairstyles. That’s a sure give-away. The other give-away is that they are sitting trying to write SQL badly so that they can get the data for their beloved Data Load Tool.

So what is Data Load? Data Load is a tool that basically takes all your data and plays it back into an Oracle ERP screen. It’s uses are endless from typing in Account Combinations, Suppliers, Inventory Setup, etc. It can work on both Core/Professional and Self Service screens. Sure there are others ways to do some of the stuff listed, but that requires heavy technical ability, and sometimes on projects that just ain’t there, so that’s when Functional guys turn to this tool, plus all the stuff that technically cannot be done without violating your support license.

Now you imagine. Even with all the “alien technology” I doubt Oracle has everything covered in terms of setup, etc. So you imagine having a tool that can pretty much load into a very large number of screens, where no API, FNDLOAD or Interface can go……….and then be able to source control everything and when you get a new environment during implementation you can simply run Data Load to create a lot of what would take you days to setup. And it’s so easy even a Functional guy can use it………

I’ve seen a lot of people rave about this product and use it during implementations. It’s definitely worth a look and you can find it at the Data Load Site.

FSG Export/Import

Apart from the funny hairstyle and dataload, another way to spot a Functional guy is that they generally are writing FSG Reports. For those that don’t know, FSG Reporting provides the ability for end users to write various accounting reports with no programming in Oracle General Ledger. Now if you’ve ever written one of these, you’ll know they can involve zillions of rowsets, column sets, etc. They take a long time to define and setup.

An excellent blog on the details of FSG Transfer can be found here.

Oracle ISetup

iSetup has been targetted to deploy a lot of the functional setup. Now I looked at this product quite a few years back and found it disappointing to be honest.

However it does seem to have moved on and it does now support more in terms of setup.

iSetup now supports:

Application Object Library

Oracle Financials

Oracle Human Resources Management System

Another link I checked is stating that the coverage is much wider across General Ledger, Accounts Payable, Accounts Receivable, Fixed Assets, Cash Management, Purchasing, Inventory, Bill of Material Engineering, WIP, Costing, Order Management, Shipping, HRMS and Payroll. Its worth checking Metalink for the latest list of objects that iSetup supports, together with re-use of API’s.

From an Application Object Library perspective, iSetup supports a similar list to FNDLOAD.

From a Financials perspective, as of 12.1 iSetup supports various General Ledger setup including Chart of Accounts, Currencies, Accounting Calendars, Budgets, Ledger Sets, etc.

iSetup has some support for Cash Management including System Parameters and Transaction Codes.

From an HRMS perspective, as of 12.1 iSetup supports various HRMS Setups including Locations, Business Groups, Organizations, Organization Structures, Job Groups, Jobs, Grades, Position Structures, Position Hierarchies and Positions. It also supports Employee Migration.

From a Payroll perspective, as of 12.1 iSetup supports various Payroll Setups including Balance Types, Defined Balances, Balance Attributes, Element Classifications, Fast Formula, Element Types, Input Values, Balance Feeds, etc.

One interesting option is the possibility to download and then modify and change. Allowing for instance setting up once and then changing for a different country and reloading. Has a lot of time-saving potential.

iSetup can also migrate XML Publisher data, Personalizations and Workflow Definitions, so it is well worth investigating as an additional tool to your deployment strategy.

Another nice feature of iSetup is the ability to compare various objects across databases. Handy when somethings stops working and you are looking for a needle in the proverbial haystack with a small piece of missing/mistyped setup.

Further details can be found on the websites below:

Steven Chan – Ten Ways of Using iSetup

iSetup Data Sheet

Frontier Consulting – iSetup

Frontier Consulting – iSetup Best Kept Secrets

Solution Beacon – Some new Thoughts for an Old Friend – iSetup

Trutek – Migating your Customs with iSetup

And with that it’s definitely time for another movie review. So a column all about aliens and their technology…..well if this column is all about Aliens and Sci-fi movies then one of the best Alien movies of all time has to be Aliens 2. I give this an absolute 10 out of 10. And you can probably pick it up on DVD pretty cheap given it’s a real old movie now, but still incredibly good.

OA Framework Personalizations

A lot of sites are using OA Framework Personalizations to modify OA Framework Pages, with a view to limiting impact of future upgrades/patches. Some of these personalizations can be quite involved and re-applying these involves a huge amount of work. Luckily again Oracle provides a handy method of exporting and importing these personalizations across databases.

More details can be found in the excellent blogs below:

Anil Passi – OA Framework Personalization Migrations

R12 Form Personalizations
 A little known feature of R12 is Form Personalizations. This allows you to replace quite a chunk of CUSTOM.PLL (or other Library code) with code that you define directly in R12. A great article covering this can be found at

R12 Form Personalizations

Now this not only allows you to remove custom code from CUSTOM.PLL or attached libraries (which is great), it also allows you to extract and import these Personalizations.

Simply using FNDLOAD again does the trick:

FNDLOAD user/pass 0 Y DOWNLOAD affrmcus.lct output_file FND_FORM_CUSTOM_RULES form_name 

Using R12 Form Personalizations is a more flexible and more visible way of approaching some forms based customizations. It’s also much more supportable.

Putting all the Alien Technology Together

There are a couple of interesting options with all the tools listed above that could radically improve your processes, make your ERP Instance(s) a lot safer and save you piles of time.

Firstly most of them produce files. Files can be put into Source Control such as Subversion (it’s free and great). If you Source Control your files, you can implement proper Release Control, guaranteeing that what you deploy is what you really want to deploy.

With all your files for setup in source control you can write automated deployment scripts, that will deploy all your setup in a matter of minutes (or if you have huge amounts of setup, a couple of hours). This will allow you to practise deployment in a repeatable manner until it works with ZERO failures. (Use the database flashback allows you to run deployment and then restore to a previous state in a few minutes, allowing multiple deployment runs/adjustments). Once you have that deployment release perfect, your Production deployment will be vastly safer than 99% of other companies on the planet.

Now Oracle’s heading in a very interesting direction with all this using the Application Change Management Pack. This, as Oracle puts it “provides a centralized view to monitor and orchestrate changes (both functional and technical) across multiple Oracle E-Business Suite systems.”

Read this for some more details on the Application Change Management Pack.

Now I don’t know just how good this is, but imagine being able to put large parts of your Functional and Technical Setup and your code into a Release Patch, just like Oracle Corporation. Imagine being able to clone Production, apply your patch and get to the point of repeating this until you had zero failures on your very complex R12 Upgrade Customization Release Patch.

This is certainly a very interesting development and the Change Management Pack is certainly worth looking at.

So time for a final movie review and a wrap-up. A word of warning. Not all alien technology is entirely friendly. In War of the Worlds, Alien technology proved to be extremely deadly. So my final movie pick is War of the Worlds. On reflection I’ll give this a 9 out of 10. It’s a superb remake. And a word of advice, when using the FNDLOAD Alien Technology, just be careful with your parameters or you may download and then upload into Production a whole lot more than you want………

The point of these tools isn’t primarily to save you time, although they will definitely save huge amounts of time on any project. Imagine having to setup say 100 custom programs in 12 instances during a project implementation. Or 200 messages. Or thousands of menus and entries. Or module setup. A large chunk of project time is wasted by very expensive consultants doing very menial setup, rather than delivering the end solution. And it doesn’t need to be that way with the tools that Oracle provides.

The primary reason everyone should be using these tools (either in implementation or Post-Production Support or R12 Upgrades) are because these tools introduce a repeatable, testable and verifiable process. These tools vastly reduce the mistakes you make.

For all those doing R12 Upgrade, these tools allow you to be successful on cutover weekend even when the timescales look utterly impossible. (Have a look at my R12 columns, with thousands of custom objects, 600 functional setups, all to be done in a matter of a day, with everyone under serious stress. Impossible? Not if you plan ahead and fully utilize all the tools that Oracle provides to make your life easier………

Till next time when I hope to publish the Secrets of the Holy Grail and the connection to R12. Just don’t tell the Feds where Oracle technology really came from. The secret is out there.

So I guess as Arnie would say from Terminator,  Hasta La Vista Baby………

Further secret blogs and prophecies can be found at:

Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt3)

October 31, 2010

Welcome back to the R12 Upgrade House of Terror.

All I can promise you is pain, heart-ache (that Oracle Support girl for the AP Trial Balance broke mine…..ahhhhhhhhh) and impending doom. Abandon all hope on your R12 for those who read on……..

So you’ve been mad enough to start an R12 project (they’ll lock you up in the asylum they will !!!) and you’ve made it through Implementation (designing any new stuff, build, test, UAT, etc) and you are sitting there smugly thinking you’re smarter than your average 5th grader and that this column was just scare-mongering……..Well lets go to the Transition Phase and see how confident you are………

 Cutover is complex

Deployment involves high number of objects

Deployment involves significant application setup

Deployment carries high risk areas

Users are not familiar with the new software

Large number of users require training

Users are distributed across countries

Help Desk Staff are not trained to take calls

Transition is poorly documented

Transition is not rehearsed

Change in Business Procedures for R12

Before you start the cutover, just wondering if you actually trained all your users? Yes R12 is mostly the same, but there are significant variations between R11 and R12 in Payments, TCA, SLA, etc. Now if those are spread across countries, it’s going to be even worse. A new system with new features suddenly appearing will lead to a lot of angry users, all wanting to talk to you on Day 1 of R12 as you are inundated with other more serious issues……..Now I went through four audits on my R12 and got a clean bill of health on all, except for training (even though we did some). The auditor did a survey and the users said they would have liked MORE TRAINING. But the products almost the same I said bar a few key areas as we were doing a Technical Upgrade !!!!!!!! I got burnt at the Halloween stake on this – OK ever so slightly singed. Don’t make this mistake. At least offer some training on delta differences and some educational classes with no commitment to implementing (get your lawyer to write the disclaimer) new functionality.

Has anyone told the Helpdesk of your R12 transition? Has anyone told the business they will lose their systems for 4 days (weekend, plus two working days). Has anyone told the external users or put out appropriate announcements? Are there key business activities that expect to run during your upgrade? Have you scheduled your upgrade right in the middle of these? People are going to be seriously upset if payroll is delayed for your upgrade. Or month end closing is delayed and you have serious problems with your new R12 that goes live right before the preparation of those financial results. Wondering if you told your corporate website team that pulls out summary information from ERP to display to your company website that ERP won’t be available over the transition? Or how about the legacy team that are still sending you files to interface into ERP? I know you are feeling pretty insecure now. I guess you are wondering “Have I missed anyone in the announcements……”.

Cutover is a nightmare. Get this wrong and you see your entire project collapse in a pile of dust, on day 1 of Production, with the Board asking why your business has ceased to function and lost 5M US. Pulling the transition on R12 at the last minute is really bad, but even worse is going live and then finding out you have serious issues……..Either has a good chance of ending your career.

Has everything been tested? Has every bit of code your about to deploy to Production actually really been through UAT (or did the developer slip in that one last easy fix with no testing?). Does your Application Setup really reflect the setup you tested? Did the Consultant complete the BR.100 before he left for a contract paying 30% more? Can anyone actually follow that BR.100……?

So you have a thousand customization files (actually isn’t a lot when you count Java files, setup, etc). Have you ever taken that as a single software release and had your DBA’s (not your developers that know to run script 2 if script 1 falls over and hack script 3 to make it work, without anyone finding out) and applied to an exact clone of your Production system and had a 100% success rate? Have you repeated this multiple times as you roll towards the go-live date, just to make sure that any changes between the time you last ran it and any updates to Production didn’t cause any adverse impact? Are you sure that the DBA’s can run all your customization deployments in a very short-time frame. (Your typical go-live window is a maximum of 4 days with two of those the weekend, after that your business will scream if you go one hour over). Is your deployment manual and if so, will the DBA do the same steps each time? If your deployment is automated will it actually work?

We had 600 application changes. Have you ever checked how long it would take to deploy all of those changes? Have you ever done an entire setup and checked all the setup was valid and tested properly? Have you thought of the issues of letting knowledgeable people do the setup in Production in terms of access, SOX, etc? If new people need to do the setup, as the project team cannot have access to Production, are they familiar enough not to make mistakes? Have they ever practiced the setup on an R12? Interesting how long it takes to carefully setup and check 600 application setups. Just wondering if you ever thought how long it actually takes to setup 600 application setups for a single person. Do you know it’s impossible to do that setup with one person? Have you ever bothered to do a careful review of all setup, access needed and then balance setup streams across different people to make it achievable in the very short transition time the business has granted you whilst still ensuring segregation of duties, SOX compliance, etc?

Have you ever considered setting up some functions, menus, etc in R11 BEFORE you go-live. Or some new program definitions? Or perhaps having these as Loader files, so you can load them instead of manually setting up? Of course if you load them, have you checked that the developers have generated these LDT files correctly?

How many minutes will the R12 Upgrade run on Production hardware? (I ask for minutes because you should know very precisely the estimated time of this – if you don’t you are in serious trouble. For info an R12 will run for a very, very long time indeed, even on a small ERP database.). Have your DBA group ever run an R12 Upgrade with no failures? How long will it take to upgrade the Technology components or database? Have you ever seen a detailed step by step upgrade document from the DBA Team? Are there any hot wiring the DBA team apply during the upgrade due to known failures? Are you aware of these known failures and the implications? Do you know that an R12 Upgrade will require some very serious shift working? Do you know that your DBA’s will be seriously burned out as they have the very first task in the actual upgrade of running the hundreds of thousands of scripts that the standard R12 upgrade runs. And they need to actually be awake and watch this constantly, because if it falls over, you don’t have the time to lose 8 hours on an upgrade weekend…….

Wondering if you’ve actually even got a transition plan? Have you got an hour by hour schedule, with milestones and an effective method to monitor setup, customizations, actual upgrade, etc? On an R12 upgrade transition, you have no slippage time for anything. If you slip, the team will get stressed, you’ll get stressed and mistakes will be made.

Bottom line is have you REALLY rehearsed an R12 cutover BEFORE you actually do it? Do you really feel confident that first time you do it for real will be on your Production system……….if so you must be a clown……by the way do you think the McDonald’s clown is intrinsically evil????? (My Attorney has advised me at this point to clearly state that is a question and in no way implies anything on McDonalds whose food I have been legally advised to state I love and the picture below is in no way related to said company).

If you take a look at the Oracle Upgrade manuals, it’s interesting how many pre-req steps potentially have to be done. Ironically if you miss some of these you can have major support issues going live. I wouldn’t leave any Payments in the middle of process. I’d be carefully checking the effect of an upgrade half way through each of your workflows. Is there any effect on transactions that are left half completed? Do you even have these in your cutover procedures and does everyone in the Business know from your carefully documented plan exactly what their responsibilities are prior to bringing down their R11 ERP. If you (or the Business) makes mistakes here, you are in real trouble. Do you have scripts to check everything has been accounted, no payments are sitting halfway or are you just relying on your business to do the right thing with no checking whatsoever…….Just to add a little time pressure, your DBA’s will be screaming at YOU if you go one minute over the agreed time to shut down R11 and hand it to the DBA team……..of course you’ll have to herd dozens of business units across 27 countries to complete all activities before that time…….but then they want to do their business not your upgrade……which leaves the DBA’s screaming at you…….(The DBA’s will be working all through the night, so every hour you delay, means an hour extra in the middle of the night so a little empathy for them here folks……..Show them you care for their welfare….at least till your upgrade is over. Alternatively take the selfish approach that the DBA that is working all night might be the same one running your customization scripts after two hours of sleep because of your poor planning that could wreck your upgrade when he makes a serious mistake because he’s so tired……….)

Have you made any plans to backup the original system? What are your fallback plans if the upgrade fails? When will you be prepared to pull the plug and under what criteria, over the transition cutover to R12? Who is going to take responsibility to give the green light to go live? Will the senior management even be there at the weekend to make that decision?

Just wondering if you told your team and others (Unix, Email, Legacy, etc) that they would need to work some very stressful and long hours over the cutover weekend. Wondering if any of those key staff have their holidays planned…….?

Have you made plans to clone and test the actual R12 on the cutover weekend? If not how do you know all your setup, customizations, etc actually were promoted cleanly and your R12 actually works? You will need a large team to test in such a short period of time. Do you even have a cutdown test plan to make sure you hit the key functionality? Have you got all this planned? Probably not…….And if you find stuff missed, do you have a proper controlled mechanism to track it, test it and release it to Production in a horrifically short and highly pressured weekend in a safe manner?

What’s amazing is that everyone plans a transition, but no-one actually thinks about turning on the R12……….when you enable the workflow and the concurrent managers are you just going to open the tap, or are you carefully going to release production jobs and monitor. I’ll tell you that your single most stressful time is when you commit to switching on these elements over the weekend. I bet you that your heart will be pounding and you will be sweating at this point. The entire company ERP and Business hangs on your Go or No-Go signal. Once you’ve done that, there is no turning back, there is no backing out and you are on R12. At that point if you’ve got it wrong, you’ll be looking for another job. (Doesn’t that picture below remind you of your boss when it all goes wrong……?????). I wonder if that Oracle Support girl I dealt with for the AP Trial Balance wore red lipstick…….ahhhhhhh……..

So your live, but it’s not over……..

Serious Outages can occur

Lack of Staff to address serious support issues post go-live

Patches are required after go-live

Are you ready for serious outages? Did your transition plan have proper contingency plans for these events? If not exactly what will you tell the Board and the Business?

Do you have the appropriate staffing plan to handle what could be a very rough few days (or weeks) or months? An R12 will almost always lead to a spike in calls. Are you ready? Did you bother to give your people proper notification that they may be required to work late nights and weekends for the first few weeks after go-live? Or is it just a line item on your plan that you didn’t bother to communicate?

Did you re-synchronize all of your ERP and Legacy systems in terms of interfaces? Some of these integrations may require manually running interfaces to catch-up with transactions for the days you were down.

With major business events such as payroll and closing, do you have any plans to clone and rehearse these events, prior to these happening? Now if you can clone and simulate an R12 closing after you are live, but before the closing happens, it’s better to find out serious problems and have a few weeks to fix them right? (Of course this has you wondering if you should close in R11 and then immediately move to R12, but of course, then your Financial reporting would be straight after go-live, so you can’t win right?)

Have you got any plans for patches after go-live? What happens if you really need to apply patches? Are your project team still there? Did you set business expectations just in case? Always better to make business aware of the real risks inherent in an R12 Upgrade. That way, they’ll be incredibly happy if you manage to get it right.

I hope you remembered to get before and after reports from key areas, such as your AP Trial Balance (that Oracle Support girl was just so nice, ahhhhhhhhh), GL Trial Balance, etc. And keep a copy of your Production database on R11, whether on tape or disk. When the auditors come and take an interest in it, as they will it would be unfortunate if you’ve not done this and it causes issues with your auditor’s signing off your company accounts.

Also before you go live just have a quick review of JVM sizing and parameters. If you fail to tune and size this properly you could see your Applications crashing, intermittent crashes, etc. Getting your JVM sizing wrong leads to absolute nightmares……..Ask your DBA’s “Have you size the JVM’s and if they say what’s that, start to worry……”. If you have too many users and not enough JVM’s and OACORES then BOOM !!!!!!!!

Finally is there monitoring of key components in place? Are you touching base with all the areas of Business (and Legacy and Web and email and and DBA and Unix and other teams). Are you even aware of major production issues or is your Production setup so poor in terms of support that you’ll be the last to know……..

And lets break for another movie recommendation. The single scariest movie I have EVER seen. This was the 1970’s television mini series called Salem’s Lot. It is absolutely terrifying. Get it on DVD….it’s one creepy movie from the Master of Horror Mr. Stephen King. I saw this when I was ten, and I didn’t go out for weeks after dark……It’s a classic ten out of ten movie. But REALLY TERRIFYING.

This isn’t an exhaustive list of what can go wrong in an R12. Now note this article has been written in a strong manner to scare the crap out of you because if I do that then you’ll start to take every last point out of this article and make sure you’ve covered it. Make no mistake – R12 Upgrades are a serious risk to your career. (However don’t let some consulting company come in and scare the crap out of you. They are in it to make as much money as possible from you for an R12 by scaring you as much as possible. I write these articles for FREE and hope people can avoid the mistakes that I and others made in our R12 Upgrades). If you read this article and take everything on board, you’ll be strongly versed in R12 risks and be able to evaluate what these consulting companies are telling you and separate the fact from the fiction…….

And a final note. Premier Support for 11.5.2 ends in November 2010…..that gives you about a month left……Interestingly you also need to have minimum baselines in place for R11 if you want proper support from November 2010……….And will you get hit by fees if your still on R11 in November 2011??????

Have you read Metalink Note 1178133.1…….

“Starting on Dec 01, 2010, Oracle E-Business Suite customers on Release 11i10 will only receive extended support for new bugfixes as described in My Oracle Support, Minimum Baseline Patch Requirements for Extended Support on Oracle E-Business Suite 11.5.10.”

That’s a lovely little bit of legal small print from Oracle with major implications for you………Getting to those minimum levels will be enough to “persuade” most customers to jump to R12 instead.

It’s funny also that so many people don’t think they can start doing preparatory work in R11 that will greatly ease R12. Have a look at the references below. Work spread over R11 and R12 will greatly ease your stress in the R12 Upgrade itself, as you’ll have less to do. (Did you know Solution Beacon even has Vision instances you can play with for free, even before you install an R12 in your company’s test environment…..)

Let me tell you briefly our R12 story. We had some very serious problems with quality (12.0.4 RUP5) – an early R12 release in many respects. However despite that we got through a Finance, HRMS and Procurement Suite R12 Upgrade across 20+ countries with strong planning. Our transition finished 30 minutes ahead of the planned 96 hour time window (I wasn’t kidding when I said you should know the transition in minutes…..). Yes, I was terrified when I made the call to go-live. But due to all the planning and a great Project team, we went live with no major issues on 3rd August 2009 and I went home at 5PM two days after go-live (And for the cynics, no I didn’t spend the first two days constantly in the office 48 hours straight…..I got home at a reasonable time the first two days as well). We cleared Payroll, breezed through month end and went live on two Payroll rollouts to two new countries one month after go-live.

Of course now I’ve told you all the risks, well you’d be a fool to actually fall into those holes right? Take this article, make a list, make sure you cover each and every one. Then you’ll make your R12 a huge success.

An R12 is certainly a very achievable project, with significantly reduced stress and risk, if planned correctly.

Hope I haven’t scared you too much……..

Enjoy your Halloween.


As always, my appreciation goes to the kind authors and contributors of the following articles and resources. Some of these are especially note-worthy, because those people are putting out their failings (as well as successes) on R12 in a very public and open way, for the benefit of those about to travel the R12 Upgrade road.

General Blogs and Websites

OAUG Website

Chris Warticki’s Blog – EBS R12 Support Resources – Consolidated

Oracle Applications Upgrade Guide: Release 11i to Release 12 (B31566-01)

Oracle Financials Applications Blog – R12 Lessons Learned

Analyzing, Planning and Executing an R12 EBS Upgrade

Steven Chan Blog

Risk Management: Tricks of the Trade for Project Managers (Rita Mulcahy)

My example of an R12 Risk Spreadsheet (Excel 2007) – Available on Request



Oracle E-Business Suite Release 12.1.3 Release Update Pack (Patch 9239090)

 Oracle E-Business Suite Release 12.1.3 Readme (Note 1080973.1)    

Upgrade Manual Script (TUMS) – Patch 5120936

Metalink Note: 580299.1 – Best Practices for Adopting Oracle E-Business Suite, Release 12

Metalink Note: 394692.1 Oracle E-Business Suite Upgrade Resources

Metalink Note: 562887.1 R12: Helpful Tips for a Successful R12 Oracle Payables Implementation

Metalink Note: 437422.1 R12 Troubleshooting Period Close in Payables

Metalink Note: 73128.1 R12 Troubleshooting Accounting Issues

Metalink Note: Oracle Support Upgrade Advisors (250.1)

                             Tech Stack                     Note 253.1

                             Financials                     Note 256.1

                             HRMS HCM                    Note 257.1

                             Manufacturing              Note 258.1

Metalink Note: Extended Support Patch Level Verification in Oracle E-Business Suite Release 11.5.10 (Note 1178133.1)

Metalink Note: R12 Upgrade Considerations by Product (Note: 889733.1)

Oracle E-Business Suite R12.1 Financials Pre-Upgrade, Setup and Operational Tips (Note: 1104163.1)

Oracle E-Business Suite Release 12.1 Information Center (806593.1)

Database Preparation Guidelines for an Oracle E-Business Suite Release 12.1.1 Upgrade (Note: 761570.1)

Oracle Applications Installation and Upgrade Notes Release 12 (12.1.1)

R12: Period-End Procedures for Oracle Financials E-Business Suite (Note: 961285.1)

Oracle Open World 2010 – On Demand

For those with access to Oracle Open World 2010 On Demand there are good references at

Mission Impossible: Oracle E-Business Suite 12 Upgrade on 3 Continents in 6 Months

Success with Oracle E-Business Suite Release 12.1.2 Upgrade Drivers

Algar Telecom Upgrades to Oracle E-Business Suite R12   

Get Ready for Oracle E-Business Suite Release 12.1: Tasks to Complete Now

Oracle E-Business Suite 12 Upgrade : Best Practices to Obtain Business Value

Planning your Oracle E-Business Suite Upgrade from Release 11i to Release 12.1 (Superb presentation – highly recommended)

Oracle E-Business Suite 12 Upgrade: An Easier Ride on Nine Miles of Bad Road

Oracle E-Business Suite R12 Upgrades: Have you Thought about the Details?

Real-Time System Assessment of Oracle E-Business Suite 12 Upgrade: Case Study

And finally talk to other companies, use forums, other areas (such as and make contacts with people that have done R12. This again will help you considerably on your R12.


Some of the great graphics were the work of:


Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt 2)

October 23, 2010

So it’s heading for Halloween. Only a few days left……..And looks like you’ve come back for the 2nd part of the R12 Halloween Upgrade. So I guess you don’t scare too easy? Prepare to be afraid. Prepare to be very, very afraid.

So after a whopping amount of things that can go wrong just in the initial stages of an R12, we can finally talk about the actual R12 Implementation in terms of analyzing, designing, building, testing, new functionality, etc. It’s amazing so much can go wrong BEFORE you even start……’s a previous Project Manager that was running an R12…….(he’s the guy 3rd from the right, the one 2nd from the right did the R11 Upgrade and the one on the right did the initial ERP implementation….isn’t Oracle ERP pretty unforgiving project wise……)

Here’s a few things on Implementation to help you lose sleep.

Applications do not provide a good fit to Business Requirements

Subsequent Analysis during the Project drives Significant Change

Not possible to achieve a consistent Business Process across all units

Some modules are largely custom built

Large number of customizations

Undocumented customizations

Lack of development standards

Information is confidential

System Integration is complex

System integration is high risk

System integrates with non Oracle technology

Complex Oracle Application Configuration Management

Complex custom development configuration management

Poor Technical Designs

Customizations are delayed or lacks resources in Development Team

Functional Designs are late or incomplete

Technical Designs are late or incomplete

Patch Management is poor

Complex Setup

Testing misses requirements or scenarios

Testing is incomplete by Project Team

Testing is incomplete by Business Team

Testing is delayed or lacks resources for Project Team

Testing is delayed or lacks resource for Business Team

Testing integration is delayed by dependencies

Legacy data is of poor quality

Legacy Data requires data cleansing

Legacy Data has complex structure

Legacy data involved high data volumes

Changes in Legacy System required

Oracle does not have interfaces to support conversion

Oracle uses complex API’s for conversion

Ignore the Legacy Conversion stuff. I use this spreadsheet for more than just R12 Upgrades…..

Now if you’ve ignored the advice to aim for a R12 Technical upgrade, then you may find yourself knee deep in very long, difficult and time consuming business requirements, business change, etc. On an R12 Upgrade, trust me just don’t go there. Keep it clearly 100% out of the scope of the project. It can wipe out your project unless absolutely forced to. Which brings us nicely on to R12 Oracle Payments.

Payments module is a re-implementation. This will take some adjusting as it looks nothing like the R11. With this and other new features, you run the risk of not fitting to business requirements, or wanting new processes or wanting changes. The problem is if you show the new stuff (and with Payments you have to….) you could have significant change in your project scope and therefore significant risk. If you don’t show the new stuff you are going to be accused of not showing it, not training users, not taking advantage of the “amazing” new functionality, etc. So if you show it, you are screwed. If you don’t you are screwed. For me R12 is complex enough to pull off without adding anything other than like for like. New functionality for us could wait (and it did in 99% of cases, despite some fairly tough fights).

Of course if you are rolling out R12 across multiple countries, then it’s even worse to agree all the new functionality, legislative stuff, etc.

Then you get into your biggest nightmare of all. Customizations. Imagine someone having a decade to write customizations on R11. Imagine no discipline, documentation or standards. And you’ve got to upgrade something that you have no clue what it is you actually have to upgrade. Talk about a Pandora’s box……Or worse does anyone remember the Hellraiser horror 80’s movie flick with the box that brought out the demons…….Do you really want to open that R12 box??????

When you open an R12 Upgrade project, you are opening a Pandora’s box unless you did an advance study. Or worse you could be the guy hearing the doorbell, going to his door and seeing a burning brown paper bag with R12 written on it on his doorstep. You go to stamp it out and find something VERY unpleasant on your feet that your company that implemented your ERP left you with but didn’t tell you…….Is your current R11 a brown paper bag with a pile of crap inside waiting for you to stamp on it with your R12 Upgrade?

One single customization missed can cause a catastrophic failure. Funnily enough I saw one Upgrade where they had account generators in the hooks of a standard package for Costing. Now yes, this was supported, but totally undocumented. Funny that generated a pile of entries to General Ledger that took weeks to figure out and calculate the reversals to, after the code was fixed post go-live. So the message is, it isn’t even enough to look at the obvious custom code. What about the standard packages that support hooks? What about overwriting standard code in packages, forms, reports and other customs? What about the change of seeded menus? None of this should happen, but don’t count that it hasn’t.

If you don’t have a customization register before you start you are in very serious trouble. You are going to miss stuff, and that one thing you miss could wipe you out in Production.

Also do you have any idea what R12 will do to your customizations. We saw our Payment customizations wiped out and need completely rewritten. (BI Publisher). Any workflow changes? They’ll need to be redone. Any Self Service in the old MOD PL*SQL? They’ll need to be redone. Any interfaces to GL or reports or anything else? They’ll need to be fixed as 80% of our stuff stopped working. Small changes on a lot of customizations add up to a lot of work, time and risk that can seriously delay your schedules. Did you know there have been very significant changes to the data models between R11 and R12 in certain areas……all that will cause your existing R11 customizations to crash and burn……

If you are doing an R12 Upgrade, you’ll have frequent clones. If you don’t encrypt data you could be incurring serious risk as all those outside resources can see vast amounts of confidential information. (See R12 Security and the Auditors from Mars article to scare you some more).

From a Systems Integration viewpoint, watch out that your R12 Upgrade doesn’t just focus on Oracle ERP. When we did the upgrade, we had Legacy, email , Web, external 3rd parties (Citibank and others), Health Insurers, etc. Now us Oracle guys like to think ERP is the center of the universe. Sometimes it is, sometimes it isn’t. But your typical R12 (or R11) can have interfaces, data extractions, data imports, links to spreadsheets, etc that can number many, many hundreds of integration points. Are you aware of what they are and if not how the hell are you going to test and make sure everything works day 1? Think about missing the testing on say Payroll interface to the Bank. That would be a nice one to find out doesn’t work on say, Payroll day for 50,000 employees. Or perhaps you forgot to test the interface to your planning system that orders the parts to build the products you make, which would stop your production line for 5 days, cost your copy millions of dollars and definitely get you on the “skull decoration” shown at the start of the article in the Boardroom or hanging on your managers wall….. . Hope you sleep well thinking of your integration points to ERP and the catastrophic damage it does if you get it wrong. I bet you that your one of these people that DOESN”T EVEN HAVE an architecture diagram of your ERP integration points. And your company spent like what 50 Million on that system with your integrator and you didn’t even get a coffee mat Integration and Interface Architecture chart????? You were ripped off and it may come back to haunt you…….

By the way, speaking of integration, does all the tech stack count? Just think, adding Forms 10G, Reports 10G, JDeveloper 10G, downloading the extensions for OA Framework, your Discoverer upgraded, BI Publisher. I guess not so much integration as getting the absolute basic tools to even start your R12, as all of these change between R11…….so you’d better sort that before you hire all those developers to upgrade your customizations…..

There will be a need to update or add new functional designs unless you are really a vanilla site. We did a lot of work again around Payments. Will those designs be decent and cover the requirements? Will they be reviewed in a timely manner? Will they be signed off? Keep this to a minimum (no new functionality, that’s gold plating an R12 Project) and you’ll avoid this risk.

A lot of companies also do MD.070 Technical Specifications. I’m not a fan of these to be honest as they have too much info, take too much time, are subject to not getting updated so become quickly outdated, and are largely done by many to tick the box. If you are doing these, make sure the quality and approach of customizations is correct for R12. (And don’t build what R12 provides – that’s one area I would add to an R12 to replace customizations with standard functionality where possible. We took out a raft of custom modifications which are the worst types of customs and vastly reduced our risk both to our upgrade and future upgrades/patches).

So I saw a movie with my kid. It was called The Hole (and it was in 3D so the ticket price is like double !!!!) Now this was a PG13 which meant that it shouldn’t have been that scarey, except it scared the crap out of my kid. More to the point it scared the crap out of me even more. So what’s it about? The hole generates your darkest fears and then these monsters come out. And what’s my darkest R12 fear? It’s about managing all of the configurations on an R12 Upgrade Project – that is my worst nightmare. The Hole gets a 7 out of 10 on the movie review. Well worth seeing, but if your kids get scared easy, approach with caution.

Now managing all your environments, patches, setup and customizations is a truly epic and very complex task. Really someone could write a book on this by itself, but the point is, you lose track of this and you lose track of your project. The lack of source control, the lack of BR.100’s, the lack of configuration management spreadsheets (or software) will cause your project to descend into chaos, especially as your ERP clones spiral from a development instance, to a sandbox, to a project team test instance, to a UAT instance, to a patch instance, to a DBA instance, to an instance with a DMZ, to an instance just for one module as it’s getting piles of patches and needs isolated……..your configuration management isn’t about a single instance. We had 10 instances at one point during the upgrade…….Or to put it another way our R12 had over 600 Functional Setups between R11 and R12, dozens of patches, thousands of test scripts all requiring data/setup to be in exact scenarios and thousands of customization files, loaders, packages, etc……Just curious on how exactly you are going to manage that………..? Are you one of these people that manage this type of thing on the backs of bits of paper or a cigarette packet? If so you’ll need to become a chain smoker as you’ll need a mountain of cigarette packets to keep track of all your configurations items on an R12 Upgrade project……

What I always find most worrying on an upgrade is playing the Oracle Patch Lottery. This is cool. You’ve got your R12 say Test environment all setup. You’ve got your thousand customization files, 600 setups and all the test data ready and you apply a patch, having no idea what that patch changes. And it trashes your entire environment, wipes out your 80% test completion and trashes the modules down-line of your testing. Every patch you apply is a roll of the dice unless you very carefully manage each and every patch. So when you apply a patch during your upgrade and have no clue/plan/analysis of that patch, the question is, do you feel lucky punk?

Testing is full of pitfalls on any upgrade. Have you got all the scenarios covered? Are your test scripts complete? Is your test data valid? Have you covered month end testing? Have you covered year end testing? Have you covered integration testing? Did you try the Web ADI? What about tools like Discoverer? Have you covered testing to and from Oracle and 3rd party companies that you barely even talk to (and met in 2007 when you put in your interface to their products which run offsite – think bank interfaces). Does your new patch invalidate Test Scenario 220, 221, 222 all the way through to 230 for Payables and therefore invalidate your testing across 80% of Payments? Or what if you apply a patch on TCA? Does it hit PO, AP and Payments? Or if you apply your customization, what have you invalidated? Will the auditors pick you up for claiming a sign off from the user, when they tested on a completely different configuration of custom, patch, etc? Are you sure your environments are all the same? Or do some differ in terms of setup/customizations/etc? If so, then how valid is your testing?

Here’s another great one to keep you awake. So all your systems around ERP. I bet you one joker that owns for instance your Legacy planning system or say your Payment middleware is planning to upgrade at exactly the most inconvenient time for your R12. Now you’ve always hated that guy and he’s always hated you. I mean is he doing this at the most inconvenient time in your R12 Upgrade without telling you just to screw you over yet again to get that next promotion over you ? If your not aware of everything you need in your R12 Upgrade and everything in your company’s universe (planned upgrades, planned business initiatives, politics, etc) that can screw you over, I need to ask, why did they even give you the R12 upgrade to do?????

Even if you get all this right, does your team have enough time to thoroughly test? Did you properly track all the bugs? Are you monitoring the bugs? Are you monitoring the Service Requests? What happens if Oracle decides to give a RUP (Rollup Patch) for you in your UAT and refuses to provide a single patch? Do you also take that latest security patch? Did you know that security patches have been known to screw up the Applications…….

It’s also interesting that an R12 DOES have data migration. Think about all those scripts running for the upgrade from Oracle Corporation. They are re-arranging your ERP data pretty substantially in some areas. What if the standard upgrade scripts for changing the structure of the Bank data (to TCA) goes wrong or cause problems? Think that won’t happen? Have a look at our Risk Register for R12………Or did you know you can run a script to convert your previous data to Subledger Accounting? Did you know R12 actually does some fairly heavy data conversion for SLA as standard? Are you confident it will all work with no need to reconcile? Are you confident that the new way that R12 handles accruals will work, as you actually have to run programs to build all the Accrual data………If so have a look at our Risk Register…..we didn’t spend 4 months reconciling and talking to the guy that had written this (in ironically Redwood shores…..those conference calls time-wise between Asia and California, weren’t easy) and apply multiple patches because it was working first time or second time or third time or fourth time……..Again have a look at the risk register from us. Or how about your AP data being converted? Again R12 does some very serious data conversion in the background. Try running the AP Trial Balance and see if you can reconcile…I didn’t spend two months just trying to talk to the lovely girl at Oracle Support to get a date from her….there honestly was a problem with the AP Trial Balance although I did close the SR immediately once it was transferred to a guy……..And did you know all your supplier data is converted by Oracle’s Upgrade process to TCA……..

The good news is that newer versions of R12 are a lot better, largely thanks to “suckers” like me that spent months with Oracle making all of the above data conversions that the standard R12 does in the background work. Of course until you check and reconcile all this for yourself, how confident do you really feel trusting some guy on a blog that claims to be a prophet yet never picks decents stocks………?

On top of all that resource management across your R12 will be the usual nightmare. Competing projects, production issues and users screaming for enhancements of R11. Key Project People walking off to lucrative new contracts and positions. Lack of R12 experience in the market. The time delays finding and hiring people to replace those that walk off. Dealing with unexpected issues that you didn’t plan for that have a catastrophic effect on your resourcing across the project (try cleaning up 3,000 banks when you find out what Oracle does to these during an R12 Upgrade….)

We’ve pretty much covered the testing phase above but just a quick recap to really scare you…..

Testing misses requirements or scenarios

Testing is incomplete by Project Team

Testing is incomplete by Business Team

Testing is delayed or lacks resources for Project Team

Testing is delayed or lacks resource for Business Team

Testing integration is delayed by dependencies

Test Plans are poor

Test Scripts are poor

Reconciliation of Ledgers and Subledgers is not carried out

Test Coverage is poor

Numerous Patches are required disrupting Test Cycle

All of the above can wreck an R12 Upgrade. It’s easy to miss key scenarios. Test scripts can run into thousands. Integration testing can have hundreds of points including external parties making it very complex indeed. User testing is beyond your control in many respects, especially for those that see no need for an R12 Upgrade. Reconciliation of ledgers and data that R12 has converted in the background is a significant task. Patches create havoc on internal test cycles and formal UAT. Remember you can’t just get each group of users to test each module. Each module talks to the other modules. Data from one module flows to the other for the next scenarios. PO needs Inventory. PO feeds the receipts. AP matches on PO receipts which then feeds to assets. IRecruitment flows to HRMS which then trigger hiring which then triggers Payroll which then relies on Advanced Benefits which then drives Self Service. The beauty and wonder of a fully integrated system makes testing co-ordination far from a stand alone list of test plans for each module………Managing an R12 in terms of testing is an extremely complex and challenging endeavor. Try doing that across 45 business areas and 27 countries, each speaking a different language……What’s Halloween in Spanish……??????

With an R12 Upgrade, vendor management is unavoidable…..

Software has significant patches

Software purchased is unstable

Hardware Vendor provides poor support

Software Vendor provides poor support

Poor quality in Product (bugs)

Software purchased is recently released

You are going to spend a significant amount of time dealing with Oracle Support. Depending on who you get on your Service Request can really screw you up big-time, as quality varies significantly. Get the right person (such as the lovely Indian girl we got on our Trial Balance problem, ahhhhhhhh…….) and you could get an immediate response. Get the wrong person and you could see weeks or months wasted, with the “apply this next rollup patch, we’re sure one of the thousand+ fixes will fix your minor problem and retest your entire R12 again” attitude.

R12 was highly unstable in 2008. 2009 saw it got better. By now reports are it is solid……….but I wouldn’t make that assumption, until I had actually implemented it and got my UAT signed off 100%.

When you implement R12 you may need to upgrade your database. (most will). When you upgrade your database you may need to upgrade your Operating System. When you upgrade to R12 you may upgrade or replace your servers, depending on where they are in their lifecycle. Now companies don’t generally have massive dealings with their hardware vendors (as Unix/Linux is incredibly reliable). But during an upgrade that relaxed relationship is going to be potentially very heavily tested. Question is, can the vendor provide strong support, resolve issues quickly and get you safely onto a server that is the very foundations of your business? If you fail to install the correct O/S plus interoperability patches you could see your business crippled…….It’s funny that the Infrastructure Manager that you were fighting with two weeks ago, kind of holds your entire career in the balance and you don’t really know anything about Servers, questions to ask, etc so are 100% dependent on him for your project to succeed.

Just wondering if your R12 needs more resources in terms of CPU, disk, memory, etc? Have you checked? It would be a shame if the server collapsed on day 1 of go-live.

Lastly the million dollar question. Oracle is on R12.1.3 at time of writing. Do you implement this version (and be the one to find the bugs as no one else is using it) or do you go for say 12.1.1. At this stage of the R12 lifecycle I’d go for R12.1.3 but then again you will get some additional pain of being one of the first. Obviously by the time you get to UAT, 12.1.4 will be out……….then do you jump to that or perhaps apply that new ATG patch or a RUP Patch that has come out as it has some really needed functionality……..All of these create massive risks not just for the stability of the project but also your budget, timescales and resourcing. Larry Ellison is a little devil….always releasing new ATG, Security or RUP Patches just at key points in your project, just to tempt you and cause major indecision in your mind. Question is, is that patch a poisoned apple that will drop your project at a key time??????

All hail Larry Ellison. The True and Only Oracle Prophet !!!!!

Happy Halloween. The clock is ticking down to the midnight hour. Before Halloween is over, I will scare you. The witching hour is coming with Transitioning from your R11 to R12, the actual cutover and post go-live, with all the reference material and a prebuilt R12 Risk Spreadsheet as a treat for all the kids in the next article……

Delivered on Halloween !!!!! Enjoy………..


See the other Prophecies including Business Intelligence and the Kung Fu Dragons of Wudang at

Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt 1)

October 20, 2010

Darkness falls across the land
The midnight hour is close at hand
Creatures crawl in search of blood
To terrorize the neighborhood

Now if you don’t know that song intro, shame on you. It’s from one of the biggest hits of all time – Michael Jackson’s thriller. A very appropriate one for Halloween I’m sure you’ll agree.

So first movie recommendation of the day. This would be “This is It”. There’s something very poignant about seeing him performing in what was his final curtain call.

Now Halloween has always been a time to scare the crap out of people. Kids dress up as Ghouls, Vampires and the like and go “trick or treating”, sometimes with some very nasty tricks indeed. Either the local neighborhood pays the kids with sweets, etc or the kids do something nasty.

Back when I was a kid, there was one particularly nasty kid who didn’t bother with the treat stuff. He figured he could go beat up the other kids at the end of the evening and steal their sweets. This saved him the effort allowing him to do his favorite trick of all.

This kid had a real nasty trick. What this kid would do would be to get some dog crap, put it in a brown paper bag and then go place it on someone’s doorstep. He’d quickly set the bag on fire, ring the doorbell and wait for someone to answer.

Now what is amazing is that the immediate reaction of an adult to a small burning paper bag (even if they are not wearing shoes) is to try to stamp out the fire with their feet. Irrespective of wearing shoes or not, this had some pretty unpleasant results for the poor neighbor at the end of this “trick”.
Before we start, let’s put our other movie recommendation out. It’s the original Halloween movie. Now I saw this when I was 13 and it scared the absolute crap out of me. Well worth getting the video. A classic 70’s horror flick for Halloween, but definitely not for the kids. (Don’t see the more recent one, it sucks).

So seeing as it’s Halloween, I should scare the crap out of everyone reading this right? So everyone’s moving to an R12 Upgrade in 2011. So what better way to scare you than tell you how horribly wrong it can all go on an R12 Upgrade !!!

So R11 Oracle Apps Premier Support is ending in November. That means you are screwed. OK, not completely (but Oracle will be putting the pressure on you and telling you that you are) but seriously you should be looking at R12 now.

Now let me tell you that an R12 Upgrade will be the biggest, toughest upgrade of ERP that you will EVER have done. It can go disastrously wrong. You could corrupt your data, you could have major functional failures dropping your business to it’s knees. Or your entire ERP could disintegrate and you could lose your job because at the end of the day your boss will need a sacrificial lamb and you seem to fit the bill.

Are you starting to get scared? You will be by the time you read this article, because it contains a very long list of every single thing that I can possibly think of that can wreck your R12 upgrade.

So let’s start at the very beginning. Getting the Project Started. If you get this wrong, it’s probably not worth bothering about the rest of the project. Here’s a short list of things to worry about. (Now the purists will say these are not risks). For me I like to have everything that can go wrong on one spreadsheet. Call it risk, call it a list, doesn’t bother me, because I can quickly mentally check each month how things are going and make sure I haven’t forgotten anything. It also gives me a nice risk score and lets me manage that to hopefully give the project a fighting chance of delivery.

Let’s start with the Project Management risks and everything that can go wrong here:

Weak Business Commitment to Project

Funding is not secured for Project

Project Decisions are not Timely and On Time

Weak Business Commitment to Change

Review of Project Deliverables and Sign-Off is not timely

Weak Support of Business Case

Scope is not defined clearly or completely

Business process model is not defined clearly

Planning is of poor quality

Project involves large number of departments

Tight Time-Frames, Minimal Slack between Dependent Tasks

Go-Live Date accommodates limited UAT

Project Milestones are not achievable (Project Schedule)

Dependency on other Projects

Project requires complex organizational changes

Contract involves significant cost

Project involves time-critical Procurement

Significant Scope creep

Gold Plating by Implementation Team

Project Budget is not sufficient with Current Progress

Project Resources are no sufficient with Current Progress

Project schedule is delayed

Project Cost is overrunning

Product does not meet client expectations

The very first thing you need to do is get your most knowledgeable and trusted people in a room and do proper planning, taking into account your organizations political landscape, resources, expected funding and other commitments. If you fail to plan, you plan to fail. Your planning must give ample slack time for delays in UAT, delays in decisions, procurement delays, bugs, resource problems (only 5% of people have done R12. The other 95% of people will do R12 in 2011, so can you get resources??????)

The most important part of any project is to get high level buy-in and commitment from the senior people across your organization. From a resource viewpoint an R12 Upgrade requires intense Business and IT Commitment. Fail to get every area of Business and IT involved and agreed, then you are screwed 100% come UAT, if not even earlier. You need to make it clear when you need the business, what you need them for, how long and how many resources right at the onset of the Project Initiation. IT resources need to be equally onboard.

Securing realistic and adequate funding is critical to an R12. It is bigger. It is more complex. It does take longer. An R10 to R11 or a 11.0.3 to say 11.5.10 is NOT the same as an R12. Build in a decent contingency.

An R12 is not a basic like for like Technical Upgrade. You should ideally keep R12 as a pure technical upgrade and not a re-implementation, unless there are very good reasons, otherwise your risks and costs will escalate severely. However even doing a technical upgrade involves a pile of new modules and new functionality (albeit some can be deferred). But you’d better be prepared to implement Subledger Accounting, EBiz Tax, TCA (changes in Banks), Payments and Self Service (new functionality and removal of MOD PL*SQL). All of these will need user review, functional designs (if customizations on top), setup and decisions, as well as time for your own team to learn the implications.  This can lead to multiple risks including your own team wanting to add “cool functionality” or users not making timely decisions or being prepared to adopt new processes to account for changes in say TCA or Payments.

If you are dealing with contracts for consultants, firms (outsourcing or otherwise) or need hardware, etc, be prepared for the heavy risks of delays in Procurement. IT guys often overlook that it may take a while, despite having the cash to get resources onsite or hardware commissioned, etc.

Talking of resources this should be viewed as a serious risk throughout your project. R12 resources are still less rather than more common.

As well as all of the above, you still have to worry about all the usual Project Management stuff of quality, schedule, cost, delays, budget, etc. And think about it, we’ve not even got to the actual project yet…….

So the very first thing you need to do (before your project team arrives) or is internally deployed is to make sure they actually have an R12 Instance to work on…..

Infrastructure is often overlooked for projects. Unfortunately this can cripple the project team (and therefore your project) very rapidly indeed.

Network Speed

Performance Issues on Project hardware

Performance on Production may not be acceptable Go-Live

Deployment involves new hardware

Hardware is not sized correctly

Database is not sized correctly (do you have enough disk space for R12)

Hardware is no available for Project Use

Backups are not available for Projects

Project Environments not available

Network availability is poor (if teams distributed)

DMZ Configuration is not available

Architecture is new

R12 Instances are not available

Space for Project Instances is not available

Missing interoperability patches

Missing patches to Operating System

If you are going R12, either you have a lot of servers around spare, or you are going to need to buy (or lease) additional hardware. Otherwise what is your team going to use? Ten Consultants onsite doing nothing is some serious cash burn.

Then you have to have a plan for environments needed, dates needed, who prepares them, who sets them up, etc. DBA’s will shout at you if you need environments the next day. And what if there is no space? Did you check the disk space as part of your project? Your average R12 environment requires considerably more space than your R11.

During an R12 Upgrade you’ll need a pile more databases. We used around 8 additional (and we still needed all our R11 Development, Patch, Test as R11 Production work could not be completely halted).

Now just think when you finally deliver that R12, and it crumbles on day 1 into a performance black hole. Did anyone bother to do proper sizing either on CPU or disk or Memory? If not, try when it all fails on day 1.

Even when you are starting the project, insufficient hardware will kill you.

Did anyone ask for backups of your project databases? It would be a real shame to see your R12 go down and no backup, losing months of effort.

If your Production is a DMZ enabled configuration, I hope that someone is also going to emulate this for some of your R12 databases as you go through the project.


So lets get to the one that usually gives R12 Project Managers nightmares. Resourcing.

Poor IT Staff Skills – Functional

Poor IT Staff Skills – Business

Poor IT Staff Skills – Quality Assurance

Poor IT Staff Skills – Java

Poor IT Staff Skills – Oracle Development

Poor IT Staff Skills – ERP

Poor IT Staff Skills – Modules Implemented (or new modules)

Poor IT Staff Skills – Technology

Poor IT Staff Skills – Unix/Linux

Poor IT Staff Skills – DBA

Poor IT Staff Skills – Project Management

Poor IT Staff Skills – Senior IT Management

Poor IT Project Team Rating – Skills

Poor Business Project Team Rating – Skills

User Department Staff not available for Project

IT Staff not available for Project

Poor User Department Staff Skills

Poor or lack of Project Team Lead (s) per Functional Stream

Experience of Business Area is Low

Project has competing projects for Resources

Custom Development relies on 3rd party

Project Manager is inexperienced in Upgrades/ERP

Legacy Staff unavailable for Data Conversions

IT Staff skills – training required

Low retention of project  team during implementation

Low availability of Trained and Experienced reasonable cost alternate resources externally

No IT Staff available for existing Production Support

Oracle Support provides poor support

Have you even contemplated the vast variety of resources you are going to need to pull off an R12 Upgrade? Have a look above if you think you can use a couple of consultants over a few months.

Will your internal IT guys be good enough? Do they know the R12 functionality? Probably not. Will you train them? Probably not. So how can they deliver? And even if you want to train them, can you get the appropriate training completed before they actually do the upgrade. Training courses from Oracle can be hard to come by, especially on some of the niche modules and require booking way in advance.

Do your IT guys REALLY know the business? If not are you sure you’ll catch every scenario and every nuance during implementation.

Do you have decent Oracle skills? Do you have decent Java skills? Do you have decent Oracle ERP Technical people? If not, you are going to need to ramp up or get someone that does to do your upgrade (unless your complete vanilla ERP).

Do you have anyone that knows all the new R12 modules, such as EBiz Tax, Subledger Accounting, Payments, etc? These are not optional modules (except Payments, but if your running Payables in R11, then Payments becomes essential in R12 and it’s a new module). Most companies will require to have knowledge on these and something like Payments has heavy changes between R11 and R12 – indeed it is a rewrite between R11 and R12.

New technology components are included across the board. Are your people ready and capable?

Your typical upgrade will require some pretty heavy Unix or Linux skills. You might have to upgrade the O/S or you may take the opportunity to switch to cheaper, less proprietary hardware or Linux. Do you have any clue what’s involved? (we did this and yes it saved us a pile of cash for the future, but it ain’t easy and is littered with risks).

Did you tell the Infrastructure guys you’d be needing a full time DBA for the R12; to clone R11 Production; to work on the upgrade process; to write detailed upgrade documents that give step by step guides; to optimize the process so that it can fit into a long weekend on a huge global database? To apply all the patches? You probably thought your DBA’s sit around doing nothing and can do all this at the drop of a hat…….Count on the initial upgrade where your DBA’s find out the nuances of R12 to take quite a chunk of initial time and plan that into your environments and people coming onboard project wise.

R12 Project Management. Words fail me on this one. If you are not an ERP person don’t even think of doing an R12 without a strong R12 Project Manager, ideally someone that has done R12 Upgrades. It’s horrendously complex, extremely technical and the risks are catastrophic if you get it wrong. (I can be contacted on the following number for lucrative job offers……… 🙂   )

Are the Senior Management (both in IT and Business) in full support of both you and the R12 Upgrade, or are they ready to crucify you at the first delay? R12 will have delays, it will throw horrendous curveballs in terms of bugs and other complexities. Just make sure everyone is on your side politically before you start. The guy below thought everyone was his friend, until he forgot to order the hardware in time……

Now this is a weird one, but how well do your Business know the Business and Oracle ERP? Look for the weak spots here before you start. If a Business Unit has a lot of new people or inexperienced people or are anti-Oracle ERP (and some are), how are you going to get them to assist (or write) Test Plans and Test Scripts and do actual testing. Now if your IT and Consultants don’t test well (and they are NOT business experts) the one hope of catching everything before Go-Live is the Business. I never build 100% dependency on the Business for testing, but if your IT and Business guys can’t be relied upon for testing, you’ve got a perfect storm.

The other thing to check is everybody’s schedules. We’re not talking dinner dates, or when they plan to see a movie. Are the Business available for an R12 (which won’t add vast value to be honest) or do they have a planned move to say IFRS planned from June to December 2011. If so you won’t get a look-in resource wise for UAT and your project will be dead in the water.

Same goes for IT. If they have some major high profile initiatives, your R12 is bottom of the list, as in the priority list, it’s just one of these unpleasant tasks that needs to be done.

Watch out that your R12 Upgrade doesn’t just focus on Oracle ERP resources. When we did the upgrade, we had Legacy guys, email guys, Web guys, external 3rd parties (Citibank and others), Health Insurers, etc. Miss those guys out and who can help you complete your UAT?

What’s amazing about an R12 Upgrade is that you’ll need exactly the same resources that are getting used for your R11 Maintenance and Support. That leads to automatic conflict and resource contention and delays (and when it comes to Production problems or R12 Upgrade, you’ll lose that argument every single time). So ensure you augment your team to not be 100% reliant on goodwill.

If you are getting a company to do it for you, be careful. Have they done an R12? If yes, will they give you resources that have ACTUALLY DONE AN R12? Most outsourcing or consulting companies might have done R12’s, but they are as keen as anyone else to stack your project with their guys who can then learn R12 at your expense. Nice how they’ll happily charge you for the privilege of training their people…….Interview all people, before they deploy onsite. Hire an independent highly knowledgeable freelancer Consultant to fight your corner and keep the vendor on the right track, working in your interests, not the vendors.

Do you know the effort involved in training people for R12? The IT guys, the Business guys, etc? If not, start to worry.

The great thing about an R12 Upgrade is that your team (and maybe your consultants or outsourcers) will all get fantastic R12 skills, in the midst of the hell of an R12 project. Trust me that’s a great way to learn. Of course all those companies that have not done R12 (and that’s 95% worldwide) will be looking and admiring your very skilled people. And your loyal people that you’ve spent time and money training in R12, will of course be looking for that next payrise with their new found skills. Losing key resources (say for instance your Payments expert) is your worst nightmare. Suddenly one single module can derail your entire R12 Upgrade. If you think this is the stuff of nightmares, well during our R12 back in 2008/2009 we lost our Payments Consultant (one of the few in Asia at the time). Luckily we had him shadowed throughout the project by another staff. The demand for R12 skills will hugely increase in 2011 and your guys will be headhunted or tempted away. Just be ready. I am sure the resigning employee will feel your pain as your project is now doomed, in the same way you felt his pain when you froze his pay last year, as part of a corporate wide pay freeze…….

Our final resource issue is to prepare for the maelstrom of Production support hell. Just make sure your well staffed and haven’t let the entire project team go, just before you get deluged with Production issues. (and yes you should start thinking about this not at the end of your project when everyone is jumping ship as the project for them is over, but at the very start of your project). It’s funny as R12 projects get to go-live, most of the work is done, and you start letting consultants/contractors go. Do you have any idea the pyschology in play with the others that you expect to see remaining to do the support work and provide critical support for a few months after go-live? Why should they stay when they can get a far longer, far higher paying and far easier (i.e. not pressured go-live support) contract with their fantastic R12 skills you gave them? Sleep on that one, assuming you can still sleep……..

And I hope you’ve got a full communication plan going on – an upgrade can take time and if you are not informing and working with the users throughout…..don’t expect them to be enthusiastic about putting large amounts of time into your UAT. Of course if you run workshops and demos and mini training throughout your R12 you’ll have a whole lot more acceptance during UAT and go-live…….

And finally Oracle Support……… R12 should be stable now, but back in the early days, we had these jokers on the phone daily. We even had a few removed from our support calls because progress was atrocious. Are you ready to deal with Oracle Support? Do you know how to escalate calls effectively? Do you know how to work around Oracle stalling for time or sending you on a wild goose chase? If not hire someone that does. Good ERP people with strong experience are essential for your R12. It’s funny but I bet you didn’t even think about informing Oracle Support of your upgrade, which is a shame as they might have given you a Critical Account Manager that could have helped you push all your Service Requests….

And that’s all for now folks. Part 2 will cover more in ghost, ghouls, implementation, transition and onto go-live, together with a whole pile of great references, covering fantastic presentations from experts in R12 and great Oracle Open World links of R12 Presentations there, together with a heavily researched bunch of Metalink articles around R12.

I’ll be scaring you very shortly with Part 2 before the full moon of halloween is over.

The other blogs can be found on



The great graphics were the work of:


Looks like 12.1.3 is Imminent

August 20, 2010

With the rapidly approaching of Oracle Open World 2010, it looks like 12.1.3 is also rapidly approaching.

The Release Notes for 12.1.3 are now out:

12.1.3 Release Documents

Having a quick look through probably the most interesting change is in the Applications Technology area.

It’s a shame that as far as I can see (correct me if I’m wrong) but they haven’t put BIEE dashboards directly into any ERP screens yet (with half the screen in OAF Framework and the other half embedded BIEE), despite enabling the technology in the previous 12.1.2 Release. For me that is a little disappointing, but hopefully by R12.1.4 we’ll see that.

It is now possible to link to Oracle Application Development Framework (ADF) Release 11g applications deployed on an Oracle Fusion Middleware Release 11g container from the Oracle E-Business Suite home page. Each ADF application will have an FND Function of type ‘ADF’ defined for it. If such a function is granted to a user, then when the user logs in, the home page automatically builds a link to the external ADF application.”

This is an interesting note on ADF Integration to ERP. I’m left wondering if clients can finally move from OAF to ADF and still have the integration to ERP. The new functionality sounds as if it will allow you to call cleanly ADF Applications from the Oracle ERP homepage. It’ll be interesting to see this as it could provide those who do go to 12.1.3 another good staging point to Fusion in the future that pre 12.1.3 sites don’t have. I’ve talked to a number of people in Oracle over the last couple of years on when to start using ADF instead of OAF, but the message has always been to use OAF. Obviously this left me wondering for the future on “will I need to convert all these OAF applications in 3 or 4 years as I move to Fusion Apps”, so not an ideal situation. Maybe over the coming months the OAF/ADF will become clearer.

On the Finance, Procurement and HRMS there is the usual small changes here and there, additional changes for different countries, but nothing overall that for my organization is really ground-breaking. 12.1.1 and 12.1.2 had a lot more functionality that we would be interested in overall.

A short reminder that demo Vision instances are available and a whole lot more from the excellent Solution Beacon site, allowing you to test drive the new features for those with a CSI Number from Oracle, before you take the plunge. Currently up to 12.1.2, as of the time of this article (20th August 2010).

R12.1.x Vision Instances

Hopefully we’ll all start to see the new mythical Fusion Apps over the coming weeks.

Footnote: A short footnote courtesy of Steven Chan’s excellent blog that 12.1.3 is now available. See the latest at:

  • Oracle E-Business Suite Release 12.1.3 Release Update Pack (Patch 9239090)
  • Oracle E-Business Suite Release 12.1.3 Readme (Note 1080973.1)
  • 12 Steps to R12

    August 16, 2010

    Date: 16th August 2010

    During the Oracle OpenWorld 2009 in San Francisco, the average conference session started with “How many of you are on R12?” The show of hands was no more than 5% in any given session. Recent research suggests only 5% of companies are on R12 as of mid 2010.

    Many organizations will be starting the journey to R12 in 2010, given the end of Premier Support for in November 2010. Our organization successfully completed an Oracle E-Business Suite Release 12 Upgrade (R12) in August 2009. This article shares our experiences so that readers can benefit from what we did right and avoid what we did wrong.

    Following are the twelve steps our organization followed during its R12 upgrade, along with suggestions and insights that others might use in their own upgrade projects.

    Step 1 – R12 Study

    It is recommended that before doing an R12 Upgrade, you do an assessment/study to determine the impact an R12 would have.

    Get a little funding, do test upgrades, an impact analysis, try the new functionality and then go for full funding of the project. This leads to a highly informed and carefully scoped project.

    Many companies got burned very badly by early adoption of R12. If a study had been done first, I doubt they would have upgraded so early – saving a lot of time, money and heartache.

    Step 2 – Business Case

    The key foundation to any project is a good Business Case that everyone agrees on. It gets the funding, visibility and users onboard. Emphasize the benefits of improved Oracle Support and significant new functionality.

    I would recommend that you keep the R12 Upgrade as much as possible a Technical Upgrade (utilizing new functionality where appropriate), but it is a daunting task and the fewer new things thrown in the better.


    Step 3 – Planning and Preparation

    There are a few things you should check prior to recruiting an upgrade team.

    Make sure you have the hardware to do the upgrade. Your typical upgrade will require an extra 6-12 Instances and you still need 11.5.x instances for ongoing production support. Prior to bringing any expensive external resources in ensure you have R12 Instances to work on.

    Write a detailed project plan. Figure 1 shows a very high level approach. Doing an upgrade is not rocket science…..the devil is very much in the detail. Remember you will need to schedule time with many different groups – Legacy, Data Warehouse, Business Users, DBA, Server Team and the ERP Team. During an Upgrade all major initiatives will need to stop to allow you to focus on R12, so be sure to set user expectations in advance.

    Recruiting – Each company varies on how they staff an upgrade. Unless you have a very good in-house team, you will need help. Consultants will save you time and money. R12 has a lot of new modules and functionality, so we brought in expertise. It really helped and knowledge was transitioned to our internal resources.

    The DBA Team will take between 2-4 weeks to get an R12 instance, depending on their skills. Put this into your plan.

    Doing a PM.010 – Project Management Plan helps you to focus, together with a solid Work Breakdown Structure. Together with a good Risk Register (brainstorm with your team) you’ll protect yourself from some of the pitfalls.

    R12 is a lot bigger than any previous upgrade in Oracle ERP. Your plan should reflect this. Planning is everything for a successful R12 Upgrade.

     Figure 1 – 12 Steps to R12

    Step 4 – Review and Design

    A short amount of time reviewing what you have and what is new will vastly improve your upgrade. The Functional Team should use the R12 Upgrade Guides to identify the relevant changes in Functionality.

    The Functional Team should also try to identify with the Business and Development Teams, customizations that can be removed with new functionality provided as standard. A little time doing this could bring significant savings.

    There is significant new functionality in R12 and the Functional Team should review with the Business to show the possibilities. Keep a balance – the focus should remain on the R12 Upgrade, not new functionality.

    The Functional Team should also be writing new MD.050 Functional Designs – there will be customization needed in R12 and other Functional Designs may need updated.

    Working closely with Functional Team, identify the core customizations. Once you prioritize the customizations you have a solid basis for moving into the Build and Test Phase.

    Step 5 – Build and Test

    Functional Stream

    The Team should prepare high level Test Plans, working closely with Business. The Test Plans should be comprehensive and signed off. This should cover all Test Scenarios across all Business Areas, including external systems.

    Once the Test Plans are signed off, ensure that comprehensive Test Scripts are written. As these are being written, the application should be tested initially to throw up the bugs. The earlier you find bugs the earlier you get them to Oracle Support to get fixes.


    Technical Stream

    The Technical Team MUST have a customization register. This can be used as a very good monitoring tool. If you don’t have this, how do you know what to upgrade? Customizations should be upgraded methodically according to your plan from the Design phase.

    Technical Summary – Our Customization Upgrade Impact

    General Ledger    80%   Minor issues – Set of Books ->Ledger_ID

    Self Service            30%    MOD*PLSQL Dessupported

    Accounts Payable  30%   Change of Database Structure

    Payments                  60%    Upgrade Check Formats

    OA Framework         <5%  Minimal Issues

    HRMS                            <5%  Minimal Issues

    Payroll                          <5%  Minimal Issues

    Procurement              <5%  Minimal Issues

    Receivables/Fixed Assets <5%  Minimal Issues

    Source Control

    This is essential so if you are not using any tools, you are doing something very wrong and risking your Production System. Open Source tools such as Subversion are ideal.

    Patch Management

    One key consideration is your approach to patches – aggressive or conservative. We went on an early version of R12 (released one year previously) and we choose conservative thinking it would be relatively stable. Towards the end of the project, we saw stability still had not been reached. We moved to a highly aggressive patching strategy that really did get us to stability, although it was extremely painful with multiple regression tests and carrying considerable risk.

    Reviewing Metalink is important – when new RUP’s (Roll-Up Patch Sets) and CPC’s (Critical Patch Sets) are released, you have a very important decision to make on whether to apply. If your well into your project, the advice I would give is not to apply unless required….patches cause a lot of issues. If the patch is newly released you’ll be the poor guys finding all the new bugs. Leave that to someone else.

    Obviously if you have critical errors, pile the pressure on to Oracle Support to provide single patches. At some point you are going to have to freeze patches, otherwise you will never finish. Learn to work with Oracle Support and Oracle Management regionally to resolve issues.

    Patch Management is critical and Oracle has some great tools for Patch Assessment under System Administration – these will allow you to make informed judgments on whether to apply patches or not. This will show all the objects changed by a patch – with that list your technical people can give you well informed advice.

    Strong Patch Management is critical to any R12 Upgrade. Lose control on this and you lose control of your entire project.

    Step 6 – Internal User Acceptance Test

    An Internal UAT is basically a UAT but it is carried out by the R12 Upgrade Team, not the users.

    Focus strongly on key functionality – if one thing has to work on Day 1 of Go-Live it has to be the key stuff – entering invoices, running a payroll, etc. Too many companies focus too much on the peripheral and not enough on the core functions. That will end in tears.

    Given that some modules are going to be delivered before others into UAT, you may want to do this staged. It’s not ideal but it does compress the overall project time. You will have problems on Payables, and should that hold up your whole HRMS track? I would argue no. And I am sure some will disagree. But be pragmatic on the area of UAT and use your judgment.

    Test the entire ERP and external systems such as Legacy, your planning systems, your banks or other external systems that interface to customers/suppliers/etc. All bugs should be recorded.

    A few other tips:

          Simulate a close on your R11.5.10

          Reconcile 11.5.x and R12.x

          Simulate transactions half way through workflow

          Check your banks migrate correctly

    Step 7 – User Acceptance Testing

    Another clone of Production should be taken and all customizations/setup applied. If you have not been keeping deployment registers, setup documents, etc I would strongly suggest you start doing so. Give the users plenty of notice of when the UAT is coming and make sure you clearly set the expectations.

    UAT kickoff meetings should be held and contact lists given to users. Your team will need to work hand in hand with the Business Users to co-ordinate the UAT. It’s going to be a lot of work with new, unfamiliar screens and quite a few complaints, no matter how well you’ve done till now. The Test Plans and Test Scripts should have been agreed months ago, with full review and signoff which should lessen the surprises (and complaints) users may have. Work closely with your users throughout the project from project initiation to way after go-live. That will lead to better relations and when you do encounter problems; users will go a lot easier on you, because you kept them informed right?

    One trick here is to put the Test Plan on a shared drive and ask the user to mark Pass/Fail/To Test as they progress. As a Project Manager you can then review and summarize progress easily. Delivering some modules into UAT before others may be an option if you want to progress quicker. Others prefer to deliver the entire UAT. What I have found is that by progressing some modules quicker, it gives the project momentum and when you put out weekly progress tables to users, they don’t want to be last to finish. A little bit of Psychology certainly helps when doing an R12 upgrade.

    Make sure all Test Scripts are tested. Make sure Month End and Year End are simulated, with proper reconciliations. Make sure all the systems to and from Oracle ERP are tested and continue to work with R12. Make sure integration between modules is smooth and working – organizing this across departments is tricky. A UAT room for this can be handy. Providing simple refreshments can create a good ambience and encourage your testers to work for you.

    Record all issues and bugs as you go. Monitor closely and ensure that when your users are doing UAT, your R12 Team is there to assist. Some areas may need twice weekly meetings on certain modules with the Business Users. This will help to manage difficult UAT areas, stop user frustration and make the UAT a little less painful for all. We did this on Payables in particular. It worked extremely well.

    One point – be pragmatic on UAT. Remember users have full-time jobs so doing a UAT is a big favor and usually a lot of extra work. Avoid being too pushy especially during stress points such as month-end closing and Payroll – that will simply be counter-productive.

    The UAT can also double as a training ground, but do also be prepared to give additional training, especially in the areas of AP, Banks, Payments and Subledger Accounting/Reconciliation as there are significant differences here.

    Step 8 – User Sign Off

    On completion, ask the head of each area and other users to sign off a standard Acceptance certificate. As you achieve sign-off let everyone know – again it helps to push the other Business Areas that have not signed off. I found weekly newsletters (just a single page) kept everyone onboard. Openness and transparency is the key.

    Step 9 – Transition Planning

    As UAT goes on, you should be planning what will be a very complex and difficult cutover. The team should also be doing rehearsals by taking copies of production, upgrading, deploying, setting up and testing. All bugs should be recorded.

    You should have deployment registers for each module, including tasks for DBA, System Admin, etc. These should be very detailed and concise.

    Review the deployment registers and figure out what can be done in advance of R12. This saves time and will ease the stress of the upgrade weekend.

    Call meetings with the users and make sure that the cutover is clear.  The users will lose Production for a few business days, unless you do it on a holiday.

    We did the closing in R11.5.10 just before cutover, closing down all transactions, etc. Why? There were a couple of reasons. If data is left open in R11 (say unaccounted) it will give real problems in R12 Secondly it allows you a whole month before your first closing in R12 to resolve issues.

    Get a register of who needs does what during cutover and ensure work is balanced. Ensure every person knows their role in the cutover.

    Deployment of code is a significant task. If you have used Subversion (or another Source control system) then you can grab all changes and give to the DBA’s. A lot can be automated and this will save a lot of time on the cutover.

    Talk to Oracle Corporation and make them aware of your key dates, cutover, month end, payroll, etc. They can get you critical support management over that time.

    Try and do as many rehearsals as you can whilst users are doing UAT. The closer the rehearsal is to an exact recent copy of Production the better. We repeatedly cloned production for three weeks prior to Production and checked our deployment.

    Make sure you do patch reconciliations between your UAT and Rehearsal databases on a consistent basis. Otherwise you may be testing on completely different Patch Sets and your entire UAT would be null and void.

    As Project Manager, you should be writing a PM.030 Transition and Contingency Plan that covers all the angles.

    Make sure you go to the Change Control Board, publicize on your website and make sure you tell every single person that needs to know that Oracle Applications will not be available. The Cardinal Sin – missing someone out.

    If the cutover fails all the hard work of your team will have gone to waste. Do you really want to be telling everyone publicly (including Senior Management) that the nice, shiny R12 that was promised isn’t live and that you had to pull out the whole upgrade at the last minute due to your bad planning? 

     Rehearse, Rehearse, Rehearse……

    Step 10 – Cutover

    A cutover generally takes place over a four day window, depending on your database size/speef of servers/etc. Holidays are ideal but make sure you tell Business well in advance if their key system is going to be down during business hours. Total Upgrade time took 96 hours, working around the clock.

    Users should run key reports on R11.x and then run the same reports immediately after on R12.x. Your functional team should also do a mirror reconciliation exercise quickly during the cutover. Also keep a reference environment copy of your R11.x production.

    Wednesday evening and Thursday were days for the actual Upgrade, Friday the Upgrade Team rolled in to do setup, customizations, etc. Saturday and Sunday were for testing with a go-live decision on Sunday evening. As we were moving servers, our contingency was simple; if the cutover was a no-go we simply turned our old server back on……Don’t repeat the errors you made in rehearsals, keep a log of everything that failed and make sure it doesn’t happen on your go-live, as you simply won’t have time to troubleshoot.

    Be very careful that all transactions are accounted, workflows complete (as much as possible), payments complete. Review the Best Practices from Oracle for more detailed advice. If you don’t do this you will hit serious problems.

    Cutover is a stressful time no matter how well planned. The Project Manager should ensure everyone stays calm, ensure work is balanced and avoid burn-out for those working shifts. Organize some food during the cutover, step in to calm any friction quickly and monitor every last aspect of progress.

    Step 11 – Critical Support Cover

    There are going to be problems on anything of this scale. That’s a given. Have all teams on standby so that when issues arise, they can be addressed. As soon as possible clone production to a test system and simulate the payroll, closing and other key events in advance.

    Before the Go-Live, set the expectations of users. If you do this the user community and management will cut you a little more slack and be more understanding of issues encountered. A little bit of psychology and transparency will go a long way to making your transition a lot less stressful.

    One tip that is useful before go-live: Brief the Helpdesk on how to take the calls as you go live. Brief the users on how to log issues. Make sure there is strong reporting available to you in real-time showing all issues. Monitor this religiously. Make sure the escalation and support teams are in place, so calls are not lost or delayed. This will ensure you have a very strong support structure to deal with the significant number of issues you will have, irrespective of how well you did the upgrade.

    No-one will remember all the hard work you have done to get here if major issues are not addressed quickly. If you manage problems well and quickly the Upgrade will be remembered as a huge success. If you don’t, you’ve just thrown away thousands of hours of your team’s reputation and hard work.

    Strong and very well managed Critical Post-Production support is everything in an R12 Upgrade.

    Step 12 – Post-Production Support Cover

    After the first month end close you should be able to move to a more standard support footing.

    Don’t celebrate the day you go-live. It could get extremely embarrassing to celebrate Mission Accomplished and then hit major problems, right? We left the celebrations for 6 weeks.  If you do an R12 Upgrade, you deserve a big party. R12 is the most challenging upgrade you will ever have done in Oracle ERP.

    Those who fail to plan, plan to fail. Good Luck!!!

    About the Author

    Iain Robertson has been using Oracle Applications since 1990. Over 20 years in Oracle ERP starting in MPL7 through MPL8, 9.x, 10.x, 11.x and 12.0.4. He is currently working as a Program Manager running a team of 90+  in Asia Pacific, migrating complex Legacy Systems to Oracle Applications – Procurement, CRM, Financials, HRMS and Projects Suite. He also runs the Oracle Support Team. He has experience across Asia, Europe and the US, specializing in R12 and building world-class teams in Asia-Pacific providing Global Support.

    You may contact Iain at



    #1 IT Service in the Company – 2008

    Oracle Innovation Award Winner 2009 – ERP

    Oracle Session Presentation Winner – Oracle Open World 2010

    R12 Successes


    Reference Material

    To those authors who published articles I would like to give credit, for making an almost impossible R12 Upgrade possible by kindly sharing their knowledge, their successes and their failures for the benefit of the entire Oracle Applications community.

    R12 Resources

           OAUG Website

          Chris Warticki’s Blog – EBS R12 Support Resources – Consolidated

           Oracle Applications Upgrade Guide: Release 11i to Release 12 (B31566-01)

           Upgrade Manual Script (TUMS) – Patch 5120936

           Metalink Note: 580299.1 – Best Practices for Adopting Oracle E-Business Suite, Release 12

           Metalink Note: 394692.1 Oracle E-Business Suite Upgrade Resources

           Metalink Note: 562887.1 R12: Helpful Tips for a Successful R12 Oracle Payables Implementation

           Metalink Note: 437422.1 R12 Troubleshooting Period Close in Payables

           Metalink Note: 73128.1 R12 Troubleshooting Accounting Issues

           There are many articles from Oracle Open World in 2008/2009 for R12 available at Oracle Open World Tools -> On Demand:


           Oracle Financials Applications Blog – R12 Lessons Learned

           Analyzing, Planning and Executing an R12 EBS Upgrade

           Rochester Institute of Technology – One Year after Upgrading our Oracle E-Business Suite from 11.5.9 to R12 – An Update 

          Talk to other companies that have done R12. Use the communities and forums.

    R12 – A Detailed Transition Plan

    September 19, 2009

    So your UAT is heading for completion, you’ve got as many bugs out as you can and your ready for go-live.

    Problem is…..this is the most dangerous part of the entire project. All the hard work and success could go be wrecked over a poorly planned weekend (or long weekend if your upgrade is really tricky or big)

    Planning is the key.

    It’s recommended that everything is documented very, very heavily. Start off with a PM.030 and get into your head exactly what your going to do.  And a PMP.030 Transition Plan isn’t just what you will do over the cutover weekend; its before, during and after – how are you going to support the R12 when live? Are your teams ready for that?

    Get all your Functional Setup into a Spreadsheet, BR.100 per module or other installation documents. Give assignments to each person. Make it clear who does what, when they do it, order, dependencies, etc. On a cutover weekend there is ZERO time for mistakes.

    Have a spreadsheet of all tasks to countdown, dates, people, etc. Mark them off one by one as you go.

    Have another spreadsheet for all cutover tasks. I suggest some sort of matrix where setup is assigned and you can see each persons workload. Then try to balance that workload. If one person is too slow or too busy on the cutover, you’ll hit major delays on cutover weekend.

    Your source code should be in a Source Control System (we used Subversion – it’s FREE – it’s GREAT). You should repeatedly clone Production and Test the deployment until very close to Production. Also apply Functional Setup and retest.

    Are you confident that what was passed in UAT is what you’ll put in Production?

    Keep doing the clone, setup and test until your happy that the answer is 99.9% yes.

    Also during final approach to go-live, do a simulated financial close on the Clone. Trust me this will save you a lot of heartache. Wouldn’t you rather find disastrous stuff you need to do before you hit it in the crazy, high pressured time after go-live. Pay careful attention to AP closing. It’s tough. Run the accounting health checks.

    You must have an automated mechanism for code application and your DBA’s should be familiar with this. And by automated it can be as simple as a shell script to move stuff to the right place and a script to cut and paste and compile objects. Get this 100% before you do it.

    One tactic that worked well for us was to do setup on 11.5.10. We’re not saying everything, but some menus, some value sets, messages…..every little helps. Of course you can also do these on the day using FNDLOAD, but the point is the more you can do ahead, the less you have to do in a high pressured weekend.

    Get the meetings going with Business. Make sure everyone is aware of how and what is getting done. We closed on R11.5.10 for all subledgers prior to R12 apart from GL. This really helped reduce overall errors and risk, as all that had to be done in week one of R12 was a GL close. It went very smoothly. And then you have 4 weeks until your first real R12 close – gives you time to simulate another close 3 weeks into your financial period. Now some may disagree with this approach and prefer to go-live after month end closing. I understand this is another perfectly valid option. In our case we made the decision to jump immediately after subledger closing as we also run two large payrolls twice a month…….so our cutover windows for making the jumps were seriously limited. Also we had done this procedure many, many times (including on very recent clones), so had a level of confidence that this was safe to do. If your cutover testing isn’t giving you that confidence, suggest you close in R11 fully then jump to R12.

    Make sure you have plenty of meetings with your DBA Team, Unix Team and Management. I prefer to plan for the worst; set the expectations low and then everyone will be prepared mentally for a tough time. If it goes smooth then everyone is happy. Doing an R12 is the toughest upgrade anyone will have ever done, don’t underestimate the issues you will have. Many, many companies have had horror stories during go-live.

    So your plans are done, teams are briefed and your now ready for launch. Did you actually remember in advance to tell all users (and post notices on your website internal and external) that your system is unavailable due to upgrade? Did you remember to tell all the other systems (Legacy, data warehouse, etc) that your system is unavailable?

    Oracle people (including myself think the world is Oracle). Remember there’s more to life than Oracle…..and most companies have a lot more systems than just Oracle, although Oracle is the foundation… if you take that away and forget to announce, you’ll have some very unhappy people……

    Make sure you brief your helpdesk. Make sure they know the escalation procedures. Even if you do a great job, it’s going to be a busy time. Make sure you Oracle ERP Team is also ready to work round the clock post go-live, just in case…….

    We brought our ERP down at 5:00PM on a Wednesday after closing all subledgers. Actually it was 5:52PM and the DBA’s were already complaining like crazy.

    The DBA’s in our case brought the system down and started the export. They worked through the night to export from IBM and bring this into HP Linux. (Yes, we throw in not only an upgrade but also a migration for fun…..)

    The Upgrade started on the Thursday and was then backed up early Friday morning.

    The DBA’s automatically deployed the code (7,000+ files) onto what was effectively a live Production system. (with no one else on and of course backed up on R12 AND R11.5.10  before we started).

    The Deployment Team rolled in at 10AM on Friday, sat around for a few hours and then started at 11:45AM to do setup. Of course we screamed at the DBA’s for being 1hour and 45 minutes late, but they claimed we were 52 minutes late handing the system over on Wednesday…….things are always tense during an upgrade cutover.

    Functional Setup continued until 9PM on Friday night (remember all those new modules, Payments is a large setup, and there are loads of System Admin tasks  – Hint split some setup into Application Developer responsibility. Do others setup using FND_LOADER. I think your System Admin may be a bottle neck – that’s what happened with us as there were just so many changes between R11.5.10 and R12)

    Around 9:30PM Friday the database was ready to be backed up and cloned. DBA’s worked to clone the database and make it available at 10:00AM Saturday.

    Testing Team rolled in on Saturday and did a quick Financial Reconciliation. All well and good testing started on major key functionality. I hope in your cutover plan you detailed what would be tested, who would test, etc.  In addition each Team Lead should prepare a more detailed Test Plan for the transition and brief their respective teams. Concentrate on key functionality. If you do this, you’ll be doing well if it all works on day 1. Testing continued for about 10 hours on Saturday and started at 9:00AM Sunday.

    As we caught problems, we logged them and got our System Admins/DBA to deploy missing items to Production (with heavily controlled approval, logging of bugs, etc so we tracked every single change)

    Around about 4:00PM on Sunday we had a Team Lead meeting. Each Lead was asked for a go/no go signal. With the decision to go made, we went to Management. The current status was reported and management after discussion gave the green light.

    All workflow services were turned on (with emails being caught in our email system so we could monitor and then release – nice little trick). Once we were happy emails/workflow was OK, we released one (a Leave from one of my staff) and tried the complete cycle. Once we were happy with this, we released all workflow, all concurrent jobs (we had simulated release of all concurrent jobs on Saturday to avoid surprises) and there was no going back.

    We went live on R12.0.4 (RUP5) at 5:00PM on 2nd August 2009, some 96 hours after bringing down the R11.5.10. (Although we are on RUP5 we’re really heavily patched on Financials so RUP5 is a bit misleading)

    Teams had worked round the clock, across all areas to make the cutover a huge success – we finished 30 minutes ahead of planned time. Now that’s pretty good planning. Have you planned to that level? If not you should be.

    On 3rd August, our users logged on to R12 and we were fully operational.