Posts Tagged ‘Patch Management’

The Quest for the Holy Grail of R12 Patch Management

April 26, 2011

With new technological innovations, please wear 3D Glasses and Headphones for optimal experience whilst reading this blog. This represents the first ever blog to cover ERP Columns and Movie Columns, together with hot off the press exposes that Oracle Technology came from Alien UFO’s captured at Roswell, all in 3D and Dolby Sound. Popcorn and a coke are recommended.

Ensure your PC sound is turned down if in the office with no headphones.

In the time of King Arthur, legend had it there was a magical cup, a holy grail.

Whoever drank from the cup would have immortality. During the time of King Arthur, many knights embarked on dangerous quests to find this magical grail, always to fail. The question is, were those pursuing their quest searching for something, that perhaps did not actually exist?

Sometimes it feels like Oracle Patch Management is perhaps a search for the holy grail. To be able to stay current on patches, to be able to apply patches and still ensure that no adverse effects are inflicted on the Business seems an almost impossible quest. This column looks at some of the strategies that can be used, to perhaps not achieve the holy grail, but certainly move towards the holy grail of patching ERP.

Now previously some crazy oracle prophet had written about the Art of Zen Patching . Let’s be clear on this – the quest for the Holy Grail of R12 Patching is not such a zen-like endeavor. It is an arduous, painful and often disappointing journey, frought with failure, despair and danger.

And at this point I feel a movie review is appropriate.

One of the most famous movies (and one of my all time favorites EVER) covering the times of the Holy Grail has to be Excalibur. Highly recommended on DVD. A superb and very haunting movie. It is one of the most atmospheric movies I have ever seen. You just have to see it.

So the movie is about a search for something that people dream of but quite possibly doesn’t exist. It’s about human dreams that become all consuming leading people to eventual madness and despair.

For many years I have been searching for what can only be described as the Holy Grail of ERP Patching. I started off pretty sane, but as the years went on my sanity deteriorated until I started writing columns combining movie reviews and Oracle ERP tips. I tried to turn to Zen (The Art of Zen Patching), but lately, just maybe I found the Holy Grail of Patching. I uncovered this during an R12 Upgrade Quest. In the midst of battle, with patches coming out daily from Oracle (on 12.0.4) we perhaps found not a perfect grail, but certainly something that made the quest worthwhile.

As I explained in a previous column there are a few patch scenarios:

1. Security Patches, Database Patches and ATG Patches on a Production System. This column could be used for that, but I’d also recommend the Art of Zen Patching for these types of patches. These patches are generally stable (if left to others to apply first) and cause less problems from experience. Note however we HAVE had problems on each of these types of patches, so don’t underestimate the fact that the quality is better that you can be sloppy on your testing.

2. Patches that come out during an R12 Upgrade Project or need to be applied. I guess this column kind of applies, although depending on how far your upgrade is you can take a more aggressive patch approach. If you are early in an upgrade, you generally throw in the Rollup Patches or others as needed. The later in the project you are the more careful you need to be.

3. The Holy Grail of Patching refers to the approach taken for Patches on a Production System. It is like searching for the grail, a very slow and dangerous process, with little in the way of rewards.

As everyone knows Oracle has patches to fix the patches you applied. This isn’t good, because like Forest Gump, Oracle gives you a box of chocolates, and your never sure if there is something unpleasant in one of them…….Or as I say, our office coffee machine (I live in the Far East) is like an Oracle Patch. Occassionally you get a cockroach dispensed……..(and no I am not joking on this one – cockroaches like Nescafe too…….)

Our approach to the Holy Grail of Patching can be described as a highly conservative and disciplined approach, based largely on the principles of ITIL. Our primary aim is not one of speed, but of keeping our Production Systems safe and stable, to avoid major business disruption. It is a very slow and very painful process.

So let me walk you through the process (Apologies, it’s going to get a bit serious from here on in. Have a look at the Alien Technology blog if you need something lighter……Oracle Fusion finally went live in one company. Poor buggers………Although it is based on captured Alien Technology from Roswell……………so I am sure as a first release of an Oracle product quality is great as it always is……. )

The bug process starts either with the EBS Support Team (I assume you are proactively look for bugs/problems before they happen, by keeping an eye on Oracle Support Updates) or the business user reporting an incident. The Incident Management process should record the details and check if it’s a known problem or there is a workaround.

This should then be passed to your Problem Management Process, where you investigate the problem reported (DS10B-001).

You should then try to replicate the problem (DS10B-002).

If you can replicate, then it’s time to log on to My Oracle Support (DS10B-006). (I’ll slowly introduce some ITIL Terminology into this column. ITIL is a framework to manage IT Systems and it’s very, very good indeed, applicable not only to ERP but to any IT activities. If it was a movie I’d give it a 10 out of 10, but it’s a bunch of great books that can be found here. Introducing these ITIL concepts can seriously improve your service to your ERP users.

And onto another movie review. It has to be Monty Python and the Holy Grail. This is about as British as humor gets. A great movie and very, very original. It’s a 9. Get it on DVD.

On My Oracle Support you should check for similar bugs. It’s a great database and no doubt someone else has hit your bug before you (unless you are one of the suckers to take an early release).

You’ll get to one of two points here. You’ll either identify a patch (which may have pre-reqs) or you’ll come up blank. Lets deal with the second point.

If there is no patch (and don’t be lazy not bothering to look) then raise a Service Request (DS10B-012). Now Oracle’s game isn’t necessarily to help you fix the problem………..Typically responses are as follows, but we suggest you counter-act these by using phrases from the movie Monty Python and the Holy Grail. Assuming Oracle Support doesn’t hang up, this will certainly be a tactic they have not seen from other companies………..

Oracle Support – Good Morning. How can I help?

You – “Go and tell your master that we have been charged by God with a sacred quest.”

Oracle Support – We’re very sorry to hear that.

You – “Everytime I try to talk to someone it’s ‘sorry’ this and ‘forgive me’ that and ‘I’m not worthy’!”

Oracle Support – Sorry Sir, could you explain the problem with your system?

You – “Come and see the violence inherent in the system!”

(We recommend you explain the problem, maintaining the Monty Python style and English accent – pretend you are an old English queen)

Oracle Support – That’s standard functionality Sir. The accounting on the Payables isn’t supposed to do proper exchange rate variance calculations, otherwise your General Ledger would be correct for the 30M Dollars you spent on Oracle. Now that sounds perfectly plausible, doesn’t it Sir?

You – “Your mother was a hamster and your father smelled of elderberries.”

Oracle Support – Sorry Sir, let me just check (By this point Oracle Support is a little concerned)

Oracle Support – Ah, it’s a known bug. It’s in development………..should be out in six months…….

You – “I’ll bite your legs off!”

Oracle Support – Wait Sir, Wait Sir, let me check. I’ll check again. You need to go to R12.0.6 (By this time the support person is probably thinking “nutcase on the phone, better help him quick”……)

You – “Jesus Christ!”

Oracle Support – This small patch will definitely fix your problem !!!! (Now Oracle Support knows this has nothing to do with your problem but ask you to apply it anyway just so you get off the phone and they can close the call………)

You – “Pull the other one!”

Oracle Support – OK we suggest you apply the latest Rollup Patch

You – “I fart in your general direction.”

Oracle Support – OK how about this patch, which has 300M of Pre-Requisites. That will do right?

You – “Aaarrrggghhh!”

Oracle Support – What about a one off patch that no-one else has that will trash your system and is cobbled together by our developer?

You – “You tiny-brained wipers of other people’s bottoms!”

Oracle Support – But we’ll need to spend 4 weeks of burueacracy giving you a hard time to justify our development team doing this for you, even though it’s a serious bug we should fix……..

You – “You can’t expect to wield supreme executive power just ’cause some watery tart threw a sword at you.”

Oracle Support – Wait I’ve actually done my job and found a correct patch………

You – “I feel happy!”

A very Monty Python approach, but every scenario above represents some actual conversations with Oracle Support……(bar the Monty Python quotes – I normally pretend to be Harry Potter when talking to Oracle Support).

They may also pretend it’s a problem for another team (e.g. AP blamed the SLA Team and vice versa) or point to the good old technology stack components or database components. Common tactics by Oracle Support.

There’s a very serious point to the above. It shows the strategies that Oracle Support uses against you when it comes to patches. Your end game with Oracle Support is to get you to the minimum patches required to fix your problem. You and Oracle have very different aims sometimes (Oracle’s current aim is to move everyone onto minimum of 12.0.6…..), so you have to be very careful in how you manage the Oracle Support Analyst because that seriously affects your patch outcome and therefore testing work and eventual risk to your system.

And it’s got to be time for a movie review. This is another favourite Indiana Jones and the Last Crusade. Great movie for all the family. It’s a 9 out of 10 and keeps our Grail movie theme moving along nicely. This makes me feel old, because when the first Indiana Jones movie came out I was 10. Now Indiana Jones (Harrisson Ford) is a pensioner…….

So you’ve got that patch sitting on your server waiting to be applied. Do you feel lucky punk? You’ve reached DS10B-016 in the Patch Management Process.

A quick excerpt from a previous conversation I had with a person in one company:

Techie Guy -We need to apply the payment patches this weekend.

Oracle Prophet – Why?

Techie Guy – My manager told me to

Oracle Prophet – What did he say?

Techie Guy –“If it’s not done by sunrise, I’ll cut your balls off.”

Techie Guy – So we had real problems with the application of patches in a Test System that knocked out all Payment functionality.

Oracle Prophet – So you’ve never had a clean run of applying this patch?

Techie Guy – No, that’s right. Never been applied correctly in any test system. (Stated not with concern, but an unbelievable stupidness that is rarely found). So can we go ahead and apply it in Production this weekend?

Oracle Prophet – “I’ve got no option but to sell you all for scientific experiments.”

This was a true conversation of one person considering applying patches that he had never gotten to work during a testing phase… our Global Payments Systems……….truly a new way of testing patches. “Yep it didn’t work in any test environments, but maybe it will behave differently in Production when we apply……….”.

Our first step for Patch Application is normally to look at the size of the patch. Now a lot of people say size does not matter – many woman beg to differ. In this case, the smaller the better – the patch I mean. It gives an indication of just how many objects it’s going to hit potentially. Not scientific but a quick check gives a reasonable impact assessment. However……….

Applying patches is about a horrible and painful quest with little reward. But going back to King Arthur the Oracle Prophet recommends you seek a mystical wizard. Yep, your thinking I’ve lost the plot again on this one………..Just trust me. Go into your Manager, now obviously your credibility is shot to pieces after suggesting using the Art of Zen on corporate systems. Obviously he thinks you are a bit mental after suggesting using captured alien technology for deployments of ERP customizations.

But trust me, when you tell him you should seek out a mystical wizard to improve patching, you will truly have made those other suggestions look almost coherent………

So King Arthur had Merlin as his wizard. On your quest, we suggest that you take a look at the Patch Wizard from Oracle. (I surpass myself in corny literally prose this time……..)

Oracle has had a Patch Wizard for a considerable amount of time. I am not here to teach you all there is in this (see the excellent authors at the end of this column). I am here to raise awareness of these tools, what they do and how they fit into the Holy Grail of R12 Patching. The Patch Wizard can be found under the System Administrator Responsibility.

Once we have assessed the patch size, read me, etc, we normally try to apply the patch on a temporary database (we usually have one we can apply on). (We’re now on DS10B-007 in the Process Chart previously).

The Patch Screens below are a very useful impact assessment tool. Now size isn’t always an indication of how satisfying the patch will be for you………After applying the patch, bring up the Applied Patch screen and query the patch applied.

So a patch may be 300M. But it may not apply many files, if your patch levels are already recent.

Looking at the files copied and Action Summary will give you a very good idea of just how satisfying that big patch has been for you.

If a patch shows that it is replacing database objects, forms, etc then it certainly has had potentially an impact on your system.

We’ve faced large patches, but when looking at the files applied, we may only end up with a handful of files. This allows us to then use our technical guys to put a reasonable assessment on roughly what we have to test.

With an idea of what we have to test, we can now draw up a reasonable test plan for the patch. Without the patch screens we would be looking at guessing the best approach to testing. Now it is not easy to work out from the files exactly what to test – you need to be careful in that respect.

Now a message to Oracle. Why don’t you enhance the Patch Wizard to do a mapping between code files and Functionality. This would allow Functional Users/Support Teams to far more easily assess the patches, then draw up a far more accurate test plan. This would be an excellent enhancement for Patch Management for companies globally.

One of the problems with patches is where they overwrite your customizations (customizaton by modification). I suggest that where you modify shipped Oracle code to make new customizations, you record these in the appcust.txt file. To quote the manual “Oracle Applications uses this file during patch processes to generate warning messages that customizations are being overwritten or may need to be replaced after the patch”. It’s a handy tip. A missed customization that is overwritten by a patch can cause immense damage later……….

Our approach to testing patches is comprehensive.

We apply the patch onto a DBA Instance. (Step DS10B-008). This lets the DBA’s check the pre-reqs, make sure the patch applies cleanly, makes sure objects are not invalidated, etc. We then do a very quick sanity test.

We apply the patch onto a PTH Instance. (Step DS10B-009). Our test here is not thorough, but gives us confidence that the patch has fixed the problem and does not break any other key functionality.

We apply the patch onto a DEV Instance. (Step DS10B-010). Testing becomes more thorough at this point from our internal team. (Make sure it is a clean DEV instance, else use another clean environment). Once complete by our IT ERP Team, we release the Patch to our users.

We also test critical functionality when applying a patch. The drop dead functionality such as entering an invoice and paying. We also do an end to end around the patch when necessary (e.g. a quick cycle through procure to pay). Testing patches well in the IT area of ERP ensures quality patches are delivered to the Business. The messy dealings with Oracle and cycle of patches to fix patches is kept out of the Business Users view, as sometimes this could undermine confidence in the patches themselves otherwise.

We apply the patch onto a TST Instance. (Step DS10B-011). Our users then independently test this patch, using their own test plan and test cases. Keeping the testing independent (separate IT and Business Testing) gives more chance of any issues with the patch being caught.

For testing patches, we have suites of comprehensive test scripts/test plans across our application. We built these up as part of our previous R11 and then R12 Upgrade. This allows us to test our scenarios easily and quickly. It also covers integration to other systems. With freehand testing, formal test script usage, business user testing and IT Testing, we avoid mistakes.

With sign-off from the Business Users (DS10B-018), the patch is passed to our Release Management process for scheduling into Production.

With the use of the Patch Wizard and Patch Screens in R12, our testing becomes much more targeted, minimizing testing effort but re-utilizing our test script library selectively.

By having multiple databases with multiple testing cycles from both business and IT, our testing becomes much safer, reducing risk of a bad patch reaching production. It also protects other environments including your Development and Test Instances. You wouldn’t want to wreck these with a bad patch usually.

There are a few options we are looking at to further improve our Patch Management:

We are looking at Oracle Configuration Manager as a potential to alert us to patches that need to be applied, moving to a more proactive patch management policy.

We are looking at Testing Automation. Our preference is the HP Tools with pre-built testing scripts for ERP, but we know there is a huge amount of work before we could get to proper automation. Oracle also has a Testing Tool, but I still prefer HP’s to be honest. Automated Testing does seem to be the holy grail eventually – to be able to press a button at 5:00PM, have all the test scripts executed and come in at 8:00AM and see the results really is something that I think companies should aim for. However it’s expensive, difficult and often a huge failure when companies try to do this.

Finally we just bought Oracle RUEI. I have written about this in the past. It is a support revolution. We can now see the errors that users are encountering, the system performance that they are experiencing and so much more. Again it’s a move to a much more proactive incident/problem management. It even has an option to playback user errors. As said no more having a helpdesk asking “And what step did you do next. What data did you enter, blah, blah, blah and failing to take the right information for the Support Analysts”. It has pre-built integration into ERP and Oracle had this up and running for us in a matter of days. It even works with our custom modules……… I’d recommend this product to anyone running ERP. It’s a super product, quite revolutionary in terms of support.

And the final movie recommendation. Dan Brown’s Da Vinci Code. I read the book (also highly recommended) on a long haul flight to the Far East. Then I watched the movie on the TV a few years later. The movie is pretty interesting and I’d say it’s an 8.

Personally I always thought Mona Lisa looked absolutely miserable……

But Dan Brown’s book caused a huge amount of controversy. It’s a full circle from where I started the column. Many looked for the holy grail yet never found it. Dan Brown’s book was based on the premise that the Grail wasn’t really an object, but the grail was actually a person.

I’d say that to find a Holy Grail of ERP patching isn’t perhaps looking for an object or a magic bullet. I’d say that the Holy Grail of ERP Patching is also, actually a person. A person that puts in place the strong processes, controls and procedures, whilst thinking smart to reduce the work to enable a safe and reasonable patching strategy that supports the business goals. That is the true Holy Grail of ERP patching.

One last question – why are you wearing those 3D Glasses? You look pretty stupid in those in the office, given this is just an ordinary web page……..

And to end on an equally silly Monty Python note:

Woman -“He’s the Messiah !!!!”

Oracle Prophet – “I am not the Messiah!”

Man – “He’s not the Messiah. He’s a very naughty boy.”

Until next time when we look at R12 Patching – Friday 13th – A Horror Movie Special.

Further secret blogs and prophecies can be found at:


This column was based on some very hard-won experience. However other parts of this column came from research from generous authors in the Oracle Community. Below are some excellent articles that further shine light on the difficult process of Patching ERP safely.

Oracle E-Business Suite Patch Wizard – Path to Less Errors

Release 12 eBusiness Suite Codelines, Codelevels and PatchWizard

The 11i Patch Wizard

Managing R12 EBS

Tips and Tricks for Patching R12 EBS

Best Practises for Patching and Maintaining EBS R12

Real User Experience Insight