R12 Patching and the Art of Zen

Reading through Wikipedia, I found an interesting article on the concepts of Zen. Now I’m not really into that type of stuff myself (each to their own), but I thought it would make an original way to present this article 🙂

“One practice of Zen Buddhist’s is Koan Inquiry. A koan is a question, or statement, the meaning of which cannot be understood by rational thinking but may be accessible through intuition. The answer can occur during meditation or during your typical daily life with all the mundane tasks you do.

To Zen Buddhist’s the Koan is “the place and the time and the event where truth reveals itself”. It is a way to induce an experience of enlightenment or realization, not through rational reasoning, but through intuition.

Answering a Koan requires a student to let go of conceptual thinking and of the logical way we order the world, so that like creativity in art, the appropriate insight and response arises naturally and spontaneously in the mind.”

Or to quote from a very non-Zen perspective, you think about a problem very hard all day. You fail to make any breakthrough. During the next morning, in the shower, without even thinking of the problem, you suddenly think of the idea. Ironically perhaps we are all practicing Koan Inquiry as a natural state of mind to solve difficult problems, without even having to think about the problem at hand.

Now let’s look at ERP Patching in relation to Zen 🙂 We need to be clear from the outset that this form of Zen Patching applies only to the following patches below. This is extremely important to keep in mind.

Security Patches

ATG Patches

Database Patches

This form of Zen does not apply to other ERP Patches

Applying this form of Zen Patching to any other types of patches will cause you some serious grief in your career when you report to your boss that your ERP for your entire organization worldwide is trashed because you read some amazing article by some “new age ERP guy called the Oracle Prophet” on a radical new method using the Art of Zen for ERP Patching and thought it was worth a try on your Production System……….

Do note that this form of Zen Patching does work on both R11 and R12, but not on R10. It also works on 9i, 10G and 11G databases. Please check Oracle Certification matrices and raise the question to Oracle Support if in doubt.

Oracle Support – Good morning. Can I help you?

Reader – Yes could you tell me if the Art of Zen patching is certified against R12 Apps please?

If the phone goes dead at this point, we suggest you assume Oracle Support is not aware of the Art of Zen patching and you should not pursue your question with them……….We also suggest you give your colleagues name during any telephone calls with Oracle,  in case Oracle raises a complaint for nuisance phone calls to your company…..:-)

So where does the Art of Zen fit into Oracle ERP patching?

 Let’s use a typical a koan to provide an illustration.

“We will test the patch by not testing the patch. Only then will we know that the patch has worked.”

Now at this point in time, you are probably thinking I’ve been hitting some fairly strong stuff to get to this state of mind, or I’ve completely lost the plot.

I can hear everyone thinking “So let me get this straight. You are going to test the patch by not testing the patch, so that you know the patch is working”. To which I’d reply, great you’ve got it. You are certainly a quick learner on this Zen Patching stuff!!! 🙂

Our R12 Patching Philosophy actually made our auditors jaws drop, not in terms of the Zen stuff (trust me, keep this stuff between yourself and myself please and maybe better not mention to your management or auditors……), but on the thoroughness of approach.

We always have five databases for our patching (at a minimum). This is probably a lot more than most have but let me explain why and you’ll probably want to then copy this model.

Our DBA Environment. This is where the patches are applied to make sure, well they actually apply. Believe it or not some patches from Oracle don’t even apply cleanly.

Our Patch Environment. This is where they are applied with a little bit of testing. OK we deviate a little from the Zen stuff, but give me a break……This makes sure they at least do what they say on the box without major functional failure.

Our development environment, which is always busy with daily activity by our development team , functional team and testers.

Our test environment which is always busy with daily activity by our users.

Our Production environment.  I’ve been pushing our company to drop this as it uses a lot of space and we hear most of our complaints from this database, but management insist it is important and needed. 🙂

We should also state our databases are pretty heavily used so application flows naturally are being used throughout development and test databases. We also apply any patches onto any other instances we have at that point in time, so that the patch is naturally tested by the simple day to day activity in as many places as possible, with a careful rollout to each environment.

 

The Art of Zen Patching

The point is simple on these types of patches. Oracle does release patches that should be applied at some point.

The Security Patches typically come quarterly and we try to apply 3-6 months after they come out. Security patches represent a serious risk to apply, although generally apply well. However security patches also represent a risk if you do not apply. You need to find the balance, but you SHOULD apply these regularly.

The ATG Patches are less frequent but provide critical updates to Browsers (especially if you have DMZ applications) and other technology components, including diagnostics.

The Database patches (and we’re talking 10G to 11G for instance) do come out periodically and at some point you need to decide to keep at least supported, although we’re very picky on applying these, but are in the process of an 11G Upgrade. (Various 10G database versions are losing or have lost premier support). This activity is every couple of years or so.

We’re not talking about applying every patch. No company in their right mind can achieve this. We’re talking about keeping your head above water and staying supportable.

The point is this. To test every time on these types of patches across every last item is impossible. The conventional way is to get the patch, apply, test everything and then move to Production in a few weeks. That’s a very logical way to order the world of Oracle ERP. But unfortunately this is not a very practical or safe way. These patches are by their very nature too broad and silently hit too many areas to be open to a logical, standardized testing process. The conventional approach actually increases risk with these types of patches because it is by no means obvious what could be impacted.

There must be a better way where you find that balance. This is the Art of R12 Zen Patching.

Our philosophy is simple. We plan carefully on all these types of patches well ahead of applying.

We do not apply these patches immediately they come out. We are kind enough to give others the opportunity to be the heroes or unenlightened who find the bugs, log them in My Oracle Support and make our life so much simpler because we heavily research each patch to find the problems the unenlightened logged. This way we avoid the bulk of the problems. Are you one of the unenlightened? If so we appreciate you finding the bugs for us, causing issues for your users on Production systems and generally making our life so much easier and less stressful.

Our philosophy then rests correctly on a peace of mind that these patches are largely stable, largely trusted and tested by others around the world. This isn’t just a philosophy, it’s backed up by hard facts based on an incredibly low failure rate of patches we have applied. The patch types listed are generally very mature, very stable and very reliable. The quality of these types of patches is far higher than the Oracle ERP patches for the application modules.

Our key philosophy can be defined by the koan below. (As you raise a smile, remember this is used in leadership teaching by guys that make more in a month than you make in 5 years and sell books by the truckload at Amazon 🙂

“Once upon a time in ancient Japan, a young man was studying martial arts under a famous teacher. Every day the young man would practice in a courtyard along with the other students. One day, as the master watched, he could see that the other students were consistently interfering with the young man’s technique. Sensing the student’s frustration, the master approached the student and tapped him on the shoulder. “What is wrong?” inquired the teacher. “I cannot execute my technique and I do not understand why,” replied the student. “This is because you do not understand harmony. Please follow me,” said the master. Leaving the practice hall, the master and student walked a short distance into the woods until they came upon a stream. After standing silently beside the streambed for a few minutes, the master spoke. “Look at the water,” he instructed. “It does not slam into the rocks and stop out of frustration, but instead flows around them and continues down the stream. Become like the water and you will understand harmony.” Soon, the student learned to move and flow like the stream, and none of the other students could keep him from executing his techniques” – Timothy H. Warneka

Now I’m not into all this stuff and I’m as skeptical as anyone else, but maybe they have a point. Too many companies are simply slamming into the rocks with patches, rather than working with the flow of Oracle Corporation. Working with Oracle, you often feel that you are not talking of a stream but more a raging torrent of patches. The problem is you are always fighting against the flood of patches, rather than finding what these guys would refer to as “harmony”.

The very essence of our philosophy and the koan itself can now be answered 🙂

“We will test the patch by not testing the patch. Only then will we know that the patch has worked.”

After rolling the patch through DBA and Patch environments very carefully over many weeks, we are ready to proceed to our main development and testing environments.

We typically roll patches into our development environment for a minimum of 4 weeks. We observe the behavior of the environment and record any bugs. We carefully investigate all bug occurrences.

Once we are comfortable at that point, we do run testing. OK so we broke our mantra, but nowhere near the testing that would normally be required. Why? Because we have seen the bugs naturally arise through our normal daily activity (as a Project Manager you’ll know typically what is going on and where the gaps may be I would hope). So to quote the Art of Zen,” the appropriate insight and response arises naturally.” This is the beauty of the Art of Zen Patching 🙂 You do your daily stuff to get to the answer of whether the patch causes major grief.

At this point we normally release the patches to our Test Instance, again allowing patches to settle for 4-8 weeks. Again using normal user activity, we gain further appropriate insight and responses, in terms of stability of the patch and subsequent bugs, arising from the natural process of user activity.

We do ask our users to test and hit the key functionality, but again, with the insight given from normal daily use, we have achieved ” the appropriate insight and response which arises naturally” as a Zen Master or Leadership or Lifestyle coach would tell you for quite a lot of cash 🙂

Even our DBA Team reaches a relaxed Zen like state and if you know your average DBA guys……… With planning comes time for our DBA Team to work and document carefully the steps needed for each patch. The timeframes create space for many, many practice runs, so that on the day of application to Production, they know exactly what to do and what to expect. It also creates the space and time for good old fashioned research on My Oracle Support. (A tip is that as we’re doing this approach over a number of months, the DBA’s always get copies of Production on a regular basis to run through the patching process, so any production specific issues are always encountered early).

In addition, we carefully plan for releases. So if you take our last security patches, these were rolled into an ATG RUP6 and a minor database point release (to stay supported). This reduces a constant patch cycle to a more manageable ITIL Release concept, reducing your workload overall. The raging torrent of patches becomes a much more manageable stream.

Most companies have huge stress over these types of patches. Most companies don’t even bother applying, much to the detriment in terms of support, future upgrades and security.

We are like every other company in many respects. We are highly conservative on applying patches. We like to stick on what we know.

But we do pay attention and plan for security, de-support of databases, new browser support in ATG RUP’s, etc in a very careful manner well ahead of time, allowing us to practice the Art of Zen Patching 🙂

Companies stress out, rush patches and therefore make mistakes. That is not the Art of Zen Patching. Zen Patching stresses the very opposite approach. Put the patch into your environments, slowly observe and watch over many months, then test and finally you will see the appropriate insight and responses that it has worked. Now looking at Wiki again, Monks meditate over many months or even years to answer a Koan. There is no difference in the approach of Zen Patching 🙂 The time the patches spend in your environments can be thought of as your meditation period over many months (typically 3-6 months depending on risk assessment) to find the answer to the koan of “how to test the patch without testing the patch to know the patch has worked”.

But with our R12 Zen Patching, we’ve reached an almost Zen like state 🙂 Patches are simply a natural part of the lifecycle of ERP. We have accepted that. They are planned and are allowed to settle for several months, to give insight into their nature and risk. Patches still require testing, but to a far lesser extent than a fully focused, high risk “lets test this patch and apply this patch in two weeks time”, which is similar to slamming into the rocks in the stream.

So where are we today with such an approach?

R12.0.4 RUP5 (yes this was a nightmare of testing the old fashioned way, definitely slamming into the rocks in the stream, but our go-live was incredibly smooth. Zen Patching doesn’t work here I’m afraid for those embarking on an R12 Upgrade. It’s the good old fashioned conventional testing approach that is needed here).

Security Patches to April 2010

ATG RUP6

11G and July Security Patches are currently under the Zen method as is a SUSE Linux Upgrade

We have never had a failure or serious outage as a result of Zen Patching (should I trademark this perhaps and make a lot of cash like those leadership guys??????), although as with all Oracle patches, it is a very serious business, with serious risks, so there is no place for complacency.

So what about the opinions of others on our ERP and how up to date we are overall with patches?  To quote one Senior Consultant DBA recently, we are way ahead of most companies in terms of patching, and have an “aggressive patching policy”.

I would say that the Art of Zen Patching can never be described with words like “aggressive” 🙂 . In fact it is quite the opposite. It is a very slow and considered process, stressing great patience, over many months, waiting for insight as a part of a natural process to reduce the risks we face, as the real Zen guys would put it 🙂

We simply achieve a lot more than most companies, with a lot less effort and a lot less risk. I think the Zen guys and leadership/lifestyle gurus would term it as “simply learning to move and flow with the Oracle stream creating harmony and peace of mind”. Obviously they’d be charging a thousand US Bucks an hour for this type of advice. (I remember we had one such IT Guru in our company. Cost us 6 figures for six weeks of work – we ended up using his laminated b*llshit as coffee mats……that was about the only value we got……). Now maybe the Consultant our company hired wasn’t too hot except for the coffee mats, but with some of these leadership and other philosophies, well maybe there’s something in it after all……

Call it Zen. Call it Lifestyle (or is it Patch) Management :-), but to have a safe, low risk and stress free approach to this type of patching which works with reduced effort (rather than increased effort) is  not a bad place to be, as an ERP Manager…….

Health Warning

This article was designed to present a very serious subject in a hopefully entertaining and educational manner utilizing both conventional approaches of testing, in conjunction with a more unconventional approach. However applying any patches on a Production Database is a very serious business. Patches do need testing and this should never be underestimated. However the point of this article is that by allowing patches to settle into various instances over time, you vastly increase the chances of spotting serious issues and vastly reduce the risk of issues in production that conventional testing can never address. This is the safest way to apply such patches that I have found, using a conventional testing approach, together with a far less conventional approach. 21st Century Testing meets 5th Century Zen Philosophy.

Disclaimer

(By the way, I haven’t been smoking anything….the intention of this article is to present a very serious subject – Oracle Patching – in hopefully what some will find a very funny and original manner, that can then be remembered and applied to help all of us that face the very serious risks of patching global ERP Systems, so don’ take all the Zen references for Patching too seriously………otherwise someone may think you’ve been smoking something………). Think Monty Python British humour as you read it……..

If you remember the article, change patching from a rush to a planned perspective and patch carefully over a period of months, then the article did it’s job 🙂

I hope you find it as funny to read as I did writing it 🙂

Advertisements

Tags: , , , , , , , , , ,

9 Responses to “R12 Patching and the Art of Zen”

  1. Kathy Jacobs Says:

    We are currently reviewing with our patching process after recent application of a patch broke other functionality not directly addressed by that patch. Not all patch documentation is equally useful. I’m going to bring this to the attention of our Patch team.

    • irroberts Says:

      If you let me know the patch type I can comment – Is it:

      1) Application Patch
      2) Database Patch
      3) Security Patch
      4) ATG Patch
      5) Other

      Depending on the Patch type there are a few things you can do, but Application patches especially can be very, very difficult. (and even the ATG normally require a few patches on top, as they do break things and that’s really, really hard to find, given the pure scale of these patches and the fact that it’s really not obvious where these patches hit home).

      Thanks

  2. Kathy Jacobs Says:

    Our most recent problem was with an Application Patch (Project Accounting).

    Do you think there’s any way to get Oracle to better communicate about what is involved in these patches?

  3. irroberts Says:

    Unfortunately the Art of Zen patching isn’t really for application patches. (although if you leave it long enough you might have seen the problem, but obviously when Oracle gives a patch for a production problem, I understand fully the pressures you are under).

    On Apps patches, if you are on R12, there is a great patch screen which tells you the objects that this hits. Now this isn’t perfect, but if you have a good technical team, then it can certainly narrow down the objects and therefore feedback to your functional team or users. (An example was we applied an AR patch the other day. By checking the patch screen, we saw it was mainly hitting around the Transaction workbench, although it hit a few other objects. But from that I was reasonably confident that the patch told us it was hitting this, specifically for credit memos, our technical check validated that and we proceeded focusing on that screen with our various scenarios.

    Now I know this isn’t the answer you want, but you should build up a library of test scripts. We did that as we went from 11.5.5 to 11.5.10 and then updated them from 11.5.10 to 12.0.4 RUP5. You then either use these in a cut down mode (don’t obviously do a full UAT, but do try to identify key flows around the functionality you think is hit) to validate the key areas.

    Like most sites, we don’t have automated testing, but we are moving towards this. But it takes a huge amount of time, but eventually would kill this scenario, as you press a button and test the whole thing. But we’re years off that……..

    A lot of people say size doesn’t matter…….but I do look at patch size, as less objects mean less damage as it changes less stuff. OK not a wholesale safe argument, but that plus a technical view of objects changed using the patch screen does again give a guide to your functional testing. (although I have seen small patches cause catastrophic problems also….)

    If Oracle gives you a huge patch start to worry. (The jokers in Oracle Support asked us to upgrade from 12.0.4 to 12.1.2 to solve one bug in Irecruitment. Our answer to them was pretty heavy……if you can I’d suggest if facing a large patch, try to insist on a one off fix. Now this carries it’s own risks (they aren’t heavily tested by Oracle). However the balance is the patch is targetted, changes fewer objects and will cause less damage. However getting a one off patch takes time, patience and a lot of difficult conversations with Oracle Support.

    Sometimes also we do try to force Oracle to give alternates. It’s a hard move as the Support guy wants to close the call as he’s given you a patch, and he probably gets performance ratings based on this, but I’ve had too many Oracle Support guys take this attitude and I spot it a mile away.

    Do also consider if the patch is needed or workarounds exist. I know this is a hard one to sell, but we’re like every other Oracle site with apps patches……we really really avoid applying if we can. They do cause major issues and they do take a huge effort to check they cause no other problems. We spent huge time in our UAT on R12 to get to a point where we were actually very happy and to a large extent this has limited our patches needed after go-live. We still have problems, but we have workarounds, our users are happy, although we have had to apply limited patches. The point is when companies do a major upgrade, get it as right as you can, sit on it and upgrade three years later……avoid major patches if you can.

    I’ll also sometimes have a trawl through My Oracle Support, looking for references to the patch number. Sometimes you’ll see patches related or mentioned in this context. A little research can tell you the patch your about to apply caused other people issues…….without you having to apply it……..

    My final advice (again probably not great) is to think of what you really cannot afford to lose and then have a quick test of that if you don’t have the resources to do really full testing. We’ll normally do major functional flows and a quick end to end (say procure to pay – requisition-po-receipt-invoice-payment) and check especially things like workflow, approval via email/notification, especially opening forms from these (these are your archetypal stuff that get hit during ATG patching/Security Patching – we had custom stuff knocked out on this). No matter what Apps patch, I’ll always use caution and try and live with the bug, to allow a decent time for testing. It’s just too risky otherwise.

    Goes without saying that all patches should be tested on a recent clone of PRD. If we have a very specific problem, we’ll always take the time to clone and test on that, before we decide on moving it even to a DEV environment.

    And a final answer, I doubt Oracle will ever produce proper documentation on patches. It is frustrating, but all you can do is read between the lines, use the above outlined techniques and pursuade your users to test as much as possible, preferably without rushing although I know how hard that is. We were immensely frustrated with 12.0.4 as we tried to go live and could not for 4 months, due to quality of patches. (We had conf calls between India, US and Malaysia on a weekly basis and we made our feelings very clear on the quality of stuff being given to us patchwise which was shocking – an AP patched knocked out all Employee Invoice functionality………) None of this really made a difference (and we raised it to Oracle VP level in Asia-Pac). So working with Oracle on this, your just one client in a sea of 70,000+ clients, so worth a try, but doubt you’ll see much success.

    I am I am sure all readers relate to your patching issue (and projects is one tough module…..)

  4. Kathy Jacobs Says:

    Would you consider joining the OAUG CM SIG Linked In group and post this document there. Several of our members have expressed a need to focus our Change Management perspective on Patch Management.

  5. irroberts Says:

    Sure. Very much appreciate the invite. Can you send me an invite via linkedin please as I cannot see the group.

    (I’ll be publishing an ITIL style Change Management process on Patch Management when I get a little time, based on what we do here.)

  6. lee Says:

    How do you check patches prerequisites before you apply them?
    Now we started to use Patchdepends(www.patchdepends.com) to speed up the process.

    • irroberts Says:

      Unfortunately we still use the good old fashioned manual way but I’ll certainly take a look at this. Appreciate the contribution and I am sure others will also find it interesting.

  7. Aejaz Says:

    Most apps patch pre-requisites can be checked using admsi.pl which is supplied with release 12 onwards

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: