Posts Tagged ‘ERP Patches’

Global Thermonuclear War – The New Oracle R12 Feature

January 18, 2011

This column will be short and sweet, explaining how someone can launch Global Thermonuclear War on you, completely wiping you out. Nice topic and not any jokes this month I’m afraid for such a serious subject. Of course the movie reviews will still be included, otherwise I’d lose my rating of the only combined Oracle ERP and Movie Rating Column on the Web……….

Now Oracle ERP has grown immensely over the years, adding module after module. Perhaps this column is about a new module that controls nuclear missiles? Computers (and Oracle ERP these days) seems to control everything else but thankfully Oracle hasn’t quite got to the point of having a module to do this. Worrying if they ever did, given the number of bugs in the early R12’s.

Oracle Support: Good morning. Can I help you?

User: Yes, we implemented Oracle’s Global Nuclear Missile Control module in Fusion Apps and it’s launched a nuclear missile accidentally against a large city that will kill millions in two minutes from now.”

Oracle Support: Yes, that’s a known bug. We will work on a patch and get back to you in a few days.

User: The missile will hit in two minutes. We need to escalate.

Oracle Support: We’ll have the duty manager phone you within 3 hours. Goodbye.

Ironically we’ve had a few Severity One Service Requests that make this conversation horribly familiar………

But what would happen if a hacker managed to get into your ERP system? Access to your Payroll. Access to your Financial results. Access to your HRMS System. Access to Payments. Imagine a hacker being INSIDE your system. Have a look at the column R12 and the Auditors from Mars. That will give you an idea of the horrible consequences of someone being inside your systems………..

Which brings us nicely to a movie recommendation. A hacker inside the system? It has to be the original Tron which I would give 9 out of 10 for it’s vision, way ahead of it’s time. The more recent version was OK, but lacked something I felt. But still worth seeing.

Now to get down to the serious business.  R12 did have a rather nasty payload, of thermonuclear proportions. I don’t normally write (or disclose) hacking vulnerabilities, but given this is already out on the web and represents a serious threat to you, I thought it now appropriate to warn everyone about what is a real global thermonuclear device, just waiting to go off in your ERP System with potentially catastrophic results.

In R12 a JSP file was shipped – jtfwcpnt.jsp. This JSP takes a query that executes against your database opening you up to SQL Injection based attacks………..Now let me see if I had access to an ERP Database as a hacker where would I want to start…….??????

I am not going to go into the details of how this is exploited, but you should strongly check if this file is used and then remove it if not. This warning is applicable for anyone who is using products such as iRecruitment, iSupplier or other DMZ based products in R12. (although an internal attack could equally be done).

This vulnerability seems to be across all R12 releases judging by other reports on the web. (We’re currently upgrading R12.1.3 and will be checking this also shortly).

This file represents a very serious risk to your entire ERP and therefore to your company

And to end this rather serious column, we need a movie recommendation. Well that has to be the original War Games movie. A great story from 1983 and decades ahead of it’s time. It’s all about how computers controlling everything internally are accessed from external sources to almost start a nuclear war. For me given it’s relevant 30 years later, it’s a 9 out of 10.

There’s quite a number of very serious points to this column.

We have to wonder what Oracle was doing shipping stuff like this, whilst busily shipping security patches quarterly. I am just utterly stunned this ever got out as part of the shipped R12 product.

I’d also suggest that companies start looking very seriously at security of their ERP, especially those running products in the DMZ.

Review the papers on Metalink on the best practises for DMZ. Review Steven Chan’s column as there is always great information, but most of all, out of this learning experience, google regularly for security vulnerabilities on the web about R12 or R11 – I know most people don’t do this, which is why I published this vulnerability in this column. Solution Beacon also provides some good security information. Also make sure you have decent firewalls (Oracle has released a new product just recently) and software to protect against SQL Injection and other similar attacks.

Also do keep your ATG and Quarterly Security patches up to date. I know how difficult that is, but it is critical. (A previous security patch closed a hole in iRecruitment that could be exploited from outside). See R12 Patching and the Art of Zen for an approach that makes this less painful.

Security is very much a multi-layered approach and your ERP needs heavy protection like any other corporate system. (and arguably even heavier than most).

The hacking days of Windows and Internet trojans will continue as they have done for many years, but there’s a new age of hacking dawning and there is a real awareness from hackers on other areas, and that now includes ERP Systems such as SAP and Oracle.

This is a real wake-up call in terms of security with ERP and I hope that everyone really starts looking at ERP security as a priority in their companies, over and above anything else.

The dawn of the ERP Hacking Wars is beginning……

Further Prophecies can be found at https://oracleprophet.wordpress.com

Advertisements

Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt 1)

October 20, 2010

Darkness falls across the land
The midnight hour is close at hand
Creatures crawl in search of blood
To terrorize the neighborhood

Now if you don’t know that song intro, shame on you. It’s from one of the biggest hits of all time – Michael Jackson’s thriller. A very appropriate one for Halloween I’m sure you’ll agree.

So first movie recommendation of the day. This would be “This is It”. There’s something very poignant about seeing him performing in what was his final curtain call.

Now Halloween has always been a time to scare the crap out of people. Kids dress up as Ghouls, Vampires and the like and go “trick or treating”, sometimes with some very nasty tricks indeed. Either the local neighborhood pays the kids with sweets, etc or the kids do something nasty.

Back when I was a kid, there was one particularly nasty kid who didn’t bother with the treat stuff. He figured he could go beat up the other kids at the end of the evening and steal their sweets. This saved him the effort allowing him to do his favorite trick of all.

This kid had a real nasty trick. What this kid would do would be to get some dog crap, put it in a brown paper bag and then go place it on someone’s doorstep. He’d quickly set the bag on fire, ring the doorbell and wait for someone to answer.

Now what is amazing is that the immediate reaction of an adult to a small burning paper bag (even if they are not wearing shoes) is to try to stamp out the fire with their feet. Irrespective of wearing shoes or not, this had some pretty unpleasant results for the poor neighbor at the end of this “trick”.
Before we start, let’s put our other movie recommendation out. It’s the original Halloween movie. Now I saw this when I was 13 and it scared the absolute crap out of me. Well worth getting the video. A classic 70’s horror flick for Halloween, but definitely not for the kids. (Don’t see the more recent one, it sucks).

So seeing as it’s Halloween, I should scare the crap out of everyone reading this right? So everyone’s moving to an R12 Upgrade in 2011. So what better way to scare you than tell you how horribly wrong it can all go on an R12 Upgrade !!!

So R11 Oracle Apps Premier Support is ending in November. That means you are screwed. OK, not completely (but Oracle will be putting the pressure on you and telling you that you are) but seriously you should be looking at R12 now.

Now let me tell you that an R12 Upgrade will be the biggest, toughest upgrade of ERP that you will EVER have done. It can go disastrously wrong. You could corrupt your data, you could have major functional failures dropping your business to it’s knees. Or your entire ERP could disintegrate and you could lose your job because at the end of the day your boss will need a sacrificial lamb and you seem to fit the bill.

Are you starting to get scared? You will be by the time you read this article, because it contains a very long list of every single thing that I can possibly think of that can wreck your R12 upgrade.

So let’s start at the very beginning. Getting the Project Started. If you get this wrong, it’s probably not worth bothering about the rest of the project. Here’s a short list of things to worry about. (Now the purists will say these are not risks). For me I like to have everything that can go wrong on one spreadsheet. Call it risk, call it a list, doesn’t bother me, because I can quickly mentally check each month how things are going and make sure I haven’t forgotten anything. It also gives me a nice risk score and lets me manage that to hopefully give the project a fighting chance of delivery.

Let’s start with the Project Management risks and everything that can go wrong here:

Weak Business Commitment to Project

Funding is not secured for Project

Project Decisions are not Timely and On Time

Weak Business Commitment to Change

Review of Project Deliverables and Sign-Off is not timely

Weak Support of Business Case

Scope is not defined clearly or completely

Business process model is not defined clearly

Planning is of poor quality

Project involves large number of departments

Tight Time-Frames, Minimal Slack between Dependent Tasks

Go-Live Date accommodates limited UAT

Project Milestones are not achievable (Project Schedule)

Dependency on other Projects

Project requires complex organizational changes

Contract involves significant cost

Project involves time-critical Procurement

Significant Scope creep

Gold Plating by Implementation Team

Project Budget is not sufficient with Current Progress

Project Resources are no sufficient with Current Progress

Project schedule is delayed

Project Cost is overrunning

Product does not meet client expectations

The very first thing you need to do is get your most knowledgeable and trusted people in a room and do proper planning, taking into account your organizations political landscape, resources, expected funding and other commitments. If you fail to plan, you plan to fail. Your planning must give ample slack time for delays in UAT, delays in decisions, procurement delays, bugs, resource problems (only 5% of people have done R12. The other 95% of people will do R12 in 2011, so can you get resources??????)

The most important part of any project is to get high level buy-in and commitment from the senior people across your organization. From a resource viewpoint an R12 Upgrade requires intense Business and IT Commitment. Fail to get every area of Business and IT involved and agreed, then you are screwed 100% come UAT, if not even earlier. You need to make it clear when you need the business, what you need them for, how long and how many resources right at the onset of the Project Initiation. IT resources need to be equally onboard.

Securing realistic and adequate funding is critical to an R12. It is bigger. It is more complex. It does take longer. An R10 to R11 or a 11.0.3 to say 11.5.10 is NOT the same as an R12. Build in a decent contingency.

An R12 is not a basic like for like Technical Upgrade. You should ideally keep R12 as a pure technical upgrade and not a re-implementation, unless there are very good reasons, otherwise your risks and costs will escalate severely. However even doing a technical upgrade involves a pile of new modules and new functionality (albeit some can be deferred). But you’d better be prepared to implement Subledger Accounting, EBiz Tax, TCA (changes in Banks), Payments and Self Service (new functionality and removal of MOD PL*SQL). All of these will need user review, functional designs (if customizations on top), setup and decisions, as well as time for your own team to learn the implications.  This can lead to multiple risks including your own team wanting to add “cool functionality” or users not making timely decisions or being prepared to adopt new processes to account for changes in say TCA or Payments.

If you are dealing with contracts for consultants, firms (outsourcing or otherwise) or need hardware, etc, be prepared for the heavy risks of delays in Procurement. IT guys often overlook that it may take a while, despite having the cash to get resources onsite or hardware commissioned, etc.

Talking of resources this should be viewed as a serious risk throughout your project. R12 resources are still less rather than more common.

As well as all of the above, you still have to worry about all the usual Project Management stuff of quality, schedule, cost, delays, budget, etc. And think about it, we’ve not even got to the actual project yet…….

So the very first thing you need to do (before your project team arrives) or is internally deployed is to make sure they actually have an R12 Instance to work on…..

Infrastructure is often overlooked for projects. Unfortunately this can cripple the project team (and therefore your project) very rapidly indeed.

Network Speed

Performance Issues on Project hardware

Performance on Production may not be acceptable Go-Live

Deployment involves new hardware

Hardware is not sized correctly

Database is not sized correctly (do you have enough disk space for R12)

Hardware is no available for Project Use

Backups are not available for Projects

Project Environments not available

Network availability is poor (if teams distributed)

DMZ Configuration is not available

Architecture is new

R12 Instances are not available

Space for Project Instances is not available

Missing interoperability patches

Missing patches to Operating System

If you are going R12, either you have a lot of servers around spare, or you are going to need to buy (or lease) additional hardware. Otherwise what is your team going to use? Ten Consultants onsite doing nothing is some serious cash burn.

Then you have to have a plan for environments needed, dates needed, who prepares them, who sets them up, etc. DBA’s will shout at you if you need environments the next day. And what if there is no space? Did you check the disk space as part of your project? Your average R12 environment requires considerably more space than your R11.

During an R12 Upgrade you’ll need a pile more databases. We used around 8 additional (and we still needed all our R11 Development, Patch, Test as R11 Production work could not be completely halted).

Now just think when you finally deliver that R12, and it crumbles on day 1 into a performance black hole. Did anyone bother to do proper sizing either on CPU or disk or Memory? If not, try www.jobserve.com when it all fails on day 1.

Even when you are starting the project, insufficient hardware will kill you.

Did anyone ask for backups of your project databases? It would be a real shame to see your R12 go down and no backup, losing months of effort.

If your Production is a DMZ enabled configuration, I hope that someone is also going to emulate this for some of your R12 databases as you go through the project.

 

So lets get to the one that usually gives R12 Project Managers nightmares. Resourcing.

Poor IT Staff Skills – Functional

Poor IT Staff Skills – Business

Poor IT Staff Skills – Quality Assurance

Poor IT Staff Skills – Java

Poor IT Staff Skills – Oracle Development

Poor IT Staff Skills – ERP

Poor IT Staff Skills – Modules Implemented (or new modules)

Poor IT Staff Skills – Technology

Poor IT Staff Skills – Unix/Linux

Poor IT Staff Skills – DBA

Poor IT Staff Skills – Project Management

Poor IT Staff Skills – Senior IT Management

Poor IT Project Team Rating – Skills

Poor Business Project Team Rating – Skills

User Department Staff not available for Project

IT Staff not available for Project

Poor User Department Staff Skills

Poor or lack of Project Team Lead (s) per Functional Stream

Experience of Business Area is Low

Project has competing projects for Resources

Custom Development relies on 3rd party

Project Manager is inexperienced in Upgrades/ERP

Legacy Staff unavailable for Data Conversions

IT Staff skills – training required

Low retention of project  team during implementation

Low availability of Trained and Experienced reasonable cost alternate resources externally

No IT Staff available for existing Production Support

Oracle Support provides poor support

Have you even contemplated the vast variety of resources you are going to need to pull off an R12 Upgrade? Have a look above if you think you can use a couple of consultants over a few months.

Will your internal IT guys be good enough? Do they know the R12 functionality? Probably not. Will you train them? Probably not. So how can they deliver? And even if you want to train them, can you get the appropriate training completed before they actually do the upgrade. Training courses from Oracle can be hard to come by, especially on some of the niche modules and require booking way in advance.

Do your IT guys REALLY know the business? If not are you sure you’ll catch every scenario and every nuance during implementation.

Do you have decent Oracle skills? Do you have decent Java skills? Do you have decent Oracle ERP Technical people? If not, you are going to need to ramp up or get someone that does to do your upgrade (unless your complete vanilla ERP).

Do you have anyone that knows all the new R12 modules, such as EBiz Tax, Subledger Accounting, Payments, etc? These are not optional modules (except Payments, but if your running Payables in R11, then Payments becomes essential in R12 and it’s a new module). Most companies will require to have knowledge on these and something like Payments has heavy changes between R11 and R12 – indeed it is a rewrite between R11 and R12.

New technology components are included across the board. Are your people ready and capable?

Your typical upgrade will require some pretty heavy Unix or Linux skills. You might have to upgrade the O/S or you may take the opportunity to switch to cheaper, less proprietary hardware or Linux. Do you have any clue what’s involved? (we did this and yes it saved us a pile of cash for the future, but it ain’t easy and is littered with risks).

Did you tell the Infrastructure guys you’d be needing a full time DBA for the R12; to clone R11 Production; to work on the upgrade process; to write detailed upgrade documents that give step by step guides; to optimize the process so that it can fit into a long weekend on a huge global database? To apply all the patches? You probably thought your DBA’s sit around doing nothing and can do all this at the drop of a hat…….Count on the initial upgrade where your DBA’s find out the nuances of R12 to take quite a chunk of initial time and plan that into your environments and people coming onboard project wise.

R12 Project Management. Words fail me on this one. If you are not an ERP person don’t even think of doing an R12 without a strong R12 Project Manager, ideally someone that has done R12 Upgrades. It’s horrendously complex, extremely technical and the risks are catastrophic if you get it wrong. (I can be contacted on the following number for lucrative job offers……… 🙂   )

Are the Senior Management (both in IT and Business) in full support of both you and the R12 Upgrade, or are they ready to crucify you at the first delay? R12 will have delays, it will throw horrendous curveballs in terms of bugs and other complexities. Just make sure everyone is on your side politically before you start. The guy below thought everyone was his friend, until he forgot to order the hardware in time……

Now this is a weird one, but how well do your Business know the Business and Oracle ERP? Look for the weak spots here before you start. If a Business Unit has a lot of new people or inexperienced people or are anti-Oracle ERP (and some are), how are you going to get them to assist (or write) Test Plans and Test Scripts and do actual testing. Now if your IT and Consultants don’t test well (and they are NOT business experts) the one hope of catching everything before Go-Live is the Business. I never build 100% dependency on the Business for testing, but if your IT and Business guys can’t be relied upon for testing, you’ve got a perfect storm.

The other thing to check is everybody’s schedules. We’re not talking dinner dates, or when they plan to see a movie. Are the Business available for an R12 (which won’t add vast value to be honest) or do they have a planned move to say IFRS planned from June to December 2011. If so you won’t get a look-in resource wise for UAT and your project will be dead in the water.

Same goes for IT. If they have some major high profile initiatives, your R12 is bottom of the list, as in the priority list, it’s just one of these unpleasant tasks that needs to be done.

Watch out that your R12 Upgrade doesn’t just focus on Oracle ERP resources. When we did the upgrade, we had Legacy guys, email guys, Web guys, external 3rd parties (Citibank and others), Health Insurers, etc. Miss those guys out and who can help you complete your UAT?

What’s amazing about an R12 Upgrade is that you’ll need exactly the same resources that are getting used for your R11 Maintenance and Support. That leads to automatic conflict and resource contention and delays (and when it comes to Production problems or R12 Upgrade, you’ll lose that argument every single time). So ensure you augment your team to not be 100% reliant on goodwill.

If you are getting a company to do it for you, be careful. Have they done an R12? If yes, will they give you resources that have ACTUALLY DONE AN R12? Most outsourcing or consulting companies might have done R12’s, but they are as keen as anyone else to stack your project with their guys who can then learn R12 at your expense. Nice how they’ll happily charge you for the privilege of training their people…….Interview all people, before they deploy onsite. Hire an independent highly knowledgeable freelancer Consultant to fight your corner and keep the vendor on the right track, working in your interests, not the vendors.

Do you know the effort involved in training people for R12? The IT guys, the Business guys, etc? If not, start to worry.

The great thing about an R12 Upgrade is that your team (and maybe your consultants or outsourcers) will all get fantastic R12 skills, in the midst of the hell of an R12 project. Trust me that’s a great way to learn. Of course all those companies that have not done R12 (and that’s 95% worldwide) will be looking and admiring your very skilled people. And your loyal people that you’ve spent time and money training in R12, will of course be looking for that next payrise with their new found skills. Losing key resources (say for instance your Payments expert) is your worst nightmare. Suddenly one single module can derail your entire R12 Upgrade. If you think this is the stuff of nightmares, well during our R12 back in 2008/2009 we lost our Payments Consultant (one of the few in Asia at the time). Luckily we had him shadowed throughout the project by another staff. The demand for R12 skills will hugely increase in 2011 and your guys will be headhunted or tempted away. Just be ready. I am sure the resigning employee will feel your pain as your project is now doomed, in the same way you felt his pain when you froze his pay last year, as part of a corporate wide pay freeze…….

Our final resource issue is to prepare for the maelstrom of Production support hell. Just make sure your well staffed and haven’t let the entire project team go, just before you get deluged with Production issues. (and yes you should start thinking about this not at the end of your project when everyone is jumping ship as the project for them is over, but at the very start of your project). It’s funny as R12 projects get to go-live, most of the work is done, and you start letting consultants/contractors go. Do you have any idea the pyschology in play with the others that you expect to see remaining to do the support work and provide critical support for a few months after go-live? Why should they stay when they can get a far longer, far higher paying and far easier (i.e. not pressured go-live support) contract with their fantastic R12 skills you gave them? Sleep on that one, assuming you can still sleep……..

And I hope you’ve got a full communication plan going on – an upgrade can take time and if you are not informing and working with the users throughout…..don’t expect them to be enthusiastic about putting large amounts of time into your UAT. Of course if you run workshops and demos and mini training throughout your R12 you’ll have a whole lot more acceptance during UAT and go-live…….

And finally Oracle Support………..now R12 should be stable now, but back in the early days, we had these jokers on the phone daily. We even had a few removed from our support calls because progress was atrocious. Are you ready to deal with Oracle Support? Do you know how to escalate calls effectively? Do you know how to work around Oracle stalling for time or sending you on a wild goose chase? If not hire someone that does. Good ERP people with strong experience are essential for your R12. It’s funny but I bet you didn’t even think about informing Oracle Support of your upgrade, which is a shame as they might have given you a Critical Account Manager that could have helped you push all your Service Requests….

And that’s all for now folks. Part 2 will cover more in ghost, ghouls, implementation, transition and onto go-live, together with a whole pile of great references, covering fantastic presentations from experts in R12 and great Oracle Open World links of R12 Presentations there, together with a heavily researched bunch of Metalink articles around R12.

I’ll be scaring you very shortly with Part 2 before the full moon of halloween is over.

The other blogs can be found on https://oracleprophet.wordpress.com

Enjoy.

Footnote:

The great graphics were the work of:

WWW.MYSPACEGRAPHICSANDANIMATIONS.COM

R12 Patching and the Art of Zen

September 18, 2010

Reading through Wikipedia, I found an interesting article on the concepts of Zen. Now I’m not really into that type of stuff myself (each to their own), but I thought it would make an original way to present this article 🙂

“One practice of Zen Buddhist’s is Koan Inquiry. A koan is a question, or statement, the meaning of which cannot be understood by rational thinking but may be accessible through intuition. The answer can occur during meditation or during your typical daily life with all the mundane tasks you do.

To Zen Buddhist’s the Koan is “the place and the time and the event where truth reveals itself”. It is a way to induce an experience of enlightenment or realization, not through rational reasoning, but through intuition.

Answering a Koan requires a student to let go of conceptual thinking and of the logical way we order the world, so that like creativity in art, the appropriate insight and response arises naturally and spontaneously in the mind.”

Or to quote from a very non-Zen perspective, you think about a problem very hard all day. You fail to make any breakthrough. During the next morning, in the shower, without even thinking of the problem, you suddenly think of the idea. Ironically perhaps we are all practicing Koan Inquiry as a natural state of mind to solve difficult problems, without even having to think about the problem at hand.

Now let’s look at ERP Patching in relation to Zen 🙂 We need to be clear from the outset that this form of Zen Patching applies only to the following patches below. This is extremely important to keep in mind.

Security Patches

ATG Patches

Database Patches

This form of Zen does not apply to other ERP Patches

Applying this form of Zen Patching to any other types of patches will cause you some serious grief in your career when you report to your boss that your ERP for your entire organization worldwide is trashed because you read some amazing article by some “new age ERP guy called the Oracle Prophet” on a radical new method using the Art of Zen for ERP Patching and thought it was worth a try on your Production System……….

Do note that this form of Zen Patching does work on both R11 and R12, but not on R10. It also works on 9i, 10G and 11G databases. Please check Oracle Certification matrices and raise the question to Oracle Support if in doubt.

Oracle Support – Good morning. Can I help you?

Reader – Yes could you tell me if the Art of Zen patching is certified against R12 Apps please?

If the phone goes dead at this point, we suggest you assume Oracle Support is not aware of the Art of Zen patching and you should not pursue your question with them……….We also suggest you give your colleagues name during any telephone calls with Oracle,  in case Oracle raises a complaint for nuisance phone calls to your company…..:-)

So where does the Art of Zen fit into Oracle ERP patching?

 Let’s use a typical a koan to provide an illustration.

“We will test the patch by not testing the patch. Only then will we know that the patch has worked.”

Now at this point in time, you are probably thinking I’ve been hitting some fairly strong stuff to get to this state of mind, or I’ve completely lost the plot.

I can hear everyone thinking “So let me get this straight. You are going to test the patch by not testing the patch, so that you know the patch is working”. To which I’d reply, great you’ve got it. You are certainly a quick learner on this Zen Patching stuff!!! 🙂

Our R12 Patching Philosophy actually made our auditors jaws drop, not in terms of the Zen stuff (trust me, keep this stuff between yourself and myself please and maybe better not mention to your management or auditors……), but on the thoroughness of approach.

We always have five databases for our patching (at a minimum). This is probably a lot more than most have but let me explain why and you’ll probably want to then copy this model.

Our DBA Environment. This is where the patches are applied to make sure, well they actually apply. Believe it or not some patches from Oracle don’t even apply cleanly.

Our Patch Environment. This is where they are applied with a little bit of testing. OK we deviate a little from the Zen stuff, but give me a break……This makes sure they at least do what they say on the box without major functional failure.

Our development environment, which is always busy with daily activity by our development team , functional team and testers.

Our test environment which is always busy with daily activity by our users.

Our Production environment.  I’ve been pushing our company to drop this as it uses a lot of space and we hear most of our complaints from this database, but management insist it is important and needed. 🙂

We should also state our databases are pretty heavily used so application flows naturally are being used throughout development and test databases. We also apply any patches onto any other instances we have at that point in time, so that the patch is naturally tested by the simple day to day activity in as many places as possible, with a careful rollout to each environment.

 

The Art of Zen Patching

The point is simple on these types of patches. Oracle does release patches that should be applied at some point.

The Security Patches typically come quarterly and we try to apply 3-6 months after they come out. Security patches represent a serious risk to apply, although generally apply well. However security patches also represent a risk if you do not apply. You need to find the balance, but you SHOULD apply these regularly.

The ATG Patches are less frequent but provide critical updates to Browsers (especially if you have DMZ applications) and other technology components, including diagnostics.

The Database patches (and we’re talking 10G to 11G for instance) do come out periodically and at some point you need to decide to keep at least supported, although we’re very picky on applying these, but are in the process of an 11G Upgrade. (Various 10G database versions are losing or have lost premier support). This activity is every couple of years or so.

We’re not talking about applying every patch. No company in their right mind can achieve this. We’re talking about keeping your head above water and staying supportable.

The point is this. To test every time on these types of patches across every last item is impossible. The conventional way is to get the patch, apply, test everything and then move to Production in a few weeks. That’s a very logical way to order the world of Oracle ERP. But unfortunately this is not a very practical or safe way. These patches are by their very nature too broad and silently hit too many areas to be open to a logical, standardized testing process. The conventional approach actually increases risk with these types of patches because it is by no means obvious what could be impacted.

There must be a better way where you find that balance. This is the Art of R12 Zen Patching.

Our philosophy is simple. We plan carefully on all these types of patches well ahead of applying.

We do not apply these patches immediately they come out. We are kind enough to give others the opportunity to be the heroes or unenlightened who find the bugs, log them in My Oracle Support and make our life so much simpler because we heavily research each patch to find the problems the unenlightened logged. This way we avoid the bulk of the problems. Are you one of the unenlightened? If so we appreciate you finding the bugs for us, causing issues for your users on Production systems and generally making our life so much easier and less stressful.

Our philosophy then rests correctly on a peace of mind that these patches are largely stable, largely trusted and tested by others around the world. This isn’t just a philosophy, it’s backed up by hard facts based on an incredibly low failure rate of patches we have applied. The patch types listed are generally very mature, very stable and very reliable. The quality of these types of patches is far higher than the Oracle ERP patches for the application modules.

Our key philosophy can be defined by the koan below. (As you raise a smile, remember this is used in leadership teaching by guys that make more in a month than you make in 5 years and sell books by the truckload at Amazon 🙂

“Once upon a time in ancient Japan, a young man was studying martial arts under a famous teacher. Every day the young man would practice in a courtyard along with the other students. One day, as the master watched, he could see that the other students were consistently interfering with the young man’s technique. Sensing the student’s frustration, the master approached the student and tapped him on the shoulder. “What is wrong?” inquired the teacher. “I cannot execute my technique and I do not understand why,” replied the student. “This is because you do not understand harmony. Please follow me,” said the master. Leaving the practice hall, the master and student walked a short distance into the woods until they came upon a stream. After standing silently beside the streambed for a few minutes, the master spoke. “Look at the water,” he instructed. “It does not slam into the rocks and stop out of frustration, but instead flows around them and continues down the stream. Become like the water and you will understand harmony.” Soon, the student learned to move and flow like the stream, and none of the other students could keep him from executing his techniques” – Timothy H. Warneka

Now I’m not into all this stuff and I’m as skeptical as anyone else, but maybe they have a point. Too many companies are simply slamming into the rocks with patches, rather than working with the flow of Oracle Corporation. Working with Oracle, you often feel that you are not talking of a stream but more a raging torrent of patches. The problem is you are always fighting against the flood of patches, rather than finding what these guys would refer to as “harmony”.

The very essence of our philosophy and the koan itself can now be answered 🙂

“We will test the patch by not testing the patch. Only then will we know that the patch has worked.”

After rolling the patch through DBA and Patch environments very carefully over many weeks, we are ready to proceed to our main development and testing environments.

We typically roll patches into our development environment for a minimum of 4 weeks. We observe the behavior of the environment and record any bugs. We carefully investigate all bug occurrences.

Once we are comfortable at that point, we do run testing. OK so we broke our mantra, but nowhere near the testing that would normally be required. Why? Because we have seen the bugs naturally arise through our normal daily activity (as a Project Manager you’ll know typically what is going on and where the gaps may be I would hope). So to quote the Art of Zen,” the appropriate insight and response arises naturally.” This is the beauty of the Art of Zen Patching 🙂 You do your daily stuff to get to the answer of whether the patch causes major grief.

At this point we normally release the patches to our Test Instance, again allowing patches to settle for 4-8 weeks. Again using normal user activity, we gain further appropriate insight and responses, in terms of stability of the patch and subsequent bugs, arising from the natural process of user activity.

We do ask our users to test and hit the key functionality, but again, with the insight given from normal daily use, we have achieved ” the appropriate insight and response which arises naturally” as a Zen Master or Leadership or Lifestyle coach would tell you for quite a lot of cash 🙂

Even our DBA Team reaches a relaxed Zen like state and if you know your average DBA guys……… With planning comes time for our DBA Team to work and document carefully the steps needed for each patch. The timeframes create space for many, many practice runs, so that on the day of application to Production, they know exactly what to do and what to expect. It also creates the space and time for good old fashioned research on My Oracle Support. (A tip is that as we’re doing this approach over a number of months, the DBA’s always get copies of Production on a regular basis to run through the patching process, so any production specific issues are always encountered early).

In addition, we carefully plan for releases. So if you take our last security patches, these were rolled into an ATG RUP6 and a minor database point release (to stay supported). This reduces a constant patch cycle to a more manageable ITIL Release concept, reducing your workload overall. The raging torrent of patches becomes a much more manageable stream.

Most companies have huge stress over these types of patches. Most companies don’t even bother applying, much to the detriment in terms of support, future upgrades and security.

We are like every other company in many respects. We are highly conservative on applying patches. We like to stick on what we know.

But we do pay attention and plan for security, de-support of databases, new browser support in ATG RUP’s, etc in a very careful manner well ahead of time, allowing us to practice the Art of Zen Patching 🙂

Companies stress out, rush patches and therefore make mistakes. That is not the Art of Zen Patching. Zen Patching stresses the very opposite approach. Put the patch into your environments, slowly observe and watch over many months, then test and finally you will see the appropriate insight and responses that it has worked. Now looking at Wiki again, Monks meditate over many months or even years to answer a Koan. There is no difference in the approach of Zen Patching 🙂 The time the patches spend in your environments can be thought of as your meditation period over many months (typically 3-6 months depending on risk assessment) to find the answer to the koan of “how to test the patch without testing the patch to know the patch has worked”.

But with our R12 Zen Patching, we’ve reached an almost Zen like state 🙂 Patches are simply a natural part of the lifecycle of ERP. We have accepted that. They are planned and are allowed to settle for several months, to give insight into their nature and risk. Patches still require testing, but to a far lesser extent than a fully focused, high risk “lets test this patch and apply this patch in two weeks time”, which is similar to slamming into the rocks in the stream.

So where are we today with such an approach?

R12.0.4 RUP5 (yes this was a nightmare of testing the old fashioned way, definitely slamming into the rocks in the stream, but our go-live was incredibly smooth. Zen Patching doesn’t work here I’m afraid for those embarking on an R12 Upgrade. It’s the good old fashioned conventional testing approach that is needed here).

Security Patches to April 2010

ATG RUP6

11G and July Security Patches are currently under the Zen method as is a SUSE Linux Upgrade

We have never had a failure or serious outage as a result of Zen Patching (should I trademark this perhaps and make a lot of cash like those leadership guys??????), although as with all Oracle patches, it is a very serious business, with serious risks, so there is no place for complacency.

So what about the opinions of others on our ERP and how up to date we are overall with patches?  To quote one Senior Consultant DBA recently, we are way ahead of most companies in terms of patching, and have an “aggressive patching policy”.

I would say that the Art of Zen Patching can never be described with words like “aggressive” 🙂 . In fact it is quite the opposite. It is a very slow and considered process, stressing great patience, over many months, waiting for insight as a part of a natural process to reduce the risks we face, as the real Zen guys would put it 🙂

We simply achieve a lot more than most companies, with a lot less effort and a lot less risk. I think the Zen guys and leadership/lifestyle gurus would term it as “simply learning to move and flow with the Oracle stream creating harmony and peace of mind”. Obviously they’d be charging a thousand US Bucks an hour for this type of advice. (I remember we had one such IT Guru in our company. Cost us 6 figures for six weeks of work – we ended up using his laminated b*llshit as coffee mats……that was about the only value we got……). Now maybe the Consultant our company hired wasn’t too hot except for the coffee mats, but with some of these leadership and other philosophies, well maybe there’s something in it after all……

Call it Zen. Call it Lifestyle (or is it Patch) Management :-), but to have a safe, low risk and stress free approach to this type of patching which works with reduced effort (rather than increased effort) is  not a bad place to be, as an ERP Manager…….

Health Warning

This article was designed to present a very serious subject in a hopefully entertaining and educational manner utilizing both conventional approaches of testing, in conjunction with a more unconventional approach. However applying any patches on a Production Database is a very serious business. Patches do need testing and this should never be underestimated. However the point of this article is that by allowing patches to settle into various instances over time, you vastly increase the chances of spotting serious issues and vastly reduce the risk of issues in production that conventional testing can never address. This is the safest way to apply such patches that I have found, using a conventional testing approach, together with a far less conventional approach. 21st Century Testing meets 5th Century Zen Philosophy.

Disclaimer

(By the way, I haven’t been smoking anything….the intention of this article is to present a very serious subject – Oracle Patching – in hopefully what some will find a very funny and original manner, that can then be remembered and applied to help all of us that face the very serious risks of patching global ERP Systems, so don’ take all the Zen references for Patching too seriously………otherwise someone may think you’ve been smoking something………). Think Monty Python British humour as you read it……..

If you remember the article, change patching from a rush to a planned perspective and patch carefully over a period of months, then the article did it’s job 🙂

I hope you find it as funny to read as I did writing it 🙂