Posts Tagged ‘EBS Security’

21st Century Time Bandits of Accounts Payable

May 9, 2011

When I walk into an Accounts Payable department, it feels like I have almost stepped back in time, from the 21st Century to the 1990’s. Think about this – the 1990’s saw the dawn of ERP systems. Companies implemented these ERP Systems to handle amongst other things, the process of managing and paying invoices from suppliers.

Companies got large amounts of invoices, sent via the good old postal service. These lie on desks in huge piles, until some Payables clerk is ready to pick it up, put their head down and key it in. This process goes on endlessly, in many companies around the globe. Very little has changed in the last 20 years………it is as if, we got into a time machine and went back 20 years.

So a quick intro and a quick movie recommendation before we get into the content. First up, another favorite (and directed by an Ex-Monty Python actor), Time Bandits was a very original movie, all about, you guessed it, Time Travelling. It’s a really original and very entertaining movie. It’s an 8 and great for your kids.

Now a lot of people instantly say outsource Accounts Payable. To those people I say, outsource precisely what? I am a firm believer in using technology to automate certain areas. Freeing up those people to then add far more value to the Business in other areas of Accounts Payable. Too many people outsource business functions in a knee-jerk unimaginative reaction, effectively keeping the 90’s Business processes, whilst really they should be looking at new ways of working and transforming their organization with 21st century technology.

Now I sat in one Oracle OpenWorld session a few years back and listened as one company explained how they processed 1 million invoices a year, with a handful of staff in their Accounts Payable function. My company, if it had the same volume of invoices, would have needed a staff of well over twenty times that. So how did this company manage? Because they changed the way they did business. They didn’t blindly outsource 1990’s Accounts Payable business processes.

They were a 21st century Accounts Payable function whilst their competitors remained firmly in the 90’s.

This caused me to take a very good look at exactly what was the Accounts Payable function in my company processing. I’m going to go through a breakdown of exactly what went through our Accounts Payables subledger, throwing in a few ideas of what we did, what we automated and a few future directions. Hopefully it might trigger you to start questioning some of what you are doing in Accounts Payable.

I’ve been through a lot of Self Service applications in previous blogs and this blog continues to pursue this key technology with a view to automating business processes.

What struck me about my company was that our Payables unit was putting in all sorts of staff related invoices for payment to staff. Not only were they manually entering these but they were also checking all the accounting, figures, etc. Could we outsource? Why on earth would we outsource a function that had absolutely no business value to be done manually and actually shouldn’t exist asw a manual process?

Our company gave loans to staff. When a staff applied on paper, it went through massive approval processes, then finally hit the Accounts Payable for processing. We simply replaced this with an online application form. Staff applied, everything in terms of checking was automated, and we put an invoice through the AP Open Interface (already pre-approved with all the correct accounting) ready for Payment. One function of Accounts Payable removed, with over 2,500 invoices automated.

Our company gave Home Country Travel allowances. Same process as above. Staff applied on paper, massive approval processes, then finally hit Accounts Payable for processing. Online application form, pre-approval, automation, AP Open Interface, neeed I say more. 850 complicated invoices gone, completely automated.

Housing allowances. Same procedure again. Full automation. 850 complicated manual invoices gone, completely automated.

Staff go on a lot of travel given the nature of my company’s business. Same approach again. Online application, straight through Accounts Payable processing (pre-approved, proper accounting, etc). This removed 5,000 manual invoices per year for payment of travel allowances, going completely 100% automated.

Some of our staff have Payroll paid through Accounts Payable. (Don’t ask, it’s complicated). Linking our Oracle Payroll to Payables resulted in over 5,000 manual invoices being removed from Accounts Payable. A similar approach for Pensioners (don’t ask, it’s even more complicated than staff Payroll…….) removed 12,000 manual invoices.

The point is simple. We are a small company. Our company was processing manually over 30,000 invoices per year that it simply didn’t even need to process. Automation led to not only vastly reduced workload, but also 100% error free, paper free invoice processing and vastly faster and better service to our employees.

A short point to finish up. If your Accounts Payable function is being overloaded due to Employee “invoices”, do look at what can be automated by online application.

Another perfectly valid approach is to look at iExpenses to reduce the overall manual effort and avoid your Accounts Payable function (which is supposed to be processing supplier invoices) becoming swamped with Employee expenses. This is another good approach to streamlining your overall Employee Expense Management processes.

By looking very carefully at our business processes for handling Employee Expenses, we automated over 20% of our entire Accounts Payable function, removing completely the need for manual intervention.

And with that it’s definitely time for a movie recommendation. Back in the 80’s, a great movie (actually series of movies) covering time travel was Back to the Future. These are well worth a rent for the kids. A 7 out of 10. It’s so sad to see Michael J Fox, the star of these movies, battling a terrible disease when he was still relatively very young.

Moving on to the next area of Accounts Payable. Do you have any very large, very frequent suppliers? Depending on your business, a significant percentage of your Accounts Payable invoice processing may be down to a small number of suppliers. In our case this was certainly true. Let me go through the examples of how we transformed this area.

My company does a lot of travel. But we deal with two travel agents only, giving competition but also good control of our overall travel spend. We process around 10,000 travel invoices per year. Now these companies were not happy with us. Our Accounts Payable was failing to process invoices in a timely manner. Disputed items were confusing. Volumes were high. Invoices required vast amounts of checking. Vice versa we were not overly happy with them, as we found equally the Accounts Payable process very time consuming, confusing and seriously expensive.

Now our Travel process is fully automated, including bookings to these travel agencies. We asked the Travel agents, given everything is automated when staff travel, can you give us an electronic billing file? They were very keen indeed – this was going to save them a huge amount of time as they no longer needed to prepare vast quantities of paper invoices.

So look at our processing of Travel Invoices today. Two Travel agents give us a flat file monthly each. These two files are loaded and automatically matched to our Travel system. Can you imagine how much time that saves us just in reconciling an invoice to actual travel. 95% of the invoices sent are automatically matched by our automation programs. These invoices are then automatically imported using the Payables Open Interface and paid, with no intervention by our Accounts Payable department. Our processing times for these invoices have been drastically reduced (98%). Our time is spent dealing with the exceptions, not every single invoice. And the Travel Agents are happy because for the first time ever, they actually get paid on time. (And the 95% matching is a work in progress. It’s early days and we aim to hit 98% first time matching).

Next example. Our telecoms companies. Again we have two providers for competition. We have a huge number of staff with Blackberries. Every month, we receive bills for each staff, each with a different account number. Now that’s how the service providers work, so we need to deal with that. Each month, we would circulate these paper bills to staff, get the feedback, pay each invoice……………a truly dreadful process. Huge amounts of time and effort, bills were never paid on time. Service provider was unhappy. We were unhappy…..everyone was unhappy. Simple solution – service providers gave us consolidated billing files. These were uploaded and matched to staff, sent as notifications in Self Service and uploaded directly to Accounts Payable. A very complex, highly manual process, became highly automated, again with levels of matching approaching 100% with minimal human intervention. This removed 10,000 manual invoices from our Accounts Payable function. With the simple provision of two flat files that the suppliers were extremely happy to provide.

Final example. Our company, by the nature of it’s business, has an outsourced caterer, both for the staff canteen and for functions where food is provided. Our functions are very, very frequent, leading to a very large volume of invoices. Now our caterer used to prepare each invoice and send it to us. All paper based. These would then be circulated around our company, finally hit our Accounts Payable and finally be paid. We requested a monthly file from the supplier. This is then uploaded again very simply into Accounts Payable, with automated matching again approaching 100%. Again this supplier had never actually been paid on time, wasn’t too happy but accepted it as we put a lot of business their way. From our side the process was a nightmare. But with simple invoice files, monthly, manual invoices and manual checking completely disappeared. This removed more than 6,000 manual invoices per year.

We also do this with  a few of our other large service providers. They are usually all happy to work to a very similar template for the flat files they provide to us. We’ve taken over 30,000 manually entered invoices out of our Accounts Payable function. Our matching rates are above 98%, with very few of these invoices requiring manual intervention.

Simple approach. Look for the suppliers with the most invoices. Ask those suppliers if they want to reduce paper work, reduce burueacracy and get paid a lot easier. Most will say “Yes please”. Then automate.

Now we’ve not referred to Oracle’s EDI product – E-Commerce Gateway. But effectively that is what we are doing on a very simple, but very effective approach. Oracle does have a module that allows for large-scale electronic data interchange, including for Accounts Payable. EDI has been around a very long time (I remember implementing this in the 90’s for a very large customer). Now Oracle supports what are industry standard transactions, making it possible to link companies and their suppliers together using the same language, in terms of file formats/contents, etc. A few links can be found in the Reference Section relating to this. (Note Oracle provides EDI not just for Payables, but for other modules including Purchasing, Order Management and Receivables).

The use of EDI (either Oracle’s module or simple, effective home-grown EDI) can transform your Accounts Payable process.

And now another movie review. Actually I’m going to do something a little different and recommend a TV Series (although there have also been movies) for a change. Doctor Who is a TV show broadcast by the British Broadcasting Corporation. Now this show is done relatively cheaply, but always has original stories and is highly entertaining. It’s about a time traveller that travels through time and space, fighting evil aliens such as Cybermen, Daleks and other such monsters, whilst protecting the human race. He travels around using a blue British Police Box which is actually a time machine. His main enemy are the Daleks (shown below in a poster). These are robot monsters that move around on wheels taking over the galaxy, unless that Galaxy has too many stairs, as with wheels they cannot go up stairs…….Honestly, get a few DVD’s as the series are now all available. Kids will love it and adults too…….It’s a great British institution – been around since the sixties and I’d give it a 10 out of 10.

So the next option for transforming your Accounts Payables department. I’d say this is only for the large suppliers, but Oracle has an option called Payment on Receipt (ERS). The idea here is simple – you have large suppliers that provide you with goods. Now if your Shipping department (or whoever receives and accepts the goods) has checked the goods, accepted the goods, quality checked the goods and you have a validated Purchase Order, then exactly why do you then need a supplier to send you a paper invoice (or even electronic invoice) to your Payables department to process that invoice, check it again, etc. Why do you not just get Oracle ERP to create an invoice automatically? You remove vast amounts of bureacracy, vast amounts of unnecessary work and the reliability of this method trumps even EDI – this is a huge benefit to both you and the Supplier. Details can be found at the end of this blog.

Another interesting option is that Oracle can generate Invoices based on milestone payment in Services Procurement. A very similar concept to the ERS functionality, again further removing the need for manual invoices. If you have agreed to pay a supplier on a milestone, based on a deliverable, then once a manager has accepted that milestone has been achieved, an invoice can be automatically generated. (Hopefully as you read this blog, your also starting to see just how important a good Purchasing function is when it comes to Accounts Payable – one function cannot reach it’s full potential without the other also working highly efficiently).

The idea of companies still receiving paper invoices is a curious one for many areas of their business. One step better than you paying people in your company to enter invoices is for the company wanting paid to do the work for you.

Our company hires a lot of consultants. Those consultants work on contracts, to deliver certain tangible reports/studies/other pieces of work. The project manager signs off these deliverables and the consultant gets paid. Our approach to this is to provide a simple screen for the Consultants to enter their claim (very similar in many respects to iExpenses). With the consultant entering their own data, automatically validated against a contract, workflow notification for approval to their manager and then an invoice generated automatically for Payment (based on accepted deliverables), we will remove approximately 40,000 manual invoices from our Accounts Payable department.

Our other approach for more standard goods and services is the introduction of iSupplier. This module provides many facilities including suppliers entering/updating their details/views of Purchase Orders/updated shipping information, etc. One of the facilities is for the supplier to enter invoices, removing the need for your company to do so.

Next up……… many calls and how much time do you waste talking between your Accounts Payable Department and your supplier? You don’t know because no-one ever actually records the time………..but I bet it’s significant. The average company spends a lot of time dealing with calls that are really unnecessary. “Did you receive my invoice” or “when will my invoice be paid” or “what was this payment actually for?” We’ve replaced a very large number of these calls, again by implementing iSupplier. This module gives the supplier a full end to end view from Procurement to Payment of all activities. It effectively gives a real time window into your company to the supplier, eliminating the need to inquire on invoices or payments. Invoice and payment information can be downloaded by the Supplier for easy reconciliation.

Interestingly, if your sending out Remittance advices, this can also be replaced either by iSupplier Portal or simply letting the Payments function automatically email the supplier with a detailed remittance advice slip. Every invoice we process generates a payment and every payment has an email remittance. That’s over 150,000 manual remittance advice slips that we no longer have to prepare, print, put in an envelope and send.

This has been a busy blog. Time for a break. Time for a movie recommendation. Probably one of the first and most famous stories of Time Travel was by H. G. Wells. Now his books have been made into countless movies and the most recent The Time Machine wasn’t a bad interpretation. Overall I’d say a 7 out of 10 and worth a rent, although probably not a buy.

Now I don’t need a time machine nor some weirdo Oracle prophet to predict the future of Accounts Payables departments. The future has actually been around for quite a few years already with products that can read, scan and workflow invoices, significantly streamlining and automating your Accounts Payable function. But the one I am now looking at is Oracle’s own Imaging and Process Management. So here’s my prediction. I believe this piece of middleware (bought by Oracle, as all the best stuff seems to be) will become a central pillar, not just of AP, but will be used everywhere in future versions of R12 and Oracle Fusion. If you want to get on the Fusion express train, this is one product you should be seriously looking at.

Irrespective of product, the future, already used by many visionary companies, is to be able to receive either paper invoices or scanned emailed invoices into a single shared service center. The technology is already smart enough to scan these invoices and identify the information in a reliable manner.It can also generate invoices and match those invoices. Further these products can then workflow the invoices to the correct recipient for approval.

So what else does the Oracle Prophet see in the future? Again it’s an easy question to answer, as the future is definitely already here. I’ve done a blog previously about Oracle Pre-Built analytics. Well one of the major functionalities of the Oracle Pre-Built Analytics is the Payables module. I watched Oracle install all the tools, database, populate the data warehouse with 7 years of corporate AP data – in seven hours (as a proof of concept). The dashboards were then ready to use and we could instantly gain insight into our Accounts Payable functions. All the KPI’s were there to see invoice aging, discounts given, invoices processed, methods to process, etc. This is another fabulous product from Oracle and rounds off nicely this 21st Century Accounts Payable blog. Imagine having a complete, instant view of the health of your entire Payables process, with the ability to drill down to any single transaction, or aggregate at any level or instantly see who your problem suppliers are.

Now my company made a mistake in early 2000 by decentralizing the Accounts Payable function. In many respects this was organized chaos. If I had been with my company I could have agreed with the person doing it, except then we’d both be wrong. What I’d like to see my company do next is to move to a shared service, 100% centralized function that is 100% decentralized…………yep I’ve totally lost you, but we’re back to the concept of total federated services at the point they are used using shared service centers where appropriate. The technology is there today to allow a shared service to receive and quickly scan invoices, doing most of the keying automatically. The shared service provides a centralized function for dealing with difficult invoices (ones that don’t match, ones that don’t automatically go into the system, etc). It provides a Governance, Control and Analytical function. along the lines of a standard Accounts Payable department. However once in the system, the invoices should be routed directly to those who need to approve payment. The line managers in all the various departments worldwide who actually received the goods or services. Hence you have a shared service center, but a totally federated approval process. The technology today is very much capable of supporting this process. Throw in the automation for straight through processing on as much of your Accounts Payable invoices as possible, a sprinkle or heavy serving of EDI depending on your taste, and using your time machine, you’ve arrived in the 21st Century for Accounts Payable processing.

So out of our 150,000 Payables transactions a year, we have automated (or will shortly automate) almost 120,000.  Now I make that a whopping 80% of all Accounts Payables transactions. Now the not so smart companies out there (and there are plenty of them) would have simply perhaps have  outsourced their 1990’s business process. We will have achieved 80% automation by 2012 and taken a 20 year leap in technology and business process, all within a couple of years of the initial idea being floated.

We do intend to keep plugging away at the last 20% looking at potentially automation of invoice scanning, etc using Oracle’s Image and Process Management. We do want to continue to review what isn’t going through as EDI transactions and why. We do want to see where spend is avoiding the standard procurement processes and understand why. It’s still a work in progress.

The benefits?

1. An 80% reduction in manual effort. That saves our company a fortune. Now if you count each manual invoice costs an average company between 20-40 US Dollars to process……go do the math.

2.  Decreased supplier phone inquiries significantly

3. Improved Invoice processing times (up to 98% in some instances)

4. Improved approval processing times (and in many cases automated matching removes approval completely)

5. Vastly reduced paper, paper handling and paper storage

6. Easier to see and take effective discounts

7. Ensures compliance with Payment terms, avoiding penalties

8. Improves your company image as a good company to do business with – always pays on time

9. Absolute visibility of all Payables activities and transactions

10. Easier to audit, easier to comply with SOX/etc – the scanned invoice is available directly from the ERP Transaction

It has to be said that not everything can be automated. Some countries still need the paper invoice as a legal requirement. Some invoices just won’t have anything to automate matching to. Sometimes your supplier won’t play ball on EDI. There are limitations. Accounts Payable is still a very important and critical function that needs staffed correctly. But really, any company could look to taking a significant percentage of their manual Accounts Payable and automating to some degree.

And finally a movie recommendation to finish up our Time Travelling 21st Century Time Bandits of Accounts Payable. It has to be Bill and Teds Excellent Adventure. A movie about a couple of college kids that go through time in a telephone box to complete a history assignment by collecting famous historical figures to do speeches at their review. Along the way they play games with Death and win…..a lot of fun and a 9 out of 10. It’s the sort of no brainer nonsense that I find intellectually very challenging. Definitely not a deep movie, but MOST EXCELLENT !!!!!

As I end the column I feel it appropriate to do it in the spirit of the movie Bill and Teds Excellent Adventure:

Bill –  “Be excellent to each other. “

Ted “Party on dudes !!!!”

Until next time, when we reveal the secrets of Harry Potter and the Wizards of ERP Change Management………

Further secret blogs and prophecies can be found at:


Below are some of the references collected as part of the research for this blog. As usual I appreciate the authors making this information available on the internet, for the benefit of everyone. Also enclosed are a few of the guides/manuals from Oracle that go into greater depth of the possibilities for improving your Accounts Payable process.

IExpense – Back to Basics

iExpenses – Ohio Valley OAUG

Oracle Internet Expenses

EDI and E-Business Suite

Oracle E-Commerce Gateway

Payment on Receipt

Oracle iSupplier

Oracle Purchasing

AP Invoice Automation – Bottomline Technologies

AP Invoice Automation – Readsoft

AP Invoice Automation – Basware

AP Invoice Automation – Kofax

Accounts Payable Invoice Automation

Oracle Imaging and Process Management

Oracle AP Automation IPM Solution

Wikipedia – Oracle Imaging and Process Management

Oracle Imaging and Process Management 11g

Accounts Payable Analytics

Global Thermonuclear War – The New Oracle R12 Feature

January 18, 2011

This column will be short and sweet, explaining how someone can launch Global Thermonuclear War on you, completely wiping you out. Nice topic and not any jokes this month I’m afraid for such a serious subject. Of course the movie reviews will still be included, otherwise I’d lose my rating of the only combined Oracle ERP and Movie Rating Column on the Web……….

Now Oracle ERP has grown immensely over the years, adding module after module. Perhaps this column is about a new module that controls nuclear missiles? Computers (and Oracle ERP these days) seems to control everything else but thankfully Oracle hasn’t quite got to the point of having a module to do this. Worrying if they ever did, given the number of bugs in the early R12’s.

Oracle Support: Good morning. Can I help you?

User: Yes, we implemented Oracle’s Global Nuclear Missile Control module in Fusion Apps and it’s launched a nuclear missile accidentally against a large city that will kill millions in two minutes from now.”

Oracle Support: Yes, that’s a known bug. We will work on a patch and get back to you in a few days.

User: The missile will hit in two minutes. We need to escalate.

Oracle Support: We’ll have the duty manager phone you within 3 hours. Goodbye.

Ironically we’ve had a few Severity One Service Requests that make this conversation horribly familiar………

But what would happen if a hacker managed to get into your ERP system? Access to your Payroll. Access to your Financial results. Access to your HRMS System. Access to Payments. Imagine a hacker being INSIDE your system. Have a look at the column R12 and the Auditors from Mars. That will give you an idea of the horrible consequences of someone being inside your systems………..

Which brings us nicely to a movie recommendation. A hacker inside the system? It has to be the original Tron which I would give 9 out of 10 for it’s vision, way ahead of it’s time. The more recent version was OK, but lacked something I felt. But still worth seeing.

Now to get down to the serious business.  R12 did have a rather nasty payload, of thermonuclear proportions. I don’t normally write (or disclose) hacking vulnerabilities, but given this is already out on the web and represents a serious threat to you, I thought it now appropriate to warn everyone about what is a real global thermonuclear device, just waiting to go off in your ERP System with potentially catastrophic results.

In R12 a JSP file was shipped – jtfwcpnt.jsp. This JSP takes a query that executes against your database opening you up to SQL Injection based attacks………..Now let me see if I had access to an ERP Database as a hacker where would I want to start…….??????

I am not going to go into the details of how this is exploited, but you should strongly check if this file is used and then remove it if not. This warning is applicable for anyone who is using products such as iRecruitment, iSupplier or other DMZ based products in R12. (although an internal attack could equally be done).

This vulnerability seems to be across all R12 releases judging by other reports on the web. (We’re currently upgrading R12.1.3 and will be checking this also shortly).

This file represents a very serious risk to your entire ERP and therefore to your company

And to end this rather serious column, we need a movie recommendation. Well that has to be the original War Games movie. A great story from 1983 and decades ahead of it’s time. It’s all about how computers controlling everything internally are accessed from external sources to almost start a nuclear war. For me given it’s relevant 30 years later, it’s a 9 out of 10.

There’s quite a number of very serious points to this column.

We have to wonder what Oracle was doing shipping stuff like this, whilst busily shipping security patches quarterly. I am just utterly stunned this ever got out as part of the shipped R12 product.

I’d also suggest that companies start looking very seriously at security of their ERP, especially those running products in the DMZ.

Review the papers on Metalink on the best practises for DMZ. Review Steven Chan’s column as there is always great information, but most of all, out of this learning experience, google regularly for security vulnerabilities on the web about R12 or R11 – I know most people don’t do this, which is why I published this vulnerability in this column. Solution Beacon also provides some good security information. Also make sure you have decent firewalls (Oracle has released a new product just recently) and software to protect against SQL Injection and other similar attacks.

Also do keep your ATG and Quarterly Security patches up to date. I know how difficult that is, but it is critical. (A previous security patch closed a hole in iRecruitment that could be exploited from outside). See R12 Patching and the Art of Zen for an approach that makes this less painful.

Security is very much a multi-layered approach and your ERP needs heavy protection like any other corporate system. (and arguably even heavier than most).

The hacking days of Windows and Internet trojans will continue as they have done for many years, but there’s a new age of hacking dawning and there is a real awareness from hackers on other areas, and that now includes ERP Systems such as SAP and Oracle.

This is a real wake-up call in terms of security with ERP and I hope that everyone really starts looking at ERP security as a priority in their companies, over and above anything else.

The dawn of the ERP Hacking Wars is beginning……

Further Prophecies can be found at

R12 Patching and the Art of Zen

September 18, 2010

Reading through Wikipedia, I found an interesting article on the concepts of Zen. Now I’m not really into that type of stuff myself (each to their own), but I thought it would make an original way to present this article 🙂

“One practice of Zen Buddhist’s is Koan Inquiry. A koan is a question, or statement, the meaning of which cannot be understood by rational thinking but may be accessible through intuition. The answer can occur during meditation or during your typical daily life with all the mundane tasks you do.

To Zen Buddhist’s the Koan is “the place and the time and the event where truth reveals itself”. It is a way to induce an experience of enlightenment or realization, not through rational reasoning, but through intuition.

Answering a Koan requires a student to let go of conceptual thinking and of the logical way we order the world, so that like creativity in art, the appropriate insight and response arises naturally and spontaneously in the mind.”

Or to quote from a very non-Zen perspective, you think about a problem very hard all day. You fail to make any breakthrough. During the next morning, in the shower, without even thinking of the problem, you suddenly think of the idea. Ironically perhaps we are all practicing Koan Inquiry as a natural state of mind to solve difficult problems, without even having to think about the problem at hand.

Now let’s look at ERP Patching in relation to Zen 🙂 We need to be clear from the outset that this form of Zen Patching applies only to the following patches below. This is extremely important to keep in mind.

Security Patches

ATG Patches

Database Patches

This form of Zen does not apply to other ERP Patches

Applying this form of Zen Patching to any other types of patches will cause you some serious grief in your career when you report to your boss that your ERP for your entire organization worldwide is trashed because you read some amazing article by some “new age ERP guy called the Oracle Prophet” on a radical new method using the Art of Zen for ERP Patching and thought it was worth a try on your Production System……….

Do note that this form of Zen Patching does work on both R11 and R12, but not on R10. It also works on 9i, 10G and 11G databases. Please check Oracle Certification matrices and raise the question to Oracle Support if in doubt.

Oracle Support – Good morning. Can I help you?

Reader – Yes could you tell me if the Art of Zen patching is certified against R12 Apps please?

If the phone goes dead at this point, we suggest you assume Oracle Support is not aware of the Art of Zen patching and you should not pursue your question with them……….We also suggest you give your colleagues name during any telephone calls with Oracle,  in case Oracle raises a complaint for nuisance phone calls to your company…..:-)

So where does the Art of Zen fit into Oracle ERP patching?

 Let’s use a typical a koan to provide an illustration.

“We will test the patch by not testing the patch. Only then will we know that the patch has worked.”

Now at this point in time, you are probably thinking I’ve been hitting some fairly strong stuff to get to this state of mind, or I’ve completely lost the plot.

I can hear everyone thinking “So let me get this straight. You are going to test the patch by not testing the patch, so that you know the patch is working”. To which I’d reply, great you’ve got it. You are certainly a quick learner on this Zen Patching stuff!!! 🙂

Our R12 Patching Philosophy actually made our auditors jaws drop, not in terms of the Zen stuff (trust me, keep this stuff between yourself and myself please and maybe better not mention to your management or auditors……), but on the thoroughness of approach.

We always have five databases for our patching (at a minimum). This is probably a lot more than most have but let me explain why and you’ll probably want to then copy this model.

Our DBA Environment. This is where the patches are applied to make sure, well they actually apply. Believe it or not some patches from Oracle don’t even apply cleanly.

Our Patch Environment. This is where they are applied with a little bit of testing. OK we deviate a little from the Zen stuff, but give me a break……This makes sure they at least do what they say on the box without major functional failure.

Our development environment, which is always busy with daily activity by our development team , functional team and testers.

Our test environment which is always busy with daily activity by our users.

Our Production environment.  I’ve been pushing our company to drop this as it uses a lot of space and we hear most of our complaints from this database, but management insist it is important and needed. 🙂

We should also state our databases are pretty heavily used so application flows naturally are being used throughout development and test databases. We also apply any patches onto any other instances we have at that point in time, so that the patch is naturally tested by the simple day to day activity in as many places as possible, with a careful rollout to each environment.


The Art of Zen Patching

The point is simple on these types of patches. Oracle does release patches that should be applied at some point.

The Security Patches typically come quarterly and we try to apply 3-6 months after they come out. Security patches represent a serious risk to apply, although generally apply well. However security patches also represent a risk if you do not apply. You need to find the balance, but you SHOULD apply these regularly.

The ATG Patches are less frequent but provide critical updates to Browsers (especially if you have DMZ applications) and other technology components, including diagnostics.

The Database patches (and we’re talking 10G to 11G for instance) do come out periodically and at some point you need to decide to keep at least supported, although we’re very picky on applying these, but are in the process of an 11G Upgrade. (Various 10G database versions are losing or have lost premier support). This activity is every couple of years or so.

We’re not talking about applying every patch. No company in their right mind can achieve this. We’re talking about keeping your head above water and staying supportable.

The point is this. To test every time on these types of patches across every last item is impossible. The conventional way is to get the patch, apply, test everything and then move to Production in a few weeks. That’s a very logical way to order the world of Oracle ERP. But unfortunately this is not a very practical or safe way. These patches are by their very nature too broad and silently hit too many areas to be open to a logical, standardized testing process. The conventional approach actually increases risk with these types of patches because it is by no means obvious what could be impacted.

There must be a better way where you find that balance. This is the Art of R12 Zen Patching.

Our philosophy is simple. We plan carefully on all these types of patches well ahead of applying.

We do not apply these patches immediately they come out. We are kind enough to give others the opportunity to be the heroes or unenlightened who find the bugs, log them in My Oracle Support and make our life so much simpler because we heavily research each patch to find the problems the unenlightened logged. This way we avoid the bulk of the problems. Are you one of the unenlightened? If so we appreciate you finding the bugs for us, causing issues for your users on Production systems and generally making our life so much easier and less stressful.

Our philosophy then rests correctly on a peace of mind that these patches are largely stable, largely trusted and tested by others around the world. This isn’t just a philosophy, it’s backed up by hard facts based on an incredibly low failure rate of patches we have applied. The patch types listed are generally very mature, very stable and very reliable. The quality of these types of patches is far higher than the Oracle ERP patches for the application modules.

Our key philosophy can be defined by the koan below. (As you raise a smile, remember this is used in leadership teaching by guys that make more in a month than you make in 5 years and sell books by the truckload at Amazon 🙂

“Once upon a time in ancient Japan, a young man was studying martial arts under a famous teacher. Every day the young man would practice in a courtyard along with the other students. One day, as the master watched, he could see that the other students were consistently interfering with the young man’s technique. Sensing the student’s frustration, the master approached the student and tapped him on the shoulder. “What is wrong?” inquired the teacher. “I cannot execute my technique and I do not understand why,” replied the student. “This is because you do not understand harmony. Please follow me,” said the master. Leaving the practice hall, the master and student walked a short distance into the woods until they came upon a stream. After standing silently beside the streambed for a few minutes, the master spoke. “Look at the water,” he instructed. “It does not slam into the rocks and stop out of frustration, but instead flows around them and continues down the stream. Become like the water and you will understand harmony.” Soon, the student learned to move and flow like the stream, and none of the other students could keep him from executing his techniques” – Timothy H. Warneka

Now I’m not into all this stuff and I’m as skeptical as anyone else, but maybe they have a point. Too many companies are simply slamming into the rocks with patches, rather than working with the flow of Oracle Corporation. Working with Oracle, you often feel that you are not talking of a stream but more a raging torrent of patches. The problem is you are always fighting against the flood of patches, rather than finding what these guys would refer to as “harmony”.

The very essence of our philosophy and the koan itself can now be answered 🙂

“We will test the patch by not testing the patch. Only then will we know that the patch has worked.”

After rolling the patch through DBA and Patch environments very carefully over many weeks, we are ready to proceed to our main development and testing environments.

We typically roll patches into our development environment for a minimum of 4 weeks. We observe the behavior of the environment and record any bugs. We carefully investigate all bug occurrences.

Once we are comfortable at that point, we do run testing. OK so we broke our mantra, but nowhere near the testing that would normally be required. Why? Because we have seen the bugs naturally arise through our normal daily activity (as a Project Manager you’ll know typically what is going on and where the gaps may be I would hope). So to quote the Art of Zen,” the appropriate insight and response arises naturally.” This is the beauty of the Art of Zen Patching 🙂 You do your daily stuff to get to the answer of whether the patch causes major grief.

At this point we normally release the patches to our Test Instance, again allowing patches to settle for 4-8 weeks. Again using normal user activity, we gain further appropriate insight and responses, in terms of stability of the patch and subsequent bugs, arising from the natural process of user activity.

We do ask our users to test and hit the key functionality, but again, with the insight given from normal daily use, we have achieved ” the appropriate insight and response which arises naturally” as a Zen Master or Leadership or Lifestyle coach would tell you for quite a lot of cash 🙂

Even our DBA Team reaches a relaxed Zen like state and if you know your average DBA guys……… With planning comes time for our DBA Team to work and document carefully the steps needed for each patch. The timeframes create space for many, many practice runs, so that on the day of application to Production, they know exactly what to do and what to expect. It also creates the space and time for good old fashioned research on My Oracle Support. (A tip is that as we’re doing this approach over a number of months, the DBA’s always get copies of Production on a regular basis to run through the patching process, so any production specific issues are always encountered early).

In addition, we carefully plan for releases. So if you take our last security patches, these were rolled into an ATG RUP6 and a minor database point release (to stay supported). This reduces a constant patch cycle to a more manageable ITIL Release concept, reducing your workload overall. The raging torrent of patches becomes a much more manageable stream.

Most companies have huge stress over these types of patches. Most companies don’t even bother applying, much to the detriment in terms of support, future upgrades and security.

We are like every other company in many respects. We are highly conservative on applying patches. We like to stick on what we know.

But we do pay attention and plan for security, de-support of databases, new browser support in ATG RUP’s, etc in a very careful manner well ahead of time, allowing us to practice the Art of Zen Patching 🙂

Companies stress out, rush patches and therefore make mistakes. That is not the Art of Zen Patching. Zen Patching stresses the very opposite approach. Put the patch into your environments, slowly observe and watch over many months, then test and finally you will see the appropriate insight and responses that it has worked. Now looking at Wiki again, Monks meditate over many months or even years to answer a Koan. There is no difference in the approach of Zen Patching 🙂 The time the patches spend in your environments can be thought of as your meditation period over many months (typically 3-6 months depending on risk assessment) to find the answer to the koan of “how to test the patch without testing the patch to know the patch has worked”.

But with our R12 Zen Patching, we’ve reached an almost Zen like state 🙂 Patches are simply a natural part of the lifecycle of ERP. We have accepted that. They are planned and are allowed to settle for several months, to give insight into their nature and risk. Patches still require testing, but to a far lesser extent than a fully focused, high risk “lets test this patch and apply this patch in two weeks time”, which is similar to slamming into the rocks in the stream.

So where are we today with such an approach?

R12.0.4 RUP5 (yes this was a nightmare of testing the old fashioned way, definitely slamming into the rocks in the stream, but our go-live was incredibly smooth. Zen Patching doesn’t work here I’m afraid for those embarking on an R12 Upgrade. It’s the good old fashioned conventional testing approach that is needed here).

Security Patches to April 2010


11G and July Security Patches are currently under the Zen method as is a SUSE Linux Upgrade

We have never had a failure or serious outage as a result of Zen Patching (should I trademark this perhaps and make a lot of cash like those leadership guys??????), although as with all Oracle patches, it is a very serious business, with serious risks, so there is no place for complacency.

So what about the opinions of others on our ERP and how up to date we are overall with patches?  To quote one Senior Consultant DBA recently, we are way ahead of most companies in terms of patching, and have an “aggressive patching policy”.

I would say that the Art of Zen Patching can never be described with words like “aggressive” 🙂 . In fact it is quite the opposite. It is a very slow and considered process, stressing great patience, over many months, waiting for insight as a part of a natural process to reduce the risks we face, as the real Zen guys would put it 🙂

We simply achieve a lot more than most companies, with a lot less effort and a lot less risk. I think the Zen guys and leadership/lifestyle gurus would term it as “simply learning to move and flow with the Oracle stream creating harmony and peace of mind”. Obviously they’d be charging a thousand US Bucks an hour for this type of advice. (I remember we had one such IT Guru in our company. Cost us 6 figures for six weeks of work – we ended up using his laminated b*llshit as coffee mats……that was about the only value we got……). Now maybe the Consultant our company hired wasn’t too hot except for the coffee mats, but with some of these leadership and other philosophies, well maybe there’s something in it after all……

Call it Zen. Call it Lifestyle (or is it Patch) Management :-), but to have a safe, low risk and stress free approach to this type of patching which works with reduced effort (rather than increased effort) is  not a bad place to be, as an ERP Manager…….

Health Warning

This article was designed to present a very serious subject in a hopefully entertaining and educational manner utilizing both conventional approaches of testing, in conjunction with a more unconventional approach. However applying any patches on a Production Database is a very serious business. Patches do need testing and this should never be underestimated. However the point of this article is that by allowing patches to settle into various instances over time, you vastly increase the chances of spotting serious issues and vastly reduce the risk of issues in production that conventional testing can never address. This is the safest way to apply such patches that I have found, using a conventional testing approach, together with a far less conventional approach. 21st Century Testing meets 5th Century Zen Philosophy.


(By the way, I haven’t been smoking anything….the intention of this article is to present a very serious subject – Oracle Patching – in hopefully what some will find a very funny and original manner, that can then be remembered and applied to help all of us that face the very serious risks of patching global ERP Systems, so don’ take all the Zen references for Patching too seriously………otherwise someone may think you’ve been smoking something………). Think Monty Python British humour as you read it……..

If you remember the article, change patching from a rush to a planned perspective and patch carefully over a period of months, then the article did it’s job 🙂

I hope you find it as funny to read as I did writing it 🙂