Archive for the ‘ERP Management’ Category

The Quest for the Holy Grail of R12 Patch Management

April 26, 2011

With new technological innovations, please wear 3D Glasses and Headphones for optimal experience whilst reading this blog. This represents the first ever blog to cover ERP Columns and Movie Columns, together with hot off the press exposes that Oracle Technology came from Alien UFO’s captured at Roswell, all in 3D and Dolby Sound. Popcorn and a coke are recommended.

Ensure your PC sound is turned down if in the office with no headphones.

In the time of King Arthur, legend had it there was a magical cup, a holy grail.

Whoever drank from the cup would have immortality. During the time of King Arthur, many knights embarked on dangerous quests to find this magical grail, always to fail. The question is, were those pursuing their quest searching for something, that perhaps did not actually exist?

Sometimes it feels like Oracle Patch Management is perhaps a search for the holy grail. To be able to stay current on patches, to be able to apply patches and still ensure that no adverse effects are inflicted on the Business seems an almost impossible quest. This column looks at some of the strategies that can be used, to perhaps not achieve the holy grail, but certainly move towards the holy grail of patching ERP.

Now previously some crazy oracle prophet had written about the Art of Zen Patching . Let’s be clear on this – the quest for the Holy Grail of R12 Patching is not such a zen-like endeavor. It is an arduous, painful and often disappointing journey, frought with failure, despair and danger.

And at this point I feel a movie review is appropriate.

One of the most famous movies (and one of my all time favorites EVER) covering the times of the Holy Grail has to be Excalibur. Highly recommended on DVD. A superb and very haunting movie. It is one of the most atmospheric movies I have ever seen. You just have to see it.


So the movie is about a search for something that people dream of but quite possibly doesn’t exist. It’s about human dreams that become all consuming leading people to eventual madness and despair.

For many years I have been searching for what can only be described as the Holy Grail of ERP Patching. I started off pretty sane, but as the years went on my sanity deteriorated until I started writing columns combining movie reviews and Oracle ERP tips. I tried to turn to Zen (The Art of Zen Patching), but lately, just maybe I found the Holy Grail of Patching. I uncovered this during an R12 Upgrade Quest. In the midst of battle, with patches coming out daily from Oracle (on 12.0.4) we perhaps found not a perfect grail, but certainly something that made the quest worthwhile.

As I explained in a previous column there are a few patch scenarios:

1. Security Patches, Database Patches and ATG Patches on a Production System. This column could be used for that, but I’d also recommend the Art of Zen Patching for these types of patches. These patches are generally stable (if left to others to apply first) and cause less problems from experience. Note however we HAVE had problems on each of these types of patches, so don’t underestimate the fact that the quality is better that you can be sloppy on your testing.

2. Patches that come out during an R12 Upgrade Project or need to be applied. I guess this column kind of applies, although depending on how far your upgrade is you can take a more aggressive patch approach. If you are early in an upgrade, you generally throw in the Rollup Patches or others as needed. The later in the project you are the more careful you need to be.

3. The Holy Grail of Patching refers to the approach taken for Patches on a Production System. It is like searching for the grail, a very slow and dangerous process, with little in the way of rewards.

As everyone knows Oracle has patches to fix the patches you applied. This isn’t good, because like Forest Gump, Oracle gives you a box of chocolates, and your never sure if there is something unpleasant in one of them…….Or as I say, our office coffee machine (I live in the Far East) is like an Oracle Patch. Occassionally you get a cockroach dispensed……..(and no I am not joking on this one – cockroaches like Nescafe too…….)

Our approach to the Holy Grail of Patching can be described as a highly conservative and disciplined approach, based largely on the principles of ITIL. Our primary aim is not one of speed, but of keeping our Production Systems safe and stable, to avoid major business disruption. It is a very slow and very painful process.

So let me walk you through the process (Apologies, it’s going to get a bit serious from here on in. Have a look at the Alien Technology blog if you need something lighter……Oracle Fusion finally went live in one company. Poor buggers………Although it is based on captured Alien Technology from Roswell……………so I am sure as a first release of an Oracle product quality is great as it always is……. )

The bug process starts either with the EBS Support Team (I assume you are proactively look for bugs/problems before they happen, by keeping an eye on Oracle Support Updates) or the business user reporting an incident. The Incident Management process should record the details and check if it’s a known problem or there is a workaround.

This should then be passed to your Problem Management Process, where you investigate the problem reported (DS10B-001).

You should then try to replicate the problem (DS10B-002).

If you can replicate, then it’s time to log on to My Oracle Support (DS10B-006). (I’ll slowly introduce some ITIL Terminology into this column. ITIL is a framework to manage IT Systems and it’s very, very good indeed, applicable not only to ERP but to any IT activities. If it was a movie I’d give it a 10 out of 10, but it’s a bunch of great books that can be found here. Introducing these ITIL concepts can seriously improve your service to your ERP users.

And onto another movie review. It has to be Monty Python and the Holy Grail. This is about as British as humor gets. A great movie and very, very original. It’s a 9. Get it on DVD.

On My Oracle Support you should check for similar bugs. It’s a great database and no doubt someone else has hit your bug before you (unless you are one of the suckers to take an early release).

You’ll get to one of two points here. You’ll either identify a patch (which may have pre-reqs) or you’ll come up blank. Lets deal with the second point.

If there is no patch (and don’t be lazy not bothering to look) then raise a Service Request (DS10B-012). Now Oracle’s game isn’t necessarily to help you fix the problem………..Typically responses are as follows, but we suggest you counter-act these by using phrases from the movie Monty Python and the Holy Grail. Assuming Oracle Support doesn’t hang up, this will certainly be a tactic they have not seen from other companies………..

Oracle Support – Good Morning. How can I help?

You – “Go and tell your master that we have been charged by God with a sacred quest.”

Oracle Support – We’re very sorry to hear that.

You – “Everytime I try to talk to someone it’s ‘sorry’ this and ‘forgive me’ that and ‘I’m not worthy’!”

Oracle Support – Sorry Sir, could you explain the problem with your system?

You – “Come and see the violence inherent in the system!”

(We recommend you explain the problem, maintaining the Monty Python style and English accent – pretend you are an old English queen)

Oracle Support – That’s standard functionality Sir. The accounting on the Payables isn’t supposed to do proper exchange rate variance calculations, otherwise your General Ledger would be correct for the 30M Dollars you spent on Oracle. Now that sounds perfectly plausible, doesn’t it Sir?

You – “Your mother was a hamster and your father smelled of elderberries.”

Oracle Support – Sorry Sir, let me just check (By this point Oracle Support is a little concerned)

Oracle Support – Ah, it’s a known bug. It’s in development………..should be out in six months…….

You – “I’ll bite your legs off!”

Oracle Support – Wait Sir, Wait Sir, let me check. I’ll check again. You need to go to R12.0.6 (By this time the support person is probably thinking “nutcase on the phone, better help him quick”……)

You – “Jesus Christ!”

Oracle Support – This small patch will definitely fix your problem !!!! (Now Oracle Support knows this has nothing to do with your problem but ask you to apply it anyway just so you get off the phone and they can close the call………)

You – “Pull the other one!”

Oracle Support – OK we suggest you apply the latest Rollup Patch

You – “I fart in your general direction.”

Oracle Support – OK how about this patch, which has 300M of Pre-Requisites. That will do right?

You – “Aaarrrggghhh!”

Oracle Support – What about a one off patch that no-one else has that will trash your system and is cobbled together by our developer?

You – “You tiny-brained wipers of other people’s bottoms!”

Oracle Support – But we’ll need to spend 4 weeks of burueacracy giving you a hard time to justify our development team doing this for you, even though it’s a serious bug we should fix……..

You – “You can’t expect to wield supreme executive power just ’cause some watery tart threw a sword at you.”

Oracle Support – Wait I’ve actually done my job and found a correct patch………

You – “I feel happy!”

A very Monty Python approach, but every scenario above represents some actual conversations with Oracle Support……(bar the Monty Python quotes – I normally pretend to be Harry Potter when talking to Oracle Support).

They may also pretend it’s a problem for another team (e.g. AP blamed the SLA Team and vice versa) or point to the good old technology stack components or database components. Common tactics by Oracle Support.

There’s a very serious point to the above. It shows the strategies that Oracle Support uses against you when it comes to patches. Your end game with Oracle Support is to get you to the minimum patches required to fix your problem. You and Oracle have very different aims sometimes (Oracle’s current aim is to move everyone onto minimum of 12.0.6…..), so you have to be very careful in how you manage the Oracle Support Analyst because that seriously affects your patch outcome and therefore testing work and eventual risk to your system.

And it’s got to be time for a movie review. This is another favourite Indiana Jones and the Last Crusade. Great movie for all the family. It’s a 9 out of 10 and keeps our Grail movie theme moving along nicely. This makes me feel old, because when the first Indiana Jones movie came out I was 10. Now Indiana Jones (Harrisson Ford) is a pensioner…….

So you’ve got that patch sitting on your server waiting to be applied. Do you feel lucky punk? You’ve reached DS10B-016 in the Patch Management Process.

A quick excerpt from a previous conversation I had with a person in one company:

Techie Guy -We need to apply the payment patches this weekend.

Oracle Prophet – Why?

Techie Guy – My manager told me to

Oracle Prophet – What did he say?

Techie Guy –“If it’s not done by sunrise, I’ll cut your balls off.”

Techie Guy – So we had real problems with the application of patches in a Test System that knocked out all Payment functionality.

Oracle Prophet – So you’ve never had a clean run of applying this patch?

Techie Guy – No, that’s right. Never been applied correctly in any test system. (Stated not with concern, but an unbelievable stupidness that is rarely found). So can we go ahead and apply it in Production this weekend?

Oracle Prophet – “I’ve got no option but to sell you all for scientific experiments.”

This was a true conversation of one person considering applying patches that he had never gotten to work during a testing phase…..to our Global Payments Systems……….truly a new way of testing patches. “Yep it didn’t work in any test environments, but maybe it will behave differently in Production when we apply……….”.

Our first step for Patch Application is normally to look at the size of the patch. Now a lot of people say size does not matter – many woman beg to differ. In this case, the smaller the better – the patch I mean. It gives an indication of just how many objects it’s going to hit potentially. Not scientific but a quick check gives a reasonable impact assessment. However……….

Applying patches is about a horrible and painful quest with little reward. But going back to King Arthur the Oracle Prophet recommends you seek a mystical wizard. Yep, your thinking I’ve lost the plot again on this one………..Just trust me. Go into your Manager, now obviously your credibility is shot to pieces after suggesting using the Art of Zen on corporate systems. Obviously he thinks you are a bit mental after suggesting using captured alien technology for deployments of ERP customizations.

But trust me, when you tell him you should seek out a mystical wizard to improve patching, you will truly have made those other suggestions look almost coherent………

So King Arthur had Merlin as his wizard. On your quest, we suggest that you take a look at the Patch Wizard from Oracle. (I surpass myself in corny literally prose this time……..)

Oracle has had a Patch Wizard for a considerable amount of time. I am not here to teach you all there is in this (see the excellent authors at the end of this column). I am here to raise awareness of these tools, what they do and how they fit into the Holy Grail of R12 Patching. The Patch Wizard can be found under the System Administrator Responsibility.

Once we have assessed the patch size, read me, etc, we normally try to apply the patch on a temporary database (we usually have one we can apply on). (We’re now on DS10B-007 in the Process Chart previously).

The Patch Screens below are a very useful impact assessment tool. Now size isn’t always an indication of how satisfying the patch will be for you………After applying the patch, bring up the Applied Patch screen and query the patch applied.

So a patch may be 300M. But it may not apply many files, if your patch levels are already recent.

Looking at the files copied and Action Summary will give you a very good idea of just how satisfying that big patch has been for you.

If a patch shows that it is replacing database objects, forms, etc then it certainly has had potentially an impact on your system.

We’ve faced large patches, but when looking at the files applied, we may only end up with a handful of files. This allows us to then use our technical guys to put a reasonable assessment on roughly what we have to test.

With an idea of what we have to test, we can now draw up a reasonable test plan for the patch. Without the patch screens we would be looking at guessing the best approach to testing. Now it is not easy to work out from the files exactly what to test – you need to be careful in that respect.

Now a message to Oracle. Why don’t you enhance the Patch Wizard to do a mapping between code files and Functionality. This would allow Functional Users/Support Teams to far more easily assess the patches, then draw up a far more accurate test plan. This would be an excellent enhancement for Patch Management for companies globally.

One of the problems with patches is where they overwrite your customizations (customizaton by modification). I suggest that where you modify shipped Oracle code to make new customizations, you record these in the appcust.txt file. To quote the manual “Oracle Applications uses this file during patch processes to generate warning messages that customizations are being overwritten or may need to be replaced after the patch”. It’s a handy tip. A missed customization that is overwritten by a patch can cause immense damage later……….

Our approach to testing patches is comprehensive.

We apply the patch onto a DBA Instance. (Step DS10B-008). This lets the DBA’s check the pre-reqs, make sure the patch applies cleanly, makes sure objects are not invalidated, etc. We then do a very quick sanity test.

We apply the patch onto a PTH Instance. (Step DS10B-009). Our test here is not thorough, but gives us confidence that the patch has fixed the problem and does not break any other key functionality.

We apply the patch onto a DEV Instance. (Step DS10B-010). Testing becomes more thorough at this point from our internal team. (Make sure it is a clean DEV instance, else use another clean environment). Once complete by our IT ERP Team, we release the Patch to our users.

We also test critical functionality when applying a patch. The drop dead functionality such as entering an invoice and paying. We also do an end to end around the patch when necessary (e.g. a quick cycle through procure to pay). Testing patches well in the IT area of ERP ensures quality patches are delivered to the Business. The messy dealings with Oracle and cycle of patches to fix patches is kept out of the Business Users view, as sometimes this could undermine confidence in the patches themselves otherwise.

We apply the patch onto a TST Instance. (Step DS10B-011). Our users then independently test this patch, using their own test plan and test cases. Keeping the testing independent (separate IT and Business Testing) gives more chance of any issues with the patch being caught.

For testing patches, we have suites of comprehensive test scripts/test plans across our application. We built these up as part of our previous R11 and then R12 Upgrade. This allows us to test our scenarios easily and quickly. It also covers integration to other systems. With freehand testing, formal test script usage, business user testing and IT Testing, we avoid mistakes.

With sign-off from the Business Users (DS10B-018), the patch is passed to our Release Management process for scheduling into Production.

With the use of the Patch Wizard and Patch Screens in R12, our testing becomes much more targeted, minimizing testing effort but re-utilizing our test script library selectively.

By having multiple databases with multiple testing cycles from both business and IT, our testing becomes much safer, reducing risk of a bad patch reaching production. It also protects other environments including your Development and Test Instances. You wouldn’t want to wreck these with a bad patch usually.

There are a few options we are looking at to further improve our Patch Management:

We are looking at Oracle Configuration Manager as a potential to alert us to patches that need to be applied, moving to a more proactive patch management policy.

We are looking at Testing Automation. Our preference is the HP Tools with pre-built testing scripts for ERP, but we know there is a huge amount of work before we could get to proper automation. Oracle also has a Testing Tool, but I still prefer HP’s to be honest. Automated Testing does seem to be the holy grail eventually – to be able to press a button at 5:00PM, have all the test scripts executed and come in at 8:00AM and see the results really is something that I think companies should aim for. However it’s expensive, difficult and often a huge failure when companies try to do this.

Finally we just bought Oracle RUEI. I have written about this in the past. It is a support revolution. We can now see the errors that users are encountering, the system performance that they are experiencing and so much more. Again it’s a move to a much more proactive incident/problem management. It even has an option to playback user errors. As said no more having a helpdesk asking “And what step did you do next. What data did you enter, blah, blah, blah and failing to take the right information for the Support Analysts”. It has pre-built integration into ERP and Oracle had this up and running for us in a matter of days. It even works with our custom modules……… I’d recommend this product to anyone running ERP. It’s a super product, quite revolutionary in terms of support.

And the final movie recommendation. Dan Brown’s Da Vinci Code. I read the book (also highly recommended) on a long haul flight to the Far East. Then I watched the movie on the TV a few years later. The movie is pretty interesting and I’d say it’s an 8.

Personally I always thought Mona Lisa looked absolutely miserable……

But Dan Brown’s book caused a huge amount of controversy. It’s a full circle from where I started the column. Many looked for the holy grail yet never found it. Dan Brown’s book was based on the premise that the Grail wasn’t really an object, but the grail was actually a person.

I’d say that to find a Holy Grail of ERP patching isn’t perhaps looking for an object or a magic bullet. I’d say that the Holy Grail of ERP Patching is also, actually a person. A person that puts in place the strong processes, controls and procedures, whilst thinking smart to reduce the work to enable a safe and reasonable patching strategy that supports the business goals. That is the true Holy Grail of ERP patching.

One last question – why are you wearing those 3D Glasses? You look pretty stupid in those in the office, given this is just an ordinary web page……..

And to end on an equally silly Monty Python note:

Woman -“He’s the Messiah !!!!”

Oracle Prophet – “I am not the Messiah!”

Man – “He’s not the Messiah. He’s a very naughty boy.”

Until next time when we look at R12 Patching – Friday 13th – A Horror Movie Special.

Further secret blogs and prophecies can be found at:

https://oracleprophet.wordpress.com

References

This column was based on some very hard-won experience. However other parts of this column came from research from generous authors in the Oracle Community. Below are some excellent articles that further shine light on the difficult process of Patching ERP safely.

Oracle E-Business Suite Patch Wizard – Path to Less Errors

Release 12 eBusiness Suite Codelines, Codelevels and PatchWizard

The 11i Patch Wizard

Managing R12 EBS

Tips and Tricks for Patching R12 EBS

Best Practises for Patching and Maintaining EBS R12

Real User Experience Insight

Global Thermonuclear War – The New Oracle R12 Feature

January 18, 2011

This column will be short and sweet, explaining how someone can launch Global Thermonuclear War on you, completely wiping you out. Nice topic and not any jokes this month I’m afraid for such a serious subject. Of course the movie reviews will still be included, otherwise I’d lose my rating of the only combined Oracle ERP and Movie Rating Column on the Web……….

Now Oracle ERP has grown immensely over the years, adding module after module. Perhaps this column is about a new module that controls nuclear missiles? Computers (and Oracle ERP these days) seems to control everything else but thankfully Oracle hasn’t quite got to the point of having a module to do this. Worrying if they ever did, given the number of bugs in the early R12’s.

Oracle Support: Good morning. Can I help you?

User: Yes, we implemented Oracle’s Global Nuclear Missile Control module in Fusion Apps and it’s launched a nuclear missile accidentally against a large city that will kill millions in two minutes from now.”

Oracle Support: Yes, that’s a known bug. We will work on a patch and get back to you in a few days.

User: The missile will hit in two minutes. We need to escalate.

Oracle Support: We’ll have the duty manager phone you within 3 hours. Goodbye.

Ironically we’ve had a few Severity One Service Requests that make this conversation horribly familiar………

But what would happen if a hacker managed to get into your ERP system? Access to your Payroll. Access to your Financial results. Access to your HRMS System. Access to Payments. Imagine a hacker being INSIDE your system. Have a look at the column R12 and the Auditors from Mars. That will give you an idea of the horrible consequences of someone being inside your systems………..

Which brings us nicely to a movie recommendation. A hacker inside the system? It has to be the original Tron which I would give 9 out of 10 for it’s vision, way ahead of it’s time. The more recent version was OK, but lacked something I felt. But still worth seeing.

Now to get down to the serious business.  R12 did have a rather nasty payload, of thermonuclear proportions. I don’t normally write (or disclose) hacking vulnerabilities, but given this is already out on the web and represents a serious threat to you, I thought it now appropriate to warn everyone about what is a real global thermonuclear device, just waiting to go off in your ERP System with potentially catastrophic results.

In R12 a JSP file was shipped – jtfwcpnt.jsp. This JSP takes a query that executes against your database opening you up to SQL Injection based attacks………..Now let me see if I had access to an ERP Database as a hacker where would I want to start…….??????

I am not going to go into the details of how this is exploited, but you should strongly check if this file is used and then remove it if not. This warning is applicable for anyone who is using products such as iRecruitment, iSupplier or other DMZ based products in R12. (although an internal attack could equally be done).

This vulnerability seems to be across all R12 releases judging by other reports on the web. (We’re currently upgrading R12.1.3 and will be checking this also shortly).

This file represents a very serious risk to your entire ERP and therefore to your company

And to end this rather serious column, we need a movie recommendation. Well that has to be the original War Games movie. A great story from 1983 and decades ahead of it’s time. It’s all about how computers controlling everything internally are accessed from external sources to almost start a nuclear war. For me given it’s relevant 30 years later, it’s a 9 out of 10.

There’s quite a number of very serious points to this column.

We have to wonder what Oracle was doing shipping stuff like this, whilst busily shipping security patches quarterly. I am just utterly stunned this ever got out as part of the shipped R12 product.

I’d also suggest that companies start looking very seriously at security of their ERP, especially those running products in the DMZ.

Review the papers on Metalink on the best practises for DMZ. Review Steven Chan’s column as there is always great information, but most of all, out of this learning experience, google regularly for security vulnerabilities on the web about R12 or R11 – I know most people don’t do this, which is why I published this vulnerability in this column. Solution Beacon also provides some good security information. Also make sure you have decent firewalls (Oracle has released a new product just recently) and software to protect against SQL Injection and other similar attacks.

Also do keep your ATG and Quarterly Security patches up to date. I know how difficult that is, but it is critical. (A previous security patch closed a hole in iRecruitment that could be exploited from outside). See R12 Patching and the Art of Zen for an approach that makes this less painful.

Security is very much a multi-layered approach and your ERP needs heavy protection like any other corporate system. (and arguably even heavier than most).

The hacking days of Windows and Internet trojans will continue as they have done for many years, but there’s a new age of hacking dawning and there is a real awareness from hackers on other areas, and that now includes ERP Systems such as SAP and Oracle.

This is a real wake-up call in terms of security with ERP and I hope that everyone really starts looking at ERP security as a priority in their companies, over and above anything else.

The dawn of the ERP Hacking Wars is beginning……

Further Prophecies can be found at https://oracleprophet.wordpress.com

The Oracle Grinch Who Stole the R12 Christmas

December 30, 2010

Every Who Down in Whoville Liked Christmas a lot…
But the Grinch, Who lived just north of Whoville, Did NOT!
The Grinch hated Christmas! The whole Christmas season!
Now, please don’t ask why. No one quite knows the reason.
It could be his head wasn’t screwed on just right.
It could be, perhaps, that his shoes were too tight.
But I think that the most likely reason of all,
May have been that his heart was two sizes too small.
Whatever the reason, His heart or his shoes,
He stood there on Christmas Eve, hating the Whos,
Staring down from his cave with a sour, Grinchy frown,
At the warm lighted windows below in their town.

The Grinch was a particularly mean spirited creature, who’s prime aim was to make sure that everyone had a very miserable holiday season.

This will be a short but important column for the “Heroes of R12”, the companies that adopted R12 early, took all the pain and whom Oracle even honored as the best R12 Upgrade sites. To all those companies that took the encouragement from Oracle to upgrade early to R12.  Or to the 95% of companies that didn’t, the suckers that found out all the bugs.

Now every knows the story of the Grinch that hated all things Christmas.

Well as much as I love Steven Chan’s columns, Oracle gave a nasty Christmas surprise for all those R12 early adopters.

From 2012, February 1st, Oracle R12 will move from Premier to Extended Support. Great no problem, as Oracle has always kept extended support for all those on the relatively new versions such as 12.03, 12.04, 12.05 (OK not totally recent but still on recent releases than most companies)

Unfortunately as with R11, Oracle has now added a particularly Grinch like clause onto R12 Extended Support…….

“To be eligible for Extended Support, [EBS 12.0] customers will need to have applied at minimum the 12.0.6 Release Update Pack (Note ID 743368.1) and the Financials CPC July 2009 (Note ID 557869.1).   Additional minimum requirements for Oracle E-Business Suite Release 12.0 have not been finalized yet. “

So your options?

Upgrade to 12.0.6, but that’s going to be a significant upgrade to do, so may as well go to 12.1.3.

Upgrade to 12.1.3 (which is the most sensible, or 12.1.4 if it comes out in Q1/Q2 2011)

Or take your chances of not getting support (sure I fancy that one as I still have accounting issues and AP transactions getting stuck so no support will leave me in real trouble, but applying patches is massively risky and my users will scream at another upgrade in a 2-3 year time span).

Anyway the full story is here (and don’t blame Steven, he’s just the messenger), but I can’t help but feeling that Oracle has really screwed those early customers that made R12 the success and stable product that it now is.

Head’s Up – Preparing for E-Business Suite 12.0 Extended Support

If there’s a Grinch this Christmas, it’s definitely sitting in Oracle HQ, Redwood Shores and it’s delivered a very nasty Christmas present indeed to R12 early adoptors.

That gives you 13 months to get off R12.01-12.05 and move to R12.1.3. That’s not long to get the funding, get the support, get the project running, deliver it to users, through UAT and into Production.

And for the movie recommendation? I’m afraid I can’t recommend the Grinch (movie poster below). Sorry, but it sucks. On the plus side there were some good movies this holiday season including Harry Potter and the Deathly Hallows Part 1, Narnia – The Voyage of the Dawn Treader and Tron was kind of average. The Little Fockers is coming out and the same words come to mind to describe my thoughts on R12 Support and Oracle……….

And on that note, I’d just like to wish everyone all the very best in 2011. I hope it is a successful and prosperous year, for all those doing R12 Upgrades (and perhaps a few Fusion Apps implementations).

 

 

R12 Deployment using Captured Alien Technology from Roswell

December 26, 2010

On June 13th 1947, residents of Roswell reported seeing bright lights in the sky, moving at remarkable speed and then a huge explosion. 1947. The next day Mac Brazel found the crash site, and the biggest coverup in human history began. Today I am pleased to reveal a WORLD EXCLUSIVE of what really happened and how that technology worked it’s way into Oracle technology you use today. I promise with all the thousands of Oracle blogs on the internet, you will read this nowhere else……..

The crashed  UFO was taken to Area 51 (the photo above was smuggled out and is 100% genuine we believe)  and housed in a large top secret highly restricted warehouse. For over two decades scientists tried to break into the UFO’s computers with no success. From this 200 million dollar research program funded by the US Taxpayers, the only success was the discovery from the UFO of how to make non-stick frying pans. It was at this point during the famous alien autopsy that scientists also found out that the famous grey aliens keep their brains in their ass, explaining their relatively low intelligence compared with the sentinel aliens. Further there is strong evidence that a recent US President was indeed an alien, as many stated “he had his brains in his ass”.

At the same time, a bright but underachieving student named Larry Ellison, was flunking university with some style. In 1964 the government inadvertently invited Chicago students (including Larry) to the highly secure, non-existent area 51 for a field trip (Normally intruders are shot on sight but a clerical error led to 100 students, including those with far-left tendencies to be invited in). During this field trip, as the scientists were showing the non-stick frying pans Larry lost interest and wandered off, stumbling on the UFO (shown above in the 100% genuine photo). He started to mess around with the UFO computer and stumbled upon tape drives, which the 200 Million dollar research program scientists did not understand just what these were for almost twenty years. Luckily Larry had 20 large 10 inch tapes with him and he downloaded the entire UFO software. As he walked out Larry thought the game was up as security stopped him. “Do you need a hand with those tapes young man?” said the security guards. The kind guards got these dropped off at Larry’s house from the highly classified, non-existent area 51 site.

Larry spent the next ten years working out exactly how aliens were using advanced software technology using these tapes, before formally starting Oracle in 1977.

The truth is that Oracle software is based on alien technology

Now using alien technology was not all plain sailing (Larry can you stop spending so much on this sailing stuff – great pun and literally transition eh? – and start paying more in dividends….???). When Larry used alien technology to create Oracle Apps 10.7SC he did not realize that the aliens had tried this exact same strategy on their home planet of Ursula Minor Quadrant 5 with an ERP product called Grey Apps SC (note the naming similiarity) and everyone was laughing at how crap it was………The same thing happened here on earth to Larry’s home grown 10.7SC version. Larry eventually apologized for this ERP Release.

However the Oracle Database, ERP and now Fusion (even sounds alien) can now be revealed as being based on the software from the captured UFO in Roswell. And of course with such software, other vendors such as IBM and SAP (that 2 Billion dollar fine must sure have hurt), offering vastly inferior software are doomed….you may as well sell their shares now. It’s interesting to note it took a team of Oracle scientists a further 20 years to break the alien tapes for Fusion software. Oracle claims to have thousands of people working on this but this is just disinformation. Think about how little information is out there. Why is Oracle so secretive on Fusion Apps? When it does come out look for the alien code, that has not been taken out amongst the 50 million lines of source code.

Fusion Apps was delayed because Oracle couldn’t break the alien code. Fusion Apps is actually being developed not by a team of thousands as Oracle claim, but by a single programmer named Steve in his bedroom using the captured alien technology from Roswell.

And I feel a movie recommendation coming on. So captured alien technology that humans then re-engineer for their own ends? Well this one absolutely fits the bill. Terminator was a classic (as were all the Terminator movies) and they score a 10 out of 10. All are great movies to rent or buy.

Interestingly the new Exadata is similiary based on blueprints from those tapes. When you turn one of these Exadata servers on it uses less power than a lightbulb (it has alien fusion power sources and chips and a special plug from Home Depot). When you turn on the IBM servers with the same performance, all the lights in New York go out as it drains the power grid (allegedly according to a guy I talked to in a bar at 2:00AM. The lights of the bar went out and that’s when he stated someone turned on the IBM server again……..). It’s pretty conclusive proof of the use of captured alien technology from Roswell.

At this point I’d just like to say I don’t believe in conspiracy theories, or alien abductions, etc. So if you’re some UFO nut that stumbled on this column and think this is all true, please don’t stalk me for photos and email me for further stories, as I’ve got good lawyers and I’ll take out a restraining order on you……..Now onto the serious business of the column – Teleportation. (Your thinking I’ve really lost the plot today right? Stay with me on this one for one more paragraph)

Look at the definition of teleportation. It breaks down physical structures and moves them from one place to another. I’d like to claim the Nobel Science prize for showing we already have this today and finally get to the point of today’s column (some are happy because they get hopefully useful information after reading like 10 pages of crap, others probably think here come the boring stuff and it was amazing Oracle based all their products on captured Alien Technology….we suggest those people get a life…….). Don’t worry, I’ll try to keep it interesting as I’ve absolutely got to do some sci-fi movie recommendations given the title of this column right?

So the topic for today is migration of all your R12 setup, customiztions, etc. (Well think about it. You take physical information, sitting on a disk, transfer it over a network which might even be some fibre optics channel, so actually converting physical info to light…..now that is super advanced ….. and then reconstruct on the other disk. Now that’s the very definition of teleportation).

Now there are two approaches. Either you manually do this (and risk all the screw ups) or you use teleportation. I’d love to see your bosses face as two weeks ago you were telling him of Zen Patching (See the article on the Art of Zen Patching) and today in your management meeting when you are asked how to solve deployment issues you suggest TELEPORTATION with a straight face, which you got from some guy called the Oracle Prophet on the web…….

When you do an Oracle implementation, or an Upgrade, or a new module or a single project or even a simple customization, there is potentially a lot of information that needs teleported. I wonder how many people have been fired as a result of my columns so far……

So today is a high level overview of the options you have to safely move the various components around your R12 instances with the minimum of fuss and the minimum risk. This is going to be a handy column for all those doing an R12 Upgrade in 2011 (given Premium Support has ended and R11 Support is dependent on some nasty small print of applying minimum patch levels…….)

FNDLOAD

Lets start with one which I hope just about everyone knows. FNDLOAD. Now this has been around since the first aliens came down and built the Pyramids……..So to quote Oracle Oracle Applications System Administrator’s Guide – Configuration Manual:

“The Generic Loader can download data from an application entity into a portable, editable text file”. How cool is that? You can download lots of your setup and edit it, giving you the ideal opportunity to screw things up bigtime.

But FNDLOAD is an amazing utility. With a single command line you can download an incredible array of setup including:

Applications

Attachment Setup

Concurrent Program Executables

Concurrent Program Definitions

Descriptive Flexfields

Folders

Forms

Functions

Key Flexfields

Lookup Types

Lookup Values

Menus

Messages

Printers

Printer Styles Definitions

Profile Options

Profile Values

Request Groups

Responsibilities

Values

Value Sets

(and a few more not documented………)

Using FNDLOAD is a breeze. Simply type in a one line command to download and a one line command to upload. Now if your doing 1,000 Menus across 12 Project Instances during Implementation, the time savings are vast….

FNDLOAD apps/apps_psswd 0 Y Mode configfile datafile entity [param]

It really is as simple as a single one line command.

A great blog entry on the details can be found at Anil Passi Blog. This gives a much more detailed overview than this column can and is a great blog with lots of other info too.

So which movie made teleportation famous? Well that has to be Star Trek, which started way back in 1966. Now I’m not a Star Trek fan, but the the 2009 Star Trek Movie was pretty good and I’d give that an 8 out of 10. Definitely worth a rental on DVD in 2011.

Generic File Manager Access Utility

Well here’s one I’ve never even heard off – Generic File Manager Access Utility. Now in a manner befitting Star Trek and during research for this article I came across this and we should boldly go where no man has gone before (Note Star Trek finally became gender aware and change this to a more acceptable politically correct sentence…..)

As said can’t say much on the Generic File Manager Access Utility as never used it, but its covered again in the Oracle Applications System Administrator’s Guide – Configuration Manual. From what I can see it’s used to transfer Help Files between Systems, but as with everything in an Oracle manual, a lot is left unstated………

Oracle Alerts

Now Oracle has the ability to define Alerts across pretty much any business area. These are very useful and used by a lot of companies. But they can take a fair chunk of time to define and migrate. Not many people know this, but alerts can also be transferred between databases. Having a quick look at Oracle Alert User’s Guide it’s a pretty straightforward procedure:

Firstly go to the Alert Screen.

Simply choose from the Tools Menu the Transfer Alert option and fill out the screen above. Very straightforward and saves a lot of time.

Now apparently there is also an option to transfer alerts using FNDLOAD

FNDLOAD apps_user_name/apps_password O Y DOWNLOAD $ALR_TOP/patch/115/import/alr.lct my_file.ldt ALR_ALERTS APPLICATION_SHORT_NAME=”INV” ALERT_NAME=”Alert_to_download”

See Meta Link Note: 400295.1 for more details.

Now the interesting point is that you can download all your configuration files and once your happy, store them in a Source Control System such as Subversion or PVCS. Once you are ready to deploy to a new instance all the files can be automatically loaded using a Shell script (or some other newer options I’ll discuss) in a matter of minutes. Interestingly if you combine using a database flashback option and an automated shell script you can practise deployments many times in quick succession until you get it right. That keeps your Production very safe, as you will have removed all the bugs from deployment by repeateded practise runs. The Database Flashback option lets you go back to a previous version of the database very quickly indeed.

So time for another movie recommendation. Back in the 80’s, when I was a kid, I saw one of the most magical and memorable movies of my life. E.T. This movie is timeless and one your kids will definitely love. It’s funny how ET used a Texas Instruments Speak and Spell and a few bits of wire to make an intergalactic telephone to call across the Universe for help. Ironically putting a bunch of very simple tools together (FNDLOAD, ISetup, etc) can equally help you in either your R12 Upgrades or R12 Implementations………..

Data Load

So how do you spot a Functional Consultant? One way is that generally Functional Consultants have weird hairstyles. That’s a sure give-away. The other give-away is that they are sitting trying to write SQL badly so that they can get the data for their beloved Data Load Tool.

So what is Data Load? Data Load is a tool that basically takes all your data and plays it back into an Oracle ERP screen. It’s uses are endless from typing in Account Combinations, Suppliers, Inventory Setup, etc. It can work on both Core/Professional and Self Service screens. Sure there are others ways to do some of the stuff listed, but that requires heavy technical ability, and sometimes on projects that just ain’t there, so that’s when Functional guys turn to this tool, plus all the stuff that technically cannot be done without violating your support license.

Now you imagine. Even with all the “alien technology” I doubt Oracle has everything covered in terms of setup, etc. So you imagine having a tool that can pretty much load into a very large number of screens, where no API, FNDLOAD or Interface can go……….and then be able to source control everything and when you get a new environment during implementation you can simply run Data Load to create a lot of what would take you days to setup. And it’s so easy even a Functional guy can use it………

I’ve seen a lot of people rave about this product and use it during implementations. It’s definitely worth a look and you can find it at the Data Load Site.

FSG Export/Import

Apart from the funny hairstyle and dataload, another way to spot a Functional guy is that they generally are writing FSG Reports. For those that don’t know, FSG Reporting provides the ability for end users to write various accounting reports with no programming in Oracle General Ledger. Now if you’ve ever written one of these, you’ll know they can involve zillions of rowsets, column sets, etc. They take a long time to define and setup.

An excellent blog on the details of FSG Transfer can be found here.

Oracle ISetup

iSetup has been targetted to deploy a lot of the functional setup. Now I looked at this product quite a few years back and found it disappointing to be honest.

However it does seem to have moved on and it does now support more in terms of setup.

iSetup now supports:

Application Object Library

Oracle Financials

Oracle Human Resources Management System

Another link I checked is stating that the coverage is much wider across General Ledger, Accounts Payable, Accounts Receivable, Fixed Assets, Cash Management, Purchasing, Inventory, Bill of Material Engineering, WIP, Costing, Order Management, Shipping, HRMS and Payroll. Its worth checking Metalink for the latest list of objects that iSetup supports, together with re-use of API’s.

From an Application Object Library perspective, iSetup supports a similar list to FNDLOAD.

From a Financials perspective, as of 12.1 iSetup supports various General Ledger setup including Chart of Accounts, Currencies, Accounting Calendars, Budgets, Ledger Sets, etc.

iSetup has some support for Cash Management including System Parameters and Transaction Codes.

From an HRMS perspective, as of 12.1 iSetup supports various HRMS Setups including Locations, Business Groups, Organizations, Organization Structures, Job Groups, Jobs, Grades, Position Structures, Position Hierarchies and Positions. It also supports Employee Migration.

From a Payroll perspective, as of 12.1 iSetup supports various Payroll Setups including Balance Types, Defined Balances, Balance Attributes, Element Classifications, Fast Formula, Element Types, Input Values, Balance Feeds, etc.

One interesting option is the possibility to download and then modify and change. Allowing for instance setting up once and then changing for a different country and reloading. Has a lot of time-saving potential.

iSetup can also migrate XML Publisher data, Personalizations and Workflow Definitions, so it is well worth investigating as an additional tool to your deployment strategy.

Another nice feature of iSetup is the ability to compare various objects across databases. Handy when somethings stops working and you are looking for a needle in the proverbial haystack with a small piece of missing/mistyped setup.

Further details can be found on the websites below:

Steven Chan – Ten Ways of Using iSetup

iSetup Data Sheet

Frontier Consulting – iSetup

Frontier Consulting – iSetup Best Kept Secrets

Solution Beacon – Some new Thoughts for an Old Friend – iSetup

Trutek – Migating your Customs with iSetup

And with that it’s definitely time for another movie review. So a column all about aliens and their technology…..well if this column is all about Aliens and Sci-fi movies then one of the best Alien movies of all time has to be Aliens 2. I give this an absolute 10 out of 10. And you can probably pick it up on DVD pretty cheap given it’s a real old movie now, but still incredibly good.

OA Framework Personalizations

A lot of sites are using OA Framework Personalizations to modify OA Framework Pages, with a view to limiting impact of future upgrades/patches. Some of these personalizations can be quite involved and re-applying these involves a huge amount of work. Luckily again Oracle provides a handy method of exporting and importing these personalizations across databases.

More details can be found in the excellent blogs below:

Anil Passi – OA Framework Personalization Migrations

R12 Form Personalizations
 
 A little known feature of R12 is Form Personalizations. This allows you to replace quite a chunk of CUSTOM.PLL (or other Library code) with code that you define directly in R12. A great article covering this can be found at

R12 Form Personalizations

Now this not only allows you to remove custom code from CUSTOM.PLL or attached libraries (which is great), it also allows you to extract and import these Personalizations.

Simply using FNDLOAD again does the trick:

FNDLOAD user/pass 0 Y DOWNLOAD affrmcus.lct output_file FND_FORM_CUSTOM_RULES form_name 

Using R12 Form Personalizations is a more flexible and more visible way of approaching some forms based customizations. It’s also much more supportable.

Putting all the Alien Technology Together

There are a couple of interesting options with all the tools listed above that could radically improve your processes, make your ERP Instance(s) a lot safer and save you piles of time.

Firstly most of them produce files. Files can be put into Source Control such as Subversion (it’s free and great). If you Source Control your files, you can implement proper Release Control, guaranteeing that what you deploy is what you really want to deploy.

With all your files for setup in source control you can write automated deployment scripts, that will deploy all your setup in a matter of minutes (or if you have huge amounts of setup, a couple of hours). This will allow you to practise deployment in a repeatable manner until it works with ZERO failures. (Use the database flashback allows you to run deployment and then restore to a previous state in a few minutes, allowing multiple deployment runs/adjustments). Once you have that deployment release perfect, your Production deployment will be vastly safer than 99% of other companies on the planet.

Now Oracle’s heading in a very interesting direction with all this using the Application Change Management Pack. This, as Oracle puts it “provides a centralized view to monitor and orchestrate changes (both functional and technical) across multiple Oracle E-Business Suite systems.”

Read this for some more details on the Application Change Management Pack.

Now I don’t know just how good this is, but imagine being able to put large parts of your Functional and Technical Setup and your code into a Release Patch, just like Oracle Corporation. Imagine being able to clone Production, apply your patch and get to the point of repeating this until you had zero failures on your very complex R12 Upgrade Customization Release Patch.

This is certainly a very interesting development and the Change Management Pack is certainly worth looking at.

So time for a final movie review and a wrap-up. A word of warning. Not all alien technology is entirely friendly. In War of the Worlds, Alien technology proved to be extremely deadly. So my final movie pick is War of the Worlds. On reflection I’ll give this a 9 out of 10. It’s a superb remake. And a word of advice, when using the FNDLOAD Alien Technology, just be careful with your parameters or you may download and then upload into Production a whole lot more than you want………

The point of these tools isn’t primarily to save you time, although they will definitely save huge amounts of time on any project. Imagine having to setup say 100 custom programs in 12 instances during a project implementation. Or 200 messages. Or thousands of menus and entries. Or module setup. A large chunk of project time is wasted by very expensive consultants doing very menial setup, rather than delivering the end solution. And it doesn’t need to be that way with the tools that Oracle provides.

The primary reason everyone should be using these tools (either in implementation or Post-Production Support or R12 Upgrades) are because these tools introduce a repeatable, testable and verifiable process. These tools vastly reduce the mistakes you make.

For all those doing R12 Upgrade, these tools allow you to be successful on cutover weekend even when the timescales look utterly impossible. (Have a look at my R12 columns, with thousands of custom objects, 600 functional setups, all to be done in a matter of a day, with everyone under serious stress. Impossible? Not if you plan ahead and fully utilize all the tools that Oracle provides to make your life easier………

Till next time when I hope to publish the Secrets of the Holy Grail and the connection to R12. Just don’t tell the Feds where Oracle technology really came from. The secret is out there.

So I guess as Arnie would say from Terminator,  Hasta La Vista Baby………

Further secret blogs and prophecies can be found at:

https://oracleprophet.wordpress.com



Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt3)

October 31, 2010

Welcome back to the R12 Upgrade House of Terror.

All I can promise you is pain, heart-ache (that Oracle Support girl for the AP Trial Balance broke mine…..ahhhhhhhhh) and impending doom. Abandon all hope on your R12 for those who read on……..

So you’ve been mad enough to start an R12 project (they’ll lock you up in the asylum they will !!!) and you’ve made it through Implementation (designing any new stuff, build, test, UAT, etc) and you are sitting there smugly thinking you’re smarter than your average 5th grader and that this column was just scare-mongering……..Well lets go to the Transition Phase and see how confident you are………

 Cutover is complex

Deployment involves high number of objects

Deployment involves significant application setup

Deployment carries high risk areas

Users are not familiar with the new software

Large number of users require training

Users are distributed across countries

Help Desk Staff are not trained to take calls

Transition is poorly documented

Transition is not rehearsed

Change in Business Procedures for R12

Before you start the cutover, just wondering if you actually trained all your users? Yes R12 is mostly the same, but there are significant variations between R11 and R12 in Payments, TCA, SLA, etc. Now if those are spread across countries, it’s going to be even worse. A new system with new features suddenly appearing will lead to a lot of angry users, all wanting to talk to you on Day 1 of R12 as you are inundated with other more serious issues……..Now I went through four audits on my R12 and got a clean bill of health on all, except for training (even though we did some). The auditor did a survey and the users said they would have liked MORE TRAINING. But the products almost the same I said bar a few key areas as we were doing a Technical Upgrade !!!!!!!! I got burnt at the Halloween stake on this – OK ever so slightly singed. Don’t make this mistake. At least offer some training on delta differences and some educational classes with no commitment to implementing (get your lawyer to write the disclaimer) new functionality.

Has anyone told the Helpdesk of your R12 transition? Has anyone told the business they will lose their systems for 4 days (weekend, plus two working days). Has anyone told the external users or put out appropriate announcements? Are there key business activities that expect to run during your upgrade? Have you scheduled your upgrade right in the middle of these? People are going to be seriously upset if payroll is delayed for your upgrade. Or month end closing is delayed and you have serious problems with your new R12 that goes live right before the preparation of those financial results. Wondering if you told your corporate website team that pulls out summary information from ERP to display to your company website that ERP won’t be available over the transition? Or how about the legacy team that are still sending you files to interface into ERP? I know you are feeling pretty insecure now. I guess you are wondering “Have I missed anyone in the announcements……”.

Cutover is a nightmare. Get this wrong and you see your entire project collapse in a pile of dust, on day 1 of Production, with the Board asking why your business has ceased to function and lost 5M US. Pulling the transition on R12 at the last minute is really bad, but even worse is going live and then finding out you have serious issues……..Either has a good chance of ending your career.

Has everything been tested? Has every bit of code your about to deploy to Production actually really been through UAT (or did the developer slip in that one last easy fix with no testing?). Does your Application Setup really reflect the setup you tested? Did the Consultant complete the BR.100 before he left for a contract paying 30% more? Can anyone actually follow that BR.100……?

So you have a thousand customization files (actually isn’t a lot when you count Java files, setup, etc). Have you ever taken that as a single software release and had your DBA’s (not your developers that know to run script 2 if script 1 falls over and hack script 3 to make it work, without anyone finding out) and applied to an exact clone of your Production system and had a 100% success rate? Have you repeated this multiple times as you roll towards the go-live date, just to make sure that any changes between the time you last ran it and any updates to Production didn’t cause any adverse impact? Are you sure that the DBA’s can run all your customization deployments in a very short-time frame. (Your typical go-live window is a maximum of 4 days with two of those the weekend, after that your business will scream if you go one hour over). Is your deployment manual and if so, will the DBA do the same steps each time? If your deployment is automated will it actually work?

We had 600 application changes. Have you ever checked how long it would take to deploy all of those changes? Have you ever done an entire setup and checked all the setup was valid and tested properly? Have you thought of the issues of letting knowledgeable people do the setup in Production in terms of access, SOX, etc? If new people need to do the setup, as the project team cannot have access to Production, are they familiar enough not to make mistakes? Have they ever practiced the setup on an R12? Interesting how long it takes to carefully setup and check 600 application setups. Just wondering if you ever thought how long it actually takes to setup 600 application setups for a single person. Do you know it’s impossible to do that setup with one person? Have you ever bothered to do a careful review of all setup, access needed and then balance setup streams across different people to make it achievable in the very short transition time the business has granted you whilst still ensuring segregation of duties, SOX compliance, etc?

Have you ever considered setting up some functions, menus, etc in R11 BEFORE you go-live. Or some new program definitions? Or perhaps having these as Loader files, so you can load them instead of manually setting up? Of course if you load them, have you checked that the developers have generated these LDT files correctly?

How many minutes will the R12 Upgrade run on Production hardware? (I ask for minutes because you should know very precisely the estimated time of this – if you don’t you are in serious trouble. For info an R12 will run for a very, very long time indeed, even on a small ERP database.). Have your DBA group ever run an R12 Upgrade with no failures? How long will it take to upgrade the Technology components or database? Have you ever seen a detailed step by step upgrade document from the DBA Team? Are there any hot wiring the DBA team apply during the upgrade due to known failures? Are you aware of these known failures and the implications? Do you know that an R12 Upgrade will require some very serious shift working? Do you know that your DBA’s will be seriously burned out as they have the very first task in the actual upgrade of running the hundreds of thousands of scripts that the standard R12 upgrade runs. And they need to actually be awake and watch this constantly, because if it falls over, you don’t have the time to lose 8 hours on an upgrade weekend…….

Wondering if you’ve actually even got a transition plan? Have you got an hour by hour schedule, with milestones and an effective method to monitor setup, customizations, actual upgrade, etc? On an R12 upgrade transition, you have no slippage time for anything. If you slip, the team will get stressed, you’ll get stressed and mistakes will be made.

Bottom line is have you REALLY rehearsed an R12 cutover BEFORE you actually do it? Do you really feel confident that first time you do it for real will be on your Production system……….if so you must be a clown……by the way do you think the McDonald’s clown is intrinsically evil????? (My Attorney has advised me at this point to clearly state that is a question and in no way implies anything on McDonalds whose food I have been legally advised to state I love and the picture below is in no way related to said company).

If you take a look at the Oracle Upgrade manuals, it’s interesting how many pre-req steps potentially have to be done. Ironically if you miss some of these you can have major support issues going live. I wouldn’t leave any Payments in the middle of process. I’d be carefully checking the effect of an upgrade half way through each of your workflows. Is there any effect on transactions that are left half completed? Do you even have these in your cutover procedures and does everyone in the Business know from your carefully documented plan exactly what their responsibilities are prior to bringing down their R11 ERP. If you (or the Business) makes mistakes here, you are in real trouble. Do you have scripts to check everything has been accounted, no payments are sitting halfway or are you just relying on your business to do the right thing with no checking whatsoever…….Just to add a little time pressure, your DBA’s will be screaming at YOU if you go one minute over the agreed time to shut down R11 and hand it to the DBA team……..of course you’ll have to herd dozens of business units across 27 countries to complete all activities before that time…….but then they want to do their business not your upgrade……which leaves the DBA’s screaming at you…….(The DBA’s will be working all through the night, so every hour you delay, means an hour extra in the middle of the night so a little empathy for them here folks……..Show them you care for their welfare….at least till your upgrade is over. Alternatively take the selfish approach that the DBA that is working all night might be the same one running your customization scripts after two hours of sleep because of your poor planning that could wreck your upgrade when he makes a serious mistake because he’s so tired……….)

Have you made any plans to backup the original system? What are your fallback plans if the upgrade fails? When will you be prepared to pull the plug and under what criteria, over the transition cutover to R12? Who is going to take responsibility to give the green light to go live? Will the senior management even be there at the weekend to make that decision?

Just wondering if you told your team and others (Unix, Email, Legacy, etc) that they would need to work some very stressful and long hours over the cutover weekend. Wondering if any of those key staff have their holidays planned…….?

Have you made plans to clone and test the actual R12 on the cutover weekend? If not how do you know all your setup, customizations, etc actually were promoted cleanly and your R12 actually works? You will need a large team to test in such a short period of time. Do you even have a cutdown test plan to make sure you hit the key functionality? Have you got all this planned? Probably not…….And if you find stuff missed, do you have a proper controlled mechanism to track it, test it and release it to Production in a horrifically short and highly pressured weekend in a safe manner?

What’s amazing is that everyone plans a transition, but no-one actually thinks about turning on the R12……….when you enable the workflow and the concurrent managers are you just going to open the tap, or are you carefully going to release production jobs and monitor. I’ll tell you that your single most stressful time is when you commit to switching on these elements over the weekend. I bet you that your heart will be pounding and you will be sweating at this point. The entire company ERP and Business hangs on your Go or No-Go signal. Once you’ve done that, there is no turning back, there is no backing out and you are on R12. At that point if you’ve got it wrong, you’ll be looking for another job. (Doesn’t that picture below remind you of your boss when it all goes wrong……?????). I wonder if that Oracle Support girl I dealt with for the AP Trial Balance wore red lipstick…….ahhhhhhh……..

So your live, but it’s not over……..

Serious Outages can occur

Lack of Staff to address serious support issues post go-live

Patches are required after go-live

Are you ready for serious outages? Did your transition plan have proper contingency plans for these events? If not exactly what will you tell the Board and the Business?

Do you have the appropriate staffing plan to handle what could be a very rough few days (or weeks) or months? An R12 will almost always lead to a spike in calls. Are you ready? Did you bother to give your people proper notification that they may be required to work late nights and weekends for the first few weeks after go-live? Or is it just a line item on your plan that you didn’t bother to communicate?

Did you re-synchronize all of your ERP and Legacy systems in terms of interfaces? Some of these integrations may require manually running interfaces to catch-up with transactions for the days you were down.

With major business events such as payroll and closing, do you have any plans to clone and rehearse these events, prior to these happening? Now if you can clone and simulate an R12 closing after you are live, but before the closing happens, it’s better to find out serious problems and have a few weeks to fix them right? (Of course this has you wondering if you should close in R11 and then immediately move to R12, but of course, then your Financial reporting would be straight after go-live, so you can’t win right?)

Have you got any plans for patches after go-live? What happens if you really need to apply patches? Are your project team still there? Did you set business expectations just in case? Always better to make business aware of the real risks inherent in an R12 Upgrade. That way, they’ll be incredibly happy if you manage to get it right.

I hope you remembered to get before and after reports from key areas, such as your AP Trial Balance (that Oracle Support girl was just so nice, ahhhhhhhhh), GL Trial Balance, etc. And keep a copy of your Production database on R11, whether on tape or disk. When the auditors come and take an interest in it, as they will it would be unfortunate if you’ve not done this and it causes issues with your auditor’s signing off your company accounts.

Also before you go live just have a quick review of JVM sizing and parameters. If you fail to tune and size this properly you could see your Applications crashing, intermittent crashes, etc. Getting your JVM sizing wrong leads to absolute nightmares……..Ask your DBA’s “Have you size the JVM’s and if they say what’s that, start to worry……”. If you have too many users and not enough JVM’s and OACORES then BOOM !!!!!!!!

Finally is there monitoring of key components in place? Are you touching base with all the areas of Business (and Legacy and Web and email and and DBA and Unix and other teams). Are you even aware of major production issues or is your Production setup so poor in terms of support that you’ll be the last to know……..

And lets break for another movie recommendation. The single scariest movie I have EVER seen. This was the 1970’s television mini series called Salem’s Lot. It is absolutely terrifying. Get it on DVD….it’s one creepy movie from the Master of Horror Mr. Stephen King. I saw this when I was ten, and I didn’t go out for weeks after dark……It’s a classic ten out of ten movie. But REALLY TERRIFYING.

This isn’t an exhaustive list of what can go wrong in an R12. Now note this article has been written in a strong manner to scare the crap out of you because if I do that then you’ll start to take every last point out of this article and make sure you’ve covered it. Make no mistake – R12 Upgrades are a serious risk to your career. (However don’t let some consulting company come in and scare the crap out of you. They are in it to make as much money as possible from you for an R12 by scaring you as much as possible. I write these articles for FREE and hope people can avoid the mistakes that I and others made in our R12 Upgrades). If you read this article and take everything on board, you’ll be strongly versed in R12 risks and be able to evaluate what these consulting companies are telling you and separate the fact from the fiction…….

And a final note. Premier Support for 11.5.2 ends in November 2010…..that gives you about a month left……Interestingly you also need to have minimum baselines in place for R11 if you want proper support from November 2010……….And will you get hit by fees if your still on R11 in November 2011??????

Have you read Metalink Note 1178133.1…….

“Starting on Dec 01, 2010, Oracle E-Business Suite customers on Release 11i10 will only receive extended support for new bugfixes as described in My Oracle Support, Minimum Baseline Patch Requirements for Extended Support on Oracle E-Business Suite 11.5.10.”

That’s a lovely little bit of legal small print from Oracle with major implications for you………Getting to those minimum levels will be enough to “persuade” most customers to jump to R12 instead.

It’s funny also that so many people don’t think they can start doing preparatory work in R11 that will greatly ease R12. Have a look at the references below. Work spread over R11 and R12 will greatly ease your stress in the R12 Upgrade itself, as you’ll have less to do. (Did you know Solution Beacon even has Vision instances you can play with for free, even before you install an R12 in your company’s test environment…..)

Let me tell you briefly our R12 story. We had some very serious problems with quality (12.0.4 RUP5) – an early R12 release in many respects. However despite that we got through a Finance, HRMS and Procurement Suite R12 Upgrade across 20+ countries with strong planning. Our transition finished 30 minutes ahead of the planned 96 hour time window (I wasn’t kidding when I said you should know the transition in minutes…..). Yes, I was terrified when I made the call to go-live. But due to all the planning and a great Project team, we went live with no major issues on 3rd August 2009 and I went home at 5PM two days after go-live (And for the cynics, no I didn’t spend the first two days constantly in the office 48 hours straight…..I got home at a reasonable time the first two days as well). We cleared Payroll, breezed through month end and went live on two Payroll rollouts to two new countries one month after go-live.

Of course now I’ve told you all the risks, well you’d be a fool to actually fall into those holes right? Take this article, make a list, make sure you cover each and every one. Then you’ll make your R12 a huge success.

An R12 is certainly a very achievable project, with significantly reduced stress and risk, if planned correctly.

Hope I haven’t scared you too much……..

Enjoy your Halloween.

References

As always, my appreciation goes to the kind authors and contributors of the following articles and resources. Some of these are especially note-worthy, because those people are putting out their failings (as well as successes) on R12 in a very public and open way, for the benefit of those about to travel the R12 Upgrade road.

General Blogs and Websites

OAUG Website

Chris Warticki’s Blog – EBS R12 Support Resources – Consolidated

Oracle Applications Upgrade Guide: Release 11i to Release 12 (B31566-01)

Oracle Financials Applications Blog – R12 Lessons Learned

Analyzing, Planning and Executing an R12 EBS Upgrade

Steven Chan Blog

Risk Management: Tricks of the Trade for Project Managers (Rita Mulcahy)

My example of an R12 Risk Spreadsheet (Excel 2007) – Available on Request

 

Metalink

Oracle E-Business Suite Release 12.1.3 Release Update Pack (Patch 9239090)

 Oracle E-Business Suite Release 12.1.3 Readme (Note 1080973.1)    

Upgrade Manual Script (TUMS) – Patch 5120936

Metalink Note: 580299.1 – Best Practices for Adopting Oracle E-Business Suite, Release 12

Metalink Note: 394692.1 Oracle E-Business Suite Upgrade Resources

Metalink Note: 562887.1 R12: Helpful Tips for a Successful R12 Oracle Payables Implementation

Metalink Note: 437422.1 R12 Troubleshooting Period Close in Payables

Metalink Note: 73128.1 R12 Troubleshooting Accounting Issues

Metalink Note: Oracle Support Upgrade Advisors (250.1)

                             Tech Stack                     Note 253.1

                             Financials                     Note 256.1

                             HRMS HCM                    Note 257.1

                             Manufacturing              Note 258.1

Metalink Note: Extended Support Patch Level Verification in Oracle E-Business Suite Release 11.5.10 (Note 1178133.1)

Metalink Note: R12 Upgrade Considerations by Product (Note: 889733.1)

Oracle E-Business Suite R12.1 Financials Pre-Upgrade, Setup and Operational Tips (Note: 1104163.1)

Oracle E-Business Suite Release 12.1 Information Center (806593.1)

Database Preparation Guidelines for an Oracle E-Business Suite Release 12.1.1 Upgrade (Note: 761570.1)

Oracle Applications Installation and Upgrade Notes Release 12 (12.1.1)

R12: Period-End Procedures for Oracle Financials E-Business Suite (Note: 961285.1)

Oracle Open World 2010 – On Demand

For those with access to Oracle Open World 2010 On Demand there are good references at http://www.oracle.com/us/openworld/index.htm

Mission Impossible: Oracle E-Business Suite 12 Upgrade on 3 Continents in 6 Months

Success with Oracle E-Business Suite Release 12.1.2 Upgrade Drivers

Algar Telecom Upgrades to Oracle E-Business Suite R12   

Get Ready for Oracle E-Business Suite Release 12.1: Tasks to Complete Now

Oracle E-Business Suite 12 Upgrade : Best Practices to Obtain Business Value

Planning your Oracle E-Business Suite Upgrade from Release 11i to Release 12.1 (Superb presentation – highly recommended)

Oracle E-Business Suite 12 Upgrade: An Easier Ride on Nine Miles of Bad Road

Oracle E-Business Suite R12 Upgrades: Have you Thought about the Details?

Real-Time System Assessment of Oracle E-Business Suite 12 Upgrade: Case Study

And finally talk to other companies, use forums, other areas (such as http://www.linkedin.com) and make contacts with people that have done R12. This again will help you considerably on your R12.

Footnote:

Some of the great graphics were the work of:

WWW.MYSPACEGRAPHICSANDANIMATIONS.COM

R12 Security and the Auditors from Mars

October 9, 2010

Let’s face it auditors are not from this planet. Rather like the famous movie War of the Worlds, they bring fear and terror wherever they go. Are they from Mars? Probably, although the one thing certain about auditors is the uncertainty they bring with them, together with the general fear and terror they instill.

I thought the picture below was a rather appropriate one of a hand reaching out with the entire planet in its grip…..when the auditors are around you sometimes get the feeling that your entire company is under the grip of these terrifying Auditors from Mars…..

 

Now this column seems almost like a recommendation for movies these days, so if you’ve not seen War of the Worlds, get it on DVD. It’s a great movie.

Now as someone that had not one, not two, not three but FOUR different companies looking at my R12 Upgrade Project (AND FINDING NOTHING!!!!!) I have to say I’ve learned to deal with them pretty well over time. Our first R12 audit was from our internal audit guys, who wanted to learn R12. It’s important for internal staff to learn, so we were happy to help. We then had an audit from E&Y as our R12 Upgrade was part of a large Program…….we then had a pre-Attestation audit from PWC to make sure the project was OK from an attestation viewpoint. Finally we had another guy from a large Consulting Organization looking at it and never quite figured out why. (Guess they also wanted to learn how to do an R12 Upgrade). Amusing that these audits cost 30% of the actual cost of the upgrade and ended up commending the R12 Upgrade 🙂 One audit surveyed random users with 20 out of 21 commending the team, with the other neutral. It was a lot of work providing the auditors everything they needed, but I have to ask – 4 audits???????

Now here’s a question for you. If you saw an auditor drowning in a river, would you run to get the life belt 30 feet away from the river, or run to your car 50 feet away to get your video camera? Alternatively if you’re a Project Management Professional (PMP) you’d have planned ahead and kept a bag with bricks in your car beside your video camera. As you throw the bag of bricks the auditor is thinking “What a nice guy throwing a bag to keep me afloat”, whilst you are thinking “What a sucker, that’ll help him sink……”

 

I have to point out you should not take this blog too seriously, as the above may get you into some serious trouble, although with a good lawyer you’ll probably get off with a small fine as it could be seen as justifiable or reasonable……. You may be able to even make some money back to pay for your legal fees from the video on America’s favorite home movies show.

There is no question that Auditors are from Mars, causing fear and terror wherever they go………well recently they struck on an area that probably too many people just don’t take seriously enough or don’t even realize is their responsibility. This area is Data Security within your ERP regarding access of your Support Team. Now what is worse is that when auditors do raise this they generally have a very good point and your best interests at heart.

So we’re almost going back to our Koan stuff (see the Zen column). How do you do support Oracle ERP Production when your support team can’t access Production due to information and data security concerns?

That’s a difficult question and I’ve yet to see any really effective, comprehensive solution either in a blog, white paper or from Oracle themselves (at a reasonable price – I had a discussion with Oracle last week and almost fell off my chair – the Oracle Sales guy might as well have had a mask and a gun…….).

So you’ve got your auditors on the one hand, a whole raft of legislation on data security (your legally obliged to follow – it’s not a matter of choice) and a very limited IT budget. So how do you square the circle to keep every party happy?

So look at your typical ERP these days. It’s a monolithic, super system:

  • Financial performance – if you’re a listed company that causes major data security issues to prevent internal staff dealing in shares, because they know you had a record quarter looking at your General Ledger before it was announced or worse short your shares if they know there is bad news coming. That’s real company loyalty for you.
  • It probably has all your HR information in there, including performance appraisals (some none too pleasant quotes on how a staff was involved in some questionable activity), potentially medical information, salary of course and a whole lot more.
  • You might have Procurement information, so a huge amount of confidential contracts also.
  • And most ERP sites run Payables and Payments. Do you know the potential fines you can get if you lose bank information? Or worse if you lost your Supplier bank information and the damage that could do to your company’s reputation, not to mention legal liability.
  • Ever worked out that your Accounts Receivables is a very good indication of a customer’s credit rating? After all, you kind of know who’s got cash flow problems……And the Bank Account information is everywhere. Are you really looking after that securely? Your customers won’t be too impressed if you lost that and would probably never do business with you again.
  • Or are you capturing credit card information as part of say your Order Management/Accounts Receivable process…….Have a look here if you are.

Data Security is an absolute minefield for any IT department responsible for Oracle ERP.

There is just so much legislation on this area that most are blissfully unaware of:

  • Sarbanes Oxley
  • PCI
  • HIPAA
  • European Directives
  • Individual country legislation
  • And a whole lot more…..

This is serious business and can really hit you hard if things go wrong – blissful ignorance is no legal defense. And yet I bet you haven’t reviewed anything from a legal viewpoint on this……

There’s a couple of interesting links at the bottom. Choicepoint had 163,000 customer records compromised. That cost $15 million US Dollars, not to mention reputational damage and cost of implementing proper data security. The data involved? Names, Social Security Numbers, Birth Dates, Employment Information – sounds pretty much like Oracle HRMS right – they got fined for not protecting exactly the type of information you’re holding…..Or how about the California hospital that got hit with $250,000 penalties just for reporting a breach late……

Not in the US, so don’t really care? Have a look at this link. The UK has put in place legislation for fines up to GBP500,000 (equivalent to around US800,000) for data security breaches. Or HSBC, one of the largest banks in the world, that got hit with a fine of almost 5,000,000 US Dollars.

So not in the US, not in the UK, still don’t care. How about the following:

A German company fined 137,500 Euro for asking employees why they were sick and storing the information. If you run absence management, are you potentially collecting scanned medical records and are they properly secured?

Still not convinced? Forrester estimates a cost of $155 Per record for data security breaches. Lose 100,000 records and you’ll be looking at $15.5 Million US Dollars legal liability and probably looking for another job…….I guess that’s finally caught your attention? Legislation has been enacted or is being enacted worldwide. Sooner or later, it’s going to hit your country too.

Now I have to state from the outset this article does not provide all the answers. This is a highly specialized area which goes way beyond my own personal capabilities in this area. So if you’re looking for all the answers, hire a specialist. Actually read hire multiple specialists over a number of years. Security is a massive task to do well and therefore effectively.

The aim of this article is simply to show what we are doing and have done to implement a more robust security around our R12 Oracle ERP that allows us to support the Business, whilst still trying to do our best to be secure.

This purely looks at using roles to restrict database access for support people. This article does nothing more and that has to be stressed at the outset.

Hopefully it can get you at least part of the way there, giving some additional protection without costing a fortune.

 

Step 1 – Find out what access was out there historically

Our starting point was to figure out from an Applications side what access our IT guys had to Production (and note when I am talking IT I am talking ERP Support only in this column. Other areas such as DBA, System Admin, etc are beyond the scope of this article). Luckily we were already very heavily locked down. We don’t have any access to modify any part of Oracle ERP and I’d say that’s a very good thing. It makes me sleep easier at night knowing that there is no way my staff can log in to Production, “cause there’s a really easy way to fix something………”. Every change we do to Production is through a formal turnover to separate groups, fully approved, appropriately segregated and audited. That keeps our Production system safe. (All code is also in source control using Subversion just in case we ever need to rollback a change).

The next step was to see what read-only access was given to Support Teams. Now over the years, many IT systems just grow and grow, access is given for support purposes and no-one really thinks too much about it.  I was relieved to see that the Support Team had no access to Oracle ERP through the front end applications.

I thought I’d throw in a useful note at this point. We have a custom program that reports quarterly on all menus/functions/applications by person. It replaces the standard Oracle System Administrator reports that generate listings the size of War and Peace. This report is reviewed both in IT and Business to ensure that each user has the appropriate access for their job. We also tied this to staff movements within our HRMS System, so that when someone moves, this is highlighted as a potential access control violation and can be assessed before it even happens (using date-tracking in HRMS). The other useful point here is that this quarterly check also ensures complete segregation of duties. Finally by using a custom report and showing dates menus/functions were changed, it vastly reduces the work to check each menu against each data item. Actually a single maximum date at the top of the report means the users only check this and can thereby can confirm no changes from the last review. Not perfect, but keeps our auditors happy. This is a cheap and cheerful approach without buying Oracle’s GRC Suite although I think the GRC Suite looks very interesting indeed.

With our front end applications secure from our Support Team, we then started to look at the database access. Now I know some people say IT Support should have no access, especially the evil DBA Vogons from Vogsphere, who according to the book The Hitchhikers Guide to the Galaxy have “as much sex appeal as a road accident”. On the other hand others say “With no access, I can’t see anything, therefore no support’. We’re back to Koan (See R12 Patching and the Art of Zen) here – How can you fix something you cannot see? (To our DBA friends I heard the new 11G Database Release 2 is self-managing, so exactly what do you guys do???? – that should provoke a few comments 🙂  )

Our support team had read-only access using database roles to Production, but when we looked at the roles created we concluded that these were too broad and needed some serious work. Roles are simply access to database tables/views or other objects given to a specific support person by the evil DBA Vogon group.

Note: For those interested in Vogons, it’s from a movie called Hitchhikers Guide to the Galaxy. Here’s a movie recommendation. Don’t see this as it sucks big-time (although the original TV series and radio show and book were amazing).

Now our next step was pretty simple. Monthly we review database access to Production and with the count of accesses by support staff, we found that many actually just didn’t need access to Production, because the support volumes regarding database queries by support staff were low (or zero). We cut these immediately.

Again doing a review monthly (only takes ten minutes) ensures that as people leave, move job, etc your access controls remain updated and correct. Again this keeps the auditors happy.

At the end of this step, we defined a Security Access Matrix, showing each Database Role against each Support Staff, shown below. This was our first baseline.

  

 

Step 2 – Defining Your Database Security Architecture

Ignore the fancy title. What this means is simple. You need to look at:

  1. Your Support Team Structure
  2. Your Applications
  3. Your Required Access

So let’s take an example. We run a fairly large ERP footprint:

  • HRMS Suite (HRMS, Payroll, Learning, Time & Labor, Self Service, Advanced Benefits, iRecruitment)
  • Financials (General Ledger, Payables, Payments, Fixed Assets, SLA, EBiz Tax, Accounts Receivables)
  • Procurement (Purchasing,  Services, Inventory, iSupplier)
  • Projects
  • A large number of custom modules
  • We’re going to add CRM (hopefully Fusion) to that in 2011.

Our Support Team is organized along classic line of business model (Procurement, Finance, HRMS). It is further segregated so that certain support people provide support for say HRMS/Payroll, others do Self Service, etc.

Now the problem we had was two-fold:

  1. Too many Support people had too much access
  2. The data protection needed to be much stronger

Our approach was simple, but effective. Each role would be along the lines of module or modules required for logical support. So we would set up roles for Procurement, AP/Payments, General Ledger, HRMS, etc.

Now this for us still didn’t get us to where we wanted to be. What was going to be in these roles database object wise?

Our next task was to then define exactly what that role would have access to table and view wise. Your average ERP Module can have several hundred tables (or even more….). Now you can give a blanket access if you want, but I think that’s overkill. It’s hard to audit and hard to police. Your average support person needs access to only a limited number of tables. That is what you SHOULD define.

In addition, obviously there is some cross-cutting of access between modules and tables, so for instance you may have to grant GL_CODE_COMBINATIONS across various roles.

In our final step, to the horror of many, we decided to NOT give access to standard sensitive tables, but instead to build views WITHOUT KEY ELEMENTS. Now a lot of support people are going to scream at this, but I need to ask, if you have a support call, do you really need access to the guy’s date of birth or performance or disciplinary records or even Full Name? I doubt it for 99.99% of cases.

Here are a couple of examples of securing tables into masked views:

Instead of giving access on PER_ALL_PEOPLE_F we removed all information except the Employee Number. No access to Social Security, Performance, Home Phone Numbers or anything else.

Instead of giving access on the Bank Tables, we removed all information and masked the bank account.

Instead of giving access of PO_VENDORS, we removed all information and simply gave the VENDOR_NUMBER.

What you need to be careful of is the other tables that you really wouldn’t expect to hold sensitive information. A prime example of this is the workflow tables.

To secure workflow tables we ensured that each role only had access to relevant workflow ITEM_TYPES. This was a very simple, yet highly effective way to secure this by role.

The great thing about this is that we got our support people to define these roles in a simple spreadsheet. This gets you your buy-in from the very people who would be screaming about cutting access. After all they are now the ones tasked with defining it (obviously subject to reviews within IT and approvals by Business).

We then decided that we’ll make this formalized. This has a lot of benefits:

  1. It provides an agreed document of what a role has access to data-wise
  2. It provides an agreed document between Business and IT on that role/data
  3. It provides a document that can be agreed and signed by all parties
  4. It makes you think about your data and protection of said data in a disciplined manner
  5. It provides the basis for all future access to sensitive data by IT Support Personnel
  6. It provides a clear statement of who has access to what in a transparent, controlled manner
  7. It provides protection from your auditors

We decided to record other details in this spreadsheet. (Each role has a separate spreadsheet).

The spreadsheets had the following columns:

  1. Object Name
  2. Database Owner
  3. Object Type
  4. Business Owner
  5. Description
  6. Object Classification (Custom/Standard or Secure. A Secure Object was a restricted, masked view of a key table)
  7. Data Security Risk (Critical, High, Medium, Low)
  8. IT Review Date

A separate Tab (Security Details) on the Excel Spreadsheet held additional details

  1. Object Name
  2. Base Table
  3. Masked Columns (.e.g. vendor name, bank account, social security number, etc)
  4. Date Approved

The Spreadsheet was split into two sections – data objects and reference objects. (Reference objects are the pre-seeded or setup tables within R12). A further tab showing review actions was included to show that this was a serious, considered exercise to the auditors.

 

 Suddenly, without even realizing it, you’ve just created a very precise, auditable and comprehensive technical security framework for your R12 Oracle ERP that will vastly improve your security of data, whilst still allowing effective support. This also provides a very powerful data classification framework for both you and your users. Even more interesting, by putting a risk score on each object, you can actually see the overall risk access rating of each role, allowing you to make informed decisions on how widely this role is given to support personnel (or not).

If you have knowledgeable, motivated people, this doesn’t take a huge amount of time. I am extremely lucky to have such a great team of people.

Once the roles were defined, signed off and agreed, the DBA Team created the roles/access very easily.

Note that once you have these documents in place and have implemented the framework within your database, these documents (and hence the roles within the database) should be strictly controlled otherwise all your work will have been for nothing……

So after not a huge amount of work we had roles for most applications that were nicely segregated, secured and with all key information masked appropriately. This can be seen in the Security Matrix below:

 

(Note how the General Database Access role was entirely removed once we worked out roles/access. The above is only a sample, not the real resources/roles).

So in a couple of weeks we went to very fine grained security on our Production System, without spending vast sums of either cash or time.

Our overall security had vastly improved for our Support access to Production. Indeed the data had really been masked very effectively yet we still could do effective support, without disclosing any critical data to our IT Support Staff.

 

Step 3 – Defining Your Access Security Processes

Without policies and procedures any security framework will fall to bits pretty quickly. Now this article isn’t really about processes, but here are the key aspects to consider. Security is not a one-time process. It’s a critical ongoing process that needs to have strong controls to keep you secure going forward:

  • When people join our company, we ensure that non-disclosure clauses are signed the very minute they step in the door.
  • Security is not just about securing your database or applications or data once.
  • The Database Security Architecture showing each module/access to tables/views should be agreed and signed off by the Business. It is important that Business understand the access IT has to data in a transparent manner. Too many IT organizations think they own the data. They don’t, they are merely custodians.
  • When new access is required for a Support person, or additional access is required, there must be a strong process to seek authorization from the Business for that access. This process should be formalized, ideally automated and fully auditable. We are currently implementing Oracle’s Identity Management product and intend to integrate the process into this.
  • When a person changes role or position, it is important that any access is immediately discontinued. Linking your process to the HRMS System gets you part of the way there and avoids any embarrassing moments when the auditor asks why a person that was fired still has access to your HRMS database details……..
  • When the Database Security Architecture documents need adjusted to add a new table/view, this must go through a formal change control process, with appropriate sign off by the Business users.
  • The DBA Team should not update any role to add/modify any table/view unless the change has been formally signed off in the Database Security Architecture documents. All requests for changes in the database should go through a formal turnover process which is fully auditable and approved by IT Management and Business Users.
  • A monthly review of Support Personnel/Access/Roles should be done by the person managing the Team. This doesn’t take long and ensures that only those needed access have access. Documenting this process protects you when the auditors role in. (Note the Oracle Database can be configured to automatically email you the roles/people/number of accesses)
  • There is one other interesting area. If you formalize the Data Architecture documents and make that part of every project before go-live, you’ll also ensure that support for go-live is properly thought through and adding new systems does not compromise security going forward.

 Step 4 – Encrypt Non-Production Databases

I’ll cover this in another article (working title is Securing your R12 using Captured Alien Technology from Roswell), but the key point here is that there is no point in protecting your production database, if you then clone and have all the same access in copies of production……..Security is about all your databases, not just Production.

 Step 5 – Securing all Layers

Again, this is beyond the scope of this article, but keep in mind that the best security is a layered approach. Your Oracle ERP security framework must involve all of the following:

  1. Network
  2. Servers (Linux, Unix, etc)
  3. Oracle ERP (across the modules)
  4. Individual Modules (each with their own security models)
  5. Database
  6. Security Patch Policy across all areas

Security is only as strong as the weakest layer…….A couple of the best articles that take a holistically approach are found in the Related Articles Section at the end of this article.

 Step 6 – Future Directions

Our goal in this article was to make others aware of simple steps to improve security for Support Staff supporting Production systems by using role based access at the database level.

However, as we looked more and more on the security, and as we became more aware of the challenges, but also consequences of getting it wrong, we see other opportunities that remain very interesting future initiatives that we will seriously look at.

We’re interested to lock down database access to restricted IP addresses. This will stop password sharing, but also ensure that even if an account is compromised, unless you sit in the person’s chair, you’re still going to have difficulty accessing. Not bullet proof for sure, but it adds another layer in our security.

We’re interested in auditing the database statements of support personnel and perhaps moving database roles/custom built to a more secure and robust solution. It looks like Oracle’s Audit Vault may get us there, but at a heavy price. This can implement roles in a more secure manner, provide full auditing, reporting, restrict data by IP address and a whole lot more.

We do already have data scrambling and encryption on our non-Production databases, along with formalized controls and processes. However as one article points out, the days of custom scripts are coming to an end (or is it just an Oracle sales pitch to get you to buy the products?) and we are seriously looking at a couple of options:

  1. The Oracle Enterprise Manager with Data Masking Pack. This allows you to setup and execute data masking. With Grid Control you can then have central data masking across all Non-Production databases. So yes they are trying to sell you even more, but then again, it’s better and cheaper than a 15M US Dollar fine for non-compliance…..
  2. Oracle Application Management Pack for ERP – This is a pretty broad release that provides functionality for Configuration Management, Application Performance Management and Service Level Management (including related to this article obfuscating sensitive data). This will allow you to clone and scramble data in a single process.

 

 One area we are interested in looking at in the respect of functional changes, setup, etc is the GRC (Governance, Risk and Compliance) module. This provides a lot of the audit capturing and reporting capability as a by-product of the general setup process. More details can be found at the end of this column. We haven’t fully evaluated this product but it does look interesting. Again Oracle has added analytics that could potentially make this a complete, all round solution (subject to our evaluation).

Oracle Identity Management. We are so interested in this product that we actually went out and bought it. This will once fully operational, control access to all our key systems, allowing us to comply with key controls from both auditors and legislation. We intend to use this directly linked to our HRMS system to basically provide all provisioning of user accounts, access, etc. This is linked not only to the triggering HR Process, but also directly to the Business owners who grant access to the modules, all using Oracle workflow technology. We’re also having a close look at the Identity Management analytics, although have not bought this yet. The two products combined can not only provide very strong security, but save a huge amount on the cost of our compliance, which becomes more and more every year……

 

 

Our final exercise is to get every Support Person off Production completely. We’re working on nightly, automated clones with data scrambling. At that point we expect to remove almost all access to Production. After all you have everything you need to do your job as a support person, bar critical calls. This is where we really want to be, but whether we get to that point remains to be seen, but we’re certainly going to try.

Of course, we’ll have all the usual security exercises on database hardening, ERP hardening, Linux hardening, segregation of duties, access control, etc going to ensure we have continuous improvement across these key areas.

I hope you’ve enjoyed this article and hopefully it provided some useful insights into how we are improving security and easing our compliance burden overall. This article has mainly been on securing data in Production from support, but I guess I’ve gone off on a tangent into some other closely connected areas also……

 I’d be very interested also to hear comments from others, as I think security is such a vast topic that really others views (please do comment) would be very helpful.

So what are you doing on security of data encryption/access to production data?

I’d very much appreciate anyone to comment. (I have written this column in the hope of actually getting feedback and learning what others are doing to improve what I also do 🙂  )

And now, if you’ll excuse me, I’m off to apply a security patches to Production before the PWC Auditors from Hong Kong (awesome city) role in for a Penetration Test in October……..I like to call them my Loser friends (nothing like poking fun at the auditors….the loser buys lunch……), but they are immensely smart guys (as most PWC folks seem to be) and security is always a moving target requiring continuous improvement, so it’ll be interesting to see if I put one over on the auditors yet again (where they find nothing) or they finally put one over me and I need to buy them lunch…….

Of course either way, our company will be a safer and better place, security wise.

 

Related Articles

I hope that as you start to consider security in more depth, you may find the other articles relevant and useful reading also. My thanks go to the generous authors on a number of the articles below for sharing their knowledge for the wider Oracle ERP (and also Oracle Database) community.

Oracle Audit Vault

Oracle Governance, Risk and Compliance

Best Practices for Securing Oracle E-Business Suite Release 12 (Note 403537.1)

Oracle Identity Management

Oracle Identity Management Analytics

Oracle ERP Management Tools and Solutions

Oracle Applications Management Pack (with Data Scrambling)

Oracle Enterprise Manager 11G

Oracle Enterprise Manager 11G Data Masking Pack

Security Breaches – US Company Penalty

Security Breaches – UK Data Protection Penalties 1

Security Breaches – UK Data Protection Penalties 2

Security Breaches – California hospital Penalties

Security Breaches – HSBC Penalties

Germany Company Penalties

Data Masking – Strengthening Data Privacy and Security

Solution Beacon Security Column

Solution Beacon Security Best Practices

Project Lockdown – Securing Your Database

Recommendations for Leveraging the Critical Patch Updates

Oracle Security Column – Technology Network

Mask your Secrets using Oracle Enterprise Manager

Oracle Security Column – Documentation and Best Practices

Oracle Security Column – Critical Patch Updates and Security Alerts

Business Intelligence and the Kung Fu Dragons of Wudang

October 3, 2010

I recently watched an interesting program on TV about the Kung Fu Dragons in Wudang, China. The students join from a very young age and basically study 16 hours a day to become a Kung Fu Dragon Master over many, many years. The training is difficult, filled with risks of failure and many give up before ever coming close to becoming a Kung Fu Dragon master. Indeed the truth is most simply don’t have the skills, time or patience to reach the heights of a true Kung Fu Dragon Master. Very few will ever become a Kung Fu Dragon Master.

Now you look at data warehouses and Business Intelligence and ERP. It’s extremely difficult, filled with risks of failure and many give up (after spending a fortune) before they even get close after many, many years of trying. The truth is most simply don’t have the skills, time or patience to reach the heights of a true Kung Fu Dragon Master in Business Intelligence and ERP. The parallels and similarities are indeed striking.

Now the Art of Zen Patching article talked about patience and a non-aggressive approach. Not so this article. This article is really designed to allow you to become a true Kung Fu Dragon Master in Business Intelligence in 9-12 weeks (in vanilla implementation) and stresses a highly aggressive approach to Business Intelligence and ERP. (This article doesn’t ask for any money to be sent for some dodgy DVD and a certificate I printed in my bedroom. This is free, it’s not marketing hype – this is the real deal).

A cautionary tale before I start.

I’ve seen the two sides of data warehousing with Oracle ERP. One European company I know bought a bunch of expensive, state of the art tools almost a decade ago. Almost a decade on, with a lot of tears and pain  (sounds like a Kung Fu Dragon training course…..???), 2 million bucks and a change in system architecture that will require massive re-work, users simply not adopting because what has been achieved is small, fragmented and ad-hoc, and performs like a three legged dog (indeed in the user manual it instructs the person to submit the report and then get a coffee for 20 minutes before coming back or lunch if it’s a larger report) – the dreams of being a Kung Fu Dragon Master in ERP Data Warehousing lie in shattered pieces, a ruin of what should have been a shining example of 21st Century technology. This isn’t a one off – according to independent research a huge percentage of projects fail in the custom approach. Indeed it is the minority of ERP Data Warehouse projects that succeed.

Sure they bought the most expensive software, they put in piles of money, but they never sought a Kung Fu Dragon master to train them properly, preferring naively to think that they could learn the art of the Kung Fu Dragon Business Intelligence off a DVD in front of the bedroom mirror whilst their parents wondered what the weird shouts and snake hissing sounds were from upstairs………Instead of being a Kung Fu Dragon Master of Business Intelligence, they ended up as more like something of a joke reminiscent from the movie Kung Fu Panda……….Trust me if you’re a Kung Fu Business Intelligence Panda, people are laughing at you, not with you…….

 

(By the way, if you haven’t got the DVD for Kung Fu Panda movie, it’s highly recommended). As well as Oracle Fusion Apps being released in 2011, Kung Fu Panda 2 will also be out…..

Here’s the other side. I’ve seen Oracle come in with the prebuilt ERP Analytics, install the database, ETL tools, dashboards, KPI’s and reports and do all the data orchestration, extracts and imports in……..seven hours………Now I’m an utter skeptic when it comes to Oracle marketing and consultancy, but yes, I watched as one single Kung Fu Dragon Master in Business Intelligence and Pre-Built ERP Analytics achieved in 7 hours what could not be achieved in 7 years by an army of untrained people in another European company. Now this was only a demo (done for free as a demonstration of what could be achieved), but it took all our historical R12 Oracle ERP Accounts Payable data for 7 years and transformed it in a fully defined, fully functional data warehouse with dashboards in 7 hours. This guy was an Indian Kung Fu Dragon Master in Business Intelligence from Oracle Singapore. I was in utter awe of what could be achieved so easily and so quickly by one talented individual.

Have a look at Haemonetics Corporation story at Oracle Open World on Demand (Oracle On Demand Access Required) and then the results when they moved to Oracle’s prebuilt BI Applications approach.

The reason this utterly amazing complete data warehouse could be achieved is due to the pre-built Analytics packs that Oracle now provides (for of course, an additional license fee), together with of course an incredibly talented Indian Kung Fu Dragon Master.

So what’s the deal with Oracle BI and Pre-built Analytics?

Oracle is moving heavily into the BI market, taking the overall lead in terms of BI according to Gartner, much to the surprise of many.

In parallel Oracle is moving to create an unprecedented amount of pre-built content for Oracle ERP (including R11, R12, Peoplesoft, Siebel, JD Edwards with SAP in the works in 7.9.7…..) moving the task of creating a data warehouse from a vast army over many years (with a horrific failure rate) to a guy with an installation disk. Or that’s the single great hope and prophecy of Oracle Corporation. OK, we’ll never get to an installation disk, but if the guy with the installation disk can get us to 70% I won’t be complaining……..

Finally Oracle is now heading towards, probably in R12.1.4 to embedding the BI content directly into Oracle ERP screens. (The technology to do this by embedding into Java screens – not hanging a BIEE screen of an ERP menu by the way – was released as part of 12.1.2). If you also look at the new Fusion Apps, embedded dashboards are now an integral part of each module. According to Oracle at Open World 2010, there will be “hundreds of pre-built embedded analytics in (Fusion V1) or as they put it another way “Analytics are always available in Fusion Apps”.

 

 

Whatever BI tool or preference, you have to admit Oracle’s strategy and growth is certainly very impressive. With the recent release of Oracle BIEE 11G it just gets even better. The presentation layer is truly fantastic.

 

So many are probably asking, what’s the pre-built content for Oracle ERP?

Oracle has effectively written for you the following components:

  • Data Extraction from Oracle ERP (and many other ERP’s)
  • Data Orchestration (pulling dependent tables as needed with full synchronization)
  • Data Import to Data Warehouse with full management and restart capabilities
  • Data Model Design of Warehouse
  • Standard Dashboards
  • Standard KPI’s
  • Standard Reports
  • Security Models
  • Integration both to and from Oracle ERP
  • and a whole lot more

Now all of the above have been done not by a single Kung Fu Dragon Master in Business Intelligence, but by an army of Kung Fu Dragon Masters in Business Intelligence and ERP at Oracle Corporation.

These guys are experts in Business.

These guys are experts in Data Warehouse modeling.

These guys are experts in BI.

These guys are experts integrating BI to ERP.

These guys are experts at extracting data reliably from ERP to a data-warehouse.

The product line has come from a huge amount of hours collaborating with top businesses around the globe to find out what an ERP Data Warehouse should be

They have created something that no matter how much money and how deep your pockets, you will never be able to match, because Oracle Corporation is in the software business and can use economies of scale. Your company isn’t, despite the best intentions of your IT department.

The Pre-Built Analytics module list for Oracle ERP is equally impressive and is growing substantially on a consistent basis. This now covers:

Now it’s almost definite that this list will continue rapidly expanding, so that not only is there prebuilt content on ERP, but you’ll find it integrated directly into ERP (R12, Fusion, etc) and I bet into all of the other tools that Oracle has in the coming years including their database and other management tools. Basically if you are running Oracle ERP now, you WILL be running Oracle BIEE in the years to come.

Not only does Oracle Pre-Built Analytics create entire date warehouse content, they also add to it on a frequent basis. So when Pre-Built Analytics 7.9.6 was released it had a whole raft of brand new areas of HR addressed including Talent Management, Learning and iRecruitment.

So if you are still considering building using a custom approach to Oracle ERP Data warehousing, think about this foolish Kung Fu Panda……

Imagine that Oracle ERP has some 14,000 tables – do you really want to build a data warehouse from this (and will it actually perform…….????)

Imagine having to write extracts from the key tables which although much smaller is still huge. Imagine writing all the complex logic on something like Order Management for the extracts.

Imagine having to define a complex data warehouse model (and no, ERP and standard database design don’t work for data warehouses – it’s a completely different approach from a relational database model so your skills won’t work here……).

Imagine having to write processes to manage the extraction and import of data.

Imagine having to do the Security models.

Imagine having to do all the integration to and from your ERP.

Imagine trying to keep your custom built data warehouse in synch with Oracle patches, upgrades, etc

Imagine designing and building 100’s of dashboards with no idea of the proper principles of dashboard design and usability

Still fancy your chances of building a BI Apps comparable product Kung Fu Panda? Have a look at the summary data warehouse of BI Apps…….I guess you would be a whole lot less confident if it was your own money at stake on your custom build rather than your company’s……

 

The above represents years of work by an army of highly skilled, highly paid people, where the company must wait for a very significant time to get the results if you choose to custom build. Where you don’t get these highly trained people (as too many companies don’t), the consequence is simple – utter failure and massive cost.

And of course, even if you build it, what happens when you add another 2,000 users? Is it scalable or will you find major flaws in the design that are incredibly costly to fix? With Oracle Pre-Built Analytics scalability has already been designed in by the experts.

Still not convinced? Well let me give you the stats on Oracle’s prebuilt analytics:

350 Fact Tables

550 Dimension Tables

5,200 Pre-Built Metrics

15,000 Data Elements

And Kung Fu Panda, do you really think that the quality and performance and design can match that of the Kung Fu Dragon Masters from Oracle Corporation that do nothing but BI Applications? If so put down that Vodka bottle (or whatever else you are smoking) and get a reality check.

 

Or how about this Dashboard, with all the extracts, imports, data model behind it? For info this is Oracle Projects, one of the hardest niche modules in the entire ERP Suite, with links to EVERYTHING…….

 

In BI Apps just for HRMS there are 9 dashboards with a whopping 47 pages, 230 reports and 330+ metrics. And that’s just ONE MODULE OF BI APPS !

By the time you build that using a custom build approach (that’s so 1990’s in terms of approach – are you also still using Computer Punch Cards?) you’ll be retired or fired before you see it go-live……..

Today Oracle has the capability to deploy in 9-12 weeks (in vanilla implementation). Marketing slogans and nothing else? Absolutely not. We watched Oracle install everything (from database, to tools, to dashboards and migrate 7 years of data) and bring up Accounts Payable in 7 hours as a proof of concept. I can be pretty skeptical and hard on vendors, but believe me, this was truly amazing the ease with which this product could be brought up.

Now true there are some serious challenges around also, even with the Pre-built Analytics. Anyone thinking they can buy this and that’s it, well that unfortunately is naïve. I wish life was that simple……

The fit of the BI Analytics needs to be checked in terms of dashboards. What we found was that a lot of dashboards fitted very well onto the Business Requirements. Please bear in mind that dashboards are very easy to build (and BIEE is very easy to learn), especially when the Kung Fu Dragon Masters at Oracle Corporation have done the hardest part of building the entire data model and all the data extractions for you. Even if you find no use for the pre-built dashboards, simply having the data model and extracts will save you a huge amount of time. Now looking at Infosys they reckon a 70-80% fit in terms of BI Apps. I’d tend to agree that would be a ballpark figure when we assessed this against our ERP.  (And remember a fit means data extraction, import, data models, dashboards and all the other components).

If you do have custom modules, then you will need to build the extracts, data model, etc. However don’t be fooled by naïve arguments that if you’ve customized your ERP, BI Apps will not work. I’ve rarely seen a company modify base tables in ERP (add new custom tables yes) as they would be extremely foolish to do so. Therefore BI Apps will have just about every key table from the core ERP model you could possibly need. That’s a huge chunk of work already done.

You will need to define further dashboards and reports. That is a given. BI Apps will get you very far down the line, but it won’t be a silver bullet.

You will need to design data models and extracts for any custom modules you have. That can be a difficult task, but your ERP will already have all the data models, extracts provided by BI Apps. Custom modules are almost always a whole lot simpler than Oracle ERP Data models.

One of the biggest tasks that not even pre-built analytics will address is Training and user adoption. Moving from reports and Excel to Analytics is a major business change exercise and should be treated accordingly. The project can still fail even with great pre-built content, because users are not trained appropriately and there is a lack of vision in how this can really change the way the user community work.

The cost of prebuilt analytics is still a major problem I think. There are alternates out there that are much, much cheaper but then you have to consider upgrades, risk of building, integration and a whole host of other problems. I love the Oracle BIEE and the Pre-Built analytics but the cost is still pretty high. Of course compare the cost of that to the cost of building all this yourself and there is just no comparison between the two, like for like. Playing hardball with Oracle Corporation (ideally at their year end when the best deals can be achieved) can pay dividends in terms of price, although I’m not going to give away how cheaply we got offered the prebuilt analytics……There are still alternates to Oracle in many areas, solution wise and Oracle should be reminded that you can go elsewhere…..

Pre-Built analytics for ERP can get you there fast. But a big bang approach is not recommended. A fast initial deployment to a small community followed by consistent wins over a period of time, slowly building up the user base and expanding across the lines of the business is the way to go, bringing in and training champions each and every step of the journey. Then you’ll succeed where so many others fail. When everyone can see frequent wins (and with BI Apps it can be every few months), it becomes an unstoppable train that everyone wants to get on. BI Apps is beautiful eye candy that users seem to love. (Or is it that they finally can get their own answers in seconds without waiting 12 weeks for the IT Department…..)

You can buy the software, but you need to sell the vision, otherwise you fail. IT Alignment to Business Alignment and support from Business is the key to any successful project.

I’d still recommend having a BI Expert around who you can gain the knowledge from. Again Pre-Built Analytics is not a silver bullet and your staff will need to be properly trained by an expert.

I think all companies running ERP should be looking seriously at BI Apps, even those with the approach of building everything. In the 90’s companies realized the value of letting Oracle, SAP and the rest build their ERP applications. That’s now a generally accepted principal by most companies around the globe.

Over 2,500 companies have already adopted the BI Applications. If it was cheaper (a message to Oracle Corporation and Larry Ellison) the uptake I am sure would be massive.

In the coming months, I’ll write a few more articles on deep-dive into some of the core areas of BI Applications – HRMS, Financials and Procurement, time permitting.

I think in the 21st century, that same principal will gain general acceptance on the value of pre-built analytics for ERP. Put quite simply, in the cut-throat economy companies can’t wait ten years to become a Kung Fu Dragon Master of Business Intelligence with Oracle ERP.

Companies need to achieve that in 9-12 weeks (vanilla implementation) and the only way to do that is to rely on the Kung Fu Dragon Masters from Oracle Corporation that have already done this for you…………before your competitors beat you to it……..

Related Articles

Below are a number of articles. Some are your average Oracle Sales pitches, but all the same very informative overall. Others are written by generous authors whom I acknowledge and extend my thanks to for writing very interesting and useful articles, which were used as background research during writing this article. (I particularly enjoyed the articles by Jeff McQuigg and Mark Rittman, both renowned experts in their field and excellent authors. Their articles are highly recommended reading).

 For those lucky enough to have been to the Oracle Open World 2010 show (hope you enjoyed the Black Eyed Peas !!! Turn down the volume before you click that link……), the following can be accessed from Oracle Open World On Demand.

(Note there were many more sessions, but many do not yet have the content uploaded at the time of writing – 2nd October 2010. I have chosen to hyperlink only a few of my favorite session links. Others are available in Oracle Open World On Demand).

  • End-to-End Oracle Business Intelligence: From Warehouse to Advanced Analytics
  • Enterprise Sales Analytics with Oracle Business Intelligence Enterprise Edition
  • Oracle BI Applications Roadmap, Including Support for Oracle Fusion Applications
  • Oracle Business Intelligence Enterprise Edition Integration at Comcast
  • Oracle Human Resources/Oracle Financial Analytics for ResCare Decision-Making
  • What’s New in Oracle Business Intelligence Suite, Enterprise Edition Plus 11g
  • Gain the Insight You Need with Oracle Business Intelligence Applications
  • Implementation Experiences with Oracle Business Intelligence Applications
  • Implementation Experiences with Oracle BI Applications
  • Oracle Transactional Business Intelligence (OTBI)
  • Haemonetics Corporation: Better Decisions with Oracle Human Resources Analytics
  • Get Real-Time and Actionable Information with New Reporting and Analytics

R12 Patching and the Art of Zen

September 18, 2010

Reading through Wikipedia, I found an interesting article on the concepts of Zen. Now I’m not really into that type of stuff myself (each to their own), but I thought it would make an original way to present this article 🙂

“One practice of Zen Buddhist’s is Koan Inquiry. A koan is a question, or statement, the meaning of which cannot be understood by rational thinking but may be accessible through intuition. The answer can occur during meditation or during your typical daily life with all the mundane tasks you do.

To Zen Buddhist’s the Koan is “the place and the time and the event where truth reveals itself”. It is a way to induce an experience of enlightenment or realization, not through rational reasoning, but through intuition.

Answering a Koan requires a student to let go of conceptual thinking and of the logical way we order the world, so that like creativity in art, the appropriate insight and response arises naturally and spontaneously in the mind.”

Or to quote from a very non-Zen perspective, you think about a problem very hard all day. You fail to make any breakthrough. During the next morning, in the shower, without even thinking of the problem, you suddenly think of the idea. Ironically perhaps we are all practicing Koan Inquiry as a natural state of mind to solve difficult problems, without even having to think about the problem at hand.

Now let’s look at ERP Patching in relation to Zen 🙂 We need to be clear from the outset that this form of Zen Patching applies only to the following patches below. This is extremely important to keep in mind.

Security Patches

ATG Patches

Database Patches

This form of Zen does not apply to other ERP Patches

Applying this form of Zen Patching to any other types of patches will cause you some serious grief in your career when you report to your boss that your ERP for your entire organization worldwide is trashed because you read some amazing article by some “new age ERP guy called the Oracle Prophet” on a radical new method using the Art of Zen for ERP Patching and thought it was worth a try on your Production System……….

Do note that this form of Zen Patching does work on both R11 and R12, but not on R10. It also works on 9i, 10G and 11G databases. Please check Oracle Certification matrices and raise the question to Oracle Support if in doubt.

Oracle Support – Good morning. Can I help you?

Reader – Yes could you tell me if the Art of Zen patching is certified against R12 Apps please?

If the phone goes dead at this point, we suggest you assume Oracle Support is not aware of the Art of Zen patching and you should not pursue your question with them……….We also suggest you give your colleagues name during any telephone calls with Oracle,  in case Oracle raises a complaint for nuisance phone calls to your company…..:-)

So where does the Art of Zen fit into Oracle ERP patching?

 Let’s use a typical a koan to provide an illustration.

“We will test the patch by not testing the patch. Only then will we know that the patch has worked.”

Now at this point in time, you are probably thinking I’ve been hitting some fairly strong stuff to get to this state of mind, or I’ve completely lost the plot.

I can hear everyone thinking “So let me get this straight. You are going to test the patch by not testing the patch, so that you know the patch is working”. To which I’d reply, great you’ve got it. You are certainly a quick learner on this Zen Patching stuff!!! 🙂

Our R12 Patching Philosophy actually made our auditors jaws drop, not in terms of the Zen stuff (trust me, keep this stuff between yourself and myself please and maybe better not mention to your management or auditors……), but on the thoroughness of approach.

We always have five databases for our patching (at a minimum). This is probably a lot more than most have but let me explain why and you’ll probably want to then copy this model.

Our DBA Environment. This is where the patches are applied to make sure, well they actually apply. Believe it or not some patches from Oracle don’t even apply cleanly.

Our Patch Environment. This is where they are applied with a little bit of testing. OK we deviate a little from the Zen stuff, but give me a break……This makes sure they at least do what they say on the box without major functional failure.

Our development environment, which is always busy with daily activity by our development team , functional team and testers.

Our test environment which is always busy with daily activity by our users.

Our Production environment.  I’ve been pushing our company to drop this as it uses a lot of space and we hear most of our complaints from this database, but management insist it is important and needed. 🙂

We should also state our databases are pretty heavily used so application flows naturally are being used throughout development and test databases. We also apply any patches onto any other instances we have at that point in time, so that the patch is naturally tested by the simple day to day activity in as many places as possible, with a careful rollout to each environment.

 

The Art of Zen Patching

The point is simple on these types of patches. Oracle does release patches that should be applied at some point.

The Security Patches typically come quarterly and we try to apply 3-6 months after they come out. Security patches represent a serious risk to apply, although generally apply well. However security patches also represent a risk if you do not apply. You need to find the balance, but you SHOULD apply these regularly.

The ATG Patches are less frequent but provide critical updates to Browsers (especially if you have DMZ applications) and other technology components, including diagnostics.

The Database patches (and we’re talking 10G to 11G for instance) do come out periodically and at some point you need to decide to keep at least supported, although we’re very picky on applying these, but are in the process of an 11G Upgrade. (Various 10G database versions are losing or have lost premier support). This activity is every couple of years or so.

We’re not talking about applying every patch. No company in their right mind can achieve this. We’re talking about keeping your head above water and staying supportable.

The point is this. To test every time on these types of patches across every last item is impossible. The conventional way is to get the patch, apply, test everything and then move to Production in a few weeks. That’s a very logical way to order the world of Oracle ERP. But unfortunately this is not a very practical or safe way. These patches are by their very nature too broad and silently hit too many areas to be open to a logical, standardized testing process. The conventional approach actually increases risk with these types of patches because it is by no means obvious what could be impacted.

There must be a better way where you find that balance. This is the Art of R12 Zen Patching.

Our philosophy is simple. We plan carefully on all these types of patches well ahead of applying.

We do not apply these patches immediately they come out. We are kind enough to give others the opportunity to be the heroes or unenlightened who find the bugs, log them in My Oracle Support and make our life so much simpler because we heavily research each patch to find the problems the unenlightened logged. This way we avoid the bulk of the problems. Are you one of the unenlightened? If so we appreciate you finding the bugs for us, causing issues for your users on Production systems and generally making our life so much easier and less stressful.

Our philosophy then rests correctly on a peace of mind that these patches are largely stable, largely trusted and tested by others around the world. This isn’t just a philosophy, it’s backed up by hard facts based on an incredibly low failure rate of patches we have applied. The patch types listed are generally very mature, very stable and very reliable. The quality of these types of patches is far higher than the Oracle ERP patches for the application modules.

Our key philosophy can be defined by the koan below. (As you raise a smile, remember this is used in leadership teaching by guys that make more in a month than you make in 5 years and sell books by the truckload at Amazon 🙂

“Once upon a time in ancient Japan, a young man was studying martial arts under a famous teacher. Every day the young man would practice in a courtyard along with the other students. One day, as the master watched, he could see that the other students were consistently interfering with the young man’s technique. Sensing the student’s frustration, the master approached the student and tapped him on the shoulder. “What is wrong?” inquired the teacher. “I cannot execute my technique and I do not understand why,” replied the student. “This is because you do not understand harmony. Please follow me,” said the master. Leaving the practice hall, the master and student walked a short distance into the woods until they came upon a stream. After standing silently beside the streambed for a few minutes, the master spoke. “Look at the water,” he instructed. “It does not slam into the rocks and stop out of frustration, but instead flows around them and continues down the stream. Become like the water and you will understand harmony.” Soon, the student learned to move and flow like the stream, and none of the other students could keep him from executing his techniques” – Timothy H. Warneka

Now I’m not into all this stuff and I’m as skeptical as anyone else, but maybe they have a point. Too many companies are simply slamming into the rocks with patches, rather than working with the flow of Oracle Corporation. Working with Oracle, you often feel that you are not talking of a stream but more a raging torrent of patches. The problem is you are always fighting against the flood of patches, rather than finding what these guys would refer to as “harmony”.

The very essence of our philosophy and the koan itself can now be answered 🙂

“We will test the patch by not testing the patch. Only then will we know that the patch has worked.”

After rolling the patch through DBA and Patch environments very carefully over many weeks, we are ready to proceed to our main development and testing environments.

We typically roll patches into our development environment for a minimum of 4 weeks. We observe the behavior of the environment and record any bugs. We carefully investigate all bug occurrences.

Once we are comfortable at that point, we do run testing. OK so we broke our mantra, but nowhere near the testing that would normally be required. Why? Because we have seen the bugs naturally arise through our normal daily activity (as a Project Manager you’ll know typically what is going on and where the gaps may be I would hope). So to quote the Art of Zen,” the appropriate insight and response arises naturally.” This is the beauty of the Art of Zen Patching 🙂 You do your daily stuff to get to the answer of whether the patch causes major grief.

At this point we normally release the patches to our Test Instance, again allowing patches to settle for 4-8 weeks. Again using normal user activity, we gain further appropriate insight and responses, in terms of stability of the patch and subsequent bugs, arising from the natural process of user activity.

We do ask our users to test and hit the key functionality, but again, with the insight given from normal daily use, we have achieved ” the appropriate insight and response which arises naturally” as a Zen Master or Leadership or Lifestyle coach would tell you for quite a lot of cash 🙂

Even our DBA Team reaches a relaxed Zen like state and if you know your average DBA guys……… With planning comes time for our DBA Team to work and document carefully the steps needed for each patch. The timeframes create space for many, many practice runs, so that on the day of application to Production, they know exactly what to do and what to expect. It also creates the space and time for good old fashioned research on My Oracle Support. (A tip is that as we’re doing this approach over a number of months, the DBA’s always get copies of Production on a regular basis to run through the patching process, so any production specific issues are always encountered early).

In addition, we carefully plan for releases. So if you take our last security patches, these were rolled into an ATG RUP6 and a minor database point release (to stay supported). This reduces a constant patch cycle to a more manageable ITIL Release concept, reducing your workload overall. The raging torrent of patches becomes a much more manageable stream.

Most companies have huge stress over these types of patches. Most companies don’t even bother applying, much to the detriment in terms of support, future upgrades and security.

We are like every other company in many respects. We are highly conservative on applying patches. We like to stick on what we know.

But we do pay attention and plan for security, de-support of databases, new browser support in ATG RUP’s, etc in a very careful manner well ahead of time, allowing us to practice the Art of Zen Patching 🙂

Companies stress out, rush patches and therefore make mistakes. That is not the Art of Zen Patching. Zen Patching stresses the very opposite approach. Put the patch into your environments, slowly observe and watch over many months, then test and finally you will see the appropriate insight and responses that it has worked. Now looking at Wiki again, Monks meditate over many months or even years to answer a Koan. There is no difference in the approach of Zen Patching 🙂 The time the patches spend in your environments can be thought of as your meditation period over many months (typically 3-6 months depending on risk assessment) to find the answer to the koan of “how to test the patch without testing the patch to know the patch has worked”.

But with our R12 Zen Patching, we’ve reached an almost Zen like state 🙂 Patches are simply a natural part of the lifecycle of ERP. We have accepted that. They are planned and are allowed to settle for several months, to give insight into their nature and risk. Patches still require testing, but to a far lesser extent than a fully focused, high risk “lets test this patch and apply this patch in two weeks time”, which is similar to slamming into the rocks in the stream.

So where are we today with such an approach?

R12.0.4 RUP5 (yes this was a nightmare of testing the old fashioned way, definitely slamming into the rocks in the stream, but our go-live was incredibly smooth. Zen Patching doesn’t work here I’m afraid for those embarking on an R12 Upgrade. It’s the good old fashioned conventional testing approach that is needed here).

Security Patches to April 2010

ATG RUP6

11G and July Security Patches are currently under the Zen method as is a SUSE Linux Upgrade

We have never had a failure or serious outage as a result of Zen Patching (should I trademark this perhaps and make a lot of cash like those leadership guys??????), although as with all Oracle patches, it is a very serious business, with serious risks, so there is no place for complacency.

So what about the opinions of others on our ERP and how up to date we are overall with patches?  To quote one Senior Consultant DBA recently, we are way ahead of most companies in terms of patching, and have an “aggressive patching policy”.

I would say that the Art of Zen Patching can never be described with words like “aggressive” 🙂 . In fact it is quite the opposite. It is a very slow and considered process, stressing great patience, over many months, waiting for insight as a part of a natural process to reduce the risks we face, as the real Zen guys would put it 🙂

We simply achieve a lot more than most companies, with a lot less effort and a lot less risk. I think the Zen guys and leadership/lifestyle gurus would term it as “simply learning to move and flow with the Oracle stream creating harmony and peace of mind”. Obviously they’d be charging a thousand US Bucks an hour for this type of advice. (I remember we had one such IT Guru in our company. Cost us 6 figures for six weeks of work – we ended up using his laminated b*llshit as coffee mats……that was about the only value we got……). Now maybe the Consultant our company hired wasn’t too hot except for the coffee mats, but with some of these leadership and other philosophies, well maybe there’s something in it after all……

Call it Zen. Call it Lifestyle (or is it Patch) Management :-), but to have a safe, low risk and stress free approach to this type of patching which works with reduced effort (rather than increased effort) is  not a bad place to be, as an ERP Manager…….

Health Warning

This article was designed to present a very serious subject in a hopefully entertaining and educational manner utilizing both conventional approaches of testing, in conjunction with a more unconventional approach. However applying any patches on a Production Database is a very serious business. Patches do need testing and this should never be underestimated. However the point of this article is that by allowing patches to settle into various instances over time, you vastly increase the chances of spotting serious issues and vastly reduce the risk of issues in production that conventional testing can never address. This is the safest way to apply such patches that I have found, using a conventional testing approach, together with a far less conventional approach. 21st Century Testing meets 5th Century Zen Philosophy.

Disclaimer

(By the way, I haven’t been smoking anything….the intention of this article is to present a very serious subject – Oracle Patching – in hopefully what some will find a very funny and original manner, that can then be remembered and applied to help all of us that face the very serious risks of patching global ERP Systems, so don’ take all the Zen references for Patching too seriously………otherwise someone may think you’ve been smoking something………). Think Monty Python British humour as you read it……..

If you remember the article, change patching from a rush to a planned perspective and patch carefully over a period of months, then the article did it’s job 🙂

I hope you find it as funny to read as I did writing it 🙂