Archive for October, 2010

Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt3)

October 31, 2010

Welcome back to the R12 Upgrade House of Terror.

All I can promise you is pain, heart-ache (that Oracle Support girl for the AP Trial Balance broke mine…..ahhhhhhhhh) and impending doom. Abandon all hope on your R12 for those who read on……..

So you’ve been mad enough to start an R12 project (they’ll lock you up in the asylum they will !!!) and you’ve made it through Implementation (designing any new stuff, build, test, UAT, etc) and you are sitting there smugly thinking you’re smarter than your average 5th grader and that this column was just scare-mongering……..Well lets go to the Transition Phase and see how confident you are………

 Cutover is complex

Deployment involves high number of objects

Deployment involves significant application setup

Deployment carries high risk areas

Users are not familiar with the new software

Large number of users require training

Users are distributed across countries

Help Desk Staff are not trained to take calls

Transition is poorly documented

Transition is not rehearsed

Change in Business Procedures for R12

Before you start the cutover, just wondering if you actually trained all your users? Yes R12 is mostly the same, but there are significant variations between R11 and R12 in Payments, TCA, SLA, etc. Now if those are spread across countries, it’s going to be even worse. A new system with new features suddenly appearing will lead to a lot of angry users, all wanting to talk to you on Day 1 of R12 as you are inundated with other more serious issues……..Now I went through four audits on my R12 and got a clean bill of health on all, except for training (even though we did some). The auditor did a survey and the users said they would have liked MORE TRAINING. But the products almost the same I said bar a few key areas as we were doing a Technical Upgrade !!!!!!!! I got burnt at the Halloween stake on this – OK ever so slightly singed. Don’t make this mistake. At least offer some training on delta differences and some educational classes with no commitment to implementing (get your lawyer to write the disclaimer) new functionality.

Has anyone told the Helpdesk of your R12 transition? Has anyone told the business they will lose their systems for 4 days (weekend, plus two working days). Has anyone told the external users or put out appropriate announcements? Are there key business activities that expect to run during your upgrade? Have you scheduled your upgrade right in the middle of these? People are going to be seriously upset if payroll is delayed for your upgrade. Or month end closing is delayed and you have serious problems with your new R12 that goes live right before the preparation of those financial results. Wondering if you told your corporate website team that pulls out summary information from ERP to display to your company website that ERP won’t be available over the transition? Or how about the legacy team that are still sending you files to interface into ERP? I know you are feeling pretty insecure now. I guess you are wondering “Have I missed anyone in the announcements……”.

Cutover is a nightmare. Get this wrong and you see your entire project collapse in a pile of dust, on day 1 of Production, with the Board asking why your business has ceased to function and lost 5M US. Pulling the transition on R12 at the last minute is really bad, but even worse is going live and then finding out you have serious issues……..Either has a good chance of ending your career.

Has everything been tested? Has every bit of code your about to deploy to Production actually really been through UAT (or did the developer slip in that one last easy fix with no testing?). Does your Application Setup really reflect the setup you tested? Did the Consultant complete the BR.100 before he left for a contract paying 30% more? Can anyone actually follow that BR.100……?

So you have a thousand customization files (actually isn’t a lot when you count Java files, setup, etc). Have you ever taken that as a single software release and had your DBA’s (not your developers that know to run script 2 if script 1 falls over and hack script 3 to make it work, without anyone finding out) and applied to an exact clone of your Production system and had a 100% success rate? Have you repeated this multiple times as you roll towards the go-live date, just to make sure that any changes between the time you last ran it and any updates to Production didn’t cause any adverse impact? Are you sure that the DBA’s can run all your customization deployments in a very short-time frame. (Your typical go-live window is a maximum of 4 days with two of those the weekend, after that your business will scream if you go one hour over). Is your deployment manual and if so, will the DBA do the same steps each time? If your deployment is automated will it actually work?

We had 600 application changes. Have you ever checked how long it would take to deploy all of those changes? Have you ever done an entire setup and checked all the setup was valid and tested properly? Have you thought of the issues of letting knowledgeable people do the setup in Production in terms of access, SOX, etc? If new people need to do the setup, as the project team cannot have access to Production, are they familiar enough not to make mistakes? Have they ever practiced the setup on an R12? Interesting how long it takes to carefully setup and check 600 application setups. Just wondering if you ever thought how long it actually takes to setup 600 application setups for a single person. Do you know it’s impossible to do that setup with one person? Have you ever bothered to do a careful review of all setup, access needed and then balance setup streams across different people to make it achievable in the very short transition time the business has granted you whilst still ensuring segregation of duties, SOX compliance, etc?

Have you ever considered setting up some functions, menus, etc in R11 BEFORE you go-live. Or some new program definitions? Or perhaps having these as Loader files, so you can load them instead of manually setting up? Of course if you load them, have you checked that the developers have generated these LDT files correctly?

How many minutes will the R12 Upgrade run on Production hardware? (I ask for minutes because you should know very precisely the estimated time of this – if you don’t you are in serious trouble. For info an R12 will run for a very, very long time indeed, even on a small ERP database.). Have your DBA group ever run an R12 Upgrade with no failures? How long will it take to upgrade the Technology components or database? Have you ever seen a detailed step by step upgrade document from the DBA Team? Are there any hot wiring the DBA team apply during the upgrade due to known failures? Are you aware of these known failures and the implications? Do you know that an R12 Upgrade will require some very serious shift working? Do you know that your DBA’s will be seriously burned out as they have the very first task in the actual upgrade of running the hundreds of thousands of scripts that the standard R12 upgrade runs. And they need to actually be awake and watch this constantly, because if it falls over, you don’t have the time to lose 8 hours on an upgrade weekend…….

Wondering if you’ve actually even got a transition plan? Have you got an hour by hour schedule, with milestones and an effective method to monitor setup, customizations, actual upgrade, etc? On an R12 upgrade transition, you have no slippage time for anything. If you slip, the team will get stressed, you’ll get stressed and mistakes will be made.

Bottom line is have you REALLY rehearsed an R12 cutover BEFORE you actually do it? Do you really feel confident that first time you do it for real will be on your Production system……….if so you must be a clown……by the way do you think the McDonald’s clown is intrinsically evil????? (My Attorney has advised me at this point to clearly state that is a question and in no way implies anything on McDonalds whose food I have been legally advised to state I love and the picture below is in no way related to said company).

If you take a look at the Oracle Upgrade manuals, it’s interesting how many pre-req steps potentially have to be done. Ironically if you miss some of these you can have major support issues going live. I wouldn’t leave any Payments in the middle of process. I’d be carefully checking the effect of an upgrade half way through each of your workflows. Is there any effect on transactions that are left half completed? Do you even have these in your cutover procedures and does everyone in the Business know from your carefully documented plan exactly what their responsibilities are prior to bringing down their R11 ERP. If you (or the Business) makes mistakes here, you are in real trouble. Do you have scripts to check everything has been accounted, no payments are sitting halfway or are you just relying on your business to do the right thing with no checking whatsoever…….Just to add a little time pressure, your DBA’s will be screaming at YOU if you go one minute over the agreed time to shut down R11 and hand it to the DBA team……..of course you’ll have to herd dozens of business units across 27 countries to complete all activities before that time…….but then they want to do their business not your upgrade……which leaves the DBA’s screaming at you…….(The DBA’s will be working all through the night, so every hour you delay, means an hour extra in the middle of the night so a little empathy for them here folks……..Show them you care for their welfare….at least till your upgrade is over. Alternatively take the selfish approach that the DBA that is working all night might be the same one running your customization scripts after two hours of sleep because of your poor planning that could wreck your upgrade when he makes a serious mistake because he’s so tired……….)

Have you made any plans to backup the original system? What are your fallback plans if the upgrade fails? When will you be prepared to pull the plug and under what criteria, over the transition cutover to R12? Who is going to take responsibility to give the green light to go live? Will the senior management even be there at the weekend to make that decision?

Just wondering if you told your team and others (Unix, Email, Legacy, etc) that they would need to work some very stressful and long hours over the cutover weekend. Wondering if any of those key staff have their holidays planned…….?

Have you made plans to clone and test the actual R12 on the cutover weekend? If not how do you know all your setup, customizations, etc actually were promoted cleanly and your R12 actually works? You will need a large team to test in such a short period of time. Do you even have a cutdown test plan to make sure you hit the key functionality? Have you got all this planned? Probably not…….And if you find stuff missed, do you have a proper controlled mechanism to track it, test it and release it to Production in a horrifically short and highly pressured weekend in a safe manner?

What’s amazing is that everyone plans a transition, but no-one actually thinks about turning on the R12……….when you enable the workflow and the concurrent managers are you just going to open the tap, or are you carefully going to release production jobs and monitor. I’ll tell you that your single most stressful time is when you commit to switching on these elements over the weekend. I bet you that your heart will be pounding and you will be sweating at this point. The entire company ERP and Business hangs on your Go or No-Go signal. Once you’ve done that, there is no turning back, there is no backing out and you are on R12. At that point if you’ve got it wrong, you’ll be looking for another job. (Doesn’t that picture below remind you of your boss when it all goes wrong……?????). I wonder if that Oracle Support girl I dealt with for the AP Trial Balance wore red lipstick…….ahhhhhhh……..

So your live, but it’s not over……..

Serious Outages can occur

Lack of Staff to address serious support issues post go-live

Patches are required after go-live

Are you ready for serious outages? Did your transition plan have proper contingency plans for these events? If not exactly what will you tell the Board and the Business?

Do you have the appropriate staffing plan to handle what could be a very rough few days (or weeks) or months? An R12 will almost always lead to a spike in calls. Are you ready? Did you bother to give your people proper notification that they may be required to work late nights and weekends for the first few weeks after go-live? Or is it just a line item on your plan that you didn’t bother to communicate?

Did you re-synchronize all of your ERP and Legacy systems in terms of interfaces? Some of these integrations may require manually running interfaces to catch-up with transactions for the days you were down.

With major business events such as payroll and closing, do you have any plans to clone and rehearse these events, prior to these happening? Now if you can clone and simulate an R12 closing after you are live, but before the closing happens, it’s better to find out serious problems and have a few weeks to fix them right? (Of course this has you wondering if you should close in R11 and then immediately move to R12, but of course, then your Financial reporting would be straight after go-live, so you can’t win right?)

Have you got any plans for patches after go-live? What happens if you really need to apply patches? Are your project team still there? Did you set business expectations just in case? Always better to make business aware of the real risks inherent in an R12 Upgrade. That way, they’ll be incredibly happy if you manage to get it right.

I hope you remembered to get before and after reports from key areas, such as your AP Trial Balance (that Oracle Support girl was just so nice, ahhhhhhhhh), GL Trial Balance, etc. And keep a copy of your Production database on R11, whether on tape or disk. When the auditors come and take an interest in it, as they will it would be unfortunate if you’ve not done this and it causes issues with your auditor’s signing off your company accounts.

Also before you go live just have a quick review of JVM sizing and parameters. If you fail to tune and size this properly you could see your Applications crashing, intermittent crashes, etc. Getting your JVM sizing wrong leads to absolute nightmares……..Ask your DBA’s “Have you size the JVM’s and if they say what’s that, start to worry……”. If you have too many users and not enough JVM’s and OACORES then BOOM !!!!!!!!

Finally is there monitoring of key components in place? Are you touching base with all the areas of Business (and Legacy and Web and email and and DBA and Unix and other teams). Are you even aware of major production issues or is your Production setup so poor in terms of support that you’ll be the last to know……..

And lets break for another movie recommendation. The single scariest movie I have EVER seen. This was the 1970’s television mini series called Salem’s Lot. It is absolutely terrifying. Get it on DVD….it’s one creepy movie from the Master of Horror Mr. Stephen King. I saw this when I was ten, and I didn’t go out for weeks after dark……It’s a classic ten out of ten movie. But REALLY TERRIFYING.

This isn’t an exhaustive list of what can go wrong in an R12. Now note this article has been written in a strong manner to scare the crap out of you because if I do that then you’ll start to take every last point out of this article and make sure you’ve covered it. Make no mistake – R12 Upgrades are a serious risk to your career. (However don’t let some consulting company come in and scare the crap out of you. They are in it to make as much money as possible from you for an R12 by scaring you as much as possible. I write these articles for FREE and hope people can avoid the mistakes that I and others made in our R12 Upgrades). If you read this article and take everything on board, you’ll be strongly versed in R12 risks and be able to evaluate what these consulting companies are telling you and separate the fact from the fiction…….

And a final note. Premier Support for 11.5.2 ends in November 2010…..that gives you about a month left……Interestingly you also need to have minimum baselines in place for R11 if you want proper support from November 2010……….And will you get hit by fees if your still on R11 in November 2011??????

Have you read Metalink Note 1178133.1…….

“Starting on Dec 01, 2010, Oracle E-Business Suite customers on Release 11i10 will only receive extended support for new bugfixes as described in My Oracle Support, Minimum Baseline Patch Requirements for Extended Support on Oracle E-Business Suite 11.5.10.”

That’s a lovely little bit of legal small print from Oracle with major implications for you………Getting to those minimum levels will be enough to “persuade” most customers to jump to R12 instead.

It’s funny also that so many people don’t think they can start doing preparatory work in R11 that will greatly ease R12. Have a look at the references below. Work spread over R11 and R12 will greatly ease your stress in the R12 Upgrade itself, as you’ll have less to do. (Did you know Solution Beacon even has Vision instances you can play with for free, even before you install an R12 in your company’s test environment…..)

Let me tell you briefly our R12 story. We had some very serious problems with quality (12.0.4 RUP5) – an early R12 release in many respects. However despite that we got through a Finance, HRMS and Procurement Suite R12 Upgrade across 20+ countries with strong planning. Our transition finished 30 minutes ahead of the planned 96 hour time window (I wasn’t kidding when I said you should know the transition in minutes…..). Yes, I was terrified when I made the call to go-live. But due to all the planning and a great Project team, we went live with no major issues on 3rd August 2009 and I went home at 5PM two days after go-live (And for the cynics, no I didn’t spend the first two days constantly in the office 48 hours straight…..I got home at a reasonable time the first two days as well). We cleared Payroll, breezed through month end and went live on two Payroll rollouts to two new countries one month after go-live.

Of course now I’ve told you all the risks, well you’d be a fool to actually fall into those holes right? Take this article, make a list, make sure you cover each and every one. Then you’ll make your R12 a huge success.

An R12 is certainly a very achievable project, with significantly reduced stress and risk, if planned correctly.

Hope I haven’t scared you too much……..

Enjoy your Halloween.

References

As always, my appreciation goes to the kind authors and contributors of the following articles and resources. Some of these are especially note-worthy, because those people are putting out their failings (as well as successes) on R12 in a very public and open way, for the benefit of those about to travel the R12 Upgrade road.

General Blogs and Websites

OAUG Website

Chris Warticki’s Blog – EBS R12 Support Resources – Consolidated

Oracle Applications Upgrade Guide: Release 11i to Release 12 (B31566-01)

Oracle Financials Applications Blog – R12 Lessons Learned

Analyzing, Planning and Executing an R12 EBS Upgrade

Steven Chan Blog

Risk Management: Tricks of the Trade for Project Managers (Rita Mulcahy)

My example of an R12 Risk Spreadsheet (Excel 2007) – Available on Request

 

Metalink

Oracle E-Business Suite Release 12.1.3 Release Update Pack (Patch 9239090)

 Oracle E-Business Suite Release 12.1.3 Readme (Note 1080973.1)    

Upgrade Manual Script (TUMS) – Patch 5120936

Metalink Note: 580299.1 – Best Practices for Adopting Oracle E-Business Suite, Release 12

Metalink Note: 394692.1 Oracle E-Business Suite Upgrade Resources

Metalink Note: 562887.1 R12: Helpful Tips for a Successful R12 Oracle Payables Implementation

Metalink Note: 437422.1 R12 Troubleshooting Period Close in Payables

Metalink Note: 73128.1 R12 Troubleshooting Accounting Issues

Metalink Note: Oracle Support Upgrade Advisors (250.1)

                             Tech Stack                     Note 253.1

                             Financials                     Note 256.1

                             HRMS HCM                    Note 257.1

                             Manufacturing              Note 258.1

Metalink Note: Extended Support Patch Level Verification in Oracle E-Business Suite Release 11.5.10 (Note 1178133.1)

Metalink Note: R12 Upgrade Considerations by Product (Note: 889733.1)

Oracle E-Business Suite R12.1 Financials Pre-Upgrade, Setup and Operational Tips (Note: 1104163.1)

Oracle E-Business Suite Release 12.1 Information Center (806593.1)

Database Preparation Guidelines for an Oracle E-Business Suite Release 12.1.1 Upgrade (Note: 761570.1)

Oracle Applications Installation and Upgrade Notes Release 12 (12.1.1)

R12: Period-End Procedures for Oracle Financials E-Business Suite (Note: 961285.1)

Oracle Open World 2010 – On Demand

For those with access to Oracle Open World 2010 On Demand there are good references at http://www.oracle.com/us/openworld/index.htm

Mission Impossible: Oracle E-Business Suite 12 Upgrade on 3 Continents in 6 Months

Success with Oracle E-Business Suite Release 12.1.2 Upgrade Drivers

Algar Telecom Upgrades to Oracle E-Business Suite R12   

Get Ready for Oracle E-Business Suite Release 12.1: Tasks to Complete Now

Oracle E-Business Suite 12 Upgrade : Best Practices to Obtain Business Value

Planning your Oracle E-Business Suite Upgrade from Release 11i to Release 12.1 (Superb presentation – highly recommended)

Oracle E-Business Suite 12 Upgrade: An Easier Ride on Nine Miles of Bad Road

Oracle E-Business Suite R12 Upgrades: Have you Thought about the Details?

Real-Time System Assessment of Oracle E-Business Suite 12 Upgrade: Case Study

And finally talk to other companies, use forums, other areas (such as http://www.linkedin.com) and make contacts with people that have done R12. This again will help you considerably on your R12.

Footnote:

Some of the great graphics were the work of:

WWW.MYSPACEGRAPHICSANDANIMATIONS.COM

Advertisements

Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt 2)

October 23, 2010

So it’s heading for Halloween. Only a few days left……..And looks like you’ve come back for the 2nd part of the R12 Halloween Upgrade. So I guess you don’t scare too easy? Prepare to be afraid. Prepare to be very, very afraid.

So after a whopping amount of things that can go wrong just in the initial stages of an R12, we can finally talk about the actual R12 Implementation in terms of analyzing, designing, building, testing, new functionality, etc. It’s amazing so much can go wrong BEFORE you even start……..here’s a previous Project Manager that was running an R12…….(he’s the guy 3rd from the right, the one 2nd from the right did the R11 Upgrade and the one on the right did the initial ERP implementation….isn’t Oracle ERP pretty unforgiving project wise……)

Here’s a few things on Implementation to help you lose sleep.

Applications do not provide a good fit to Business Requirements

Subsequent Analysis during the Project drives Significant Change

Not possible to achieve a consistent Business Process across all units

Some modules are largely custom built

Large number of customizations

Undocumented customizations

Lack of development standards

Information is confidential

System Integration is complex

System integration is high risk

System integrates with non Oracle technology

Complex Oracle Application Configuration Management

Complex custom development configuration management

Poor Technical Designs

Customizations are delayed or lacks resources in Development Team

Functional Designs are late or incomplete

Technical Designs are late or incomplete

Patch Management is poor

Complex Setup

Testing misses requirements or scenarios

Testing is incomplete by Project Team

Testing is incomplete by Business Team

Testing is delayed or lacks resources for Project Team

Testing is delayed or lacks resource for Business Team

Testing integration is delayed by dependencies

Legacy data is of poor quality

Legacy Data requires data cleansing

Legacy Data has complex structure

Legacy data involved high data volumes

Changes in Legacy System required

Oracle does not have interfaces to support conversion

Oracle uses complex API’s for conversion

Ignore the Legacy Conversion stuff. I use this spreadsheet for more than just R12 Upgrades…..

Now if you’ve ignored the advice to aim for a R12 Technical upgrade, then you may find yourself knee deep in very long, difficult and time consuming business requirements, business change, etc. On an R12 Upgrade, trust me just don’t go there. Keep it clearly 100% out of the scope of the project. It can wipe out your project unless absolutely forced to. Which brings us nicely on to R12 Oracle Payments.

Payments module is a re-implementation. This will take some adjusting as it looks nothing like the R11. With this and other new features, you run the risk of not fitting to business requirements, or wanting new processes or wanting changes. The problem is if you show the new stuff (and with Payments you have to….) you could have significant change in your project scope and therefore significant risk. If you don’t show the new stuff you are going to be accused of not showing it, not training users, not taking advantage of the “amazing” new functionality, etc. So if you show it, you are screwed. If you don’t you are screwed. For me R12 is complex enough to pull off without adding anything other than like for like. New functionality for us could wait (and it did in 99% of cases, despite some fairly tough fights).

Of course if you are rolling out R12 across multiple countries, then it’s even worse to agree all the new functionality, legislative stuff, etc.

Then you get into your biggest nightmare of all. Customizations. Imagine someone having a decade to write customizations on R11. Imagine no discipline, documentation or standards. And you’ve got to upgrade something that you have no clue what it is you actually have to upgrade. Talk about a Pandora’s box……Or worse does anyone remember the Hellraiser horror 80’s movie flick with the box that brought out the demons…….Do you really want to open that R12 box??????

When you open an R12 Upgrade project, you are opening a Pandora’s box unless you did an advance study. Or worse you could be the guy hearing the doorbell, going to his door and seeing a burning brown paper bag with R12 written on it on his doorstep. You go to stamp it out and find something VERY unpleasant on your feet that your company that implemented your ERP left you with but didn’t tell you…….Is your current R11 a brown paper bag with a pile of crap inside waiting for you to stamp on it with your R12 Upgrade?

One single customization missed can cause a catastrophic failure. Funnily enough I saw one Upgrade where they had account generators in the hooks of a standard package for Costing. Now yes, this was supported, but totally undocumented. Funny that generated a pile of entries to General Ledger that took weeks to figure out and calculate the reversals to, after the code was fixed post go-live. So the message is, it isn’t even enough to look at the obvious custom code. What about the standard packages that support hooks? What about overwriting standard code in packages, forms, reports and other customs? What about the change of seeded menus? None of this should happen, but don’t count that it hasn’t.

If you don’t have a customization register before you start you are in very serious trouble. You are going to miss stuff, and that one thing you miss could wipe you out in Production.

Also do you have any idea what R12 will do to your customizations. We saw our Payment customizations wiped out and need completely rewritten. (BI Publisher). Any workflow changes? They’ll need to be redone. Any Self Service in the old MOD PL*SQL? They’ll need to be redone. Any interfaces to GL or reports or anything else? They’ll need to be fixed as 80% of our stuff stopped working. Small changes on a lot of customizations add up to a lot of work, time and risk that can seriously delay your schedules. Did you know there have been very significant changes to the data models between R11 and R12 in certain areas……all that will cause your existing R11 customizations to crash and burn……

If you are doing an R12 Upgrade, you’ll have frequent clones. If you don’t encrypt data you could be incurring serious risk as all those outside resources can see vast amounts of confidential information. (See R12 Security and the Auditors from Mars article to scare you some more).

From a Systems Integration viewpoint, watch out that your R12 Upgrade doesn’t just focus on Oracle ERP. When we did the upgrade, we had Legacy, email , Web, external 3rd parties (Citibank and others), Health Insurers, etc. Now us Oracle guys like to think ERP is the center of the universe. Sometimes it is, sometimes it isn’t. But your typical R12 (or R11) can have interfaces, data extractions, data imports, links to spreadsheets, etc that can number many, many hundreds of integration points. Are you aware of what they are and if not how the hell are you going to test and make sure everything works day 1? Think about missing the testing on say Payroll interface to the Bank. That would be a nice one to find out doesn’t work on say, Payroll day for 50,000 employees. Or perhaps you forgot to test the interface to your planning system that orders the parts to build the products you make, which would stop your production line for 5 days, cost your copy millions of dollars and definitely get you on the “skull decoration” shown at the start of the article in the Boardroom or hanging on your managers wall….. . Hope you sleep well thinking of your integration points to ERP and the catastrophic damage it does if you get it wrong. I bet you that your one of these people that DOESN”T EVEN HAVE an architecture diagram of your ERP integration points. And your company spent like what 50 Million on that system with your integrator and you didn’t even get a coffee mat Integration and Interface Architecture chart????? You were ripped off and it may come back to haunt you…….

By the way, speaking of integration, does all the tech stack count? Just think, adding Forms 10G, Reports 10G, JDeveloper 10G, downloading the extensions for OA Framework, your Discoverer upgraded, BI Publisher. I guess not so much integration as getting the absolute basic tools to even start your R12, as all of these change between R11…….so you’d better sort that before you hire all those developers to upgrade your customizations…..

There will be a need to update or add new functional designs unless you are really a vanilla site. We did a lot of work again around Payments. Will those designs be decent and cover the requirements? Will they be reviewed in a timely manner? Will they be signed off? Keep this to a minimum (no new functionality, that’s gold plating an R12 Project) and you’ll avoid this risk.

A lot of companies also do MD.070 Technical Specifications. I’m not a fan of these to be honest as they have too much info, take too much time, are subject to not getting updated so become quickly outdated, and are largely done by many to tick the box. If you are doing these, make sure the quality and approach of customizations is correct for R12. (And don’t build what R12 provides – that’s one area I would add to an R12 to replace customizations with standard functionality where possible. We took out a raft of custom modifications which are the worst types of customs and vastly reduced our risk both to our upgrade and future upgrades/patches).

So I saw a movie with my kid. It was called The Hole (and it was in 3D so the ticket price is like double !!!!) Now this was a PG13 which meant that it shouldn’t have been that scarey, except it scared the crap out of my kid. More to the point it scared the crap out of me even more. So what’s it about? The hole generates your darkest fears and then these monsters come out. And what’s my darkest R12 fear? It’s about managing all of the configurations on an R12 Upgrade Project – that is my worst nightmare. The Hole gets a 7 out of 10 on the movie review. Well worth seeing, but if your kids get scared easy, approach with caution.

Now managing all your environments, patches, setup and customizations is a truly epic and very complex task. Really someone could write a book on this by itself, but the point is, you lose track of this and you lose track of your project. The lack of source control, the lack of BR.100’s, the lack of configuration management spreadsheets (or software) will cause your project to descend into chaos, especially as your ERP clones spiral from a development instance, to a sandbox, to a project team test instance, to a UAT instance, to a patch instance, to a DBA instance, to an instance with a DMZ, to an instance just for one module as it’s getting piles of patches and needs isolated……..your configuration management isn’t about a single instance. We had 10 instances at one point during the upgrade…….Or to put it another way our R12 had over 600 Functional Setups between R11 and R12, dozens of patches, thousands of test scripts all requiring data/setup to be in exact scenarios and thousands of customization files, loaders, packages, etc……Just curious on how exactly you are going to manage that………..? Are you one of these people that manage this type of thing on the backs of bits of paper or a cigarette packet? If so you’ll need to become a chain smoker as you’ll need a mountain of cigarette packets to keep track of all your configurations items on an R12 Upgrade project……

What I always find most worrying on an upgrade is playing the Oracle Patch Lottery. This is cool. You’ve got your R12 say Test environment all setup. You’ve got your thousand customization files, 600 setups and all the test data ready and you apply a patch, having no idea what that patch changes. And it trashes your entire environment, wipes out your 80% test completion and trashes the modules down-line of your testing. Every patch you apply is a roll of the dice unless you very carefully manage each and every patch. So when you apply a patch during your upgrade and have no clue/plan/analysis of that patch, the question is, do you feel lucky punk?

Testing is full of pitfalls on any upgrade. Have you got all the scenarios covered? Are your test scripts complete? Is your test data valid? Have you covered month end testing? Have you covered year end testing? Have you covered integration testing? Did you try the Web ADI? What about tools like Discoverer? Have you covered testing to and from Oracle and 3rd party companies that you barely even talk to (and met in 2007 when you put in your interface to their products which run offsite – think bank interfaces). Does your new patch invalidate Test Scenario 220, 221, 222 all the way through to 230 for Payables and therefore invalidate your testing across 80% of Payments? Or what if you apply a patch on TCA? Does it hit PO, AP and Payments? Or if you apply your customization, what have you invalidated? Will the auditors pick you up for claiming a sign off from the user, when they tested on a completely different configuration of custom, patch, etc? Are you sure your environments are all the same? Or do some differ in terms of setup/customizations/etc? If so, then how valid is your testing?

Here’s another great one to keep you awake. So all your systems around ERP. I bet you one joker that owns for instance your Legacy planning system or say your Payment middleware is planning to upgrade at exactly the most inconvenient time for your R12. Now you’ve always hated that guy and he’s always hated you. I mean is he doing this at the most inconvenient time in your R12 Upgrade without telling you just to screw you over yet again to get that next promotion over you ? If your not aware of everything you need in your R12 Upgrade and everything in your company’s universe (planned upgrades, planned business initiatives, politics, etc) that can screw you over, I need to ask, why did they even give you the R12 upgrade to do?????

Even if you get all this right, does your team have enough time to thoroughly test? Did you properly track all the bugs? Are you monitoring the bugs? Are you monitoring the Service Requests? What happens if Oracle decides to give a RUP (Rollup Patch) for you in your UAT and refuses to provide a single patch? Do you also take that latest security patch? Did you know that security patches have been known to screw up the Applications…….

It’s also interesting that an R12 DOES have data migration. Think about all those scripts running for the upgrade from Oracle Corporation. They are re-arranging your ERP data pretty substantially in some areas. What if the standard upgrade scripts for changing the structure of the Bank data (to TCA) goes wrong or cause problems? Think that won’t happen? Have a look at our Risk Register for R12………Or did you know you can run a script to convert your previous data to Subledger Accounting? Did you know R12 actually does some fairly heavy data conversion for SLA as standard? Are you confident it will all work with no need to reconcile? Are you confident that the new way that R12 handles accruals will work, as you actually have to run programs to build all the Accrual data………If so have a look at our Risk Register…..we didn’t spend 4 months reconciling and talking to the guy that had written this (in ironically Redwood shores…..those conference calls time-wise between Asia and California, weren’t easy) and apply multiple patches because it was working first time or second time or third time or fourth time……..Again have a look at the risk register from us. Or how about your AP data being converted? Again R12 does some very serious data conversion in the background. Try running the AP Trial Balance and see if you can reconcile…I didn’t spend two months just trying to talk to the lovely girl at Oracle Support to get a date from her….there honestly was a problem with the AP Trial Balance although I did close the SR immediately once it was transferred to a guy……..And did you know all your supplier data is converted by Oracle’s Upgrade process to TCA……..

The good news is that newer versions of R12 are a lot better, largely thanks to “suckers” like me that spent months with Oracle making all of the above data conversions that the standard R12 does in the background work. Of course until you check and reconcile all this for yourself, how confident do you really feel trusting some guy on a blog that claims to be a prophet yet never picks decents stocks………?

On top of all that resource management across your R12 will be the usual nightmare. Competing projects, production issues and users screaming for enhancements of R11. Key Project People walking off to lucrative new contracts and positions. Lack of R12 experience in the market. The time delays finding and hiring people to replace those that walk off. Dealing with unexpected issues that you didn’t plan for that have a catastrophic effect on your resourcing across the project (try cleaning up 3,000 banks when you find out what Oracle does to these during an R12 Upgrade….)

We’ve pretty much covered the testing phase above but just a quick recap to really scare you…..

Testing misses requirements or scenarios

Testing is incomplete by Project Team

Testing is incomplete by Business Team

Testing is delayed or lacks resources for Project Team

Testing is delayed or lacks resource for Business Team

Testing integration is delayed by dependencies

Test Plans are poor

Test Scripts are poor

Reconciliation of Ledgers and Subledgers is not carried out

Test Coverage is poor

Numerous Patches are required disrupting Test Cycle

All of the above can wreck an R12 Upgrade. It’s easy to miss key scenarios. Test scripts can run into thousands. Integration testing can have hundreds of points including external parties making it very complex indeed. User testing is beyond your control in many respects, especially for those that see no need for an R12 Upgrade. Reconciliation of ledgers and data that R12 has converted in the background is a significant task. Patches create havoc on internal test cycles and formal UAT. Remember you can’t just get each group of users to test each module. Each module talks to the other modules. Data from one module flows to the other for the next scenarios. PO needs Inventory. PO feeds the receipts. AP matches on PO receipts which then feeds to assets. IRecruitment flows to HRMS which then trigger hiring which then triggers Payroll which then relies on Advanced Benefits which then drives Self Service. The beauty and wonder of a fully integrated system makes testing co-ordination far from a stand alone list of test plans for each module………Managing an R12 in terms of testing is an extremely complex and challenging endeavor. Try doing that across 45 business areas and 27 countries, each speaking a different language……What’s Halloween in Spanish……??????

With an R12 Upgrade, vendor management is unavoidable…..

Software has significant patches

Software purchased is unstable

Hardware Vendor provides poor support

Software Vendor provides poor support

Poor quality in Product (bugs)

Software purchased is recently released

You are going to spend a significant amount of time dealing with Oracle Support. Depending on who you get on your Service Request can really screw you up big-time, as quality varies significantly. Get the right person (such as the lovely Indian girl we got on our Trial Balance problem, ahhhhhhhh…….) and you could get an immediate response. Get the wrong person and you could see weeks or months wasted, with the “apply this next rollup patch, we’re sure one of the thousand+ fixes will fix your minor problem and retest your entire R12 again” attitude.

R12 was highly unstable in 2008. 2009 saw it got better. By now reports are it is solid……….but I wouldn’t make that assumption, until I had actually implemented it and got my UAT signed off 100%.

When you implement R12 you may need to upgrade your database. (most will). When you upgrade your database you may need to upgrade your Operating System. When you upgrade to R12 you may upgrade or replace your servers, depending on where they are in their lifecycle. Now companies don’t generally have massive dealings with their hardware vendors (as Unix/Linux is incredibly reliable). But during an upgrade that relaxed relationship is going to be potentially very heavily tested. Question is, can the vendor provide strong support, resolve issues quickly and get you safely onto a server that is the very foundations of your business? If you fail to install the correct O/S plus interoperability patches you could see your business crippled…….It’s funny that the Infrastructure Manager that you were fighting with two weeks ago, kind of holds your entire career in the balance and you don’t really know anything about Servers, questions to ask, etc so are 100% dependent on him for your project to succeed.

Just wondering if your R12 needs more resources in terms of CPU, disk, memory, etc? Have you checked? It would be a shame if the server collapsed on day 1 of go-live.

Lastly the million dollar question. Oracle is on R12.1.3 at time of writing. Do you implement this version (and be the one to find the bugs as no one else is using it) or do you go for say 12.1.1. At this stage of the R12 lifecycle I’d go for R12.1.3 but then again you will get some additional pain of being one of the first. Obviously by the time you get to UAT, 12.1.4 will be out……….then do you jump to that or perhaps apply that new ATG patch or a RUP Patch that has come out as it has some really needed functionality……..All of these create massive risks not just for the stability of the project but also your budget, timescales and resourcing. Larry Ellison is a little devil….always releasing new ATG, Security or RUP Patches just at key points in your project, just to tempt you and cause major indecision in your mind. Question is, is that patch a poisoned apple that will drop your project at a key time??????

All hail Larry Ellison. The True and Only Oracle Prophet !!!!!

Happy Halloween. The clock is ticking down to the midnight hour. Before Halloween is over, I will scare you. The witching hour is coming with Transitioning from your R11 to R12, the actual cutover and post go-live, with all the reference material and a prebuilt R12 Risk Spreadsheet as a treat for all the kids in the next article……

Delivered on Halloween !!!!! Enjoy………..

 

See the other Prophecies including Business Intelligence and the Kung Fu Dragons of Wudang at

https://oracleprophet.wordpress.com

Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt 1)

October 20, 2010

Darkness falls across the land
The midnight hour is close at hand
Creatures crawl in search of blood
To terrorize the neighborhood

Now if you don’t know that song intro, shame on you. It’s from one of the biggest hits of all time – Michael Jackson’s thriller. A very appropriate one for Halloween I’m sure you’ll agree.

So first movie recommendation of the day. This would be “This is It”. There’s something very poignant about seeing him performing in what was his final curtain call.

Now Halloween has always been a time to scare the crap out of people. Kids dress up as Ghouls, Vampires and the like and go “trick or treating”, sometimes with some very nasty tricks indeed. Either the local neighborhood pays the kids with sweets, etc or the kids do something nasty.

Back when I was a kid, there was one particularly nasty kid who didn’t bother with the treat stuff. He figured he could go beat up the other kids at the end of the evening and steal their sweets. This saved him the effort allowing him to do his favorite trick of all.

This kid had a real nasty trick. What this kid would do would be to get some dog crap, put it in a brown paper bag and then go place it on someone’s doorstep. He’d quickly set the bag on fire, ring the doorbell and wait for someone to answer.

Now what is amazing is that the immediate reaction of an adult to a small burning paper bag (even if they are not wearing shoes) is to try to stamp out the fire with their feet. Irrespective of wearing shoes or not, this had some pretty unpleasant results for the poor neighbor at the end of this “trick”.
Before we start, let’s put our other movie recommendation out. It’s the original Halloween movie. Now I saw this when I was 13 and it scared the absolute crap out of me. Well worth getting the video. A classic 70’s horror flick for Halloween, but definitely not for the kids. (Don’t see the more recent one, it sucks).

So seeing as it’s Halloween, I should scare the crap out of everyone reading this right? So everyone’s moving to an R12 Upgrade in 2011. So what better way to scare you than tell you how horribly wrong it can all go on an R12 Upgrade !!!

So R11 Oracle Apps Premier Support is ending in November. That means you are screwed. OK, not completely (but Oracle will be putting the pressure on you and telling you that you are) but seriously you should be looking at R12 now.

Now let me tell you that an R12 Upgrade will be the biggest, toughest upgrade of ERP that you will EVER have done. It can go disastrously wrong. You could corrupt your data, you could have major functional failures dropping your business to it’s knees. Or your entire ERP could disintegrate and you could lose your job because at the end of the day your boss will need a sacrificial lamb and you seem to fit the bill.

Are you starting to get scared? You will be by the time you read this article, because it contains a very long list of every single thing that I can possibly think of that can wreck your R12 upgrade.

So let’s start at the very beginning. Getting the Project Started. If you get this wrong, it’s probably not worth bothering about the rest of the project. Here’s a short list of things to worry about. (Now the purists will say these are not risks). For me I like to have everything that can go wrong on one spreadsheet. Call it risk, call it a list, doesn’t bother me, because I can quickly mentally check each month how things are going and make sure I haven’t forgotten anything. It also gives me a nice risk score and lets me manage that to hopefully give the project a fighting chance of delivery.

Let’s start with the Project Management risks and everything that can go wrong here:

Weak Business Commitment to Project

Funding is not secured for Project

Project Decisions are not Timely and On Time

Weak Business Commitment to Change

Review of Project Deliverables and Sign-Off is not timely

Weak Support of Business Case

Scope is not defined clearly or completely

Business process model is not defined clearly

Planning is of poor quality

Project involves large number of departments

Tight Time-Frames, Minimal Slack between Dependent Tasks

Go-Live Date accommodates limited UAT

Project Milestones are not achievable (Project Schedule)

Dependency on other Projects

Project requires complex organizational changes

Contract involves significant cost

Project involves time-critical Procurement

Significant Scope creep

Gold Plating by Implementation Team

Project Budget is not sufficient with Current Progress

Project Resources are no sufficient with Current Progress

Project schedule is delayed

Project Cost is overrunning

Product does not meet client expectations

The very first thing you need to do is get your most knowledgeable and trusted people in a room and do proper planning, taking into account your organizations political landscape, resources, expected funding and other commitments. If you fail to plan, you plan to fail. Your planning must give ample slack time for delays in UAT, delays in decisions, procurement delays, bugs, resource problems (only 5% of people have done R12. The other 95% of people will do R12 in 2011, so can you get resources??????)

The most important part of any project is to get high level buy-in and commitment from the senior people across your organization. From a resource viewpoint an R12 Upgrade requires intense Business and IT Commitment. Fail to get every area of Business and IT involved and agreed, then you are screwed 100% come UAT, if not even earlier. You need to make it clear when you need the business, what you need them for, how long and how many resources right at the onset of the Project Initiation. IT resources need to be equally onboard.

Securing realistic and adequate funding is critical to an R12. It is bigger. It is more complex. It does take longer. An R10 to R11 or a 11.0.3 to say 11.5.10 is NOT the same as an R12. Build in a decent contingency.

An R12 is not a basic like for like Technical Upgrade. You should ideally keep R12 as a pure technical upgrade and not a re-implementation, unless there are very good reasons, otherwise your risks and costs will escalate severely. However even doing a technical upgrade involves a pile of new modules and new functionality (albeit some can be deferred). But you’d better be prepared to implement Subledger Accounting, EBiz Tax, TCA (changes in Banks), Payments and Self Service (new functionality and removal of MOD PL*SQL). All of these will need user review, functional designs (if customizations on top), setup and decisions, as well as time for your own team to learn the implications.  This can lead to multiple risks including your own team wanting to add “cool functionality” or users not making timely decisions or being prepared to adopt new processes to account for changes in say TCA or Payments.

If you are dealing with contracts for consultants, firms (outsourcing or otherwise) or need hardware, etc, be prepared for the heavy risks of delays in Procurement. IT guys often overlook that it may take a while, despite having the cash to get resources onsite or hardware commissioned, etc.

Talking of resources this should be viewed as a serious risk throughout your project. R12 resources are still less rather than more common.

As well as all of the above, you still have to worry about all the usual Project Management stuff of quality, schedule, cost, delays, budget, etc. And think about it, we’ve not even got to the actual project yet…….

So the very first thing you need to do (before your project team arrives) or is internally deployed is to make sure they actually have an R12 Instance to work on…..

Infrastructure is often overlooked for projects. Unfortunately this can cripple the project team (and therefore your project) very rapidly indeed.

Network Speed

Performance Issues on Project hardware

Performance on Production may not be acceptable Go-Live

Deployment involves new hardware

Hardware is not sized correctly

Database is not sized correctly (do you have enough disk space for R12)

Hardware is no available for Project Use

Backups are not available for Projects

Project Environments not available

Network availability is poor (if teams distributed)

DMZ Configuration is not available

Architecture is new

R12 Instances are not available

Space for Project Instances is not available

Missing interoperability patches

Missing patches to Operating System

If you are going R12, either you have a lot of servers around spare, or you are going to need to buy (or lease) additional hardware. Otherwise what is your team going to use? Ten Consultants onsite doing nothing is some serious cash burn.

Then you have to have a plan for environments needed, dates needed, who prepares them, who sets them up, etc. DBA’s will shout at you if you need environments the next day. And what if there is no space? Did you check the disk space as part of your project? Your average R12 environment requires considerably more space than your R11.

During an R12 Upgrade you’ll need a pile more databases. We used around 8 additional (and we still needed all our R11 Development, Patch, Test as R11 Production work could not be completely halted).

Now just think when you finally deliver that R12, and it crumbles on day 1 into a performance black hole. Did anyone bother to do proper sizing either on CPU or disk or Memory? If not, try www.jobserve.com when it all fails on day 1.

Even when you are starting the project, insufficient hardware will kill you.

Did anyone ask for backups of your project databases? It would be a real shame to see your R12 go down and no backup, losing months of effort.

If your Production is a DMZ enabled configuration, I hope that someone is also going to emulate this for some of your R12 databases as you go through the project.

 

So lets get to the one that usually gives R12 Project Managers nightmares. Resourcing.

Poor IT Staff Skills – Functional

Poor IT Staff Skills – Business

Poor IT Staff Skills – Quality Assurance

Poor IT Staff Skills – Java

Poor IT Staff Skills – Oracle Development

Poor IT Staff Skills – ERP

Poor IT Staff Skills – Modules Implemented (or new modules)

Poor IT Staff Skills – Technology

Poor IT Staff Skills – Unix/Linux

Poor IT Staff Skills – DBA

Poor IT Staff Skills – Project Management

Poor IT Staff Skills – Senior IT Management

Poor IT Project Team Rating – Skills

Poor Business Project Team Rating – Skills

User Department Staff not available for Project

IT Staff not available for Project

Poor User Department Staff Skills

Poor or lack of Project Team Lead (s) per Functional Stream

Experience of Business Area is Low

Project has competing projects for Resources

Custom Development relies on 3rd party

Project Manager is inexperienced in Upgrades/ERP

Legacy Staff unavailable for Data Conversions

IT Staff skills – training required

Low retention of project  team during implementation

Low availability of Trained and Experienced reasonable cost alternate resources externally

No IT Staff available for existing Production Support

Oracle Support provides poor support

Have you even contemplated the vast variety of resources you are going to need to pull off an R12 Upgrade? Have a look above if you think you can use a couple of consultants over a few months.

Will your internal IT guys be good enough? Do they know the R12 functionality? Probably not. Will you train them? Probably not. So how can they deliver? And even if you want to train them, can you get the appropriate training completed before they actually do the upgrade. Training courses from Oracle can be hard to come by, especially on some of the niche modules and require booking way in advance.

Do your IT guys REALLY know the business? If not are you sure you’ll catch every scenario and every nuance during implementation.

Do you have decent Oracle skills? Do you have decent Java skills? Do you have decent Oracle ERP Technical people? If not, you are going to need to ramp up or get someone that does to do your upgrade (unless your complete vanilla ERP).

Do you have anyone that knows all the new R12 modules, such as EBiz Tax, Subledger Accounting, Payments, etc? These are not optional modules (except Payments, but if your running Payables in R11, then Payments becomes essential in R12 and it’s a new module). Most companies will require to have knowledge on these and something like Payments has heavy changes between R11 and R12 – indeed it is a rewrite between R11 and R12.

New technology components are included across the board. Are your people ready and capable?

Your typical upgrade will require some pretty heavy Unix or Linux skills. You might have to upgrade the O/S or you may take the opportunity to switch to cheaper, less proprietary hardware or Linux. Do you have any clue what’s involved? (we did this and yes it saved us a pile of cash for the future, but it ain’t easy and is littered with risks).

Did you tell the Infrastructure guys you’d be needing a full time DBA for the R12; to clone R11 Production; to work on the upgrade process; to write detailed upgrade documents that give step by step guides; to optimize the process so that it can fit into a long weekend on a huge global database? To apply all the patches? You probably thought your DBA’s sit around doing nothing and can do all this at the drop of a hat…….Count on the initial upgrade where your DBA’s find out the nuances of R12 to take quite a chunk of initial time and plan that into your environments and people coming onboard project wise.

R12 Project Management. Words fail me on this one. If you are not an ERP person don’t even think of doing an R12 without a strong R12 Project Manager, ideally someone that has done R12 Upgrades. It’s horrendously complex, extremely technical and the risks are catastrophic if you get it wrong. (I can be contacted on the following number for lucrative job offers……… 🙂   )

Are the Senior Management (both in IT and Business) in full support of both you and the R12 Upgrade, or are they ready to crucify you at the first delay? R12 will have delays, it will throw horrendous curveballs in terms of bugs and other complexities. Just make sure everyone is on your side politically before you start. The guy below thought everyone was his friend, until he forgot to order the hardware in time……

Now this is a weird one, but how well do your Business know the Business and Oracle ERP? Look for the weak spots here before you start. If a Business Unit has a lot of new people or inexperienced people or are anti-Oracle ERP (and some are), how are you going to get them to assist (or write) Test Plans and Test Scripts and do actual testing. Now if your IT and Consultants don’t test well (and they are NOT business experts) the one hope of catching everything before Go-Live is the Business. I never build 100% dependency on the Business for testing, but if your IT and Business guys can’t be relied upon for testing, you’ve got a perfect storm.

The other thing to check is everybody’s schedules. We’re not talking dinner dates, or when they plan to see a movie. Are the Business available for an R12 (which won’t add vast value to be honest) or do they have a planned move to say IFRS planned from June to December 2011. If so you won’t get a look-in resource wise for UAT and your project will be dead in the water.

Same goes for IT. If they have some major high profile initiatives, your R12 is bottom of the list, as in the priority list, it’s just one of these unpleasant tasks that needs to be done.

Watch out that your R12 Upgrade doesn’t just focus on Oracle ERP resources. When we did the upgrade, we had Legacy guys, email guys, Web guys, external 3rd parties (Citibank and others), Health Insurers, etc. Miss those guys out and who can help you complete your UAT?

What’s amazing about an R12 Upgrade is that you’ll need exactly the same resources that are getting used for your R11 Maintenance and Support. That leads to automatic conflict and resource contention and delays (and when it comes to Production problems or R12 Upgrade, you’ll lose that argument every single time). So ensure you augment your team to not be 100% reliant on goodwill.

If you are getting a company to do it for you, be careful. Have they done an R12? If yes, will they give you resources that have ACTUALLY DONE AN R12? Most outsourcing or consulting companies might have done R12’s, but they are as keen as anyone else to stack your project with their guys who can then learn R12 at your expense. Nice how they’ll happily charge you for the privilege of training their people…….Interview all people, before they deploy onsite. Hire an independent highly knowledgeable freelancer Consultant to fight your corner and keep the vendor on the right track, working in your interests, not the vendors.

Do you know the effort involved in training people for R12? The IT guys, the Business guys, etc? If not, start to worry.

The great thing about an R12 Upgrade is that your team (and maybe your consultants or outsourcers) will all get fantastic R12 skills, in the midst of the hell of an R12 project. Trust me that’s a great way to learn. Of course all those companies that have not done R12 (and that’s 95% worldwide) will be looking and admiring your very skilled people. And your loyal people that you’ve spent time and money training in R12, will of course be looking for that next payrise with their new found skills. Losing key resources (say for instance your Payments expert) is your worst nightmare. Suddenly one single module can derail your entire R12 Upgrade. If you think this is the stuff of nightmares, well during our R12 back in 2008/2009 we lost our Payments Consultant (one of the few in Asia at the time). Luckily we had him shadowed throughout the project by another staff. The demand for R12 skills will hugely increase in 2011 and your guys will be headhunted or tempted away. Just be ready. I am sure the resigning employee will feel your pain as your project is now doomed, in the same way you felt his pain when you froze his pay last year, as part of a corporate wide pay freeze…….

Our final resource issue is to prepare for the maelstrom of Production support hell. Just make sure your well staffed and haven’t let the entire project team go, just before you get deluged with Production issues. (and yes you should start thinking about this not at the end of your project when everyone is jumping ship as the project for them is over, but at the very start of your project). It’s funny as R12 projects get to go-live, most of the work is done, and you start letting consultants/contractors go. Do you have any idea the pyschology in play with the others that you expect to see remaining to do the support work and provide critical support for a few months after go-live? Why should they stay when they can get a far longer, far higher paying and far easier (i.e. not pressured go-live support) contract with their fantastic R12 skills you gave them? Sleep on that one, assuming you can still sleep……..

And I hope you’ve got a full communication plan going on – an upgrade can take time and if you are not informing and working with the users throughout…..don’t expect them to be enthusiastic about putting large amounts of time into your UAT. Of course if you run workshops and demos and mini training throughout your R12 you’ll have a whole lot more acceptance during UAT and go-live…….

And finally Oracle Support………..now R12 should be stable now, but back in the early days, we had these jokers on the phone daily. We even had a few removed from our support calls because progress was atrocious. Are you ready to deal with Oracle Support? Do you know how to escalate calls effectively? Do you know how to work around Oracle stalling for time or sending you on a wild goose chase? If not hire someone that does. Good ERP people with strong experience are essential for your R12. It’s funny but I bet you didn’t even think about informing Oracle Support of your upgrade, which is a shame as they might have given you a Critical Account Manager that could have helped you push all your Service Requests….

And that’s all for now folks. Part 2 will cover more in ghost, ghouls, implementation, transition and onto go-live, together with a whole pile of great references, covering fantastic presentations from experts in R12 and great Oracle Open World links of R12 Presentations there, together with a heavily researched bunch of Metalink articles around R12.

I’ll be scaring you very shortly with Part 2 before the full moon of halloween is over.

The other blogs can be found on https://oracleprophet.wordpress.com

Enjoy.

Footnote:

The great graphics were the work of:

WWW.MYSPACEGRAPHICSANDANIMATIONS.COM

R12 Security and the Auditors from Mars

October 9, 2010

Let’s face it auditors are not from this planet. Rather like the famous movie War of the Worlds, they bring fear and terror wherever they go. Are they from Mars? Probably, although the one thing certain about auditors is the uncertainty they bring with them, together with the general fear and terror they instill.

I thought the picture below was a rather appropriate one of a hand reaching out with the entire planet in its grip…..when the auditors are around you sometimes get the feeling that your entire company is under the grip of these terrifying Auditors from Mars…..

 

Now this column seems almost like a recommendation for movies these days, so if you’ve not seen War of the Worlds, get it on DVD. It’s a great movie.

Now as someone that had not one, not two, not three but FOUR different companies looking at my R12 Upgrade Project (AND FINDING NOTHING!!!!!) I have to say I’ve learned to deal with them pretty well over time. Our first R12 audit was from our internal audit guys, who wanted to learn R12. It’s important for internal staff to learn, so we were happy to help. We then had an audit from E&Y as our R12 Upgrade was part of a large Program…….we then had a pre-Attestation audit from PWC to make sure the project was OK from an attestation viewpoint. Finally we had another guy from a large Consulting Organization looking at it and never quite figured out why. (Guess they also wanted to learn how to do an R12 Upgrade). Amusing that these audits cost 30% of the actual cost of the upgrade and ended up commending the R12 Upgrade 🙂 One audit surveyed random users with 20 out of 21 commending the team, with the other neutral. It was a lot of work providing the auditors everything they needed, but I have to ask – 4 audits???????

Now here’s a question for you. If you saw an auditor drowning in a river, would you run to get the life belt 30 feet away from the river, or run to your car 50 feet away to get your video camera? Alternatively if you’re a Project Management Professional (PMP) you’d have planned ahead and kept a bag with bricks in your car beside your video camera. As you throw the bag of bricks the auditor is thinking “What a nice guy throwing a bag to keep me afloat”, whilst you are thinking “What a sucker, that’ll help him sink……”

 

I have to point out you should not take this blog too seriously, as the above may get you into some serious trouble, although with a good lawyer you’ll probably get off with a small fine as it could be seen as justifiable or reasonable……. You may be able to even make some money back to pay for your legal fees from the video on America’s favorite home movies show.

There is no question that Auditors are from Mars, causing fear and terror wherever they go………well recently they struck on an area that probably too many people just don’t take seriously enough or don’t even realize is their responsibility. This area is Data Security within your ERP regarding access of your Support Team. Now what is worse is that when auditors do raise this they generally have a very good point and your best interests at heart.

So we’re almost going back to our Koan stuff (see the Zen column). How do you do support Oracle ERP Production when your support team can’t access Production due to information and data security concerns?

That’s a difficult question and I’ve yet to see any really effective, comprehensive solution either in a blog, white paper or from Oracle themselves (at a reasonable price – I had a discussion with Oracle last week and almost fell off my chair – the Oracle Sales guy might as well have had a mask and a gun…….).

So you’ve got your auditors on the one hand, a whole raft of legislation on data security (your legally obliged to follow – it’s not a matter of choice) and a very limited IT budget. So how do you square the circle to keep every party happy?

So look at your typical ERP these days. It’s a monolithic, super system:

  • Financial performance – if you’re a listed company that causes major data security issues to prevent internal staff dealing in shares, because they know you had a record quarter looking at your General Ledger before it was announced or worse short your shares if they know there is bad news coming. That’s real company loyalty for you.
  • It probably has all your HR information in there, including performance appraisals (some none too pleasant quotes on how a staff was involved in some questionable activity), potentially medical information, salary of course and a whole lot more.
  • You might have Procurement information, so a huge amount of confidential contracts also.
  • And most ERP sites run Payables and Payments. Do you know the potential fines you can get if you lose bank information? Or worse if you lost your Supplier bank information and the damage that could do to your company’s reputation, not to mention legal liability.
  • Ever worked out that your Accounts Receivables is a very good indication of a customer’s credit rating? After all, you kind of know who’s got cash flow problems……And the Bank Account information is everywhere. Are you really looking after that securely? Your customers won’t be too impressed if you lost that and would probably never do business with you again.
  • Or are you capturing credit card information as part of say your Order Management/Accounts Receivable process…….Have a look here if you are.

Data Security is an absolute minefield for any IT department responsible for Oracle ERP.

There is just so much legislation on this area that most are blissfully unaware of:

  • Sarbanes Oxley
  • PCI
  • HIPAA
  • European Directives
  • Individual country legislation
  • And a whole lot more…..

This is serious business and can really hit you hard if things go wrong – blissful ignorance is no legal defense. And yet I bet you haven’t reviewed anything from a legal viewpoint on this……

There’s a couple of interesting links at the bottom. Choicepoint had 163,000 customer records compromised. That cost $15 million US Dollars, not to mention reputational damage and cost of implementing proper data security. The data involved? Names, Social Security Numbers, Birth Dates, Employment Information – sounds pretty much like Oracle HRMS right – they got fined for not protecting exactly the type of information you’re holding…..Or how about the California hospital that got hit with $250,000 penalties just for reporting a breach late……

Not in the US, so don’t really care? Have a look at this link. The UK has put in place legislation for fines up to GBP500,000 (equivalent to around US800,000) for data security breaches. Or HSBC, one of the largest banks in the world, that got hit with a fine of almost 5,000,000 US Dollars.

So not in the US, not in the UK, still don’t care. How about the following:

A German company fined 137,500 Euro for asking employees why they were sick and storing the information. If you run absence management, are you potentially collecting scanned medical records and are they properly secured?

Still not convinced? Forrester estimates a cost of $155 Per record for data security breaches. Lose 100,000 records and you’ll be looking at $15.5 Million US Dollars legal liability and probably looking for another job…….I guess that’s finally caught your attention? Legislation has been enacted or is being enacted worldwide. Sooner or later, it’s going to hit your country too.

Now I have to state from the outset this article does not provide all the answers. This is a highly specialized area which goes way beyond my own personal capabilities in this area. So if you’re looking for all the answers, hire a specialist. Actually read hire multiple specialists over a number of years. Security is a massive task to do well and therefore effectively.

The aim of this article is simply to show what we are doing and have done to implement a more robust security around our R12 Oracle ERP that allows us to support the Business, whilst still trying to do our best to be secure.

This purely looks at using roles to restrict database access for support people. This article does nothing more and that has to be stressed at the outset.

Hopefully it can get you at least part of the way there, giving some additional protection without costing a fortune.

 

Step 1 – Find out what access was out there historically

Our starting point was to figure out from an Applications side what access our IT guys had to Production (and note when I am talking IT I am talking ERP Support only in this column. Other areas such as DBA, System Admin, etc are beyond the scope of this article). Luckily we were already very heavily locked down. We don’t have any access to modify any part of Oracle ERP and I’d say that’s a very good thing. It makes me sleep easier at night knowing that there is no way my staff can log in to Production, “cause there’s a really easy way to fix something………”. Every change we do to Production is through a formal turnover to separate groups, fully approved, appropriately segregated and audited. That keeps our Production system safe. (All code is also in source control using Subversion just in case we ever need to rollback a change).

The next step was to see what read-only access was given to Support Teams. Now over the years, many IT systems just grow and grow, access is given for support purposes and no-one really thinks too much about it.  I was relieved to see that the Support Team had no access to Oracle ERP through the front end applications.

I thought I’d throw in a useful note at this point. We have a custom program that reports quarterly on all menus/functions/applications by person. It replaces the standard Oracle System Administrator reports that generate listings the size of War and Peace. This report is reviewed both in IT and Business to ensure that each user has the appropriate access for their job. We also tied this to staff movements within our HRMS System, so that when someone moves, this is highlighted as a potential access control violation and can be assessed before it even happens (using date-tracking in HRMS). The other useful point here is that this quarterly check also ensures complete segregation of duties. Finally by using a custom report and showing dates menus/functions were changed, it vastly reduces the work to check each menu against each data item. Actually a single maximum date at the top of the report means the users only check this and can thereby can confirm no changes from the last review. Not perfect, but keeps our auditors happy. This is a cheap and cheerful approach without buying Oracle’s GRC Suite although I think the GRC Suite looks very interesting indeed.

With our front end applications secure from our Support Team, we then started to look at the database access. Now I know some people say IT Support should have no access, especially the evil DBA Vogons from Vogsphere, who according to the book The Hitchhikers Guide to the Galaxy have “as much sex appeal as a road accident”. On the other hand others say “With no access, I can’t see anything, therefore no support’. We’re back to Koan (See R12 Patching and the Art of Zen) here – How can you fix something you cannot see? (To our DBA friends I heard the new 11G Database Release 2 is self-managing, so exactly what do you guys do???? – that should provoke a few comments 🙂  )

Our support team had read-only access using database roles to Production, but when we looked at the roles created we concluded that these were too broad and needed some serious work. Roles are simply access to database tables/views or other objects given to a specific support person by the evil DBA Vogon group.

Note: For those interested in Vogons, it’s from a movie called Hitchhikers Guide to the Galaxy. Here’s a movie recommendation. Don’t see this as it sucks big-time (although the original TV series and radio show and book were amazing).

Now our next step was pretty simple. Monthly we review database access to Production and with the count of accesses by support staff, we found that many actually just didn’t need access to Production, because the support volumes regarding database queries by support staff were low (or zero). We cut these immediately.

Again doing a review monthly (only takes ten minutes) ensures that as people leave, move job, etc your access controls remain updated and correct. Again this keeps the auditors happy.

At the end of this step, we defined a Security Access Matrix, showing each Database Role against each Support Staff, shown below. This was our first baseline.

  

 

Step 2 – Defining Your Database Security Architecture

Ignore the fancy title. What this means is simple. You need to look at:

  1. Your Support Team Structure
  2. Your Applications
  3. Your Required Access

So let’s take an example. We run a fairly large ERP footprint:

  • HRMS Suite (HRMS, Payroll, Learning, Time & Labor, Self Service, Advanced Benefits, iRecruitment)
  • Financials (General Ledger, Payables, Payments, Fixed Assets, SLA, EBiz Tax, Accounts Receivables)
  • Procurement (Purchasing,  Services, Inventory, iSupplier)
  • Projects
  • A large number of custom modules
  • We’re going to add CRM (hopefully Fusion) to that in 2011.

Our Support Team is organized along classic line of business model (Procurement, Finance, HRMS). It is further segregated so that certain support people provide support for say HRMS/Payroll, others do Self Service, etc.

Now the problem we had was two-fold:

  1. Too many Support people had too much access
  2. The data protection needed to be much stronger

Our approach was simple, but effective. Each role would be along the lines of module or modules required for logical support. So we would set up roles for Procurement, AP/Payments, General Ledger, HRMS, etc.

Now this for us still didn’t get us to where we wanted to be. What was going to be in these roles database object wise?

Our next task was to then define exactly what that role would have access to table and view wise. Your average ERP Module can have several hundred tables (or even more….). Now you can give a blanket access if you want, but I think that’s overkill. It’s hard to audit and hard to police. Your average support person needs access to only a limited number of tables. That is what you SHOULD define.

In addition, obviously there is some cross-cutting of access between modules and tables, so for instance you may have to grant GL_CODE_COMBINATIONS across various roles.

In our final step, to the horror of many, we decided to NOT give access to standard sensitive tables, but instead to build views WITHOUT KEY ELEMENTS. Now a lot of support people are going to scream at this, but I need to ask, if you have a support call, do you really need access to the guy’s date of birth or performance or disciplinary records or even Full Name? I doubt it for 99.99% of cases.

Here are a couple of examples of securing tables into masked views:

Instead of giving access on PER_ALL_PEOPLE_F we removed all information except the Employee Number. No access to Social Security, Performance, Home Phone Numbers or anything else.

Instead of giving access on the Bank Tables, we removed all information and masked the bank account.

Instead of giving access of PO_VENDORS, we removed all information and simply gave the VENDOR_NUMBER.

What you need to be careful of is the other tables that you really wouldn’t expect to hold sensitive information. A prime example of this is the workflow tables.

To secure workflow tables we ensured that each role only had access to relevant workflow ITEM_TYPES. This was a very simple, yet highly effective way to secure this by role.

The great thing about this is that we got our support people to define these roles in a simple spreadsheet. This gets you your buy-in from the very people who would be screaming about cutting access. After all they are now the ones tasked with defining it (obviously subject to reviews within IT and approvals by Business).

We then decided that we’ll make this formalized. This has a lot of benefits:

  1. It provides an agreed document of what a role has access to data-wise
  2. It provides an agreed document between Business and IT on that role/data
  3. It provides a document that can be agreed and signed by all parties
  4. It makes you think about your data and protection of said data in a disciplined manner
  5. It provides the basis for all future access to sensitive data by IT Support Personnel
  6. It provides a clear statement of who has access to what in a transparent, controlled manner
  7. It provides protection from your auditors

We decided to record other details in this spreadsheet. (Each role has a separate spreadsheet).

The spreadsheets had the following columns:

  1. Object Name
  2. Database Owner
  3. Object Type
  4. Business Owner
  5. Description
  6. Object Classification (Custom/Standard or Secure. A Secure Object was a restricted, masked view of a key table)
  7. Data Security Risk (Critical, High, Medium, Low)
  8. IT Review Date

A separate Tab (Security Details) on the Excel Spreadsheet held additional details

  1. Object Name
  2. Base Table
  3. Masked Columns (.e.g. vendor name, bank account, social security number, etc)
  4. Date Approved

The Spreadsheet was split into two sections – data objects and reference objects. (Reference objects are the pre-seeded or setup tables within R12). A further tab showing review actions was included to show that this was a serious, considered exercise to the auditors.

 

 Suddenly, without even realizing it, you’ve just created a very precise, auditable and comprehensive technical security framework for your R12 Oracle ERP that will vastly improve your security of data, whilst still allowing effective support. This also provides a very powerful data classification framework for both you and your users. Even more interesting, by putting a risk score on each object, you can actually see the overall risk access rating of each role, allowing you to make informed decisions on how widely this role is given to support personnel (or not).

If you have knowledgeable, motivated people, this doesn’t take a huge amount of time. I am extremely lucky to have such a great team of people.

Once the roles were defined, signed off and agreed, the DBA Team created the roles/access very easily.

Note that once you have these documents in place and have implemented the framework within your database, these documents (and hence the roles within the database) should be strictly controlled otherwise all your work will have been for nothing……

So after not a huge amount of work we had roles for most applications that were nicely segregated, secured and with all key information masked appropriately. This can be seen in the Security Matrix below:

 

(Note how the General Database Access role was entirely removed once we worked out roles/access. The above is only a sample, not the real resources/roles).

So in a couple of weeks we went to very fine grained security on our Production System, without spending vast sums of either cash or time.

Our overall security had vastly improved for our Support access to Production. Indeed the data had really been masked very effectively yet we still could do effective support, without disclosing any critical data to our IT Support Staff.

 

Step 3 – Defining Your Access Security Processes

Without policies and procedures any security framework will fall to bits pretty quickly. Now this article isn’t really about processes, but here are the key aspects to consider. Security is not a one-time process. It’s a critical ongoing process that needs to have strong controls to keep you secure going forward:

  • When people join our company, we ensure that non-disclosure clauses are signed the very minute they step in the door.
  • Security is not just about securing your database or applications or data once.
  • The Database Security Architecture showing each module/access to tables/views should be agreed and signed off by the Business. It is important that Business understand the access IT has to data in a transparent manner. Too many IT organizations think they own the data. They don’t, they are merely custodians.
  • When new access is required for a Support person, or additional access is required, there must be a strong process to seek authorization from the Business for that access. This process should be formalized, ideally automated and fully auditable. We are currently implementing Oracle’s Identity Management product and intend to integrate the process into this.
  • When a person changes role or position, it is important that any access is immediately discontinued. Linking your process to the HRMS System gets you part of the way there and avoids any embarrassing moments when the auditor asks why a person that was fired still has access to your HRMS database details……..
  • When the Database Security Architecture documents need adjusted to add a new table/view, this must go through a formal change control process, with appropriate sign off by the Business users.
  • The DBA Team should not update any role to add/modify any table/view unless the change has been formally signed off in the Database Security Architecture documents. All requests for changes in the database should go through a formal turnover process which is fully auditable and approved by IT Management and Business Users.
  • A monthly review of Support Personnel/Access/Roles should be done by the person managing the Team. This doesn’t take long and ensures that only those needed access have access. Documenting this process protects you when the auditors role in. (Note the Oracle Database can be configured to automatically email you the roles/people/number of accesses)
  • There is one other interesting area. If you formalize the Data Architecture documents and make that part of every project before go-live, you’ll also ensure that support for go-live is properly thought through and adding new systems does not compromise security going forward.

 Step 4 – Encrypt Non-Production Databases

I’ll cover this in another article (working title is Securing your R12 using Captured Alien Technology from Roswell), but the key point here is that there is no point in protecting your production database, if you then clone and have all the same access in copies of production……..Security is about all your databases, not just Production.

 Step 5 – Securing all Layers

Again, this is beyond the scope of this article, but keep in mind that the best security is a layered approach. Your Oracle ERP security framework must involve all of the following:

  1. Network
  2. Servers (Linux, Unix, etc)
  3. Oracle ERP (across the modules)
  4. Individual Modules (each with their own security models)
  5. Database
  6. Security Patch Policy across all areas

Security is only as strong as the weakest layer…….A couple of the best articles that take a holistically approach are found in the Related Articles Section at the end of this article.

 Step 6 – Future Directions

Our goal in this article was to make others aware of simple steps to improve security for Support Staff supporting Production systems by using role based access at the database level.

However, as we looked more and more on the security, and as we became more aware of the challenges, but also consequences of getting it wrong, we see other opportunities that remain very interesting future initiatives that we will seriously look at.

We’re interested to lock down database access to restricted IP addresses. This will stop password sharing, but also ensure that even if an account is compromised, unless you sit in the person’s chair, you’re still going to have difficulty accessing. Not bullet proof for sure, but it adds another layer in our security.

We’re interested in auditing the database statements of support personnel and perhaps moving database roles/custom built to a more secure and robust solution. It looks like Oracle’s Audit Vault may get us there, but at a heavy price. This can implement roles in a more secure manner, provide full auditing, reporting, restrict data by IP address and a whole lot more.

We do already have data scrambling and encryption on our non-Production databases, along with formalized controls and processes. However as one article points out, the days of custom scripts are coming to an end (or is it just an Oracle sales pitch to get you to buy the products?) and we are seriously looking at a couple of options:

  1. The Oracle Enterprise Manager with Data Masking Pack. This allows you to setup and execute data masking. With Grid Control you can then have central data masking across all Non-Production databases. So yes they are trying to sell you even more, but then again, it’s better and cheaper than a 15M US Dollar fine for non-compliance…..
  2. Oracle Application Management Pack for ERP – This is a pretty broad release that provides functionality for Configuration Management, Application Performance Management and Service Level Management (including related to this article obfuscating sensitive data). This will allow you to clone and scramble data in a single process.

 

 One area we are interested in looking at in the respect of functional changes, setup, etc is the GRC (Governance, Risk and Compliance) module. This provides a lot of the audit capturing and reporting capability as a by-product of the general setup process. More details can be found at the end of this column. We haven’t fully evaluated this product but it does look interesting. Again Oracle has added analytics that could potentially make this a complete, all round solution (subject to our evaluation).

Oracle Identity Management. We are so interested in this product that we actually went out and bought it. This will once fully operational, control access to all our key systems, allowing us to comply with key controls from both auditors and legislation. We intend to use this directly linked to our HRMS system to basically provide all provisioning of user accounts, access, etc. This is linked not only to the triggering HR Process, but also directly to the Business owners who grant access to the modules, all using Oracle workflow technology. We’re also having a close look at the Identity Management analytics, although have not bought this yet. The two products combined can not only provide very strong security, but save a huge amount on the cost of our compliance, which becomes more and more every year……

 

 

Our final exercise is to get every Support Person off Production completely. We’re working on nightly, automated clones with data scrambling. At that point we expect to remove almost all access to Production. After all you have everything you need to do your job as a support person, bar critical calls. This is where we really want to be, but whether we get to that point remains to be seen, but we’re certainly going to try.

Of course, we’ll have all the usual security exercises on database hardening, ERP hardening, Linux hardening, segregation of duties, access control, etc going to ensure we have continuous improvement across these key areas.

I hope you’ve enjoyed this article and hopefully it provided some useful insights into how we are improving security and easing our compliance burden overall. This article has mainly been on securing data in Production from support, but I guess I’ve gone off on a tangent into some other closely connected areas also……

 I’d be very interested also to hear comments from others, as I think security is such a vast topic that really others views (please do comment) would be very helpful.

So what are you doing on security of data encryption/access to production data?

I’d very much appreciate anyone to comment. (I have written this column in the hope of actually getting feedback and learning what others are doing to improve what I also do 🙂  )

And now, if you’ll excuse me, I’m off to apply a security patches to Production before the PWC Auditors from Hong Kong (awesome city) role in for a Penetration Test in October……..I like to call them my Loser friends (nothing like poking fun at the auditors….the loser buys lunch……), but they are immensely smart guys (as most PWC folks seem to be) and security is always a moving target requiring continuous improvement, so it’ll be interesting to see if I put one over on the auditors yet again (where they find nothing) or they finally put one over me and I need to buy them lunch…….

Of course either way, our company will be a safer and better place, security wise.

 

Related Articles

I hope that as you start to consider security in more depth, you may find the other articles relevant and useful reading also. My thanks go to the generous authors on a number of the articles below for sharing their knowledge for the wider Oracle ERP (and also Oracle Database) community.

Oracle Audit Vault

Oracle Governance, Risk and Compliance

Best Practices for Securing Oracle E-Business Suite Release 12 (Note 403537.1)

Oracle Identity Management

Oracle Identity Management Analytics

Oracle ERP Management Tools and Solutions

Oracle Applications Management Pack (with Data Scrambling)

Oracle Enterprise Manager 11G

Oracle Enterprise Manager 11G Data Masking Pack

Security Breaches – US Company Penalty

Security Breaches – UK Data Protection Penalties 1

Security Breaches – UK Data Protection Penalties 2

Security Breaches – California hospital Penalties

Security Breaches – HSBC Penalties

Germany Company Penalties

Data Masking – Strengthening Data Privacy and Security

Solution Beacon Security Column

Solution Beacon Security Best Practices

Project Lockdown – Securing Your Database

Recommendations for Leveraging the Critical Patch Updates

Oracle Security Column – Technology Network

Mask your Secrets using Oracle Enterprise Manager

Oracle Security Column – Documentation and Best Practices

Oracle Security Column – Critical Patch Updates and Security Alerts

Business Intelligence and the Kung Fu Dragons of Wudang

October 3, 2010

I recently watched an interesting program on TV about the Kung Fu Dragons in Wudang, China. The students join from a very young age and basically study 16 hours a day to become a Kung Fu Dragon Master over many, many years. The training is difficult, filled with risks of failure and many give up before ever coming close to becoming a Kung Fu Dragon master. Indeed the truth is most simply don’t have the skills, time or patience to reach the heights of a true Kung Fu Dragon Master. Very few will ever become a Kung Fu Dragon Master.

Now you look at data warehouses and Business Intelligence and ERP. It’s extremely difficult, filled with risks of failure and many give up (after spending a fortune) before they even get close after many, many years of trying. The truth is most simply don’t have the skills, time or patience to reach the heights of a true Kung Fu Dragon Master in Business Intelligence and ERP. The parallels and similarities are indeed striking.

Now the Art of Zen Patching article talked about patience and a non-aggressive approach. Not so this article. This article is really designed to allow you to become a true Kung Fu Dragon Master in Business Intelligence in 9-12 weeks (in vanilla implementation) and stresses a highly aggressive approach to Business Intelligence and ERP. (This article doesn’t ask for any money to be sent for some dodgy DVD and a certificate I printed in my bedroom. This is free, it’s not marketing hype – this is the real deal).

A cautionary tale before I start.

I’ve seen the two sides of data warehousing with Oracle ERP. One European company I know bought a bunch of expensive, state of the art tools almost a decade ago. Almost a decade on, with a lot of tears and pain  (sounds like a Kung Fu Dragon training course…..???), 2 million bucks and a change in system architecture that will require massive re-work, users simply not adopting because what has been achieved is small, fragmented and ad-hoc, and performs like a three legged dog (indeed in the user manual it instructs the person to submit the report and then get a coffee for 20 minutes before coming back or lunch if it’s a larger report) – the dreams of being a Kung Fu Dragon Master in ERP Data Warehousing lie in shattered pieces, a ruin of what should have been a shining example of 21st Century technology. This isn’t a one off – according to independent research a huge percentage of projects fail in the custom approach. Indeed it is the minority of ERP Data Warehouse projects that succeed.

Sure they bought the most expensive software, they put in piles of money, but they never sought a Kung Fu Dragon master to train them properly, preferring naively to think that they could learn the art of the Kung Fu Dragon Business Intelligence off a DVD in front of the bedroom mirror whilst their parents wondered what the weird shouts and snake hissing sounds were from upstairs………Instead of being a Kung Fu Dragon Master of Business Intelligence, they ended up as more like something of a joke reminiscent from the movie Kung Fu Panda……….Trust me if you’re a Kung Fu Business Intelligence Panda, people are laughing at you, not with you…….

 

(By the way, if you haven’t got the DVD for Kung Fu Panda movie, it’s highly recommended). As well as Oracle Fusion Apps being released in 2011, Kung Fu Panda 2 will also be out…..

Here’s the other side. I’ve seen Oracle come in with the prebuilt ERP Analytics, install the database, ETL tools, dashboards, KPI’s and reports and do all the data orchestration, extracts and imports in……..seven hours………Now I’m an utter skeptic when it comes to Oracle marketing and consultancy, but yes, I watched as one single Kung Fu Dragon Master in Business Intelligence and Pre-Built ERP Analytics achieved in 7 hours what could not be achieved in 7 years by an army of untrained people in another European company. Now this was only a demo (done for free as a demonstration of what could be achieved), but it took all our historical R12 Oracle ERP Accounts Payable data for 7 years and transformed it in a fully defined, fully functional data warehouse with dashboards in 7 hours. This guy was an Indian Kung Fu Dragon Master in Business Intelligence from Oracle Singapore. I was in utter awe of what could be achieved so easily and so quickly by one talented individual.

Have a look at Haemonetics Corporation story at Oracle Open World on Demand (Oracle On Demand Access Required) and then the results when they moved to Oracle’s prebuilt BI Applications approach.

The reason this utterly amazing complete data warehouse could be achieved is due to the pre-built Analytics packs that Oracle now provides (for of course, an additional license fee), together with of course an incredibly talented Indian Kung Fu Dragon Master.

So what’s the deal with Oracle BI and Pre-built Analytics?

Oracle is moving heavily into the BI market, taking the overall lead in terms of BI according to Gartner, much to the surprise of many.

In parallel Oracle is moving to create an unprecedented amount of pre-built content for Oracle ERP (including R11, R12, Peoplesoft, Siebel, JD Edwards with SAP in the works in 7.9.7…..) moving the task of creating a data warehouse from a vast army over many years (with a horrific failure rate) to a guy with an installation disk. Or that’s the single great hope and prophecy of Oracle Corporation. OK, we’ll never get to an installation disk, but if the guy with the installation disk can get us to 70% I won’t be complaining……..

Finally Oracle is now heading towards, probably in R12.1.4 to embedding the BI content directly into Oracle ERP screens. (The technology to do this by embedding into Java screens – not hanging a BIEE screen of an ERP menu by the way – was released as part of 12.1.2). If you also look at the new Fusion Apps, embedded dashboards are now an integral part of each module. According to Oracle at Open World 2010, there will be “hundreds of pre-built embedded analytics in (Fusion V1) or as they put it another way “Analytics are always available in Fusion Apps”.

 

 

Whatever BI tool or preference, you have to admit Oracle’s strategy and growth is certainly very impressive. With the recent release of Oracle BIEE 11G it just gets even better. The presentation layer is truly fantastic.

 

So many are probably asking, what’s the pre-built content for Oracle ERP?

Oracle has effectively written for you the following components:

  • Data Extraction from Oracle ERP (and many other ERP’s)
  • Data Orchestration (pulling dependent tables as needed with full synchronization)
  • Data Import to Data Warehouse with full management and restart capabilities
  • Data Model Design of Warehouse
  • Standard Dashboards
  • Standard KPI’s
  • Standard Reports
  • Security Models
  • Integration both to and from Oracle ERP
  • and a whole lot more

Now all of the above have been done not by a single Kung Fu Dragon Master in Business Intelligence, but by an army of Kung Fu Dragon Masters in Business Intelligence and ERP at Oracle Corporation.

These guys are experts in Business.

These guys are experts in Data Warehouse modeling.

These guys are experts in BI.

These guys are experts integrating BI to ERP.

These guys are experts at extracting data reliably from ERP to a data-warehouse.

The product line has come from a huge amount of hours collaborating with top businesses around the globe to find out what an ERP Data Warehouse should be

They have created something that no matter how much money and how deep your pockets, you will never be able to match, because Oracle Corporation is in the software business and can use economies of scale. Your company isn’t, despite the best intentions of your IT department.

The Pre-Built Analytics module list for Oracle ERP is equally impressive and is growing substantially on a consistent basis. This now covers:

Now it’s almost definite that this list will continue rapidly expanding, so that not only is there prebuilt content on ERP, but you’ll find it integrated directly into ERP (R12, Fusion, etc) and I bet into all of the other tools that Oracle has in the coming years including their database and other management tools. Basically if you are running Oracle ERP now, you WILL be running Oracle BIEE in the years to come.

Not only does Oracle Pre-Built Analytics create entire date warehouse content, they also add to it on a frequent basis. So when Pre-Built Analytics 7.9.6 was released it had a whole raft of brand new areas of HR addressed including Talent Management, Learning and iRecruitment.

So if you are still considering building using a custom approach to Oracle ERP Data warehousing, think about this foolish Kung Fu Panda……

Imagine that Oracle ERP has some 14,000 tables – do you really want to build a data warehouse from this (and will it actually perform…….????)

Imagine having to write extracts from the key tables which although much smaller is still huge. Imagine writing all the complex logic on something like Order Management for the extracts.

Imagine having to define a complex data warehouse model (and no, ERP and standard database design don’t work for data warehouses – it’s a completely different approach from a relational database model so your skills won’t work here……).

Imagine having to write processes to manage the extraction and import of data.

Imagine having to do the Security models.

Imagine having to do all the integration to and from your ERP.

Imagine trying to keep your custom built data warehouse in synch with Oracle patches, upgrades, etc

Imagine designing and building 100’s of dashboards with no idea of the proper principles of dashboard design and usability

Still fancy your chances of building a BI Apps comparable product Kung Fu Panda? Have a look at the summary data warehouse of BI Apps…….I guess you would be a whole lot less confident if it was your own money at stake on your custom build rather than your company’s……

 

The above represents years of work by an army of highly skilled, highly paid people, where the company must wait for a very significant time to get the results if you choose to custom build. Where you don’t get these highly trained people (as too many companies don’t), the consequence is simple – utter failure and massive cost.

And of course, even if you build it, what happens when you add another 2,000 users? Is it scalable or will you find major flaws in the design that are incredibly costly to fix? With Oracle Pre-Built Analytics scalability has already been designed in by the experts.

Still not convinced? Well let me give you the stats on Oracle’s prebuilt analytics:

350 Fact Tables

550 Dimension Tables

5,200 Pre-Built Metrics

15,000 Data Elements

And Kung Fu Panda, do you really think that the quality and performance and design can match that of the Kung Fu Dragon Masters from Oracle Corporation that do nothing but BI Applications? If so put down that Vodka bottle (or whatever else you are smoking) and get a reality check.

 

Or how about this Dashboard, with all the extracts, imports, data model behind it? For info this is Oracle Projects, one of the hardest niche modules in the entire ERP Suite, with links to EVERYTHING…….

 

In BI Apps just for HRMS there are 9 dashboards with a whopping 47 pages, 230 reports and 330+ metrics. And that’s just ONE MODULE OF BI APPS !

By the time you build that using a custom build approach (that’s so 1990’s in terms of approach – are you also still using Computer Punch Cards?) you’ll be retired or fired before you see it go-live……..

Today Oracle has the capability to deploy in 9-12 weeks (in vanilla implementation). Marketing slogans and nothing else? Absolutely not. We watched Oracle install everything (from database, to tools, to dashboards and migrate 7 years of data) and bring up Accounts Payable in 7 hours as a proof of concept. I can be pretty skeptical and hard on vendors, but believe me, this was truly amazing the ease with which this product could be brought up.

Now true there are some serious challenges around also, even with the Pre-built Analytics. Anyone thinking they can buy this and that’s it, well that unfortunately is naïve. I wish life was that simple……

The fit of the BI Analytics needs to be checked in terms of dashboards. What we found was that a lot of dashboards fitted very well onto the Business Requirements. Please bear in mind that dashboards are very easy to build (and BIEE is very easy to learn), especially when the Kung Fu Dragon Masters at Oracle Corporation have done the hardest part of building the entire data model and all the data extractions for you. Even if you find no use for the pre-built dashboards, simply having the data model and extracts will save you a huge amount of time. Now looking at Infosys they reckon a 70-80% fit in terms of BI Apps. I’d tend to agree that would be a ballpark figure when we assessed this against our ERP.  (And remember a fit means data extraction, import, data models, dashboards and all the other components).

If you do have custom modules, then you will need to build the extracts, data model, etc. However don’t be fooled by naïve arguments that if you’ve customized your ERP, BI Apps will not work. I’ve rarely seen a company modify base tables in ERP (add new custom tables yes) as they would be extremely foolish to do so. Therefore BI Apps will have just about every key table from the core ERP model you could possibly need. That’s a huge chunk of work already done.

You will need to define further dashboards and reports. That is a given. BI Apps will get you very far down the line, but it won’t be a silver bullet.

You will need to design data models and extracts for any custom modules you have. That can be a difficult task, but your ERP will already have all the data models, extracts provided by BI Apps. Custom modules are almost always a whole lot simpler than Oracle ERP Data models.

One of the biggest tasks that not even pre-built analytics will address is Training and user adoption. Moving from reports and Excel to Analytics is a major business change exercise and should be treated accordingly. The project can still fail even with great pre-built content, because users are not trained appropriately and there is a lack of vision in how this can really change the way the user community work.

The cost of prebuilt analytics is still a major problem I think. There are alternates out there that are much, much cheaper but then you have to consider upgrades, risk of building, integration and a whole host of other problems. I love the Oracle BIEE and the Pre-Built analytics but the cost is still pretty high. Of course compare the cost of that to the cost of building all this yourself and there is just no comparison between the two, like for like. Playing hardball with Oracle Corporation (ideally at their year end when the best deals can be achieved) can pay dividends in terms of price, although I’m not going to give away how cheaply we got offered the prebuilt analytics……There are still alternates to Oracle in many areas, solution wise and Oracle should be reminded that you can go elsewhere…..

Pre-Built analytics for ERP can get you there fast. But a big bang approach is not recommended. A fast initial deployment to a small community followed by consistent wins over a period of time, slowly building up the user base and expanding across the lines of the business is the way to go, bringing in and training champions each and every step of the journey. Then you’ll succeed where so many others fail. When everyone can see frequent wins (and with BI Apps it can be every few months), it becomes an unstoppable train that everyone wants to get on. BI Apps is beautiful eye candy that users seem to love. (Or is it that they finally can get their own answers in seconds without waiting 12 weeks for the IT Department…..)

You can buy the software, but you need to sell the vision, otherwise you fail. IT Alignment to Business Alignment and support from Business is the key to any successful project.

I’d still recommend having a BI Expert around who you can gain the knowledge from. Again Pre-Built Analytics is not a silver bullet and your staff will need to be properly trained by an expert.

I think all companies running ERP should be looking seriously at BI Apps, even those with the approach of building everything. In the 90’s companies realized the value of letting Oracle, SAP and the rest build their ERP applications. That’s now a generally accepted principal by most companies around the globe.

Over 2,500 companies have already adopted the BI Applications. If it was cheaper (a message to Oracle Corporation and Larry Ellison) the uptake I am sure would be massive.

In the coming months, I’ll write a few more articles on deep-dive into some of the core areas of BI Applications – HRMS, Financials and Procurement, time permitting.

I think in the 21st century, that same principal will gain general acceptance on the value of pre-built analytics for ERP. Put quite simply, in the cut-throat economy companies can’t wait ten years to become a Kung Fu Dragon Master of Business Intelligence with Oracle ERP.

Companies need to achieve that in 9-12 weeks (vanilla implementation) and the only way to do that is to rely on the Kung Fu Dragon Masters from Oracle Corporation that have already done this for you…………before your competitors beat you to it……..

Related Articles

Below are a number of articles. Some are your average Oracle Sales pitches, but all the same very informative overall. Others are written by generous authors whom I acknowledge and extend my thanks to for writing very interesting and useful articles, which were used as background research during writing this article. (I particularly enjoyed the articles by Jeff McQuigg and Mark Rittman, both renowned experts in their field and excellent authors. Their articles are highly recommended reading).

 For those lucky enough to have been to the Oracle Open World 2010 show (hope you enjoyed the Black Eyed Peas !!! Turn down the volume before you click that link……), the following can be accessed from Oracle Open World On Demand.

(Note there were many more sessions, but many do not yet have the content uploaded at the time of writing – 2nd October 2010. I have chosen to hyperlink only a few of my favorite session links. Others are available in Oracle Open World On Demand).

  • End-to-End Oracle Business Intelligence: From Warehouse to Advanced Analytics
  • Enterprise Sales Analytics with Oracle Business Intelligence Enterprise Edition
  • Oracle BI Applications Roadmap, Including Support for Oracle Fusion Applications
  • Oracle Business Intelligence Enterprise Edition Integration at Comcast
  • Oracle Human Resources/Oracle Financial Analytics for ResCare Decision-Making
  • What’s New in Oracle Business Intelligence Suite, Enterprise Edition Plus 11g
  • Gain the Insight You Need with Oracle Business Intelligence Applications
  • Implementation Experiences with Oracle Business Intelligence Applications
  • Implementation Experiences with Oracle BI Applications
  • Oracle Transactional Business Intelligence (OTBI)
  • Haemonetics Corporation: Better Decisions with Oracle Human Resources Analytics
  • Get Real-Time and Actionable Information with New Reporting and Analytics