Posts Tagged ‘Payments’

21st Century Time Bandits of Accounts Payable

May 9, 2011

When I walk into an Accounts Payable department, it feels like I have almost stepped back in time, from the 21st Century to the 1990’s. Think about this – the 1990’s saw the dawn of ERP systems. Companies implemented these ERP Systems to handle amongst other things, the process of managing and paying invoices from suppliers.

Companies got large amounts of invoices, sent via the good old postal service. These lie on desks in huge piles, until some Payables clerk is ready to pick it up, put their head down and key it in. This process goes on endlessly, in many companies around the globe. Very little has changed in the last 20 years………it is as if, we got into a time machine and went back 20 years.

So a quick intro and a quick movie recommendation before we get into the content. First up, another favorite (and directed by an Ex-Monty Python actor), Time Bandits was a very original movie, all about, you guessed it, Time Travelling. It’s a really original and very entertaining movie. It’s an 8 and great for your kids.

Now a lot of people instantly say outsource Accounts Payable. To those people I say, outsource precisely what? I am a firm believer in using technology to automate certain areas. Freeing up those people to then add far more value to the Business in other areas of Accounts Payable. Too many people outsource business functions in a knee-jerk unimaginative reaction, effectively keeping the 90’s Business processes, whilst really they should be looking at new ways of working and transforming their organization with 21st century technology.

Now I sat in one Oracle OpenWorld session a few years back and listened as one company explained how they processed 1 million invoices a year, with a handful of staff in their Accounts Payable function. My company, if it had the same volume of invoices, would have needed a staff of well over twenty times that. So how did this company manage? Because they changed the way they did business. They didn’t blindly outsource 1990’s Accounts Payable business processes.

They were a 21st century Accounts Payable function whilst their competitors remained firmly in the 90’s.

This caused me to take a very good look at exactly what was the Accounts Payable function in my company processing. I’m going to go through a breakdown of exactly what went through our Accounts Payables subledger, throwing in a few ideas of what we did, what we automated and a few future directions. Hopefully it might trigger you to start questioning some of what you are doing in Accounts Payable.

I’ve been through a lot of Self Service applications in previous blogs and this blog continues to pursue this key technology with a view to automating business processes.

What struck me about my company was that our Payables unit was putting in all sorts of staff related invoices for payment to staff. Not only were they manually entering these but they were also checking all the accounting, figures, etc. Could we outsource? Why on earth would we outsource a function that had absolutely no business value to be done manually and actually shouldn’t exist asw a manual process?

Our company gave loans to staff. When a staff applied on paper, it went through massive approval processes, then finally hit the Accounts Payable for processing. We simply replaced this with an online application form. Staff applied, everything in terms of checking was automated, and we put an invoice through the AP Open Interface (already pre-approved with all the correct accounting) ready for Payment. One function of Accounts Payable removed, with over 2,500 invoices automated.

Our company gave Home Country Travel allowances. Same process as above. Staff applied on paper, massive approval processes, then finally hit Accounts Payable for processing. Online application form, pre-approval, automation, AP Open Interface, neeed I say more. 850 complicated invoices gone, completely automated.

Housing allowances. Same procedure again. Full automation. 850 complicated manual invoices gone, completely automated.

Staff go on a lot of travel given the nature of my company’s business. Same approach again. Online application, straight through Accounts Payable processing (pre-approved, proper accounting, etc). This removed 5,000 manual invoices per year for payment of travel allowances, going completely 100% automated.

Some of our staff have Payroll paid through Accounts Payable. (Don’t ask, it’s complicated). Linking our Oracle Payroll to Payables resulted in over 5,000 manual invoices being removed from Accounts Payable. A similar approach for Pensioners (don’t ask, it’s even more complicated than staff Payroll…….) removed 12,000 manual invoices.

The point is simple. We are a small company. Our company was processing manually over 30,000 invoices per year that it simply didn’t even need to process. Automation led to not only vastly reduced workload, but also 100% error free, paper free invoice processing and vastly faster and better service to our employees.

A short point to finish up. If your Accounts Payable function is being overloaded due to Employee “invoices”, do look at what can be automated by online application.

Another perfectly valid approach is to look at iExpenses to reduce the overall manual effort and avoid your Accounts Payable function (which is supposed to be processing supplier invoices) becoming swamped with Employee expenses. This is another good approach to streamlining your overall Employee Expense Management processes.

By looking very carefully at our business processes for handling Employee Expenses, we automated over 20% of our entire Accounts Payable function, removing completely the need for manual intervention.

And with that it’s definitely time for a movie recommendation. Back in the 80’s, a great movie (actually series of movies) covering time travel was Back to the Future. These are well worth a rent for the kids. A 7 out of 10. It’s so sad to see Michael J Fox, the star of these movies, battling a terrible disease when he was still relatively very young.

Moving on to the next area of Accounts Payable. Do you have any very large, very frequent suppliers? Depending on your business, a significant percentage of your Accounts Payable invoice processing may be down to a small number of suppliers. In our case this was certainly true. Let me go through the examples of how we transformed this area.

My company does a lot of travel. But we deal with two travel agents only, giving competition but also good control of our overall travel spend. We process around 10,000 travel invoices per year. Now these companies were not happy with us. Our Accounts Payable was failing to process invoices in a timely manner. Disputed items were confusing. Volumes were high. Invoices required vast amounts of checking. Vice versa we were not overly happy with them, as we found equally the Accounts Payable process very time consuming, confusing and seriously expensive.

Now our Travel process is fully automated, including bookings to these travel agencies. We asked the Travel agents, given everything is automated when staff travel, can you give us an electronic billing file? They were very keen indeed – this was going to save them a huge amount of time as they no longer needed to prepare vast quantities of paper invoices.

So look at our processing of Travel Invoices today. Two Travel agents give us a flat file monthly each. These two files are loaded and automatically matched to our Travel system. Can you imagine how much time that saves us just in reconciling an invoice to actual travel. 95% of the invoices sent are automatically matched by our automation programs. These invoices are then automatically imported using the Payables Open Interface and paid, with no intervention by our Accounts Payable department. Our processing times for these invoices have been drastically reduced (98%). Our time is spent dealing with the exceptions, not every single invoice. And the Travel Agents are happy because for the first time ever, they actually get paid on time. (And the 95% matching is a work in progress. It’s early days and we aim to hit 98% first time matching).

Next example. Our telecoms companies. Again we have two providers for competition. We have a huge number of staff with Blackberries. Every month, we receive bills for each staff, each with a different account number. Now that’s how the service providers work, so we need to deal with that. Each month, we would circulate these paper bills to staff, get the feedback, pay each invoice……………a truly dreadful process. Huge amounts of time and effort, bills were never paid on time. Service provider was unhappy. We were unhappy…..everyone was unhappy. Simple solution – service providers gave us consolidated billing files. These were uploaded and matched to staff, sent as notifications in Self Service and uploaded directly to Accounts Payable. A very complex, highly manual process, became highly automated, again with levels of matching approaching 100% with minimal human intervention. This removed 10,000 manual invoices from our Accounts Payable function. With the simple provision of two flat files that the suppliers were extremely happy to provide.

Final example. Our company, by the nature of it’s business, has an outsourced caterer, both for the staff canteen and for functions where food is provided. Our functions are very, very frequent, leading to a very large volume of invoices. Now our caterer used to prepare each invoice and send it to us. All paper based. These would then be circulated around our company, finally hit our Accounts Payable and finally be paid. We requested a monthly file from the supplier. This is then uploaded again very simply into Accounts Payable, with automated matching again approaching 100%. Again this supplier had never actually been paid on time, wasn’t too happy but accepted it as we put a lot of business their way. From our side the process was a nightmare. But with simple invoice files, monthly, manual invoices and manual checking completely disappeared. This removed more than 6,000 manual invoices per year.

We also do this with  a few of our other large service providers. They are usually all happy to work to a very similar template for the flat files they provide to us. We’ve taken over 30,000 manually entered invoices out of our Accounts Payable function. Our matching rates are above 98%, with very few of these invoices requiring manual intervention.

Simple approach. Look for the suppliers with the most invoices. Ask those suppliers if they want to reduce paper work, reduce burueacracy and get paid a lot easier. Most will say “Yes please”. Then automate.

Now we’ve not referred to Oracle’s EDI product – E-Commerce Gateway. But effectively that is what we are doing on a very simple, but very effective approach. Oracle does have a module that allows for large-scale electronic data interchange, including for Accounts Payable. EDI has been around a very long time (I remember implementing this in the 90’s for a very large customer). Now Oracle supports what are industry standard transactions, making it possible to link companies and their suppliers together using the same language, in terms of file formats/contents, etc. A few links can be found in the Reference Section relating to this. (Note Oracle provides EDI not just for Payables, but for other modules including Purchasing, Order Management and Receivables).

The use of EDI (either Oracle’s module or simple, effective home-grown EDI) can transform your Accounts Payable process.

And now another movie review. Actually I’m going to do something a little different and recommend a TV Series (although there have also been movies) for a change. Doctor Who is a TV show broadcast by the British Broadcasting Corporation. Now this show is done relatively cheaply, but always has original stories and is highly entertaining. It’s about a time traveller that travels through time and space, fighting evil aliens such as Cybermen, Daleks and other such monsters, whilst protecting the human race. He travels around using a blue British Police Box which is actually a time machine. His main enemy are the Daleks (shown below in a poster). These are robot monsters that move around on wheels taking over the galaxy, unless that Galaxy has too many stairs, as with wheels they cannot go up stairs…….Honestly, get a few DVD’s as the series are now all available. Kids will love it and adults too…….It’s a great British institution – been around since the sixties and I’d give it a 10 out of 10.

So the next option for transforming your Accounts Payables department. I’d say this is only for the large suppliers, but Oracle has an option called Payment on Receipt (ERS). The idea here is simple – you have large suppliers that provide you with goods. Now if your Shipping department (or whoever receives and accepts the goods) has checked the goods, accepted the goods, quality checked the goods and you have a validated Purchase Order, then exactly why do you then need a supplier to send you a paper invoice (or even electronic invoice) to your Payables department to process that invoice, check it again, etc. Why do you not just get Oracle ERP to create an invoice automatically? You remove vast amounts of bureacracy, vast amounts of unnecessary work and the reliability of this method trumps even EDI – this is a huge benefit to both you and the Supplier. Details can be found at the end of this blog.

Another interesting option is that Oracle can generate Invoices based on milestone payment in Services Procurement. A very similar concept to the ERS functionality, again further removing the need for manual invoices. If you have agreed to pay a supplier on a milestone, based on a deliverable, then once a manager has accepted that milestone has been achieved, an invoice can be automatically generated. (Hopefully as you read this blog, your also starting to see just how important a good Purchasing function is when it comes to Accounts Payable – one function cannot reach it’s full potential without the other also working highly efficiently).

The idea of companies still receiving paper invoices is a curious one for many areas of their business. One step better than you paying people in your company to enter invoices is for the company wanting paid to do the work for you.

Our company hires a lot of consultants. Those consultants work on contracts, to deliver certain tangible reports/studies/other pieces of work. The project manager signs off these deliverables and the consultant gets paid. Our approach to this is to provide a simple screen for the Consultants to enter their claim (very similar in many respects to iExpenses). With the consultant entering their own data, automatically validated against a contract, workflow notification for approval to their manager and then an invoice generated automatically for Payment (based on accepted deliverables), we will remove approximately 40,000 manual invoices from our Accounts Payable department.

Our other approach for more standard goods and services is the introduction of iSupplier. This module provides many facilities including suppliers entering/updating their details/views of Purchase Orders/updated shipping information, etc. One of the facilities is for the supplier to enter invoices, removing the need for your company to do so.

Next up………..how many calls and how much time do you waste talking between your Accounts Payable Department and your supplier? You don’t know because no-one ever actually records the time………..but I bet it’s significant. The average company spends a lot of time dealing with calls that are really unnecessary. “Did you receive my invoice” or “when will my invoice be paid” or “what was this payment actually for?” We’ve replaced a very large number of these calls, again by implementing iSupplier. This module gives the supplier a full end to end view from Procurement to Payment of all activities. It effectively gives a real time window into your company to the supplier, eliminating the need to inquire on invoices or payments. Invoice and payment information can be downloaded by the Supplier for easy reconciliation.

Interestingly, if your sending out Remittance advices, this can also be replaced either by iSupplier Portal or simply letting the Payments function automatically email the supplier with a detailed remittance advice slip. Every invoice we process generates a payment and every payment has an email remittance. That’s over 150,000 manual remittance advice slips that we no longer have to prepare, print, put in an envelope and send.

This has been a busy blog. Time for a break. Time for a movie recommendation. Probably one of the first and most famous stories of Time Travel was by H. G. Wells. Now his books have been made into countless movies and the most recent The Time Machine wasn’t a bad interpretation. Overall I’d say a 7 out of 10 and worth a rent, although probably not a buy.

Now I don’t need a time machine nor some weirdo Oracle prophet to predict the future of Accounts Payables departments. The future has actually been around for quite a few years already with products that can read, scan and workflow invoices, significantly streamlining and automating your Accounts Payable function. But the one I am now looking at is Oracle’s own Imaging and Process Management. So here’s my prediction. I believe this piece of middleware (bought by Oracle, as all the best stuff seems to be) will become a central pillar, not just of AP, but will be used everywhere in future versions of R12 and Oracle Fusion. If you want to get on the Fusion express train, this is one product you should be seriously looking at.

Irrespective of product, the future, already used by many visionary companies, is to be able to receive either paper invoices or scanned emailed invoices into a single shared service center. The technology is already smart enough to scan these invoices and identify the information in a reliable manner.It can also generate invoices and match those invoices. Further these products can then workflow the invoices to the correct recipient for approval.

So what else does the Oracle Prophet see in the future? Again it’s an easy question to answer, as the future is definitely already here. I’ve done a blog previously about Oracle Pre-Built analytics. Well one of the major functionalities of the Oracle Pre-Built Analytics is the Payables module. I watched Oracle install all the tools, database, populate the data warehouse with 7 years of corporate AP data – in seven hours (as a proof of concept). The dashboards were then ready to use and we could instantly gain insight into our Accounts Payable functions. All the KPI’s were there to see invoice aging, discounts given, invoices processed, methods to process, etc. This is another fabulous product from Oracle and rounds off nicely this 21st Century Accounts Payable blog. Imagine having a complete, instant view of the health of your entire Payables process, with the ability to drill down to any single transaction, or aggregate at any level or instantly see who your problem suppliers are.

Now my company made a mistake in early 2000 by decentralizing the Accounts Payable function. In many respects this was organized chaos. If I had been with my company I could have agreed with the person doing it, except then we’d both be wrong. What I’d like to see my company do next is to move to a shared service, 100% centralized function that is 100% decentralized…………yep I’ve totally lost you, but we’re back to the concept of total federated services at the point they are used using shared service centers where appropriate. The technology is there today to allow a shared service to receive and quickly scan invoices, doing most of the keying automatically. The shared service provides a centralized function for dealing with difficult invoices (ones that don’t match, ones that don’t automatically go into the system, etc). It provides a Governance, Control and Analytical function. along the lines of a standard Accounts Payable department. However once in the system, the invoices should be routed directly to those who need to approve payment. The line managers in all the various departments worldwide who actually received the goods or services. Hence you have a shared service center, but a totally federated approval process. The technology today is very much capable of supporting this process. Throw in the automation for straight through processing on as much of your Accounts Payable invoices as possible, a sprinkle or heavy serving of EDI depending on your taste, and using your time machine, you’ve arrived in the 21st Century for Accounts Payable processing.

So out of our 150,000 Payables transactions a year, we have automated (or will shortly automate) almost 120,000.  Now I make that a whopping 80% of all Accounts Payables transactions. Now the not so smart companies out there (and there are plenty of them) would have simply perhaps have  outsourced their 1990’s business process. We will have achieved 80% automation by 2012 and taken a 20 year leap in technology and business process, all within a couple of years of the initial idea being floated.

We do intend to keep plugging away at the last 20% looking at potentially automation of invoice scanning, etc using Oracle’s Image and Process Management. We do want to continue to review what isn’t going through as EDI transactions and why. We do want to see where spend is avoiding the standard procurement processes and understand why. It’s still a work in progress.

The benefits?

1. An 80% reduction in manual effort. That saves our company a fortune. Now if you count each manual invoice costs an average company between 20-40 US Dollars to process……go do the math.

2.  Decreased supplier phone inquiries significantly

3. Improved Invoice processing times (up to 98% in some instances)

4. Improved approval processing times (and in many cases automated matching removes approval completely)

5. Vastly reduced paper, paper handling and paper storage

6. Easier to see and take effective discounts

7. Ensures compliance with Payment terms, avoiding penalties

8. Improves your company image as a good company to do business with – always pays on time

9. Absolute visibility of all Payables activities and transactions

10. Easier to audit, easier to comply with SOX/etc – the scanned invoice is available directly from the ERP Transaction

It has to be said that not everything can be automated. Some countries still need the paper invoice as a legal requirement. Some invoices just won’t have anything to automate matching to. Sometimes your supplier won’t play ball on EDI. There are limitations. Accounts Payable is still a very important and critical function that needs staffed correctly. But really, any company could look to taking a significant percentage of their manual Accounts Payable and automating to some degree.

And finally a movie recommendation to finish up our Time Travelling 21st Century Time Bandits of Accounts Payable. It has to be Bill and Teds Excellent Adventure. A movie about a couple of college kids that go through time in a telephone box to complete a history assignment by collecting famous historical figures to do speeches at their review. Along the way they play games with Death and win…..a lot of fun and a 9 out of 10. It’s the sort of no brainer nonsense that I find intellectually very challenging. Definitely not a deep movie, but MOST EXCELLENT !!!!!

As I end the column I feel it appropriate to do it in the spirit of the movie Bill and Teds Excellent Adventure:

Bill –  “Be excellent to each other. “

Ted “Party on dudes !!!!”

Until next time, when we reveal the secrets of Harry Potter and the Wizards of ERP Change Management………

Further secret blogs and prophecies can be found at:

https://oracleprophet.wordpress.com

References

Below are some of the references collected as part of the research for this blog. As usual I appreciate the authors making this information available on the internet, for the benefit of everyone. Also enclosed are a few of the guides/manuals from Oracle that go into greater depth of the possibilities for improving your Accounts Payable process.

IExpense – Back to Basics

iExpenses – Ohio Valley OAUG

Oracle Internet Expenses

EDI and E-Business Suite

Oracle E-Commerce Gateway

Payment on Receipt

Oracle iSupplier

Oracle Purchasing

AP Invoice Automation – Bottomline Technologies

AP Invoice Automation – Readsoft

AP Invoice Automation – Basware

AP Invoice Automation – Kofax

Accounts Payable Invoice Automation

Oracle Imaging and Process Management

Oracle AP Automation IPM Solution

Wikipedia – Oracle Imaging and Process Management

Oracle Imaging and Process Management 11g

Accounts Payable Analytics

Star Wars Episode IV: A New Hope – R12 Self Service Jedis

April 14, 2011

Now I can’t do the scrolling Yellow Text and the famous Star Wars fanfare, but I reckon most people will have already got the Star Wars theme for this blog. The force is obviously strong in you.

There is a serious disturbance in the force across the corporate Galaxy. Have you ever thought that your company is built up of numerous Sith Lords, all wanting to basically build a Galactic Empire of epic proportions, with no relevance to the goals of your Business.

Think of it this way. Each of those Empires is building deathstars 0f immense power over decades. Each of these wields terrible power across your company reaching to every corner of your corporate universe. At the head of these powerful empires are your typical Darth Vaders, Darth Sidius’s and other villains that steal the lifeblood from your company and quash any rebellions against them with absolute force.

However despite the absolute power, in our galactic quadrant, a small group of Rebels started a rebellion with a view to wipe out the tyranny of the Evil Empires and bring freedom to the corporate galaxy. These few were the R12 Self Service Jedis. Or perhaps you’re a Princess Leia wanna-be? (If you’re a guy and a Princess Leia wanna-be then I’d certainly find you a little strange in a long flowing white dress, but each to their own……)

So what better way to start with than a movie  recommendation. It has to be Star Wars Episode IV: A New Hope. This was a truly cutting edge movie (which a top movie critic in the UK stated was “rubbish” at the time of release – there was a truly epic screw-up……..). Well Star Wars was the thing of dreams for millions of kids (and adults) and continues to be to this very day. It’s a 10 out of 10 movie and you should buy it on DVD.

In 2001, our company built an ERP deathstar providing the various Empires powerful facilities that did everything to consolidate their power. Bar one item. Included in this deathstar was one area that could, maybe just could, undermine the Empire. There was a small part of Oracle ERP dedicated to the galactic citizens. This allowed galactic citizens to check Galactic HR records for:

Payslip

Staff Receivables (*)

Pension (*)

Company Savings Scheme (*)

Bank Account designated for Payroll

Personal Information

Staff Information

Education

(*) Indicates custom Screens we have written in OA Framework

What is interesting is that many of the above screens are pure standard functionality. Easily and very quickly enabled across the entire Galactic Empire.

Enabling the simple Staff Information gives your staff access to Employment, Salary, Performance, Training and Job Applications with a simple addition to a menu. (The screen has most of the data removed for privacy reasons).

Now perhaps you are thinking big-deal, who cares about Payslips. Well we have 2500 staff across 27 Galaxies each requiring a Payslip twice per month of 3 pages. We also have 1,000 Pensioners requiring a Payslip across 66 Galaxies. That’s over 200,000 pieces of paper that have to be printed, put in an envelope, sent to the various far flung corners of the galaxy and all that costs a huge amount of time, money and bureaucracy. Putting a Payslip online (and have it populated as a natural part of Payroll) wipes out this pointless costly burueacracy in one swift action. Start to consider the effect for a Galactic Corporation of 25,000 employees, or 50,000 employees. The savings year after year after year are immense. Not to mention that online Payslips are delivered at the speed of light, being available seconds after Payroll is completed.

(As above all the key information has been removed for privacy reason, i.e. my privacy……. although as can be seen last month I cleared 34 cents – it was better than usual 🙂

For 30 years this Payroll Empire outpost had created tyranny and bureaucracy by making citizens beg for key information that was held on the Empire’s central computers. This functionality brought a small part of freedom to the citizens in this far flung quadrant.  However the Jedis from the Oracle Corporation sect quickly left in 2001 as these were mercenaries, who would only stay whilst the galactic credits kept flowing. After that the Galactic Empire’s consolidated their power, insisting each citizen continue to file vast quantities of paper with no specific purpose.

Our journey to restore freedom to our corporate Galaxy fighting against the Empire began on the small planet of Tatooine some 6 years ago back in 2005. Surveying the corporate galaxy the Jedi’s looked for areas of maximum gain where an attack would carry minimum cost and risk.

The Empire ran a facility to give Loans to Galactic Citizens (Employees). The loans were ironically managed by an army of Galactic clones that actually managed the loans and deductions to Payroll MANUALLY using spreadsheets. There was no risk on this, given they were TRIPLED CHECKED, leading to even more clones.

The Jedi’s reviewed this facility. A citizen could loan based on their Galactic Salary and length of service to their company. The Jedi’s checked and  they could get this information from the HR System. A citizen could request for a loan to be paid into their Bank Account. This could also be found in the Payroll/Payables system. The Loan could be paid using Accounts Payable and R12 Payments modules. Finally Loan interest was deducted each month from the citizens Payroll. Check – The Jedi’s could do this through 100% automation.

So exactly what were these clones doing that they demanded citizens to fill out reams of paper, have it checked in 6 separate Empire facilities shuffling paper between each, wait for weeks or months for the approval, then finally pay the loan. This was Galactic madness, purely to justify the Empire’s existence.

With a simple OA Framework screen linked to the Empire’s Accounts Payable facility, the Jedi’s managed to remove the Empire’s clone army in a six month operation. Citizens would apply for a Loan with a few clicks of a Self Service screen. This sent an invoice to Payables (pre-approved) and fed through to R12 Payments, then through SWIFT to the citizens Bank Account. The end to end application process moved from 2 weeks to 2 minutes, with no Empire intervention.

The second part of this sytem, the Loan Engine created by the Jedi’s plugged into the Empire’s computers and automatically removed the correct interest each month from the citizen’s Payroll, with no Empire intervention from clones. The Loan Deduction fed through automatically to the Self Service Payslip.

This system now processes over 4,000 Employee Loans every year. Imagine the manual work that was done before this Self Service System was deployed.

Given this was the first attack on the Empire, everything was automated, except for the R12 Payments module, which still required manual runs of the Payment Cycle Programs in R12 Payments. The Jedi’s were careful about picking full scale battles with the much feared Sith Lord Dollarius at this early stage in case the Empire became aware of the serious challenge that was growing within. A key tactic in the Jedi Wars to bring the freedom of Self Service to the Corporate Galaxy was to carefully choose their battles, delivering quick wins, at limited cost and risk.

(Again the screen is censored, but normally you could see the Loan amounts you can borrow, current loans, etc)

The next obvious move was to take out an Empire facility that did nothing but produce Certificates for Citizens. These Certificates could be for Employment, Salary, Wages, Tax and many more. Staff used ancient technology from the days of the Old Republic to fill out Old Republic Excel sheets and sign, and send to the Empire Central Processing Facility. The Empire demanded citizens wait two weeks for their own information, whilst armies of Empire Drones ran reports, got the information, sent memos to other facilities, got returned information from said facilities, cut and paste a variety of sources into an Old Republic Excel or Word document, checked the information and waited for the return of Darth Vader, the feared Head of the HR Facility to physically sign. This was the information tyranny that was feared throughout the corporate galaxy, draining desperately needed funds from well run Republic planets to fund an army of mindless Empire drones in bureaucracy.

With the plans from Princess Leia to the facilities HRMS System, the Jedi’s worked out that a small custom system to access this could be used by the citizens of the Old Republic to access their own information, removing the need for the endless army of Empire clones.

“There is something very wrong when it takes so long to get your own information from the Empire”

Obi Wan Kenobi

With a small OA Framework screen, loading of electronically scanned signatures and a little bit of BI Publisher force, the Jedi’s slashed through the endless bureaucracy of the Empire. The Empire’s Darth Attornius questioned the validity of scanned signatures, but traders and governments throughout the Empire readily accepted these, allowing citizens to apply easily for Bank Accounts, Tax Refunds and all manner of other necessities.

A signed (scanned) certificate with the citizen’s own information could be generated within seconds by the citizen themselves simply by selecting the certificate type in a Self Service screen, pressing a button and waiting a few seconds for a PDF file to be generated, with a scanned signature attached. Imagine a process that involved many people, many departments and took weeks to deliver information that citizens often needed quickly. Imagine that process where absolutely no-one was required, the citizen pressed a button and got what they needed within 5 seconds. That was the power of the Jedi Self Service.

Another small Empire outpost fell without a fight to the R12 Self Service Jedis.

In 2006, the year we launched this Self Service feature, we had 2,500 Certificates created. By 2010 that had grown to over 6,000. (For a company of 2,500 people).

This could be delivered by any small Jedi strike team in a very short period of time at minimal cost. Scale this up to a global entity of 50,000 and savings are immense. The other immediate payback is a lot of very happy citizens who get what they need in seconds.

(Note there are confirmed rumors that the Jedi Sect from Oracle Corporation has provided Certificates in R12 which the Empire and most Jedis are unaware of. See Employment Verification in Oracle Self-Service Human Resources Deploy Self-Service Capability Guide (Part B31648-03)

This short episode outlined the initial skirmishes with the Empire, mostly which went to plan, to deliver the initial foundations of Self Service functionality and freedom to the Corporate Galaxy with rapid ROI.

The R12 Self Service Jedi’s will continue with Episode V – The Empire Strikes back shortly…….

Further Prophecies can be found at https://oracleprophet.wordpress.com

Halloween – A Time of Ghouls, Ghosts and R12 Upgrades (Pt3)

October 31, 2010

Welcome back to the R12 Upgrade House of Terror.

All I can promise you is pain, heart-ache (that Oracle Support girl for the AP Trial Balance broke mine…..ahhhhhhhhh) and impending doom. Abandon all hope on your R12 for those who read on……..

So you’ve been mad enough to start an R12 project (they’ll lock you up in the asylum they will !!!) and you’ve made it through Implementation (designing any new stuff, build, test, UAT, etc) and you are sitting there smugly thinking you’re smarter than your average 5th grader and that this column was just scare-mongering……..Well lets go to the Transition Phase and see how confident you are………

 Cutover is complex

Deployment involves high number of objects

Deployment involves significant application setup

Deployment carries high risk areas

Users are not familiar with the new software

Large number of users require training

Users are distributed across countries

Help Desk Staff are not trained to take calls

Transition is poorly documented

Transition is not rehearsed

Change in Business Procedures for R12

Before you start the cutover, just wondering if you actually trained all your users? Yes R12 is mostly the same, but there are significant variations between R11 and R12 in Payments, TCA, SLA, etc. Now if those are spread across countries, it’s going to be even worse. A new system with new features suddenly appearing will lead to a lot of angry users, all wanting to talk to you on Day 1 of R12 as you are inundated with other more serious issues……..Now I went through four audits on my R12 and got a clean bill of health on all, except for training (even though we did some). The auditor did a survey and the users said they would have liked MORE TRAINING. But the products almost the same I said bar a few key areas as we were doing a Technical Upgrade !!!!!!!! I got burnt at the Halloween stake on this – OK ever so slightly singed. Don’t make this mistake. At least offer some training on delta differences and some educational classes with no commitment to implementing (get your lawyer to write the disclaimer) new functionality.

Has anyone told the Helpdesk of your R12 transition? Has anyone told the business they will lose their systems for 4 days (weekend, plus two working days). Has anyone told the external users or put out appropriate announcements? Are there key business activities that expect to run during your upgrade? Have you scheduled your upgrade right in the middle of these? People are going to be seriously upset if payroll is delayed for your upgrade. Or month end closing is delayed and you have serious problems with your new R12 that goes live right before the preparation of those financial results. Wondering if you told your corporate website team that pulls out summary information from ERP to display to your company website that ERP won’t be available over the transition? Or how about the legacy team that are still sending you files to interface into ERP? I know you are feeling pretty insecure now. I guess you are wondering “Have I missed anyone in the announcements……”.

Cutover is a nightmare. Get this wrong and you see your entire project collapse in a pile of dust, on day 1 of Production, with the Board asking why your business has ceased to function and lost 5M US. Pulling the transition on R12 at the last minute is really bad, but even worse is going live and then finding out you have serious issues……..Either has a good chance of ending your career.

Has everything been tested? Has every bit of code your about to deploy to Production actually really been through UAT (or did the developer slip in that one last easy fix with no testing?). Does your Application Setup really reflect the setup you tested? Did the Consultant complete the BR.100 before he left for a contract paying 30% more? Can anyone actually follow that BR.100……?

So you have a thousand customization files (actually isn’t a lot when you count Java files, setup, etc). Have you ever taken that as a single software release and had your DBA’s (not your developers that know to run script 2 if script 1 falls over and hack script 3 to make it work, without anyone finding out) and applied to an exact clone of your Production system and had a 100% success rate? Have you repeated this multiple times as you roll towards the go-live date, just to make sure that any changes between the time you last ran it and any updates to Production didn’t cause any adverse impact? Are you sure that the DBA’s can run all your customization deployments in a very short-time frame. (Your typical go-live window is a maximum of 4 days with two of those the weekend, after that your business will scream if you go one hour over). Is your deployment manual and if so, will the DBA do the same steps each time? If your deployment is automated will it actually work?

We had 600 application changes. Have you ever checked how long it would take to deploy all of those changes? Have you ever done an entire setup and checked all the setup was valid and tested properly? Have you thought of the issues of letting knowledgeable people do the setup in Production in terms of access, SOX, etc? If new people need to do the setup, as the project team cannot have access to Production, are they familiar enough not to make mistakes? Have they ever practiced the setup on an R12? Interesting how long it takes to carefully setup and check 600 application setups. Just wondering if you ever thought how long it actually takes to setup 600 application setups for a single person. Do you know it’s impossible to do that setup with one person? Have you ever bothered to do a careful review of all setup, access needed and then balance setup streams across different people to make it achievable in the very short transition time the business has granted you whilst still ensuring segregation of duties, SOX compliance, etc?

Have you ever considered setting up some functions, menus, etc in R11 BEFORE you go-live. Or some new program definitions? Or perhaps having these as Loader files, so you can load them instead of manually setting up? Of course if you load them, have you checked that the developers have generated these LDT files correctly?

How many minutes will the R12 Upgrade run on Production hardware? (I ask for minutes because you should know very precisely the estimated time of this – if you don’t you are in serious trouble. For info an R12 will run for a very, very long time indeed, even on a small ERP database.). Have your DBA group ever run an R12 Upgrade with no failures? How long will it take to upgrade the Technology components or database? Have you ever seen a detailed step by step upgrade document from the DBA Team? Are there any hot wiring the DBA team apply during the upgrade due to known failures? Are you aware of these known failures and the implications? Do you know that an R12 Upgrade will require some very serious shift working? Do you know that your DBA’s will be seriously burned out as they have the very first task in the actual upgrade of running the hundreds of thousands of scripts that the standard R12 upgrade runs. And they need to actually be awake and watch this constantly, because if it falls over, you don’t have the time to lose 8 hours on an upgrade weekend…….

Wondering if you’ve actually even got a transition plan? Have you got an hour by hour schedule, with milestones and an effective method to monitor setup, customizations, actual upgrade, etc? On an R12 upgrade transition, you have no slippage time for anything. If you slip, the team will get stressed, you’ll get stressed and mistakes will be made.

Bottom line is have you REALLY rehearsed an R12 cutover BEFORE you actually do it? Do you really feel confident that first time you do it for real will be on your Production system……….if so you must be a clown……by the way do you think the McDonald’s clown is intrinsically evil????? (My Attorney has advised me at this point to clearly state that is a question and in no way implies anything on McDonalds whose food I have been legally advised to state I love and the picture below is in no way related to said company).

If you take a look at the Oracle Upgrade manuals, it’s interesting how many pre-req steps potentially have to be done. Ironically if you miss some of these you can have major support issues going live. I wouldn’t leave any Payments in the middle of process. I’d be carefully checking the effect of an upgrade half way through each of your workflows. Is there any effect on transactions that are left half completed? Do you even have these in your cutover procedures and does everyone in the Business know from your carefully documented plan exactly what their responsibilities are prior to bringing down their R11 ERP. If you (or the Business) makes mistakes here, you are in real trouble. Do you have scripts to check everything has been accounted, no payments are sitting halfway or are you just relying on your business to do the right thing with no checking whatsoever…….Just to add a little time pressure, your DBA’s will be screaming at YOU if you go one minute over the agreed time to shut down R11 and hand it to the DBA team……..of course you’ll have to herd dozens of business units across 27 countries to complete all activities before that time…….but then they want to do their business not your upgrade……which leaves the DBA’s screaming at you…….(The DBA’s will be working all through the night, so every hour you delay, means an hour extra in the middle of the night so a little empathy for them here folks……..Show them you care for their welfare….at least till your upgrade is over. Alternatively take the selfish approach that the DBA that is working all night might be the same one running your customization scripts after two hours of sleep because of your poor planning that could wreck your upgrade when he makes a serious mistake because he’s so tired……….)

Have you made any plans to backup the original system? What are your fallback plans if the upgrade fails? When will you be prepared to pull the plug and under what criteria, over the transition cutover to R12? Who is going to take responsibility to give the green light to go live? Will the senior management even be there at the weekend to make that decision?

Just wondering if you told your team and others (Unix, Email, Legacy, etc) that they would need to work some very stressful and long hours over the cutover weekend. Wondering if any of those key staff have their holidays planned…….?

Have you made plans to clone and test the actual R12 on the cutover weekend? If not how do you know all your setup, customizations, etc actually were promoted cleanly and your R12 actually works? You will need a large team to test in such a short period of time. Do you even have a cutdown test plan to make sure you hit the key functionality? Have you got all this planned? Probably not…….And if you find stuff missed, do you have a proper controlled mechanism to track it, test it and release it to Production in a horrifically short and highly pressured weekend in a safe manner?

What’s amazing is that everyone plans a transition, but no-one actually thinks about turning on the R12……….when you enable the workflow and the concurrent managers are you just going to open the tap, or are you carefully going to release production jobs and monitor. I’ll tell you that your single most stressful time is when you commit to switching on these elements over the weekend. I bet you that your heart will be pounding and you will be sweating at this point. The entire company ERP and Business hangs on your Go or No-Go signal. Once you’ve done that, there is no turning back, there is no backing out and you are on R12. At that point if you’ve got it wrong, you’ll be looking for another job. (Doesn’t that picture below remind you of your boss when it all goes wrong……?????). I wonder if that Oracle Support girl I dealt with for the AP Trial Balance wore red lipstick…….ahhhhhhh……..

So your live, but it’s not over……..

Serious Outages can occur

Lack of Staff to address serious support issues post go-live

Patches are required after go-live

Are you ready for serious outages? Did your transition plan have proper contingency plans for these events? If not exactly what will you tell the Board and the Business?

Do you have the appropriate staffing plan to handle what could be a very rough few days (or weeks) or months? An R12 will almost always lead to a spike in calls. Are you ready? Did you bother to give your people proper notification that they may be required to work late nights and weekends for the first few weeks after go-live? Or is it just a line item on your plan that you didn’t bother to communicate?

Did you re-synchronize all of your ERP and Legacy systems in terms of interfaces? Some of these integrations may require manually running interfaces to catch-up with transactions for the days you were down.

With major business events such as payroll and closing, do you have any plans to clone and rehearse these events, prior to these happening? Now if you can clone and simulate an R12 closing after you are live, but before the closing happens, it’s better to find out serious problems and have a few weeks to fix them right? (Of course this has you wondering if you should close in R11 and then immediately move to R12, but of course, then your Financial reporting would be straight after go-live, so you can’t win right?)

Have you got any plans for patches after go-live? What happens if you really need to apply patches? Are your project team still there? Did you set business expectations just in case? Always better to make business aware of the real risks inherent in an R12 Upgrade. That way, they’ll be incredibly happy if you manage to get it right.

I hope you remembered to get before and after reports from key areas, such as your AP Trial Balance (that Oracle Support girl was just so nice, ahhhhhhhhh), GL Trial Balance, etc. And keep a copy of your Production database on R11, whether on tape or disk. When the auditors come and take an interest in it, as they will it would be unfortunate if you’ve not done this and it causes issues with your auditor’s signing off your company accounts.

Also before you go live just have a quick review of JVM sizing and parameters. If you fail to tune and size this properly you could see your Applications crashing, intermittent crashes, etc. Getting your JVM sizing wrong leads to absolute nightmares……..Ask your DBA’s “Have you size the JVM’s and if they say what’s that, start to worry……”. If you have too many users and not enough JVM’s and OACORES then BOOM !!!!!!!!

Finally is there monitoring of key components in place? Are you touching base with all the areas of Business (and Legacy and Web and email and and DBA and Unix and other teams). Are you even aware of major production issues or is your Production setup so poor in terms of support that you’ll be the last to know……..

And lets break for another movie recommendation. The single scariest movie I have EVER seen. This was the 1970’s television mini series called Salem’s Lot. It is absolutely terrifying. Get it on DVD….it’s one creepy movie from the Master of Horror Mr. Stephen King. I saw this when I was ten, and I didn’t go out for weeks after dark……It’s a classic ten out of ten movie. But REALLY TERRIFYING.

This isn’t an exhaustive list of what can go wrong in an R12. Now note this article has been written in a strong manner to scare the crap out of you because if I do that then you’ll start to take every last point out of this article and make sure you’ve covered it. Make no mistake – R12 Upgrades are a serious risk to your career. (However don’t let some consulting company come in and scare the crap out of you. They are in it to make as much money as possible from you for an R12 by scaring you as much as possible. I write these articles for FREE and hope people can avoid the mistakes that I and others made in our R12 Upgrades). If you read this article and take everything on board, you’ll be strongly versed in R12 risks and be able to evaluate what these consulting companies are telling you and separate the fact from the fiction…….

And a final note. Premier Support for 11.5.2 ends in November 2010…..that gives you about a month left……Interestingly you also need to have minimum baselines in place for R11 if you want proper support from November 2010……….And will you get hit by fees if your still on R11 in November 2011??????

Have you read Metalink Note 1178133.1…….

“Starting on Dec 01, 2010, Oracle E-Business Suite customers on Release 11i10 will only receive extended support for new bugfixes as described in My Oracle Support, Minimum Baseline Patch Requirements for Extended Support on Oracle E-Business Suite 11.5.10.”

That’s a lovely little bit of legal small print from Oracle with major implications for you………Getting to those minimum levels will be enough to “persuade” most customers to jump to R12 instead.

It’s funny also that so many people don’t think they can start doing preparatory work in R11 that will greatly ease R12. Have a look at the references below. Work spread over R11 and R12 will greatly ease your stress in the R12 Upgrade itself, as you’ll have less to do. (Did you know Solution Beacon even has Vision instances you can play with for free, even before you install an R12 in your company’s test environment…..)

Let me tell you briefly our R12 story. We had some very serious problems with quality (12.0.4 RUP5) – an early R12 release in many respects. However despite that we got through a Finance, HRMS and Procurement Suite R12 Upgrade across 20+ countries with strong planning. Our transition finished 30 minutes ahead of the planned 96 hour time window (I wasn’t kidding when I said you should know the transition in minutes…..). Yes, I was terrified when I made the call to go-live. But due to all the planning and a great Project team, we went live with no major issues on 3rd August 2009 and I went home at 5PM two days after go-live (And for the cynics, no I didn’t spend the first two days constantly in the office 48 hours straight…..I got home at a reasonable time the first two days as well). We cleared Payroll, breezed through month end and went live on two Payroll rollouts to two new countries one month after go-live.

Of course now I’ve told you all the risks, well you’d be a fool to actually fall into those holes right? Take this article, make a list, make sure you cover each and every one. Then you’ll make your R12 a huge success.

An R12 is certainly a very achievable project, with significantly reduced stress and risk, if planned correctly.

Hope I haven’t scared you too much……..

Enjoy your Halloween.

References

As always, my appreciation goes to the kind authors and contributors of the following articles and resources. Some of these are especially note-worthy, because those people are putting out their failings (as well as successes) on R12 in a very public and open way, for the benefit of those about to travel the R12 Upgrade road.

General Blogs and Websites

OAUG Website

Chris Warticki’s Blog – EBS R12 Support Resources – Consolidated

Oracle Applications Upgrade Guide: Release 11i to Release 12 (B31566-01)

Oracle Financials Applications Blog – R12 Lessons Learned

Analyzing, Planning and Executing an R12 EBS Upgrade

Steven Chan Blog

Risk Management: Tricks of the Trade for Project Managers (Rita Mulcahy)

My example of an R12 Risk Spreadsheet (Excel 2007) – Available on Request

 

Metalink

Oracle E-Business Suite Release 12.1.3 Release Update Pack (Patch 9239090)

 Oracle E-Business Suite Release 12.1.3 Readme (Note 1080973.1)    

Upgrade Manual Script (TUMS) – Patch 5120936

Metalink Note: 580299.1 – Best Practices for Adopting Oracle E-Business Suite, Release 12

Metalink Note: 394692.1 Oracle E-Business Suite Upgrade Resources

Metalink Note: 562887.1 R12: Helpful Tips for a Successful R12 Oracle Payables Implementation

Metalink Note: 437422.1 R12 Troubleshooting Period Close in Payables

Metalink Note: 73128.1 R12 Troubleshooting Accounting Issues

Metalink Note: Oracle Support Upgrade Advisors (250.1)

                             Tech Stack                     Note 253.1

                             Financials                     Note 256.1

                             HRMS HCM                    Note 257.1

                             Manufacturing              Note 258.1

Metalink Note: Extended Support Patch Level Verification in Oracle E-Business Suite Release 11.5.10 (Note 1178133.1)

Metalink Note: R12 Upgrade Considerations by Product (Note: 889733.1)

Oracle E-Business Suite R12.1 Financials Pre-Upgrade, Setup and Operational Tips (Note: 1104163.1)

Oracle E-Business Suite Release 12.1 Information Center (806593.1)

Database Preparation Guidelines for an Oracle E-Business Suite Release 12.1.1 Upgrade (Note: 761570.1)

Oracle Applications Installation and Upgrade Notes Release 12 (12.1.1)

R12: Period-End Procedures for Oracle Financials E-Business Suite (Note: 961285.1)

Oracle Open World 2010 – On Demand

For those with access to Oracle Open World 2010 On Demand there are good references at http://www.oracle.com/us/openworld/index.htm

Mission Impossible: Oracle E-Business Suite 12 Upgrade on 3 Continents in 6 Months

Success with Oracle E-Business Suite Release 12.1.2 Upgrade Drivers

Algar Telecom Upgrades to Oracle E-Business Suite R12   

Get Ready for Oracle E-Business Suite Release 12.1: Tasks to Complete Now

Oracle E-Business Suite 12 Upgrade : Best Practices to Obtain Business Value

Planning your Oracle E-Business Suite Upgrade from Release 11i to Release 12.1 (Superb presentation – highly recommended)

Oracle E-Business Suite 12 Upgrade: An Easier Ride on Nine Miles of Bad Road

Oracle E-Business Suite R12 Upgrades: Have you Thought about the Details?

Real-Time System Assessment of Oracle E-Business Suite 12 Upgrade: Case Study

And finally talk to other companies, use forums, other areas (such as http://www.linkedin.com) and make contacts with people that have done R12. This again will help you considerably on your R12.

Footnote:

Some of the great graphics were the work of:

WWW.MYSPACEGRAPHICSANDANIMATIONS.COM

R12 – A Detailed Transition Plan

September 19, 2009

So your UAT is heading for completion, you’ve got as many bugs out as you can and your ready for go-live.

Problem is…..this is the most dangerous part of the entire project. All the hard work and success could go be wrecked over a poorly planned weekend (or long weekend if your upgrade is really tricky or big)

Planning is the key.

It’s recommended that everything is documented very, very heavily. Start off with a PM.030 and get into your head exactly what your going to do.  And a PMP.030 Transition Plan isn’t just what you will do over the cutover weekend; its before, during and after – how are you going to support the R12 when live? Are your teams ready for that?

Get all your Functional Setup into a Spreadsheet, BR.100 per module or other installation documents. Give assignments to each person. Make it clear who does what, when they do it, order, dependencies, etc. On a cutover weekend there is ZERO time for mistakes.

Have a spreadsheet of all tasks to countdown, dates, people, etc. Mark them off one by one as you go.

Have another spreadsheet for all cutover tasks. I suggest some sort of matrix where setup is assigned and you can see each persons workload. Then try to balance that workload. If one person is too slow or too busy on the cutover, you’ll hit major delays on cutover weekend.

Your source code should be in a Source Control System (we used Subversion – it’s FREE – it’s GREAT). You should repeatedly clone Production and Test the deployment until very close to Production. Also apply Functional Setup and retest.

Are you confident that what was passed in UAT is what you’ll put in Production?

Keep doing the clone, setup and test until your happy that the answer is 99.9% yes.

Also during final approach to go-live, do a simulated financial close on the Clone. Trust me this will save you a lot of heartache. Wouldn’t you rather find disastrous stuff you need to do before you hit it in the crazy, high pressured time after go-live. Pay careful attention to AP closing. It’s tough. Run the accounting health checks.

You must have an automated mechanism for code application and your DBA’s should be familiar with this. And by automated it can be as simple as a shell script to move stuff to the right place and a script to cut and paste and compile objects. Get this 100% before you do it.

One tactic that worked well for us was to do setup on 11.5.10. We’re not saying everything, but some menus, some value sets, messages…..every little helps. Of course you can also do these on the day using FNDLOAD, but the point is the more you can do ahead, the less you have to do in a high pressured weekend.

Get the meetings going with Business. Make sure everyone is aware of how and what is getting done. We closed on R11.5.10 for all subledgers prior to R12 apart from GL. This really helped reduce overall errors and risk, as all that had to be done in week one of R12 was a GL close. It went very smoothly. And then you have 4 weeks until your first real R12 close – gives you time to simulate another close 3 weeks into your financial period. Now some may disagree with this approach and prefer to go-live after month end closing. I understand this is another perfectly valid option. In our case we made the decision to jump immediately after subledger closing as we also run two large payrolls twice a month…….so our cutover windows for making the jumps were seriously limited. Also we had done this procedure many, many times (including on very recent clones), so had a level of confidence that this was safe to do. If your cutover testing isn’t giving you that confidence, suggest you close in R11 fully then jump to R12.

Make sure you have plenty of meetings with your DBA Team, Unix Team and Management. I prefer to plan for the worst; set the expectations low and then everyone will be prepared mentally for a tough time. If it goes smooth then everyone is happy. Doing an R12 is the toughest upgrade anyone will have ever done, don’t underestimate the issues you will have. Many, many companies have had horror stories during go-live.

So your plans are done, teams are briefed and your now ready for launch. Did you actually remember in advance to tell all users (and post notices on your website internal and external) that your system is unavailable due to upgrade? Did you remember to tell all the other systems (Legacy, data warehouse, etc) that your system is unavailable?

Oracle people (including myself think the world is Oracle). Remember there’s more to life than Oracle…..and most companies have a lot more systems than just Oracle, although Oracle is the foundation…..so if you take that away and forget to announce, you’ll have some very unhappy people……

Make sure you brief your helpdesk. Make sure they know the escalation procedures. Even if you do a great job, it’s going to be a busy time. Make sure you Oracle ERP Team is also ready to work round the clock post go-live, just in case…….

We brought our ERP down at 5:00PM on a Wednesday after closing all subledgers. Actually it was 5:52PM and the DBA’s were already complaining like crazy.

The DBA’s in our case brought the system down and started the export. They worked through the night to export from IBM and bring this into HP Linux. (Yes, we throw in not only an upgrade but also a migration for fun…..)

The Upgrade started on the Thursday and was then backed up early Friday morning.

The DBA’s automatically deployed the code (7,000+ files) onto what was effectively a live Production system. (with no one else on and of course backed up on R12 AND R11.5.10  before we started).

The Deployment Team rolled in at 10AM on Friday, sat around for a few hours and then started at 11:45AM to do setup. Of course we screamed at the DBA’s for being 1hour and 45 minutes late, but they claimed we were 52 minutes late handing the system over on Wednesday…….things are always tense during an upgrade cutover.

Functional Setup continued until 9PM on Friday night (remember all those new modules, Payments is a large setup, and there are loads of System Admin tasks  – Hint split some setup into Application Developer responsibility. Do others setup using FND_LOADER. I think your System Admin may be a bottle neck – that’s what happened with us as there were just so many changes between R11.5.10 and R12)

Around 9:30PM Friday the database was ready to be backed up and cloned. DBA’s worked to clone the database and make it available at 10:00AM Saturday.

Testing Team rolled in on Saturday and did a quick Financial Reconciliation. All well and good testing started on major key functionality. I hope in your cutover plan you detailed what would be tested, who would test, etc.  In addition each Team Lead should prepare a more detailed Test Plan for the transition and brief their respective teams. Concentrate on key functionality. If you do this, you’ll be doing well if it all works on day 1. Testing continued for about 10 hours on Saturday and started at 9:00AM Sunday.

As we caught problems, we logged them and got our System Admins/DBA to deploy missing items to Production (with heavily controlled approval, logging of bugs, etc so we tracked every single change)

Around about 4:00PM on Sunday we had a Team Lead meeting. Each Lead was asked for a go/no go signal. With the decision to go made, we went to Management. The current status was reported and management after discussion gave the green light.

All workflow services were turned on (with emails being caught in our email system so we could monitor and then release – nice little trick). Once we were happy emails/workflow was OK, we released one (a Leave from one of my staff) and tried the complete cycle. Once we were happy with this, we released all workflow, all concurrent jobs (we had simulated release of all concurrent jobs on Saturday to avoid surprises) and there was no going back.

We went live on R12.0.4 (RUP5) at 5:00PM on 2nd August 2009, some 96 hours after bringing down the R11.5.10. (Although we are on RUP5 we’re really heavily patched on Financials so RUP5 is a bit misleading)

Teams had worked round the clock, across all areas to make the cutover a huge success – we finished 30 minutes ahead of planned time. Now that’s pretty good planning. Have you planned to that level? If not you should be.

On 3rd August, our users logged on to R12 and we were fully operational.

R12 Upgrade Plan

April 9, 2009

1. The most important part prior to starting an R12 Upgrade is to build support for the project. If the Business isn’t on board, it’s going to hit real problems when you want their time – and it’ll take a lot of their time……Look for the key drivers of why your business needs an upgrade; for us we had many new Oracle modules and we felt it was better to do those projects on R12. rather than implement in 11.5.10 and upgrade in R12 later. A move to Linux and the age of 11.5.10 were other critical factors. With R12 being out for 2.5 years by the time we implement we figured it would be relatively stable overall – a lot of companies got badly burned by jumping way too early into R12.

For those companies not in R12 now is a good time to start thinking – the product is stable in most areas (even AP is getting there) and the market is depressed, so there are a lot of available resources (albeit mainly in 11.5.10, but most modules remain pretty similar overall)

2. A study approach of building a Business Case to assess the Upgrade may be a good idea. It won’t cost a lot, but will allow you to make a very informed choice ; the new functionality, the customizations you have, the estimate of effort, time and cash it going to take, pitfalls and other issues.

This approach lets you build concensus and support. It minimizes risk and allows you to build a strong business case to get necessary funding.

For us, we identified several major issues upfront; Payables was a complete re-implementation for us. Payments was a complete re-implementation. We had a LOT of customizations. General Ledger customizations require re-work; small but 60% of our customizations were hit…..Self Service screens must be migrated. All Payment formats require new build. This was going to be the biggest upgrade I have seen in a very long time…..Understand also the architecture – Oracle ERP isn’t an island. It usually talks to an awful lot of other systems and if you don’t understand this at the outset it may hit you just before go-live or worse……after.

We also used a lot of good documentation on R12. Look on Metalink, google, OAUG. People have made plenty of mistakes and published where they went wrong – with the documents available now you should avoid the pitfalls for most.

Remember this is not just an upgrade – Oracle has brand new modules so this is also a major implementation project; Payments is new and completely different, EBiz Tax is new, Subledger Accounting is new and if you are running AP , you have no choice but to implement the Payments. The other two are now core modules that must be implemented, even if your not using Tax………

Be careful of your plans for current Production during the Upgrade. Most of your resources will sink into R12, so try and reduce big new commitments.

3. Starting the project. Make sure you have enough hardware……you’ll need to run a lot of instances AND still support your existing Production on 11.5.10…….If you don’t have the hardware your project will suffer badly. Resourcing wise there are lots of options, but it will take a lot more than an upgrade from 11.5.5 to 11.5.10. I can’t stress enough how important it is to get top tier consultants – a few really good people (albeit expensive) actually saves you money. Depending on your customizations, you’ll need a fair chunk of development resources. But by doing the study you’ll be able to make good estimates. (Don’t underestimate Payables and Payments – these will take a VERY LONG TIME).

Remember that resourcing won’t just be Oracle people; you will need people from the feeder systems involved (Legacy Teams, Data Warehouse Teams – the stuff you change may create work for them), the DBA Team, the Unix Team, the Email Team, the ERP System Admins, potentially people from external (such as Citibank). Most of all , your going to need your users.

4. So your ready to start. Once your over the ubiquitous kickoff meetings (which shouldn’t have any surprises as you did all the planning, lobbying, studies in advance right :-)? ), its time to get the team going.

Obviously an R12 ERP would be helpful. Make sure that you have the hardware, know how to upgrade a copy of Production and have it available BEFORE you have your Upgrade army. It’s a lot of cash to burn through otherwise….If you’ve got a good DBA Team they should be able to get you an instance in 2-4 Week timeframe.

If you have good internal resources, then this really helps. Otherwise external consultant will need to take a bit of time to understand how you use Oracle ERP – sure most companies work in roughly the same way, but every company has its difference; some are very different indeed.

At this point I normally split off into streams; I split into modules, team leads across each module (we have 31 standard and custom modules, spanning HRMS, Procurement and Finance).

The Functional Team should be preparing an initial high level Test Plan, working closely with Business. In parallel they should use their knowledge (and the R12 Upgrade Guides) to identify the relevant changes to the company. The Test Plan should be comprehensive and signed off once complete. This should cover all Test Scenarios across all Business Areas, including external systems. The Functional Team should also try to identify, in conjunction with the Business and Development Team, customizatons that can be removed with new functionality provided as standard. (We removed 10 check formats in a single morning – imagine the saving when you need to rewrite these, do Technical MD.070, Functional MD.050, Testing, etc. A Little time doing this could bring significant savings).

The Development Team should again be split across Modules. If you have a customization register (and we have all 1,600 customizations registered so we don’t miss any) you can use this as a very good monitoring tool. Working closely with Functional Team, identify the core stuff – there is no point in working on a report first, if you can’t enter the invoice to test the report. Once you prioritize the customizations (and possibly modules if you have limited resources), start working to upgrade these, document and test.

Now based on R12 experience, from a Technical Perspective, AP and Payments are very difficult. GL customizations had 60% small updates required. Self Service needed a re-implementation to move from MOD*PLSQL to standard Java pages. Custom modules were largely unaffacted and OAF upgrades with minimal overall issues.

If the Developers update the Customization Register with flags saying Unit Tested, Fixed, Require Fix, it becomes an excellent monitoring tool providing exact progress on your project. Ensure that you use source control. There are cheap, cheerful and excellent solutions including Subversion. This is a great tool.

Once the Test Plan is signed off, Consultants and QA’s should start ensuring that comprehensive Test Scripts are available. As they are being written, the application should be tested initially to throw up the bugs. The earlier you find them the earlier you get them to Oracle Support to get fixes. R12 does have quite a chunk of bugs, especially in Payables. Keeping a close eye on Metalink is important – when new RUP’s are released, you have a very important decision to make on whether to apply. If your well into your project, the advice I would give is not to apply….they’ll cause a lot of issues, it’s newly released and you’ll be the poor guys finding all the new bugs. Leave that to someone else. Obviously if you have critical errors, pile the pressure on to Oracle to provide single patches. If you are early in the project stage, then apply the RUP’s, but do it on a Patch System and check it first…..they can cause real damage. At some point though your going to have to freeze patches, otherwise you will never finish.

Whilst all this is going on your DBA team should be working on practise upgrades and optimization. It will take many upgrades to get it right.

5. So you have your Test Plan, Test Scripts and Customizations. We run what we call an Internal UAT. Basically its a UAT but it is carried out by the R12 Upgrade Team, not the user. Try and get an exact copy of Production, upgrade fully, deploy the objects and do the setup. (Be careful to carry out pre and post upgrade steps)

Given that some modules are going to be delivered before others, you may want to do this staged. It’s not ideal but it does compress the overall project time. You will have problems on Payables, and should that hold up your HRMS track? I would argue no.

The point of the Internal UAT is to run through the entire test, as the users would do. This includes external systems such as Legacy, your planning systems, Citibank, or whatever. All bugs should be recorded. (We use a tool called JIRA which is fantastic)

Be particularly careful to look at the upgrade process itself. You should simulate a close on your R11.5.10, reconcile on 11.5.10, then do the upgrade then reconcile again on R12 and make sure the R11.5.10 and R12 add up. This is critical and we found what we think is a major issue on accruals when we did this.

6. Another clone of Production should be taken. All objects, customizations, setup should be done on the UAT environment (If you have not been keeping deployment registers, BR.100 would strongly suggest you start doing so……) Usual UAT kickoff meetings should be held and contact lists given to users. Your team will need to work hand in hand with the Business Users to co-ordinate the UAT. It’s going to be a lot of work, a lot of new and unfamiliar screens and quite a few complaints, no matter how well you’ve done till now. The Test Plans and Test Scripts should have been agreed months ago, with full review and signoff.

One trick here is to put the Test Plan on a shared drive and ask the user to mark Pass/Fail/To Test as they progress. As an Upgrade Manager you can then review and summarize easily progress across the entire ERP Suite. Now that is sweet…….Again delivering some modules into UAT before others may be an option if you want to progress quicker. Others prefer to deliver the entire UAT. What I have found is that by progressing some modules quicker, it gives the project momentum and when you put out weekly progress tables to users, they don’t want to be last to finish.

Make sure all Test Scripts are tested. Make sure Month End and Year End are similuted, with proper reconciliations. Make sure all the systems to and from Oracle ERP are tested and continue to work with R12. Make sure integration between modules is smooth and working – organizing this across departments is tricky. A UAT room for this can be handy…..providing cookies and drinks (non alcoholic please…..don’t want people testing when they are drunk….) creates a good ambience.

Record all issues and bugs as you go – again a tool like JIRA is great. Monitor closely and ensure that when your users are doing UAT, your R12 Team are there to assist. Some areas may need twice weekly meetings on certain modules. This will help to manage difficult UAT areas, stop user frustration and make the UAT a little less painful for all. We did this on Payables in particular.

On completion, ask the head of each area and other users to sign off a standard AIM Acceptance certificate.

7. Whilst the UAT was going on I hope you were also planning to do something else right? As UAT goes on, you should be planning what will be a very compex and difficult cutover. The team should also be doing rehearsals by taking copies of production, upgrading, deploying, setting up and testing. All bugs should be recorded.

Don’t underestimate the deployment.

You should have deployment registers for each module, including tasks for DBA, System Admin, etc. You should also have a big thick document on the steps the DBA’s do in the actual upgrade. Separate deployment registers with hyperlinks to relevant setup docs, BR.100, etc.

Review the deployment registers and figure out what can be done in advance of R12. We transferred new programs, menus, value sets, etc to R11.5.10 without switching on certain components. All of this saves a little bit of time and will ease the stress of the upgrade weekend.

Ensure the BR.100 (or other setup instructions) are clear, concise and complete.

Call meetings with the users and make sure that the cutover (and possibly closing processes) are clear.  We are doing the closing in R11.5.10 just before cutover, closing down all transactions, etc. Why? Couple of reasons. If data is left open in R11 (say unaccounted) it will give real problems in R12. (We worked for months to get the script from Oracle and luckily we found this prior to go-live). Secondly it allows you a whole month before your first closing in R12. You can do a couple of simulations of Production and sort out issues if needed. The users will lose Production for a few business days, unless you do it on a holiday……..its a big upgrade. Our’s runs for 42 hours.

Get a register of who needs what responsibility on the day and how many setups they need to do. Set these up in advance so you know who needs what access and are not scrambling on the day. It also lets you balance the workload. We have almost 600 setups. A lot in Payments.

Deployment of code. If you have used Subversion (or another Source control system) then you can grab all changes and give to the DBA’s. Make sure your directories match Oracle’s and Subversion as that will make deployment easier. A lot can be automated and this will save a lot of time on the cutover. Now with all the prep work the DBA’s started over the previous months maybe this is something they could also have been working on…..There are a lot of standard Loaders that can save lots of time.

Talk to Oracle and make them aware of your key dates, cutover, month end, payroll, etc. They can get you critical support management over that time.

Try and do as many rehearsals as you can whilst users are doing UAT. The closer the rehearsal is to an exact recent copy of Production the better.

Make sure you do patch reconciliations between your UAT and Rehearsal databases.

Hold a few internal meetings to ensure that everyone in IT knows exactly their role. Watch out for people booking holidays in advance. One key person out could derail your go-live.

As PM, you should be writing a PM.030 Transition and Continegency Plan that covers all the angles.

If the cutover fails all your hardwork will have gone to waste.  Rehearse, Rehearse, Rehearse……

8. Cutover

Generally over a four day window. Holidays are ideal but make sure you tell Business well in advance if their key system is going to be down during business hours. We plan to be down by 5:00PM on a Wednesday and up by 7:00AM Monday. We also have a migration to Linux thrown in. Wednesday, Thursday are days for the actual Upgrade, Friday the Upgrade Team roll in to do setup, customizations, etc. Saturday and Sunday are for testing with a go-live decision on Sunday evening. As we are moving server, our contingency is simple; if the cutover is a no-go we simply turn our old server back on……Don’t repeat the errors you made in rehearsals, keep a log of everything that failed and make sure it doesn’t happen on your go-live, as you simply won’t have time to troubleshoot.

Be very careful that all transactions are accounted, workflows complete (as much as possible), payments complete. Review the Best Practises from Oracle for more detailed advice. If you don’t do this you will hit serious problems.

9. Post Go-Live. There are going to be problems on anything of this scale. Thats a given. Have all teams on standby so that when issues arise, they can be addressed. With Oracle Critical Support you’ll hopefully be in better shape than with standard support. As soon as possible clone production to a test system and simulate the payroll, closing and other key events in advance.

10. Party? If you do an R12 Upgrade, you deserve a big party. R12 is the most challenging upgrade you will ever have done in Oracle ERP. Good Luck !!!

R12 Upgrade – Top Ten Pitfalls

March 13, 2009

If you were doing an Oracle R12 Upgrade, what Top Ten would you watch out for?

1. An R12 is a lot bigger than R11.5.x to R11.5.10. It will take considerably more time and money

2. Watch out for Oracle Payables; its had some major structural changes that have affected the stability overall. In 12.0.4 major functionality works but the accounting can still cause issues. It’s a very hard module to get through a UAT.

3. Watch out Payments; its completely new. Accounts Payable Payments functionality is now covered by this. It’s entirely in OA Framework and very, very different. All the formats will need to be redone. Basically a re-implementation so needs a lot of time and extensive user training.

4. Any customizations in Mod*PLSQL? If so, none of these will work. They will need to be upgraded to OA Framework

5. Watch out for new modules; E*Biz Tax and Subledger Accounting. These need to be configured and tested.

6. If you have customizations on General Ledger, 66% are probably going to need some re-work as Sets of Books heads to Ledgers. Minor work, but hits most customizations, custom value sets, etc.

7. Major areas such as Purchasing, HRMS, General Ledger, Payroll, Accounts Receivable are generally stable in 12.0.4. This is one you don’t need to worry quite so much on, so it’s not all bad news…….

8. In 11.5.5 to 11.5.10, 8% of our customizations failed. From 11.5.10 (CU1) to 12.0.4, 24% failed. Again give more time for the upgrade of customizations. There are some fairly large structural changes especially in Accounts Payable and General Ledger.

9. Banks – very different and there appears to still be a bug that creates duplicate banks that give an error in 12.0.4. Be prepared to spend time cleaning up your Bank Accounts. It all moves into TCA. New Rules means the Bank Number becomes mandatory for certain countries.

10. Give plenty of time to your UAT. It will take considerably longer.

So what’s your Top Ten Pitfalls?