A Successful Application of CRS Patchset on RHEL4 x86_64. So?

Upgrading CRS to on RHEL4 x86_64
It is quite likely I’m the last person to get around to updating my 10gR2 CRS—er, clusterware—with the patchset. Why? Well, upgrades always break something and since CRS was really quite stable for the specific task of node membership services (libskgxn.so), I was happy to stay with it and skip Compared to the offal we referred to as 10.1 CRS, I have been very happy with 10gR2 CRS for the main job of CRS (which is monitoring node health). Fencing is another topic as I’ve blogged about before.

Oh, Great, He’s Blogging Screen Shots of Stuff Working Fine
Well, I can’t think of anything more boring to look at than a screen shot of a successful execution of an upgrade script. With the upgrade it is root102.sh—the root script that OUI instructs you to execute in $ORA_CRS_HOME after it finishes such activities as copying pre- files over to ${ORA_CRS_HOME}/install/prepatch10203 and so on. So why am I blogging on a successful application of this patchset?

Knowing How Bad Something Has Failed—and Where
Often times when RAC installation and patch applications go awry—a very frequent ordeal—it is often nice to see what you should have seen at the point where it went wrong. Such clues can sometimes be helpful. It is for this reason that when I—and others in my group—write install guides for Oracle products on our Database Utility for Oracle clustering package I often include a lot of boring screen shots.

Testing a Rolling Application of CRS
As described later in this post it is fully supported to implement a shared ORA_CRS_HOME—as it is on OCFS2 and Red Hat GFS. In fact, there are several permutations of supported configurations to choose from:

  • Local CRS HOME, raw disk OCR/CSS
  • Shared CRS HOME, raw disk OCR/CSS

As a normal part of my testing, I wanted to make sure the storing the OCR and CSS disks on the PolyServe CFS in no way impacts the ability to perform a rolling upgrade of local ORA_CRS_HOME installations. It doesn’t. First, OUI determined is was OK for me to do so because ORA_CRS_HOME on all three nodes of this puny little cluster were installed under /opt on the internal drives. The CRS files (e.g., OCR/CSS), on the other hand, were on PolyServe:

tmr6s15:/opt/oracle/crs/install # grep u02 *
tmr6s15:/opt/oracle/crs/install # mount | grep u02
/dev/psd/psd1p3 on /u02 type psfs (rw,dboptimize,shared,data=ordered)

The first screen shot shows what to expect when OUI determines a rolling application of this patch is allowed:

NOTE: You may have to right click->view the image (e.g., with firefox I believe)


Next, OUI instructs you to stop CRS on a node and then execute the root102.sh script:


If all that goes well, you’ll see the following sort of feedback as root102.sh does its work:


I was able to move along to the other two nodes and get the same feedback from root102.sh there as well.

To Share or Not to Share ORA_CRS_HOME
Oracle and PolyServe fully support the installation of CRS in either shared or unshared filesystems. The choice is up to the administrator. There are important factors to consider when making this decision. Using a shared ORA_CRS_HOME facilitates a single, central location for maintenance and operations such as log monitoring and so on. Some administrators consider this a crucial factor on larger clusters; it eliminates the need to monitor large numbers of ORA_CRS_HOME locations, each requiring logging into a different server. When ORA_CRS_HOME is shared in the PolyServe cluster filesystem, administrators can access the files from any node in the cluster.

A shared ORA_CRS_HOME does have one important disadvantage—rolling patch application is not supported. However, a patch that manipulates the Oracle Cluster Repository cannot be applied in a rolling fashion anyway. Although is not that such a patch, it is not inconceivable that other upgrades could make format changes to the OCR that that would be incompatible with the prior-versions executing on other nodes. Oracle would, of course, inform you that such a release was not a candidate for rolling upgrade just as they do with a good number of the Critical Patch Updates (CPU).
The parallel to shared ORACLE_HOME is apparent. Many Oracle patches for the database require updates to the data dictionary, so a lot of administrators ignore the exaggerated messaging from Oracle Corporation regarding “Rolling Upgrades” of ORACLE_HOME and deploy a shared ORACLE_HOME, eliminating the need to patch several ORACLE_HOME locations whenever a patch is required. This concern is only obvious to large IT shops where there is not just one RAC database, but perhaps 10 or more. These same administrators generally apply this logic to ORA_CRS_HOME. Indeed, having only one location to patch in either the ORA_CRS_HOME or ORACLE_HOME case significantly reduces the time it takes to apply a patch. To that end, planning a very brief outage to apply patches to shared ORA_CRS_HOME and/or ORACLE_HOME for up to 16 nodes in a cluster is an acceptable situation for many applications. For those cases where downtime cannot be tolerated, Oracle Data Guard is required anyway and again the question of shared or unshared ORACLE_HOME and ORA_CRS_HOME arises. The question can only be answered on a per-application basis and the choice is yours. PolyServe finds that, in general, when an application is migrated from a single large UNIX platform to RAC on Linux, administrators do not have sufficient time to deal with the increased amount of software maintenance. These IT shops generally opt for the “single system feel” that shared software installs for ORACLE_HOME and ORA_CRS_HOME offers. In fact, PolyServe customers have used Share Oracle Home since 2002 with Oracle9i and then with Oracle10g—it has always been a staple feature of the Database Utility for Oracle. With Oracle10g the choice is yours.

15 Responses to “A Successful Application of CRS Patchset on RHEL4 x86_64. So?”

  1. 1 JASON ARNEIL January 5, 2007 at 8:56 am


    You are quick of the mark Kevin, the patchset has been out for x86_64 only 1 and a bit days – I see the 32 bit linux version has had a months longer exposure. Does not sound like you encountered anything like the following bug that hits the 32 bit linux version:

    5667023 Linux: CRS does not start after applying

    It must have been kinda depressing for the dba upgrading to the latest and greatest version to find his cluster not clustering anymore.

    Sounds like it is a fairly straightforward upgrade though.


  2. 2 kevinclosson January 5, 2007 at 3:33 pm

    Isn’t that the SELinux issue? I don’t feel like logging in to MetaLink to double check, but the thing I always do is read the list of known bugs a patchset **introduces** and when I did I saw one about CRS not starting due to SELinux. Now, there is a *lot* wrong with that. First, is an upgrade which insinuates the first DBA that tried it had a functional cluster at

  3. 3 Jeff January 11, 2007 at 5:45 pm

    Jason, just upgraded two 2 node x86_64 clusters from to This bug also exists on the x86_64 platform as well, so make sure you disable before rebooting..

  4. 4 Alex Gorbachev January 16, 2007 at 3:25 am

    “However, a patch that manipulates the Oracle Cluster Repository cannot be applied in a rolling fashion anyway.”

    Kevin, as far as I remember, CRS upgrade does involve OCR upgrade. When done on the first node of the cluster and so on until the last node, root*.sh will not upgrade OCR but keep running with older OCR format. Running on the last node of a cluster, root*.sh output will be a bit different and OCR will be upgraded. I don’t have anything handy to test but I recall that when I ran CRS -> upgrade on Linux.

  5. 5 kevinclosson January 16, 2007 at 3:32 pm

    Right, Alex, I’m saying “if”. I have not seen one that touches the OCR, but it is not out of the question.

  6. 6 Ravi S. February 13, 2007 at 5:54 pm

    Thanks Kevin for the clue “SELINUX=disabled”, the isssue i had was after the patching the CRS will not start at node reboots. Disabling did the trick. Metalink has not documented this anywhere.

  7. 7 samad July 2, 2007 at 5:59 am


    I have one question regarding Oracle 10g two nodes RAC DB, currently its version is Now I like to upgrade it to if I upgrade only DB its enough or I must upgrade CRS, ASM all if yes
    can you please send me the steps how to start first.

    I read documentation but its too long I need direct steps to upgrade
    all. I have downloaded the patchset for 10g on Sparc, do i need to download CRS or ASM patch separately.

    please help me out I am new to RAC DB

    Thanks in advance!

  8. 8 Alex Gorbachev July 2, 2007 at 11:21 pm


    Reading documentation is a must. If you don’t have time for that – I’m sorry for you but you should find yourself a job other than DBA. Hire a qualified consultant or a remote DBA that can do this task for you properly and who already read documentation. You will save yourself time and money.


  9. 9 Lenin August 2, 2007 at 3:08 pm


    Be nice and help poor guy… He is busy with real life not like us….

    Remote DBA … Berluson and Ault can help there . (Higly recommended ) for tuning 1Mb database…. with 10Gb of RAM.

  10. 10 Epistemic January 31, 2008 at 10:07 pm

    Alex, I have never seen the answer to Samad’s question documented by Oracle, yet it is a serious oversight that they do not answer it in every patch readme file. If you had read Oracle’s documentation you would know that they do not answer this question.

    Samad, Not knowing the correct answer here is what I do. We also have ASM installed into its own home with its own set of binaries. We shut down both the application Oracle instance and the ASM instance. Then we run the patch to the ASM home and then the application Oracle home. Then we start both instances and apply catcpu.sql to the application Oracle instance. We do not apply catcpu.sql to the ASM instance.

    So far, this has worked well for us. If anyone knows if this is the right way or the wrong way to do this please let us know.

  11. 11 Alex Gorbachev February 1, 2008 at 1:28 am

    The sarcasm of my previous post was directed to “I read documentation but its too long I need direct steps to upgrade

    There are reasons for long documentation and making DBA’s bored reading it is obviously not one of them.

    Regarding your question…

    Here is the quote from documentation “Oracle® Database Patch Set Notes
    10g Release 2 ( Patch Set 2 for Linux x86”:

    5.1.4 Upgrading Oracle Clusterware

    The Oracle Clusterware software must be at the same or newer level as the Oracle software in the Oracle RAC Oracle home. Therefore, you should always upgrade Oracle Clusterware before you upgrade Oracle RAC.

    Further in the same document:

    5.8.2 Stopping All Processes for an Oracle Clusterware Installation

    This section contains the following information:

    Rolling Upgrade

    Non Rolling Upgrade

    If you have a separate ASM Oracle home than you probably have read MAA guides from OTN (Maximum Availability Architecture). As far as I remember, the original MAA paper mentions the ability to run database and ASM on different patchsets as one of the reasons for having separate Oracle home for ASM. Now, if you got to Metalink and search on “asm compatibility” – the first link would be Oracle Clusterware – ASM – Database Version Compatibility.

    PS: for “Lenin” Not all consultants are the same and remote DBA service != “Remote DBA” but I’m biased. 😉

  12. 12 Epistemic February 4, 2008 at 5:38 pm

    Good point. Samad did say the documentation is too long. As a DBA you just have to get used to the requirement that you are going to have to read all the documentation before making any major database change. Even then, you have no guarentee that all your questions and situations will be covered.

    On upgrading the ASM home you proved my point that Oracle does not explicitly state that one should upgrade the ASM home when upgrading the application database home. It is implied, however, by the quotes you referenced. I have never seen direction, for example, on whether apply catcpu.sql to the ASM instance.

  13. 13 Alex Gorbachev February 23, 2008 at 10:17 pm

    Generally, you don’t have to apply patchset to ASM home if you apply it to database home providing versions are compatible acording to the compatibility matrix I referenced – Oracle Clusterware – ASM – Database Version Compatibility (Note 337737.1). So if you read it again it says:

    You can have different release of Oracle Clusterware, ASM and database software on your cluster. The intent of the note is to show the version compatibility between Oracle Clusterware, ASM and database releases.

    I’m failing to see what is not clear in this note and why you should imply something. Perhaps, I misunderstood your question.

    Note, that some one-off patches could have requirements to patch both homes but that would be in the README for this patch.

    I should emphasize again – I also don’t want to say that Oracle documentation is absolutely complete – it’s not. Not even with Metalink. But it’s still a must to read it and it can be long, indeed.

  14. 14 adzuan badawi April 29, 2009 at 11:29 am

    I run the root102.sh script and it’s still not finish doing the “Waiting for the patched CRS daemons to start.
    This may take a while on some systems.”

    I wait almost an hour but still nothing happened…

    Is it normal ??

    Kindly can u please reply to my email :


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


I work for Amazon Web Services but all of the words on this blog are purely my own. Not a single word on this blog is to be mistaken as originating from any Amazon spokesperson. You are reading my words this webpage and, while I work at Amazon, all of the words on this webpage reflect my own opinions and findings. To put it another way, "I work at Amazon, but this is my own opinion." To conclude, this is not an official Amazon information outlet. There are no words on this blog that should be mistaken as official Amazon messaging. Every single character of text on this blog originates in my head and I am not an Amazon spokesperson.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,900 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories


All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: