Upgrading CRS to 10.2.0.3 on RHEL4 x86_64
It is quite likely I’m the last person to get around to updating my 10gR2 CRS—er, clusterware—with the 10.2.0.3 patchset. Why? Well, upgrades always break something and since 10.2.0.1 CRS was really quite stable for the specific task of node membership services (libskgxn.so), I was happy to stay with it and skip 10.2.0.2. Compared to the offal we referred to as 10.1 CRS, I have been very happy with 10gR2 CRS for the main job of CRS (which is monitoring node health). Fencing is another topic as I’ve blogged about before.
Oh, Great, He’s Blogging Screen Shots of Stuff Working Fine
Well, I can’t think of anything more boring to look at than a screen shot of a successful execution of an upgrade script. With the 10.2.0.3 upgrade it is root102.sh—the root script that OUI instructs you to execute in $ORA_CRS_HOME after it finishes such activities as copying pre-10.2.0.3 files over to ${ORA_CRS_HOME}/install/prepatch10203 and so on. So why am I blogging on a successful application of this patchset?
Knowing How Bad Something Has Failed—and Where
Often times when RAC installation and patch applications go awry—a very frequent ordeal—it is often nice to see what you should have seen at the point where it went wrong. Such clues can sometimes be helpful. It is for this reason that when I—and others in my group—write install guides for Oracle products on our Database Utility for Oracle clustering package I often include a lot of boring screen shots.
Testing a Rolling Application of 10.2.0.3 CRS
As described later in this post it is fully supported to implement a shared ORA_CRS_HOME—as it is on OCFS2 and Red Hat GFS. In fact, there are several permutations of supported configurations to choose from:
- Local CRS HOME, raw disk OCR/CSS
- Local CRS HOME, CFS OCR/CSS
- Local CRS HOME, NFS OCR/CSS
- Shared CRS HOME, raw disk OCR/CSS
- Shared CRS HOME, CFS OCR/CSS
- Shared CRS HOME, NFS OCR/CSS
As a normal part of my testing, I wanted to make sure the storing the OCR and CSS disks on the PolyServe CFS in no way impacts the ability to perform a 10.2.0.3 rolling upgrade of local ORA_CRS_HOME installations. It doesn’t. First, OUI determined is was OK for me to do so because ORA_CRS_HOME on all three nodes of this puny little cluster were installed under /opt on the internal drives. The CRS files (e.g., OCR/CSS), on the other hand, were on PolyServe:
tmr6s15:/opt/oracle/crs/install # grep u02 *
paramfile.crs:CRS_OCR_LOCATIONS=/u02/crs/ocr.dbf
paramfile.crs:CRS_VOTING_DISKS=/u02/crs/css1.dbf,/u02/crs/css2.dbf,/u02/crs/css3.dbf
rootconfig:CRS_OCR_LOCATIONS=/u02/crs/ocr.dbf
rootconfig:CRS_VOTING_DISKS=/u02/crs/css1.dbf,/u02/crs/css2.dbf,/u02/crs/css3.dbf
tmr6s15:/opt/oracle/crs/install # mount | grep u02
/dev/psd/psd1p3 on /u02 type psfs (rw,dboptimize,shared,data=ordered)
The first screen shot shows what to expect when OUI determines a rolling application of this patch is allowed:
NOTE: You may have to right click->view the image (e.g., with firefox I believe)
Next, OUI instructs you to stop CRS on a node and then execute the root102.sh script:
If all that goes well, you’ll see the following sort of feedback as root102.sh does its work:
I was able to move along to the other two nodes and get the same feedback from root102.sh there as well.
To Share or Not to Share ORA_CRS_HOME
Oracle and PolyServe fully support the installation of CRS in either shared or unshared filesystems. The choice is up to the administrator. There are important factors to consider when making this decision. Using a shared ORA_CRS_HOME facilitates a single, central location for maintenance and operations such as log monitoring and so on. Some administrators consider this a crucial factor on larger clusters; it eliminates the need to monitor large numbers of ORA_CRS_HOME locations, each requiring logging into a different server. When ORA_CRS_HOME is shared in the PolyServe cluster filesystem, administrators can access the files from any node in the cluster.
A shared ORA_CRS_HOME does have one important disadvantage—rolling patch application is not supported. However, a patch that manipulates the Oracle Cluster Repository cannot be applied in a rolling fashion anyway. Although 10.2.0.3 is not that such a patch, it is not inconceivable that other upgrades could make format changes to the OCR that that would be incompatible with the prior-versions executing on other nodes. Oracle would, of course, inform you that such a release was not a candidate for rolling upgrade just as they do with a good number of the Critical Patch Updates (CPU).
The parallel to shared ORACLE_HOME is apparent. Many Oracle patches for the database require updates to the data dictionary, so a lot of administrators ignore the exaggerated messaging from Oracle Corporation regarding “Rolling Upgrades” of ORACLE_HOME and deploy a shared ORACLE_HOME, eliminating the need to patch several ORACLE_HOME locations whenever a patch is required. This concern is only obvious to large IT shops where there is not just one RAC database, but perhaps 10 or more. These same administrators generally apply this logic to ORA_CRS_HOME. Indeed, having only one location to patch in either the ORA_CRS_HOME or ORACLE_HOME case significantly reduces the time it takes to apply a patch. To that end, planning a very brief outage to apply patches to shared ORA_CRS_HOME and/or ORACLE_HOME for up to 16 nodes in a cluster is an acceptable situation for many applications. For those cases where downtime cannot be tolerated, Oracle Data Guard is required anyway and again the question of shared or unshared ORACLE_HOME and ORA_CRS_HOME arises. The question can only be answered on a per-application basis and the choice is yours. PolyServe finds that, in general, when an application is migrated from a single large UNIX platform to RAC on Linux, administrators do not have sufficient time to deal with the increased amount of software maintenance. These IT shops generally opt for the “single system feel” that shared software installs for ORACLE_HOME and ORA_CRS_HOME offers. In fact, PolyServe customers have used Share Oracle Home since 2002 with Oracle9i and then with Oracle10g—it has always been a staple feature of the Database Utility for Oracle. With Oracle10g the choice is yours.
Kevin Closson's Blog: Platforms, Databases and Storage 
Hello,
You are quick of the mark Kevin, the 10.2.0.3 patchset has been out for x86_64 only 1 and a bit days – I see the 32 bit linux version has had a months longer exposure. Does not sound like you encountered anything like the following bug that hits the 32 bit linux version:
5667023 Linux: CRS does not start after applying 10.2.0.3
It must have been kinda depressing for the dba upgrading to the latest and greatest version to find his cluster not clustering anymore.
Sounds like it is a fairly straightforward upgrade though.
jason.
Isn’t that the SELinux issue? I don’t feel like logging in to MetaLink to double check, but the thing I always do is read the list of known bugs a patchset **introduces** and when I did I saw one about CRS not starting due to SELinux. Now, there is a *lot* wrong with that. First, 10.2.0.3 is an upgrade which insinuates the first DBA that tried it had a functional cluster at
Jason, just upgraded two 2 node x86_64 clusters from 10.2.0.1 to 10.2.0.3. This bug also exists on the x86_64 platform as well, so make sure you disable before rebooting..
“However, a patch that manipulates the Oracle Cluster Repository cannot be applied in a rolling fashion anyway.”
Kevin, as far as I remember, CRS upgrade does involve OCR upgrade. When done on the first node of the cluster and so on until the last node, root*.sh will not upgrade OCR but keep running with older OCR format. Running on the last node of a cluster, root*.sh output will be a bit different and OCR will be upgraded. I don’t have anything handy to test but I recall that when I ran CRS 10.2.0.1 -> 10.2.0.2 upgrade on Linux.
Right, Alex, I’m saying “if”. I have not seen one that touches the OCR, but it is not out of the question.
Thanks Kevin for the clue “SELINUX=disabled”, the isssue i had was after the patching the CRS will not start at node reboots. Disabling did the trick. Metalink has not documented this anywhere.
Hello,
I have one question regarding Oracle 10g two nodes RAC DB, currently its version is 10.2.0.1 Now I like to upgrade it to 10.2.0.3 if I upgrade only DB its enough or I must upgrade CRS, ASM all if yes
can you please send me the steps how to start first.
I read documentation but its too long I need direct steps to upgrade
all. I have downloaded the patchset for 10g on Sparc 10.2.0.3, do i need to download CRS or ASM patch separately.
please help me out I am new to RAC DB
Thanks in advance!
Samad,
Reading documentation is a must. If you don’t have time for that – I’m sorry for you but you should find yourself a job other than DBA. Hire a qualified consultant or a remote DBA that can do this task for you properly and who already read documentation. You will save yourself time and money.
Alex
Alex,
Be nice and help poor guy… He is busy with real life not like us….
Remote DBA … Berluson and Ault can help there . (Higly recommended ) for tuning 1Mb database…. with 10Gb of RAM.
Alex, I have never seen the answer to Samad’s question documented by Oracle, yet it is a serious oversight that they do not answer it in every patch readme file. If you had read Oracle’s documentation you would know that they do not answer this question.
Samad, Not knowing the correct answer here is what I do. We also have ASM installed into its own home with its own set of binaries. We shut down both the application Oracle instance and the ASM instance. Then we run the patch to the ASM home and then the application Oracle home. Then we start both instances and apply catcpu.sql to the application Oracle instance. We do not apply catcpu.sql to the ASM instance.
So far, this has worked well for us. If anyone knows if this is the right way or the wrong way to do this please let us know.
The sarcasm of my previous post was directed to “I read documentation but its too long I need direct steps to upgrade
all”.
There are reasons for long documentation and making DBA’s bored reading it is obviously not one of them.
Regarding your question…
Here is the quote from documentation “Oracle® Database Patch Set Notes
10g Release 2 (10.2.0.3) Patch Set 2 for Linux x86”:
5.1.4 Upgrading Oracle Clusterware
The Oracle Clusterware software must be at the same or newer level as the Oracle software in the Oracle RAC Oracle home. Therefore, you should always upgrade Oracle Clusterware before you upgrade Oracle RAC.
Further in the same document:
5.8.2 Stopping All Processes for an Oracle Clusterware Installation
This section contains the following information:
Rolling Upgrade
Non Rolling Upgrade
If you have a separate ASM Oracle home than you probably have read MAA guides from OTN (Maximum Availability Architecture). As far as I remember, the original MAA paper mentions the ability to run database and ASM on different patchsets as one of the reasons for having separate Oracle home for ASM. Now, if you got to Metalink and search on “asm compatibility” – the first link would be Oracle Clusterware – ASM – Database Version Compatibility.
PS: for “Lenin” Not all consultants are the same and remote DBA service != “Remote DBA” but I’m biased. 😉
Good point. Samad did say the documentation is too long. As a DBA you just have to get used to the requirement that you are going to have to read all the documentation before making any major database change. Even then, you have no guarentee that all your questions and situations will be covered.
On upgrading the ASM home you proved my point that Oracle does not explicitly state that one should upgrade the ASM home when upgrading the application database home. It is implied, however, by the quotes you referenced. I have never seen direction, for example, on whether apply catcpu.sql to the ASM instance.
Generally, you don’t have to apply patchset to ASM home if you apply it to database home providing versions are compatible acording to the compatibility matrix I referenced – Oracle Clusterware – ASM – Database Version Compatibility (Note 337737.1). So if you read it again it says:
You can have different release of Oracle Clusterware, ASM and database software on your cluster. The intent of the note is to show the version compatibility between Oracle Clusterware, ASM and database releases.
I’m failing to see what is not clear in this note and why you should imply something. Perhaps, I misunderstood your question.
Note, that some one-off patches could have requirements to patch both homes but that would be in the README for this patch.
I should emphasize again – I also don’t want to say that Oracle documentation is absolutely complete – it’s not. Not even with Metalink. But it’s still a must to read it and it can be long, indeed.
hi…kevin
I run the root102.sh script and it’s still not finish doing the “Waiting for the patched CRS daemons to start.
This may take a while on some systems.”
I wait almost an hour but still nothing happened…
Is it normal ??
Kindly can u please reply to my email :
adzuan@nc.com.my
It is not normal. I do recommend a support call. Something specific to your environment is halting progress.