A few days ago there was a thread on the oracle-l email list about adding nodes in an Oracle Database 10g Real Application Clusters environment. The original post showed a problem that Alex Gorbachev reports he’s only seen with shared Oracle Home installs. I found that odd because I’ve done dozens, upon dozens of RAC installs on shared Oracle Homes with both CFS and NFS and haven’t seen this error:
Remote 'UpdateNodeList' failed on node: 'af-xxx2'. Refer to
'/apps/oracle/oraInventory/logs/addNodeActions2007-07-08_09-36-12PM.log'
for details.
You can manually re-run the following command on the failed nodes after the
installation:
/apps/oracle/product/10.2/oui/bin/runInstaller -updateNodeList -noClusterEnabled
ORACLE_HOME=/apps/oracle/product/10.2 CLUSTER_NODES=af-xxx1,af-xxx2,af-xxx6
CRS=false "INVENTORY_LOCATION=/apps/oracle/oraInventory" LOCAL_NODE=
<node on which command is to be run>
I never have any problems with shared Oracle Home and I blog about the topic a lot as can be seen in in this list of posts. Nonetheless, Alex pointed out that the error has to do with the Oracle Inventory being on a shared filesystem. Another list participant followed up with the following comment about placing the inventory on a shared drive:
Sharing the oraInventory across nodes is not a good practice in my opinion. It runs counter to the whole concept of redundancy in an HA configuration and RAC was not written to support it.
Well, the Oracle Inventory is not a RAC concept, it is an Oracle Universal Installer concept, but I think I know what this poster was saying. However, the topic at hand is shared Oracle Home. When people use the term shared Oracle Home, they don’t mean shared ORACLE_BASE, they mean shared Oracle Home. Nonetheless, I have routinely shared the 10g inventory without problems, but then my software environments might not be as complex as those maintained by the poster of this comment.
Shared Inventory with Oracle Database 11g
No can do! Well, sort of. Today I was installing 11g RAC on one of my RHEL 4 x86 clusters. In the fine form of not practicing what I preach, I mistakenly pointed Oracle Universal Installer to a shared location (NFS) for the inventory when I was installing CRS. I got CRS installed just fine on 2 nodes and proceeded to install the database with the RAC option. It didn’t take long for OUI to complain as follows:
Ugh. This is just a test cluster that I need to set up quick and dirty. So I figured I’d just change the contents of /etc/oraInst.loc to point to some new non-shared location-aren’t I crafty. Well, that got me past the error, but without an inventory with CRS in it, Oracle11g OUI does not detect the cluster during the database install! No node selection screen, no RAC.
I proceeded to blow away all the CRS stuff (ORA_CRS_HOME, inittab entries, /etc/oracle/* and /etc/oraInst.loc) and reinstalled CRS using a non-shared locale for the inventory. The CRS install went fine and subsequently OUI detected the cluster when I went to install the database.
This is a significant change from 10g where the inventory content regarding CRS was not needed for anything. With 10g, the cluster is detected based on what /etc/oracle/ocr.loc tells OUI.
Summary
Shared Oracle Home is an option, shared Oracle Home means shared Oracle Home not shared Oracle Inventory. Oracle11g enforces this best practice nicely!
Kevin Closson's Blog: Platforms, Databases and Storage 
Hi Kevin,
I should have said with shared locations for Oracle inventory. Thanks for correcting this.
Btw, instead of cleaning all up you could have copied inventory to another location and change it in /etc/oraInst.loc as well as… I think one file in every Oracle home (can’t recall exactly which).
Now back to catching up with your blog that has blown up while I haven’t paid attention for few days. 🙂
Cheers,
Alex
What are your thoughts on using CDSLs to make the 11g directories node specific, but still live on a shared clustered file system.
We are currently running a shared 9i on a Polyserve clustered file system, and are using CDSL directories for many of the our configuration files and logs. We are planning our migration to 11g and would like to consider restructuring our RAC cluster to use non-shared Oracle Homes, but wanted to get an expert opinion on using CDSLs to still take advantage of our shared disks.
Thanks for your time,
Jen
Hi Jen,
CDSLs were important back in the 9i days. I see no need for them with 10g later. Shared Home and CDSL are not synonymous these days.
We need a solution that allows us to continue using the shared storage that we currently have available through Polyserve. We have been told that getting local storage for our 50+ linux Oracle servers is not an option, and we need an alternative plan.
So how would you suggest setting up these separate ORACLE HOME locations on shared disks? Our thought was to use the CDSLs and create a CDSL from the ORACLE_BASE directory with directories for each node under there, and each node directory would hold a “local” ORACLE_HOME, CRS_HOME, ASM_HOME, central inventory, etc, etc…
In my head it seems like this will work, but there isn’t much chatter about this topic on the forums, so I wanted to see how others are setting their systems up.
Thanks,
Jen
Hi Jen,
Readers sure would be interested in hearing just how big this setup is of yours. A significantly high percentage of Oracle Linux deployments are so small that readers must wonder why you don’t just back it up onto a few USB sticks and fidget away all the PolyServe stuff and restore from USB.
🙂
That aside, I see what you are saying. You want to use shared storage but make it so Oracle is not aware it is shared storage. I can see the value in that…definitely.
You could have a OFA head like /u01/app/oracle and have all assets under there by CDSLs starting with the directories. That would be simple and would work…
CDSLs are very helpful…I wish NFS supported them.
Well, I have an 11g RAC with one inventory and am in the process of creating another today. I simply click ‘local installation’ from the installer then move to the create database.
It’s been a bit since I created the first setup so I may be forgetting something rather relevant. I’ll update if it’s different.
Woop, sorry, ignore my last – I did split the inventory. moderate it to the bin. Not sure what I was thinking but what I wrote is not accurate. I think I hit this point of fault (click local install) forgetting that it didn’t work. Sorry. \r