Tim Hall has stepped up to the plate to document a step-by-step recipe for setting up Oracle10g RAC on NFS mounts. In Tim’s blog entry, he points out that for testing and training purposes it is true that you can simply export some Ext3 filesystem from a Linux server and use it for all things Oracle. Tim only had 2 systems, so what he did was use one of the servers as the NFS server. The NFS server exported a filesystem and both the servers mounted the filesystem. In this model, you have 2 NFS clients and one is acting as both an NFS client and an NFS server.
This is the link to Tim’s excellent step-by-step guide.
How Simple
If you’ve ever had a difficult time getting RAC going, I think you’d be more than happy with how simple it is with NFS and using Tim’s guide and a couple of low-end test servers would prove that out.
Recently I blogged about the fact that most RAC difficulties are in fact storage difficulties. That is not the case with NFS/NAS.
Thanks Tim!
Kevin Closson's Blog: Platforms, Databases and Storage 
I have a Manly Man question for you. This Manly Man Wanna Be (MMWB) runs a 2-node 10g RAC on Dell 2850s with 2 dual-core Xeon CPUs (total of 4 CPUs). Each server has 16 GB of memory. While MMWB was installing this last year, he struggled mightily with 64-bit RAC on 64-bit Red Hat Linux 4.0. MMWB finally got it working after learning a lot of things about RPMs and such.
However, Boss Of Manly Man Wanna Be (BOMMWB) was nervous about 64-bit being “new,” and all of the difficulties that MMWB had with it, so we reinstalled with 32-bit RAC running on 32-bit Red Hat Linux 4.0.
BOMMWB was happy, but this leaves MMWB wondering, is he truly a Manly Man if he’s running x86 Oracle on hardware that’s x86-64 capable? Is 16 GB of memory manly enough? We do use a SAN, does that make me a Manly Man?
Seriously, what are your thoughts about x86 vs. x86-64 Linux, in relation to Oracle and RAC? I’d appreciate a blog entry if you could.
Hi Rob, or should I say MMWB? I love the tongue-in-cheekedness! Odd you should start this thread since I was just about to make a 32bit/64bit post…
Kevin,
How can NFS substitute the functionality that a cluster file system offers? I am confused about this whole nfs with rac. If two instances are trying to write a file, how can nfs protect?
Thanks.
Pedro,
A CFS would be in as much of the way as NFS if Oracle were to use either without Direct I/O. However, RAC uses Direct I/O with both. It is a non-issue.
Kevin,
First – let me thank you for maintaining this excellent blog.
I was thinking about NFSv4, now that that’s pretty much supported by almost all OS vendors. Have you had an opportunity to explore NFSv4 and Oracle RAC? If so then what impact do you think it’ll have on O10g RAC customers wanting to migrate to V4?
thanks,
Uday Shet
Rob Johnson: I would have thought it was obvious. The number 64 is twice the size of 32, so it must be twice as good, making you twice as manly! 😉
Cheers
Tim…
PS. I think politically correct names for this series of blog entries are:
Personally Persons only deploy…
Humanly Humans only deploy…
🙂
I’m surprised nobody has brought up “That Manly Feeling” that Irish Spring brings on! But then who would have seen those commercials? Time to blog it.
Does Manly Man drive an old landrover? Is it a requirement?
Keep up the great blogging about RAC and NFS. I will be trying it out in August.
Philip! If you are talking about my avatar, that is actually a 1940s Willey Jeep!
Uday,
I have not explored NFSV4. Since Oracle Direct NFS is here, I’m not sure there would be much value…but I don’t know.
Hi.
What do you set your filesystemio_options init.ora parameter to when using Oracle over NFS?
Based on what you’ve written before I know NFS supports direct I/O, but I’m struggling to find a specific statement about NFS and asynchronous I/O. So should I use:
filesystemio_options=directIO
or
filesystemio_options=setall
I guess “setall” is fine because Oracle only uses direct and/or synchronous IO if it can, but it would be nice to get some confirmation.
Cheers
Tim…
I found my answer. Asynchronous I/O is not supported on NFS:
http://download.oracle.com/docs/cd/B19306_01/server.102/b15658/appc_linux.htm#sthref892
Cheers
Tim…
i’m confused as Pedro Alvarez in is previous question about CFS and NFS, if Oracle uses direct i/o RAC can work also with shared datafiles on a non-cluster filesystem if this supports direct i/o? Is Oracle RAC that coordinates writes by different instances to same datafiles?
Thanks,
Cristian
Pedro, Christian,
NFS is not full filesystem so it is neither a cluster filesystem nor a non-cluster filesystem. I is cluster-able in that you can mount the same NFS export on more than one system. Try not to make it more complex than it is, OK.
Hi Kevin, great blog!
I have been combing over Tim’s Blog as well as some of the the details you provide here. Very informative.
While Tim’s guide is very good, I am trying to extrapolate a bit and apply his solution to a single instance to RAC migration (of an existing DB whose data must remain pssst…it’s a secret E-Business suite thing for a development project I am working on).
I believe that I am running into an issue related to the data file storage location of the NFS mount and it being an EXT3 file system and NOT ASM or OCFS but I am uncertain.
rconfig pukes everytime I try to convert 10gR2 (ORA-1034 & ORA-1078). I logged a bug with an am awaiting a reply – 6631213).
I have tried manual conversion as well which fails with ORA-29702: error occurred in Cluster Group Service operation after I relinked the oracle with rac_on.
It’s an interesting project I am working on and I would really like to make this work. Time frame is an issue and after trying one final time to see if my NAS mount can handle a standard new RAC install and config via dbca…if it can’t even handle that, I know something is wrong with my config.
CRS runs and OCR and voting disks are good. But the RAC db migration fails at every turn.
Any thoughts?
Thanks,
Mike
Hi Mike,
Thanks for stopping by…
You are using 10gR2 but no more specifics than that. With 10gR2 you’ll need to inform us of what the NAS device is, and what your mount options are. Actually, it is not clear what the NFS client (DB server) platform is either. Also, what is this type of migration? RMAN? I’ve heard rumors of 10gR2 RMAN NFS peculiarities so I need to know that …
Hey Kevin, thanks for the prompt reply.
Sorry for being vague…wasn’t intentional. My bad.
Openfiler 2.2 is the NAS.
Oracle’s EL 4U5 is the OS for the two “clients” or rac nodes if you prefer.
The mount options I am using are as follows:
openfiler:/mnt/nfsrac_data/crs_ocr_voting/crs_configuration
/u01/crs_configuration nfs
rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0
0 0
openfiler:/mnt/nfsrac_data/crs_ocr_voting/voting_disk
/u01/voting_disk nfs
rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0
0 0
openfiler:/mnt/nfsrac_data/nfs_db_data/data_files /u02/data_files
nfs
rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0
0 0
As to 10gR2..
10.2.0.2 for the single instance as well as the software only new RAC enabled $ORACLE_HOME.
The migration is using the newer utility rconfig which I believe calls rman in the background as archive logging is enabled on the source DB (single instance originating DB that is).
Manual conversion attempts were done by dumping the dbf’s to the NFS shared storage area and recreating the control via a ouicli front end that apps uses called RapidClone. Once the single instance had the dbf’s relocated, I relinked the oracle kernel with rac_on and attempted to add another UNDO_TS…that’s when I got the ORA-29702.
Any thoughts or suggestions are welcomed.
Thanks again,
Mike
Hey Kevin,
I wanted to close the loop and update you and your readers on my quest seeking NFS Apps RAC nirvana. 🙂
As mentioned above, I filed a bug related to the rconfig utility and it’s usage when CFS was the chosen file system versus ASM. Even though CFS isn’t realy a good definition for the EXT3 file system defined for an NFS RAC data file storage…this was the most sensible choice of the two options for the target file system type. Metalink note 388577.1 discusses how one might use the utility rconfig to convert single instance to RAC however the note falls short in the exact details required when NOT using ASM. In this particular case, when not using ASM, there is an ASMInfo stanza in the ConvertToTac.xml file used by the utility. This entire stanza must be commented out. This is not documented anywhere clearly.
A second issue in my specific case, was that while I had setup rsh host equivalence, which was enough for CRS to install and work fine over NFS between the rac nodes, rconfig requires ssh equivalence. Without ssh host and user equivalence, this rconfig utility spits and error saying you don’t have a defined shared storage area in the xml format log and terminal message during execution.
After working directly with the developer who owns the Metalink document 388577.1 for 2 days regarding my conversion, I am happy to say that Apps RAC with NFS shared storage is in fact working or rather, mostly so. I have several small issues to resolve, but nothing specific to the RAC on NFS deployment (apps related issues really)!!
My thanks and appreciation go out to you and Tim for your informative and detailed blog postings.
Best Regards,
Mike
Dear Friend..
While CRS installation ( as I am following Tim’s procedure) .. after running root.sh I get stuck as … on either node on error comes “Failed to initialize crs…” in error log it is asking that nfs not mounted with correct option (rsize=0,wsize=0)… while it is properly mounted and many times rebooted also …
Any help will be appreciated .
Kanishka