Archive for the 'Unbreakable Linux' Category

The Most Horribly Botched Oracle Database 10g Real Application Clusters Install Attempt.

Now don’t get me wrong. What I’m blogging about is not really an Oracle Database 10g Real Application Clusters (RAC) problem. All of the problems I will mention in this entry were clearly related to a botched configuration at the OS level. Since Oracle10g will be around for a long time, I suspect that some day someone else might run into this sort of a problem and I aim to make their lives easier. That said, there could be some goodies in here for all the regular readers of this blog.

The Scenario
I took the brand new 4-node RAC cluster I have in the lab and aimed to see how much the oracle-validated-100-4el4x86_64.rpm package assists in setting up the system in preparation for a RAC install. I was excited since this was the first 2-socket Xeon “Cloverdale” 5355-based cluster I’ve had the chance to test. These processors are murderously fast so I was chomping at the bits to give them a whirl.

The systems were loaded with RHEL4 U4 x86_64. I executed the oracle-validated-100-4el4x86_64.rpm package and here is what it returned:

# rpm -ivh oracle-validated-1.0.0-4.el4.x86_64.rpm
warning: oracle-validated-1.0.0-4.el4.x86_64.rpm: V3 DSA signature: NOKEY, key ID b38a8516
error: Failed dependencies:
        /usr/lib/libc.so is needed by oracle-validated-1.0.0-4.el4.x86_64
        control-center is needed by oracle-validated-1.0.0-4.el4.x86_64
        fontconfig >= 2.2.3-7.0.1 is needed by oracle-validated-1.0.0-4.el4.x86_64
        gnome-libs is needed by oracle-validated-1.0.0-4.el4.x86_64
        libdb.so.3()(64bit) is needed by oracle-validated-1.0.0-4.el4.x86_64
        libstdc++.so.5()(64bit) is needed by oracle-validated-1.0.0-4.el4.x86_64
        xscreensaver is needed by oracle-validated-1.0.0-4.el4.x86_64
    Suggested resolutions:
        compat-db-4.1.25-9.x86_64.rpm
        compat-libstdc++-33-3.2.3-47.3.x86_64.rpm
        control-center-2.8.0-12.rhel4.5.x86_64.rpm
        gnome-libs-1.4.1.2.90-44.1.x86_64.rpm
        xscreensaver-4.18-5.rhel4.11.x86_64.rpm

So I chased down what was recommended and installed those RPMs as well:

 # cat > /tmp/list
        compat-db-4.1.25-9.x86_64.rpm
        compat-libstdc++-33-3.2.3-47.3.x86_64.rpm
        control-center-2.8.0-12.rhel4.5.x86_64.rpm
        gnome-libs-1.4.1.2.90-44.1.x86_64.rpm
        xscreensaver-4.18-5.rhel4.11.x86_64.rpm

# rpm -ivh --nodeps `cat /tmp/list | xargs echo`
warning: compat-db-4.1.25-9.x86_64.rpm: V3 DSA signature: NOKEY, key ID db42a60e
Preparing...                ########################################### [100%]
   1:xscreensaver           ########################################### [ 20%]
   2:compat-db              ########################################### [ 40%]
   3:compat-libstdc++-33    ########################################### [ 60%]
   4:control-center         ########################################### [ 80%]
   5:gnome-libs             ########################################### [100%]

At this point I think I should have been ready to install Oracle Clusterware (CRS). Well, the install failed miserably during the linking phase and-shame on me-I didn’t save any of the logs or screen shots. No matter, I can still make a good blog entry out of this-so long as you are willing to believe that the linking phase of the CRS install failed. I decided to clean up the botched installation and use my normal installation process.

The approach I generally take is to look at a comparable Oracle Validated Configuration and pull the list of RPMs specified. And that is precisely what I did:

$ cat > /tmp/list
binutils-2.15.92.0.2-21.x86_64.rpm
        compat-db-4.1.25-9.x86_64.rpm
        compat-libstdc++-33-3.2.3-47.3.x86_64.rpm
        control-center-2.8.0-12.rhel4.5.x86_64.rpm
        gcc-3.4.6-3.x86_64.rpm
        gcc-c++-3.4.6-3.x86_64.rpm
        glibc-2.3.4-2.25.i686.rpm
        glibc-2.3.4-2.25.x86_64.rpm
        glibc-devel-2.3.4-2.25.i386.rpm
        glibc-devel-2.3.4-2.25.x86_64.rpm
        glibc-common-2.3.4-2.25.x86_64.rpm
        glibc-headers-2.3.4-2.25.x86_64.rpm
        glibc-kernheaders-2.4-9.1.98.EL.x86_64.rpm
        gnome-libs-1.4.1.2.90-44.1.x86_64.rpm
        libgcc-3.4.6-3.x86_64.rpm
        libstdc++-3.4.6-3.x86_64.rpm
        libstdc++-devel-3.4.6-3.x86_64.rpm
        libaio-0.3.105-2.x86_64.rpm
        make-3.80-6.EL4.x86_64.rpm
        pdksh-5.2.14-30.3.x86_64.rpm
        sysstat-5.0.5-11.rhel4.x86_64.rpm
        xorg-x11-deprecated-libs-6.8.2-1.EL.13.36.x86_64.rpm
        xorg-x11-deprecated-libs-6.8.2-1.EL.13.36.i386.rpm
        xscreensaver-4.18-5.rhel4.11.x86_64.rpm

# rpm -ivh --nodeps `cat /tmp/list | xargs echo`

[…output deleted…]

# rpm -ivh oracle-validated-1.0.0-4.el4.x86_64.rpm
warning: oracle-validated-1.0.0-4.el4.x86_64.rpm: V3 DSA signature: NOKEY, key ID b38a8516
error: Failed dependencies:
        /usr/lib/libc.so is needed by oracle-validated-1.0.0-4.el4.x86_64
        fontconfig >= 2.2.3-7.0.1 is needed by oracle-validated-1.0.0-4.el4.x86_64

# rpm -ivh fontconfig-2.2.3-7.x86_64.rpm
warning: fontconfig-2.2.3-7.x86_64.rpm: V3 DSA signature: NOKEY, key ID db42a60e
Preparing...                ########################################### [100%]
        package fontconfig-2.2.3-7 is already installed

So, once again I should have been ready to go. That was not the case. I got the following stream of error output when I ran vipca:

# sh ./vipca
PRKH-1010 : Unable to communicate with CRS services.
  [PRKH-1000 : Unable to load the SRVM HAS shared library
  [PRKN-1008 : Unable to load the shared library "srvmhas10"
  or a dependent library, from

LD_LIBRARY_PATH="/opt/oracle/crs/jdk/jre/lib/i386/client:/opt/oracle/crs/jdk/jre/lib/i386:/opt/oracle/crs/jdk/jre/..
/lib/i386:/opt/oracle/crs/lib32:/opt/oracle/crs/srvm/lib32:/opt/oracle/crs/lib:/opt/oracle/crs/srvm/lib:"
  [java.lang.UnsatisfiedLinkError: /opt/oracle/crs/lib32/libsrvmhas10.so: libclntsh.so.10.1: 

cannot open shared object file: No such file or directory]]]
PRKH-1010 : Unable to communicate with CRS services.
  [PRKH-1000 : Unable to load the SRVM HAS shared library
  [PRKN-1008 : Unable to load the shared library "srvmhas10"
  or a dependent library, from

LD_LIBRARY_PATH="/opt/oracle/crs/jdk/jre/lib/i386/client:/opt/oracle/crs/jdk/jre/lib/i386:/opt/oracle/crs/jdk/jre/..
/lib/i386:/opt/oracle/crs/lib32:/opt/oracle/crs/srvm/lib32:/opt/oracle/crs/lib:/opt/oracle/crs/srvm/lib:"
  [java.lang.UnsatisfiedLinkError: /opt/oracle/crs/lib32/libsrvmhas10.so: libclntsh.so.10.1: 

cannot open shared object file: No such file or directory]]]
PRKH-1010 : Unable to communicate with CRS services.
  [PRKH-1000 : Unable to load the SRVM HAS shared library
  [PRKN-1008 : Unable to load the shared library "srvmhas10"
  or a dependent library, from

LD_LIBRARY_PATH="/opt/oracle/crs/jdk/jre/lib/i386/client:/opt/oracle/crs/jdk/jre/lib/i386:/opt/oracle/crs/jdk/jre/..
/lib/i386:/opt/oracle/crs/lib32:/opt/oracle/crs/srvm/lib32:/opt/oracle/crs/lib:/opt/oracle/crs/srvm/lib:"
  [java.lang.UnsatisfiedLinkError: /opt/oracle/crs/lib32/libsrvmhas10.so: libclntsh.so.10.1:

Egad! What I’m about to tell you should prove beyond the shadow of a doubt that I am not a DBA! I don’t need VIPs or gsd or srvctl for that matter. My test harnesses do not use any of that-at least not the test harness I was intending to use for this specific battery of testing. So, I ignored the vipca problem and figured I’d just install the database and get to work. I’ve never before seen what I’m about to show you.

During the installation of the database, OUI detected all nodes of the cluster so I thought I might be able to sneak this one through. however, during the installation of the database, OUI presented a dialogue for database updates. Now, this is seriously odd since this was a freshly built cluster. There were no other databases installed, but that isn’t the most peculiar aspect of this dialogue. Check it out:

botched.jpg

That is strangely beyond strange! I’m not sure if you can tell what that is but the cells in the dialogue are populated with error output (PRKH-* and LD_LIBRARY_PATH, etc). Wow, that was crazy. What I’m about to tell you should be proof positive that not only am I not a DBA, I do things that are only done by someone who has no idea what a DBA is! I needed the database installed and I didn’t have any databases to upgrade, so I thought I’d ignore this database upgrade mess and push on. Bad idea.

Of course the install was a total failure. I spent some time figuring out what was going on and then it dawned on me. I forgot to install the 32bit glibc-devel RPM. After doing so I cleaned up the mess, walked through the install again and viola-no problems.

The Moral of the Story
Don’t forget the required 32bit library when installing 10g on x86_64 Linux and pay particular attention to the fact that neither the Validated Configuration list of RPMS, nor the oracle-validated-100-4el4x86_64.rpm package had any mention of this library.

I hope it helps someone, someday. Nonetheless, that was a weird screen shot wasn’t it?

Oracle Enterprise Linux: Making Oracle on Linux Even Simpler? Introducing the oracle-validated-1.0.0-4.el4.x86_64.rpm Package

I currently don’t have any systems I can point at Oracle’s Unbreakable Linux Network. If I did, I’d get a copy of the new oracle-validated-1.0.0-4.el4.x86_64.rpm package. This package aims to collect all the necessary system RPMs and install them as well as configure all the sundry system parameters required for Oracle. According to this Oracle Linux Validated Configuration description, the package does the following:


 Additional packages from Oracle:

  # This package automatically pulls in package dependencies and sets
  # system parameters recommended above.  It is not required but is
  # recommended in order to save time in getting the system setup.  It is
  # available from ULN by subscribing to the "Oracle Software" channel and
  # then using up2date to install it.  The dependent packages will be
  # installed and parameters will be adjusted as part of this up2date process.
  oracle-validated-1.0.0-4.el4.x86_64.rpm [ Optional package, but recommended ]

If anyone happens to have a copy laying around and wouldn’t mind shooting me a copy, I’d like to rpm2cpio(8) it and take a peek. I’ve been hoping for quite some time that Oracle Enterprise Linux will include some things that make Oracle on Linux simpler. This could be a start.


DISCLAIMER

I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 3,010 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: