Testing RAC Failover: Be Evil, Make New Friends.

In Alejandro Vargas’ blog entry about RAC & ASM, Crash and Recovery Test Scenarios, some tests were described that would cause RAC failovers. Unfortunately, none of the faults described were the of the sort that put clusterware to the test. The easiest types of failures for clusterware to handle are complete, clean outages. Simply powering of a server, for instance, is no challenge for any clusterware to deal with. The other nodes in the cluster will be well aware that the node is dead. The difficult scenarios for clusterware to respond to are states of flux and compromised participation in the cluster. That is, a server that is alive but not participating. The topic of Alejandro’s blog entry was not a definition of a production readiness testing plan by any means, but it was a good segway into the comment I entered:

These are good tests, yes, but they do not truly replicate difficult scenarios for clusterware to resolve. It is always important to perform manual fault-injection testing such as physically severing storage and network connectivity paths and doing so with simultaneous failures and cascading failures alike. Also, another good test to [run] is a forced processor starvation situation by forking processes in a loop until there are no [process] slots [remaining]. These […] situations are a challenge to any clusterware offering.

Clusterware is Serious Business
As I pointed out in my previous blog entry about Oracle Clusterware, processor saturation is a bad thing for Oracle Clusterware—particularly where fencing is concerned. Alejandro had this to say:

These scenarios were defined to train a group of DBA’s to perform recovery, rather than to test the clusterware itself. When we introduced RAC & ASM we did run stress & resilience tests. The starvation test you suggest is a good one, I have seen that happening at customer sites on production environments. Thanks for your comments.

Be Mean!
If you are involved with a pre-production testing effort involving clustered Oracle, remember, be evil! Don’t force failover by doing operational things like shutting down a server or killing Oracle clusterware processes. You are just doing a functional test when you do that. Instead, create significant server load with synthetic tests such as wild loops of dd(1) to /dev/null using absurdly large values assigned to the ibs argument or shell scripts that fork children but don’t wait for them. Run C programs that wildly malloc(2) memory, or maybe a little stack recursion is your flavor—force the system into swapping, etc. Generate these loads on the server you are about to isolate from the network for instance. See what the state of the cluster is afterwards. Of course, you can purposefully execute poorly tuned Parallel Query workloads to swamp a system as well. Be creative.

Something To Think About
For once, it will pay off to be evil. Just make sure whatever you accept as your synthetic load generator is consistent and reproducible because once you start this testing, you’ll be doing it again and again—if you find bugs. You’ll be spending a lot of time on the phone making new friends.

6 Responses to “Testing RAC Failover: Be Evil, Make New Friends.”

  1. 1 lhe December 9, 2006 at 9:00 pm

    regarding your comment about some people being crazy about open source products: generalizing that “they” don’t look either in the bug database nor the source code is just lame. first there is no “they” out there. second, it seems you have some misconception about open source.
    the point is that you _can_ look in bug database and source code when _you_ want to, so you get some additional freedom in choice with open source. open source software certainly isn’t “better” than proprietary software just because it is open source, it rather depends on your needs.
    so please correct the prejudice you display here, it just doesn’t fit your engineering views. personally i prefer to see all bugs and make an informed decision.

  2. 2 kevinclosson December 9, 2006 at 9:58 pm


    I have personally spoken to people that have deployed open source products without ever having read the bug database. I maintain that doing so is, without a doubt, completely crazy. That is my viewpoint and thus I share it in *my* blog. I should hope that people can agree to disagree with me on that view if they need to. I agree to disagree with people all the time. At least it is on readers’ minds now.

    Taking the thought further, I also think it is crazy to not spend a reasonable amount of time reading Oracle bugs in Metalink. As they say, “An ounce of prevention is better than a pound of cure.”

    Finally, I do get your point about the fact that one _can_ read source and bugs for open source goodies when one wants to. I should think a **really** good time to do so is **before** one crams the stuff into a solution.

    Thanks for reading.

  3. 3 lhe December 11, 2006 at 12:16 am

    Seems i misunderstood your posting. I read the paragraph about open-source crazy people as if you were talking about _anyone_. But your answer makes it clear that you had a significantly smaller group of people in mind. So i got you wrong and thought of you as another one of these FUD-types. Sorry for that.

  4. 4 kevinclosson December 11, 2006 at 1:39 am


    No problem. I appreciate your participation in the blog. I had a feeling you thought I was fingering all adopters of open source as crazy and that couldn’t be further from my view! On the other hand, where open source is concerned, I am totally against three mentalities:

    1. The Belligerently Ignorant
    2. The Irrationally Exuberant
    3. The Self-Indulgently Dishonest

    I see #3 preying on #1. When that happens, and doesn’t work out, #1 seeks emotional comfort from the #2 types.

    I am a proponent of using any particular technology **if** it solves a problem. If it doesn’t, and some #3 type tries to tell me it does, I get a very upset.

    How bad would I have to hate myself to work so hard for over 5 years for a company that provides platform software technology in the Linux environment if I was indeed completely against open source—just because it is open source. That would be pitiful.

  5. 5 Mike Ault March 22, 2012 at 7:02 am

    Starvation testing is critical. The problem of starvation doesn’t manifest itself until the system is under heavy load, the one time you don’t want it to fail. Thanks for enlightening people! Great job as usual!

    • 6 kevinclosson March 22, 2012 at 7:26 am

      Thanks for the kind words, Mike. You did notice that is a 6 year old blog entry, right? 🙂

      The absolutely, positively most difficult thing for clusterware to get right is handling nodes that are transitioning in and out of the cluster because of “fizzling” components…such as popping into the network and out before a full transition into the cluster as a member. Race conditions…tough. The totally re-written Oracle clusterware in 11.2 probably comes the closest to getting it right. All prior Oracle clusterware was quite weak. That’s why they had to totally re-write it in a “dot two” release.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.


I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 743 other subscribers
Oracle ACE Program Status

Click It

website metrics

Fond Memories


All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: