Kevin Closson's Blog: Platforms, Databases and Storage

Fun Stuff to Play With
I thought I’d post a few photos of the lab gear allocated to my projects. I have the 2 racks right next to the chair. There is a 2 node cluster of heavily loaded HP DL585s, an 8 node cluster of HP DL145s, a 2 node cluster of HP DL385s, a 2 node cluster of IBM xSeries x366 , some Chaparall SAN, Imperial Solid State Disk, MSA 1500, DS4x00, and other goodies (e.g., FC and Ethernet switches, etc) … all running Oracle on Redhat and SuSE Linux.

And the DL585s are cabled to some LUNs in the Data Direct Network cab:

And, when I schedule it, I am granted LUNs from the following storage arrays:

That is about 10% of the lab gear here… lots of AC.

Not pictured is the Texas Memory System I have on loan from the nice guys at TMS.

BLOG UPDATE 2011.08.11 : For years my criticism of Oracle Clusterware fencing methodology brought ire from many who were convinced I was merely a renegade. The ranks of “the many” in this case were generally well-intended but overly convinced that Oracle was the only proven clustering technology in existence.  It took many years for Oracle to do so, but they did finally offer support for IPMI fencing integration in the 11.2 release of Oracle Database. It also took me a long time to get around to updating this post.  Whether by graces of capitulation or a reinvention of the wheel, you too can now, finally, enjoy a proper fencing infrastructure. For more information please see: http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/admin.htm#CHDGIAAD

PolyServe to Open Source Products in Wake of “New” Oracle Unbreakable Linux

You all know I currently work for PolyServe. Over the last 5 years I cannot count how many times it has been recommended to us that we open source PolyServe Database Utility for Oracle.Back when Dave Dargo’s office was pushing the Unbreakable Linux program (circa 2002), the ecosystem for Oracle on Linux started to get dicey really fast. You see, when Unbreakable Linux started, it was a program that:

• Mandated that all software on your system—except Oracle of course—be open source
• You still have a support contract with Redhat or SuSE

Meeting these criteria allowed you to call Oracle Support with purely Operating System issues. I’m still waiting for anyone to comment on just how helpful that program was for you. It seemed like very dubious value add to me.

What about Open Source Cluster File Systems?

Over the years, we’ve been asked why our customers pay for our product when OCFS is free and GFS is available. Historically, it was because it worked and the others were seen as emerging technology. I question that supposition because GFS just about died in the hands of Sistina and OCFS has always been too rudimentary for general purpose use. I know OCFS2 is reportedly a general purpose filesystem, if that is true for you in production, that’s good. If you have only heard that rumor, I recommend my philosophy that belief should only be borne of testing.

SuSE was the only distro to adopt OCFS2 since Redhat was working on GFS. It may seem like trivial pursuit, but Novell Corporate IT actually run Oracle on PolyServe—but I digress.

Dark Dirty Secrets About Free Stuff

Oracle has been stating that OCFS is “good enough” for years, and after OCFS2 was touted as “general purpose”, SuSE decided to be nice and include it in their distribution—but they never told anyone that it doesn’t work. The voice of authority on what OCFS2 can do is actually SuSE, and Lars Marowsky-Brée is the voice of voices there since he works in the group that is trying to make free clustering solutions work. In this suse-oracle email list installment he writes:

So two nodes is not really sufficient with OCFS2 to protect against node failures – you need three at least for the majority quorum to be meaningful.

That was on 11 April 2006. I bet you haven’t heard that two node clusters with OCFS2 are split-brain poster children from anyone but this blog, right? What’s the big deal? If the stuff doesn’t work, why are you being told it does?

Of these two, GFS is at least close to pole position. It has a quality fencing model and actually works—at least their STONITH model that is. I wrote about fencing in this paper, and will go into the topic further here once the dust settles from Oracle OpenWorld. But just because it is functional doesn’t mean it performs. Sistina had all sorts of problems working out the locking model with GFS. Redhat didn’t inherit any favors in that regard. Here is an independent study of Linux CFS alternatives for unstructured data—the real kingpin performance metric for file systems.

What About Oracle Support

Now that is a great generic topic—but not for this post. After Oracle shook things up and muddied the water back on 2002 with Unbreakable Linux 1.0, they changed their stance on database support—when using third party clustering solutions—about 42 times. Eventually, Oracle instituted a program for third party cluster filesystems and clusterware compatibility which is discussed here. The RAC Technology Compatibility Matrix in Oracle Metalink is the final say on such matters.

The Linux ecosystem has been a train wreck not due to technology interoperability so much as the constant heavy-handedness by such companies as Oracle when it comes to partnering. In fact, Redhat has consistently tried to freeze out non-open source partners. Maybe this will change their attitude.

Is it really Redhat support that is slowing Linux adoption in the enterprise as Larry says? I think not.

1-800-Call-Larry—Who Ya Gonna Call? TSANet To the Rescue.

Oh, I know, we all want a single-source provider for all support. Uh huh, nothing like feeling really, truly, alone in the world. Have a complex problem and only a single 1-800 number to call? Good luck. Isn’t that why proprietary solutions were so despised? Isn’t the openness of Open Systems why Oracle is where it is today? What happened?

If this trend toward solutions consolidation continues, we are all going to sorely miss the days when there were multiple providers in a given solution who were fighting for your business and success. That is, after all, why TSANet is so crucial and why you need to know more about it. See, if you have a multi-provider solution where the providers are signatories to TSANet, there is no “finger pointing”. Ahh, yes, the fabled “finger pointing”. Unless you get a single provider—soup to nuts—there are going to be multiple players. In a problem resolution scenario, the only finger pointers are the big players. They are the only ones that can afford to lose your business. Does anyone think that, say, a small infrastructure player in a multi-vendor solution can actually get away with being the finger pointer? Heavens no. It is always the biggest player that tries to dismiss off their problem to the motivated, smaller new comer. Alas, TSANet has always been the protection from such poor business practice.

Deployment Standards

Imagine a datacenter that had both Windows (with SQL Server) and Linux running critical applications. Imagine further the need to consolidate and provide high availability at the same time. Now, quick, pick your solution. How is Unbreakable Linux (redux) or GFS or OCFS supposed to help at all?I’ve had dozens of you readers ask me why PolyServe. The answer is that only PolyServe solves this problem on both Linux and Windows. Our customers want a cluster deployment model that works for both Windows and Linux—imagine that!Doing things the same way regardless of operating system sounds like a good idea to me.

Open Source—The Perfect Business Model

Well, PolyServe missed the chance to open source our products when the timing was right. Had we, there is a chance that the same thing that just happened to Redhat could have happened to us. What a great business model.You get venture capital funding, build a world-class product, build a support ecosystem, open source it, get hundreds of customers in production, then someone like Oracle takes over for you. That almost makes the add-revenue model of the burning-piles-of-cash.com startups of the 1990’s like pretty attractive.

So, no, we won’t be doing the open source thing. After Larry’s announcement, it looks less and less attractive every day.

Solaris 10 on AMD anyone? Hmmm…

BLOG UPDATE – 2012.06.07 – Wayward Googlers resurrected this old post. Using my not-so-canny speed-reading skills I jumped in with comments. A reader emailed me to point out I bit myself with nomenclature (“B” vs “b”). At this point I think the 100Mb per 1GHz is prime for more scrutiny. I held fast to that ratio in the time frame of this original post. However, my work with Westmere and Sandy Bridge Xeons leads me to believe the ratio is in dire need for updating. I’ll address that topic in an up-coming post and link back to this post.

Calling My Friends Liars–How Fun

I can’t remember the last time I disagreed with Jeff Needham. I realized about 15 years ago (IIRC) that it doesn’t make sense to do so because he is always (5 “nines”) right. In a quick chat today he said:

Polyserve can sell up the notion that the gateway does break the 1Gbps barrier (which mostly people falsely believe is not enough I/O for “them”)

The “gateway” Jeff is referring to is the File Serving Utility for Oracle but that is not the topic of this post. I want to cowardly disagree with Jeff about his assertion that Oracle IT people are erroneously concerned that the most common NAS bandwidth (1GbE) is not sufficient for their needs. I assert that such a concern may in fact be warranted. The point I want to make, and therefore left-handedly disagreeing with Jeff about, is that it doesn’t matter.

Yes, 1 Gigabit Ethernet is the most common NAS connectivity medium today, and with very little tuning you can get a realizable payload of roughly 112MB/s—I do. If 112/MBs is starving your CPUs, all is not lost.

A Safe Rule of Thumb
There is a rule of thumb that has stood the test of time regarding the balance between processor capability and I/O bandwidth. Now, I know that sometimes man bites dog, but the highest majority of systems will strike a balance between CPU and I/O according to the following formula:

100Mb I/O bandwidth for each 1GHz of CPU

This formula leans towards DSS-style workloads, so it is certainly a safe bet for OLTP. Oh, by the way, Mb is not MB. I see such notation horribly interchanged all to often and it makes a big difference when you are trying to stuff 100 pounds (how many kilos is that?) of rocks into a 10 pound bag…

Memories
The last “really big” system I had dedicated to my projects (it was my “personal” lab system) was a Sequent NUMA-Q 2000 with 32 700MHz processors, 32GB RAM and 396 4GB hard disk drives. The formula was true then. It doesn’t sound like much by today’s standards, but about 280MB/s would saturate the system if I was doing heavy lifting such as index creation or complex queries with Parallel Query Option. On the other hand, I assure you that the processors nearly burst into flames running OLTP long before I hit ~280MB/s random 4KB transfers. After all, using the formula, that system would be able to deliver roughly 70,000 4KB IOps—and that was a lot in those days. On the contrary, I’ll blog soon about how uneventful that I/O rate is with modern commodity servers (and I still hate that term, need to blog on it—note to self).

The moral of the story is that if you are running on a legacy Unix system that is near the end of its lease it is quite likely that the compute power it offers can be replaced by an 8 core AMD Opteron system running 64-bit Oracle. Put that thought on the back burner if you had a short lease on, say, an IBM RS/6000 Regatta though J(hey, I still have my favorites). If you are planning a deployment that can be handled by an 8 core AMD Opteron system (very likely), I can all but guarantee you that triple-bonded NFS paths with client-side O_DIRECT will not starve your processors one bit.

Now, if you think a single-headed NAS device Filer) can really feed a 3-way triple bonded NFS data path for reads and writes, you need to do some testing and then read this.

So, in the end, I didn’t really disagree at all with Jeff, and for that, I feel good and safe!

Hello Folks,

This is the start of my Blog about topics that involve Oracle performance and availability at the platform (OS, server hardware) and storage level. I hope the topics I bring to the community will be informative, helpful and fit for discussion.

Recently I was exchanging email with fellow OakTable Network member, Tom Kyte, regarding some photos on his flickr page. Tom travels a lot (much more than I could handle) and captures a lot of the “feel” of the places he visits. There were a lot of photos there that had great composition so I commented about “fundamentals” and how having a background with an old rickety SLR like my Nikon FE2 (I actually love that thing) has helped me even with my inexpensive 6 megapixel digital. Tom made an entry in his blog about this. Carrying this train of thought to Oracle, I think a lot of the Oracle-related topics I cover in my papers and speaking engagements are rooted in this appreciation for the fundamentals. I hope as this Blog matures we both can figure out what that really means.

So, in concert with this thread about photography and fundamentals, and to get my Blog going, I thought I’d also post up a photo or two and introduce the spirit of this Blog. Nobody is perfect—most certainly not me. I will do the best I can to bring ideas and topics to this Blog that are as accurate and helpful as possible. I suppose one of the first things I should do is update my profile so readers can get an idea who I am, where I’ve been, and why I would have any business discussing Oracle-related topics from a platform and storage perspective. Hopefully my first real installment will make that point just as well.