Kevin Closson's Blog: Platforms, Databases and Storage

I thought a comment on one of my recent blog entries deserved handling in a blog entry. A reader posted:

Have you done any comparisons of the HP DL585 with an HP DL580? Is the DL580 a NUMA machine? Which one would you by today for a RAC cluster?

I’ll answer these out of order. The DL580 is not a NUMA system. Although it stands to reason that if HP continues the DL580 product line into the future to the point where they bake in the CSI interconnect then at that time the DL580 would be a NUMA system. So, the short answer to whether or not a DL580 is a NUMA system is no, it is not. I think long answers are more fun.

In my series of posts about Oracle on NUMA, I think I must have said it about umpteen times, but I’ll say it again concisely in this post. I’m talking about what “NUMA-aware” software means. I routinely hear that Oracle is NUMA-aware. It is, and it isn’t. The reason I say this is because there are widely varying degrees of NUMA-awareness that varies between hardware platforms and Oracle ports. I made the point in my recent post about Oracle Database 10g 10.2.0.4 that 10.2.0.4 contains NUMA-related fixes, and it does. However, that isn’t saying it is the fullness of NUMA-aware, because it isn’t. However, the only question that matters is whether it is sufficiently NUMA-aware for today’s NUMA systems, and I’d have to say that it is.

I’ll give a hint: No Linux Oracle release can be fully NUMA-aware until processes (e.g., shadow processes, PQO slaves, etc) can quickly and cheaply detect what CPU they are currently executing on and prefer memory resources based on that locality. Way back in 1996 I was in Advanced Oracle Engineering at Sequent and we were in the late stages of producing the first commercial NUMA system. It was my early Oracle work on Sequent’s NUMA that begat the GETENGNO(3SEQ) API, which was an extremely inexpensive call for processes to check what CPU they are executing on.

Let’s fast forward to today. The Linux development folks are considering the Linux corollary for Sequent’s GETENGNO() with the vgetcpu() call. The problem is that the call is very, very slow compared to the 4-6 cycles that Sequent required to inform a process what CPU it was executing on. Nonetheless, the point is that until vgetcpu() works, and Oracle exploits it, the pinnacle of NUMA-awareness has not been met. And while that may not matter given today’s AMD situation, it will certainly matter when Intel system are NUMA (e.g., CSI based). I guess I shouldn’t equate Linux NUMA with AMD since IBM’s x3950 is a building block for large NUMA systems, and there are others as well. But I was focusing on commodity-level NUMA systems which the x3950 most certainly is not.

There are a lot of factors in selecting hardware, but since I’m asked about DL585 vs DL580, I’d say DL580-so long as it is a DL580 G5. I have tested DL585 and DL580 side by side. However, that was a pretty old DL580 G3 (1066 MHz FSB). I see that the DL580 is now fit with the “Penryn” Xeons (e.g., 5460), which have a front-side bus speed of 1333MHz. There are G5’s that are fit with “Tigerton” Xeons which are 1066 MHz FSB. I’ve seen benchmark results that suggest there is some 21% to gain from going with a 5460-based G5 over a 7350-based G5. So, look closely at the specification. Also, I think a shrewd shopper would try to read the crystal ball to see when the DL580 G5 will be fit with the Xeon 5462 which has a 1600 MHz FSB. As always, with Oracle you want big pipes.

I try not to make announcements about past posts, however, I noticed that my index page of 11g related posts was incomplete. Those of you who pick up my posts by RSS will have seen this stuff already. New readers that have been using the indexed pages might care to go to my index of 11g related posts. Parts II and III of the 11g Automatic Memory Management series were not linked. Now they are.

I’ve gotten many emails over the last several months that I’ve been blogging from IT shops inquiring as to whether or not I can consult in their datacenter on Oracle related performance problems or planning situations. Of course I have to turn such opportunities down since my current gig is with Oracle Server Technologies, but I’ve never been an independent. Fortunately I know really good people that are—many of whom are fellow members of the Oaktable Network. Once such independent is Kyle Hailey. If you are in the right geography ( or perhaps geography doesn’t matter, I don’t know) and need a real heavy hitter, I’d recommend contacting  PerfVision.com and have a chat with Kyle.

The blogging platform I use, WordPress, allows me to see from whence readers are being referred to my blog. I’ve gotten a couple of hits from this post on Jeff Browning’s blog. I’m not sure if that particular post is a critique or a caricature of my paper about Oracle Database 11g Direct NFS. Nonetheless, shortly after I first saw the post I privately informed Jeff of four inaccuracies contained in the post. They were:

1) Jeff’s assertion that Direct NFS will work on any NAS device is incorrect. I know of one NAS device in particular that DNFS will not function on whatsoever. Well, that’s not exactly true. It will function to the point of creating a database, but will corrupt the control file in doing so. I’m not going to state the particular manufacturer, because it is still unclear if there is a design incompatibility problem or simply a bug in that prevents that particular NAS device from functioning with DNFS.  Nonetheless, Jeff still hasn’t corrected the following quote, so I am here and now. Jeff said:

As such dNFS will work in exactly the same manner, with identical performance benefits, on any NAS device from any vendor.

That is incorrect.

2) Jeff uses incorrect terminology to explain one of the benefits of Direct NFS. That term is “context swapping”, and I quote:

Here is the theory behind dNFS. I/O on a database server occurs in a combination of user space and kernel space. Context swaps between the two spaces are expensive in terms of CPU cost. If you can move a part of that activity from kernel space into user space, you can save CPU cost due to reduced context swapping.

I’m not playing word games. Transitions from user to kernel mode are routinely, and inaccurately, referred to as context switching and while that is one of my pet peeves I suspect from the context of Jeff’s post that context switching is the term he actually meant. The problem is that system calls do not result in a context switch.

A context switch is the stopping of one process and switching to another for the sake of process scheduling. If a system call does not block, there is no context switch. An example would be getpid(2). On the other hand, a system call that does block (e.g., synchronous physical I/O) will count as a context switch because the process parked itself on a blocking system call. The scheduler will switch to a runable process according to such criteria as mode, priority, processor affinity and so on. The sum total of context switch categories is voluntary and involuntary. The former is when a process calls a blocking system call (or any other voluntary yield such as sched_yield(2)), the latter is when a user mode process executes to the end of its time slice at such a time as there is a runable process for the kernel to switch to.

And yes, I say all this about context switching versus context swapping with total disregard for this patent. In Unix/Linux, the term is context switch and knowing what that really means makes the output of monitoring commands such as vmstat(1) a little more meaningful I should think.

3) He stated 11g x86_64 was not available yet at the time of that blog post it was indeed available for download.

4) He misspelled my name in the blog post.

What is This Blog Post Really About?
Now, all of that aside, none of that is really what this blog post is about. This blog post is about a comment on Jeff’s blog. The reader posted a comment as follows:

One would assume that an OS vendor would be better at making NFS and multipathing work rather than a database vendor. Oracle has a disadvantage in that it has to test its client on all the various flavors of OS out there.

One would indeed assume just that-correctly in fact. Oracle Database 11g doesn’t make NFS better. Direct NFS is a replacement for NFS. While NFS is a great storage presentation model for Oracle (as I’ve said so many times), it does much more than Oracle requires. So, DNFS strips all that overhead out. Direct NFS is essentially an RPC shot straight at the NAS device-no file-related system calls (e.g., open(2), pread(2), io_submit(2), etc). And, oh, while I’m at it, I’ll point out that open(2), read(2) and io_submit(2) are system calls that do not result in a context switch (unless the read suffers a page cache miss or is O_DIRECT or raw(8)). But that is not what this blog post is about.

Some time back I recommended folks give Greg Rahn’s blog a visit. The blog is at structureddata.org and I’ve been reading it for some time. I haven’t known Greg for that long, but I have the utmost respect for the group he is in since it is lead by Andrew Holdsworth. That means I have respect for Greg as well! J

Greg and I have had some short chats, the latest of which took place at Mogens Norgaard’s impromptu San Franscisco office of MiracleAS (Chevy’s across the street from Moscone south) where Mogens (as co-founder of the Oaktable Network) showed his ever-so-typical hospitality to fellow Oaktable member and those stray cats we happened to bring along with us. In my case, that was Greg Rahn and my old friend Randy from EMC (Federal) and Matt Zito of GridApp. As always that was a nice chat since Anjo Kolk, Doug Burns and Jared Still (Oaktable members) were there at one point or another as well. Of course we talked about Oracle, but I recall we spent more than a few moments talking about Data Warehouse 2.0 technology Data Warehouse Appliances-a topic I shall blog about in great detail soon enough.

I’m adding Greg to my blogroll.

I just did a Google blog search for information on the HP Smart Array Controller tool called hpaducli and didn’t find anything, so I’ll make this quick blog entry.

I just finished doing a small performance test on some storage I have access to and realized something was wrong. Granted, I’m doing a fairly wacky test, but something was wrong nonetheless. I am testing read throughput but limiting my dataset to an amount that fits within the HP Smart Array controller cache configured on the SB40c storage blade I am using. The timings I was getting were not coming in as a cached workload. I used the hpaducli command to find out that the cache was disabled. After the following grep(1) command told me I had no Smart Array cache, I used the ACU (cpqacuxe -R) to enable it and verified using the hpaducli -f command.

# grep -i 'Array Accelerator' /tmp/P400_status_before
      Array Accelerator is not configured.
      Array Accelerator is not configured.
      Array Accelerator is not configured.
      Array Accelerator is not configured.
      Array Accelerator is not configured.
      Array Accelerator is not configured.
# hpaducli -f /tmp/P400_status_after
# grep 'Array Accelerator'/tmp/P400_status_after
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.

The moral of the story is when someone hands you some hardware, you never quite know what you are getting. What’s that line from Forrest Gump? You never know, this post might help someone, somewhere, someday.

Instead of FAX communication, why don’t organizations just ask for 80 column punch cards or cuneiform on papyrus reed? These days I get so agitated when, say, a customer service department requires a FAX to clear up an issue. Does anyone else feel the way I do, or am I just (un?)characteristically crabby this morning?

Instead of wandering around looking for a FAX machine, I thought I’d email the following reasonable anti-FAX position to see where it would get me:

Regarding […customer service issue…], do you really want a FAX? Anymore that is becoming so outdated. How about a scanned PDF emailed? That would surely take the same amount of human resources (perhaps less since you wont be changing paper), cuts down on filing space, saves trees and keeps me from trying to find a  FAX machine.

And their brilliant reply? Of course:

Regarding your inquiry, the preferred methods of documentation submission are via postal mail [or] fax to our office at […]

Doesn’t it stand to reason I already knew their preferred method? Oh, hold it, there is new information there after all: I can simply stuff it in an envelope and snail mail it. Brilliant!

I’d like to recommend Christian Bilien’s Blog. He covers Solaris and SAN-related topics (and more) that pertain to Oracle very, very nicely. One more for the blog roll.

Well, it’s official. I am a completely delinquent blogger. It isn’t that I don’t have a lot of stuff to blog about, because I do. I just don’t know what is “safe” to blog about. I’m still trying to get a feel for things. I’ve taken a role as a Performance Architect in Oracle’s Systems Technology group, which is a division of the Database Server Technologies organization. I’m working on future products that I cannot talk about whatsoever. That is proving to be a bit of a bummer. Even if I blog about some of the lower-level stuff I’m seeing that is platform generic, it would still tip folks off on the sorts of things we are doing on this project, so as they say, “Mums the word.”

I know there are all sorts of folks I’ve wanted to meet up with at OOW. Send me an email at ora_kclosson at yahoo dot com , ok?

I do have a Linux block layer I/O related blog entry that is just about ready. I’ll do my best.

Oops, there was a false alarm about the availability of Oracle Database 11g (11.1.0.6) for x86_64 (AMD/EM64T). I just checked and it is now downloadable on OTN.

Not much to blog about. No, actually, I have an Olympic-sized swimming pool full of things to blog about, but I can’t. No time.

Since I’m working on a future Oracle product I’ve got RASUI on my mind. I thought this video seemed appropriate.

Polls Are Useless Without Critical Mass
Have all you readers participated in my (yet another) RAC poll?  By the numbers I’d say not. Please take a moment and visit the poll. Thanks.

Well, I’ve been in Oracle Server Technologies for a whopping week and in that short time I’m reassured of two constants:

• Blogging takes time (that I don’t have)
• Focusing on a single product, and more specifically a feature within a single product suite, makes a guy very pin-point minded

But, alas, I still need to stay well-rounded. So, thanks to a friend (and VP at a company I once worked for) for alerting me to the fact that Michael Stonebraker and others are blogging at The Database Column.

Yes, I’ll be reading that one. I don’t know if I’d recommend it to my readers though. That may sound odd, but most of my readers are practitioners of real, existing technology filling a production purpose. I expect the stuff on that blog to be laced with theory. Now, having said that, I have to remind myself that I believe well over 90% of my blog content is not exactly what one would call “ready to use” information.

All that aside, I just thought I’d poke my head up from the hole I’m in and make a quick blog entry. OK, there, I did…now it is back into the dungeon…but wait.

First Installment
Just because I said I was going to read it doesn’t imply I’m going to take it as some gulp from the fountain of omniscience.

For instance, in Michael Stonebraker’s post about row versus column orientation, he states:

[…] Vertica can be set-up and data loaded, typically in one day. The major vendors require weeks. Hence, the “out of box” experience is much friendlier. Also, Vertica beats all row stores on the planet – typically by a factor of 50. This statement is true for software only row stores as well as row stores with specialized hardware (e.g. Netezza, Teradata, Datallegro). The only engines that come closer are other column stores, which Vertica typically beats by around a factor of 10.

In my opinion that looks cut and pasted from a Vertica data sheet-but I don’t know. One thing I do know is that it seems unwise to say things like “all” and “typically by a factor of” in the same sentence. And this bit about “set-up and loaded” taking “weeks” for the “major vendors” is just plain goofy.

That’s my opinion. No, hold it, that’s my experience. I’ve loaded Oracle databases sized in the tens of terabytes that certainly didn’t take weeks!

This is a link to an Oracle ACE who didn’t nominate himself.