Kevin Closson's Blog: Platforms, Databases and Storage

I try not to make announcements about past posts, however, I noticed that my index page of 11g related posts was incomplete. Those of you who pick up my posts by RSS will have seen this stuff already. New readers that have been using the indexed pages might care to go to my index of 11g related posts. Parts II and III of the 11g Automatic Memory Management series were not linked. Now they are.

I’ve gotten many emails over the last several months that I’ve been blogging from IT shops inquiring as to whether or not I can consult in their datacenter on Oracle related performance problems or planning situations. Of course I have to turn such opportunities down since my current gig is with Oracle Server Technologies, but I’ve never been an independent. Fortunately I know really good people that are—many of whom are fellow members of the Oaktable Network. Once such independent is Kyle Hailey. If you are in the right geography ( or perhaps geography doesn’t matter, I don’t know) and need a real heavy hitter, I’d recommend contacting  PerfVision.com and have a chat with Kyle.

The blogging platform I use, WordPress, allows me to see from whence readers are being referred to my blog. I’ve gotten a couple of hits from this post on Jeff Browning’s blog. I’m not sure if that particular post is a critique or a caricature of my paper about Oracle Database 11g Direct NFS. Nonetheless, shortly after I first saw the post I privately informed Jeff of four inaccuracies contained in the post. They were:

1) Jeff’s assertion that Direct NFS will work on any NAS device is incorrect. I know of one NAS device in particular that DNFS will not function on whatsoever. Well, that’s not exactly true. It will function to the point of creating a database, but will corrupt the control file in doing so. I’m not going to state the particular manufacturer, because it is still unclear if there is a design incompatibility problem or simply a bug in that prevents that particular NAS device from functioning with DNFS.  Nonetheless, Jeff still hasn’t corrected the following quote, so I am here and now. Jeff said:

As such dNFS will work in exactly the same manner, with identical performance benefits, on any NAS device from any vendor.

That is incorrect.

2) Jeff uses incorrect terminology to explain one of the benefits of Direct NFS. That term is “context swapping”, and I quote:

Here is the theory behind dNFS. I/O on a database server occurs in a combination of user space and kernel space. Context swaps between the two spaces are expensive in terms of CPU cost. If you can move a part of that activity from kernel space into user space, you can save CPU cost due to reduced context swapping.

I’m not playing word games. Transitions from user to kernel mode are routinely, and inaccurately, referred to as context switching and while that is one of my pet peeves I suspect from the context of Jeff’s post that context switching is the term he actually meant. The problem is that system calls do not result in a context switch.

A context switch is the stopping of one process and switching to another for the sake of process scheduling. If a system call does not block, there is no context switch. An example would be getpid(2). On the other hand, a system call that does block (e.g., synchronous physical I/O) will count as a context switch because the process parked itself on a blocking system call. The scheduler will switch to a runable process according to such criteria as mode, priority, processor affinity and so on. The sum total of context switch categories is voluntary and involuntary. The former is when a process calls a blocking system call (or any other voluntary yield such as sched_yield(2)), the latter is when a user mode process executes to the end of its time slice at such a time as there is a runable process for the kernel to switch to.

And yes, I say all this about context switching versus context swapping with total disregard for this patent. In Unix/Linux, the term is context switch and knowing what that really means makes the output of monitoring commands such as vmstat(1) a little more meaningful I should think.

3) He stated 11g x86_64 was not available yet at the time of that blog post it was indeed available for download.

4) He misspelled my name in the blog post.

What is This Blog Post Really About?
Now, all of that aside, none of that is really what this blog post is about. This blog post is about a comment on Jeff’s blog. The reader posted a comment as follows:

One would assume that an OS vendor would be better at making NFS and multipathing work rather than a database vendor. Oracle has a disadvantage in that it has to test its client on all the various flavors of OS out there.

One would indeed assume just that-correctly in fact. Oracle Database 11g doesn’t make NFS better. Direct NFS is a replacement for NFS. While NFS is a great storage presentation model for Oracle (as I’ve said so many times), it does much more than Oracle requires. So, DNFS strips all that overhead out. Direct NFS is essentially an RPC shot straight at the NAS device-no file-related system calls (e.g., open(2), pread(2), io_submit(2), etc). And, oh, while I’m at it, I’ll point out that open(2), read(2) and io_submit(2) are system calls that do not result in a context switch (unless the read suffers a page cache miss or is O_DIRECT or raw(8)). But that is not what this blog post is about.

Some time back I recommended folks give Greg Rahn’s blog a visit. The blog is at structureddata.org and I’ve been reading it for some time. I haven’t known Greg for that long, but I have the utmost respect for the group he is in since it is lead by Andrew Holdsworth. That means I have respect for Greg as well! J

Greg and I have had some short chats, the latest of which took place at Mogens Norgaard’s impromptu San Franscisco office of MiracleAS (Chevy’s across the street from Moscone south) where Mogens (as co-founder of the Oaktable Network) showed his ever-so-typical hospitality to fellow Oaktable member and those stray cats we happened to bring along with us. In my case, that was Greg Rahn and my old friend Randy from EMC (Federal) and Matt Zito of GridApp. As always that was a nice chat since Anjo Kolk, Doug Burns and Jared Still (Oaktable members) were there at one point or another as well. Of course we talked about Oracle, but I recall we spent more than a few moments talking about Data Warehouse 2.0 technology Data Warehouse Appliances-a topic I shall blog about in great detail soon enough.

I’m adding Greg to my blogroll.

I just did a Google blog search for information on the HP Smart Array Controller tool called hpaducli and didn’t find anything, so I’ll make this quick blog entry.

I just finished doing a small performance test on some storage I have access to and realized something was wrong. Granted, I’m doing a fairly wacky test, but something was wrong nonetheless. I am testing read throughput but limiting my dataset to an amount that fits within the HP Smart Array controller cache configured on the SB40c storage blade I am using. The timings I was getting were not coming in as a cached workload. I used the hpaducli command to find out that the cache was disabled. After the following grep(1) command told me I had no Smart Array cache, I used the ACU (cpqacuxe -R) to enable it and verified using the hpaducli -f command.

# grep -i 'Array Accelerator' /tmp/P400_status_before
      Array Accelerator is not configured.
      Array Accelerator is not configured.
      Array Accelerator is not configured.
      Array Accelerator is not configured.
      Array Accelerator is not configured.
      Array Accelerator is not configured.
# hpaducli -f /tmp/P400_status_after
# grep 'Array Accelerator'/tmp/P400_status_after
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.
      Array Accelerator is enabled for this logical drive.

The moral of the story is when someone hands you some hardware, you never quite know what you are getting. What’s that line from Forrest Gump? You never know, this post might help someone, somewhere, someday.

Instead of FAX communication, why don’t organizations just ask for 80 column punch cards or cuneiform on papyrus reed? These days I get so agitated when, say, a customer service department requires a FAX to clear up an issue. Does anyone else feel the way I do, or am I just (un?)characteristically crabby this morning?

Instead of wandering around looking for a FAX machine, I thought I’d email the following reasonable anti-FAX position to see where it would get me:

Regarding […customer service issue…], do you really want a FAX? Anymore that is becoming so outdated. How about a scanned PDF emailed? That would surely take the same amount of human resources (perhaps less since you wont be changing paper), cuts down on filing space, saves trees and keeps me from trying to find a  FAX machine.

And their brilliant reply? Of course:

Regarding your inquiry, the preferred methods of documentation submission are via postal mail [or] fax to our office at […]

Doesn’t it stand to reason I already knew their preferred method? Oh, hold it, there is new information there after all: I can simply stuff it in an envelope and snail mail it. Brilliant!

I’d like to recommend Christian Bilien’s Blog. He covers Solaris and SAN-related topics (and more) that pertain to Oracle very, very nicely. One more for the blog roll.

Well, it’s official. I am a completely delinquent blogger. It isn’t that I don’t have a lot of stuff to blog about, because I do. I just don’t know what is “safe” to blog about. I’m still trying to get a feel for things. I’ve taken a role as a Performance Architect in Oracle’s Systems Technology group, which is a division of the Database Server Technologies organization. I’m working on future products that I cannot talk about whatsoever. That is proving to be a bit of a bummer. Even if I blog about some of the lower-level stuff I’m seeing that is platform generic, it would still tip folks off on the sorts of things we are doing on this project, so as they say, “Mums the word.”

I know there are all sorts of folks I’ve wanted to meet up with at OOW. Send me an email at ora_kclosson at yahoo dot com , ok?

I do have a Linux block layer I/O related blog entry that is just about ready. I’ll do my best.

Oops, there was a false alarm about the availability of Oracle Database 11g (11.1.0.6) for x86_64 (AMD/EM64T). I just checked and it is now downloadable on OTN.