Kevin Closson's Blog: Platforms, Databases and Storage

Some rambling thoughts…

Oracle Database Administrator or Data Administrator?
Oracle continues to add features/functionality that makes the dutiful Database Administrator look a lot more like a Data Administrator. Some would argue it started with ASM since volume management was placed into the hands of the DBA. I’d argue it started with RMAN since that feature was one of the most significant, early features that took functionality away from the OS (and 3^rd Party software) and put control in the hands of the DBA. After all, Oracle backups before RMAN had very little to do with Oracle (version 7 and earlier). I’d argue further that features like SecureFiles also make the Database Administrator more of a Data Administrator-or at least it seems it should make a DBA start to think like a “Data Administrator.” I think that trend will continue. I think that increases the value proposition of the Oracle DBA in the datacenter. I think it is a good thing.

DBAs might even find themselves performing lower-level storage operations such as creating OS-level logical drives. One such example would be HP Smart Array logical drives and the associated CLI is the hpacucli command on Linux.

I’ve been Meaning to Blog This One…
Quite some time ago I was fiddling with hpacucli. Yes, it was late at night. Yes, I was tired and crabby. Why would what’s happened in the following screen output make me crabby?

# hpacucli  
Array Configuration Utility CLI 7.15.19.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.
>  ctrl all show status
Error: Expecting "controller" in target. Type "help" for more information.
> ctrl all show status
Smart Array P400 in Slot 1
   Controller Status: OK
   Cache Status: Not Configured
   Battery Status: OK

… the CLI doesn’t parse out leading whitespaces.

Maybe somebody, someday will be battling the hpacucli CLI and google their way to this post.

P.S., For goodness sake, please don’t go running to your local HP rep to tell him I’m bashing HP. Yes, HP Smart Array is great technology. I just don’t think whitespaces are as significant as hpacucli does 🙂

Yes, I admit, sometimes little things make me crabby. Or is it that being crabby makes me easily irritated at small things? I think it’s the former rather than the latter.

Angry Email Makes Me Crabby
I am surprised at how many emails I’ve received from angry Storage Administrators about my position against Fibre Channel. I generally just ask them to not take it so personally. After all, there is more to storage than whatever storage networking protocol any particular server is plumbed with. Anyone who reads my Manly Man series about Fibre Channel will see that I’m just not a big fan of the current misapplication of Fibre Channel technology. As I point out in installment number two in the Manly Man series, Fibre Channel was originally intended to solve the difficulties associated with connecting hundreds (or thousands) of low capacity drives to a small number of large servers. These days it is being used to provide storage connectivity for hundreds of servers to a couple of huge FC storage arrays. And these hundreds of servers have both Ethernet and FC connectivity-overkill, plain and simple.

A lot is changing in the storage world where Oracle is concerned. I hope to come out of the foxhole to blog more about it over the next few months. Well, maybe I should say a few months from now. In the meantime I’ll be making some datacenter visits. I always learn something interesting when I get a chance to do that.

Little Changes Make Me Crabby
Like, for instance, the following example of the change to iostat -x output from RHEL4 to RHEL5. Notice how the rkB/s and wkB/s columns are missing? Yes, it is simple math to divide [rw]sec/s by 2 to get the same data, but there must be at least 42 of you folks that are feeding iostat -x output through some text processing, right? Time to adjust your scripts. Or is it? I wonder how many folks are really diving into OEL5/RHEL5? It seems a conservative IT shop with a functional RHEL4 configuration would most likely stay put. Any readers care to voice their experience with OEL5/RHEL5 adoption? Is it happening? Are folks waiting for their 11g adoption with plans to team that up with a move to OEL5/RHEL5?

 
# uname -r
2.6.18-53.ELsmp
# iostat -x 1 1
Linux 2.6.18-53.el5 (host1)        03/07/2008

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.08    0.00    0.04    0.03    0.00   99.85

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0        0.16    10.38  0.33  0.95    10.55    90.64    79.47     0.02   14.12   0.86   0.11
sda               0.00     0.00  0.00  0.00     0.01     0.00    21.05     0.00    3.18   2.87   0.00
dm-0              0.00     0.00  0.45 11.33    10.47    90.64     8.58     0.39   32.67   0.09   0.11
dm-1              0.00     0.00  0.00  0.00     0.01     0.00     8.00     0.00    5.05   0.51   0.00
cciss/c0d1        0.00     0.00  0.00  0.00     0.01     0.00    10.12     0.00    4.56   2.63   0.00
cciss/c0d2        0.00     0.00  0.00  0.00     0.01     0.00    10.38     0.00    4.30   2.52   0.00
cciss/c0d3        0.00     0.00  0.00  0.00     0.01     0.00    10.38     0.00    6.17   3.35   0.00
cciss/c0d4        0.00     0.00  0.00  0.00     0.01     0.00    10.38     0.00    5.50   3.22   0.00
cciss/c0d5        0.00     0.00  0.00  0.00     0.01     0.00    10.64     0.00    3.38   2.59   0.00
cciss/c0d6        0.00     0.00  0.00  0.00     0.01     0.00    10.64     0.00    4.05   3.31   0.00
cciss/c0d7        0.00     0.00  0.00  0.00     0.01     0.00    10.12     0.00    4.44   2.29   0.00

$ uname -r
2.6.9-34.ELsmp
$ iostat -x 1 1
Linux 2.6.9-34.ELsmp (host2)    03/07/2008

avg-cpu:  %user   %nice    %sys %iowait   %idle
           1.79    0.00    1.28    0.30   96.63

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0   0.06  11.83  0.14  3.21   23.07  120.39    11.53    60.19    42.77     0.68  201.33   4.60   1.54

…just a quick blog entry to plug Daniel Morgan’s (of Puget Sound Oracle User Group) very useful collection of scripts and other goodies:

Morgan’s Library

I’ve been reading a lot of blogosphere content about Data Warehousing these days. I’ve taken a lot of interest in such technology as Netezza, GreenPlum, DATAllegro and others and blog reading proves to be an interesting way to augment one’s knowledge. Who’d have thought I’d learn so much about OLTP through this reading.

Memory is Faster than Disk, So Let’s Do a Complete Rewrite
Why, just today I found out that it is time for a total rewrite of commercial RDBMS products. Uh huh. More interestingly, though, I learned:

• Memory is faster than disk. Really, truely, it is!
• A dual-core (2.8GHz) server with 4GB memory and 4 250GB SATA drives can perform 51,000 TpmC
• Disabling transaction logging entirely in a commercial RDBMS will increase throughput (TpmC) about three-fold

I found these pearls of wisdom while reading a Stonebraker paper referred to on this blog post. Yes, I know that blog is basically a store-front for Vertica, but I like to learn about different things that are going on in database technology. Unfortunately this time I was wasting my time. The URL in that blog post points to the VLDB front page, but a little sleuthing found the paper posted here: The End of an Architectural Era (It’s Time for a Complete Rewrite).

Recite after me:

If you get two orders of magnitude performance gain, you are either not doing it or you’ve moved it closer to the processor.

Dang, and I ain’t even got no too pretty good pedigree. Pshaw, I dasn’t fidget ‘mungst the quality!

Central versus De-centralized versus Shared-Nothing
No, it isn’t time for a re-write, especially one that requires a complete shared-nothing database approach. Now don’t get me wrong, I’m all for de-coupling and grid architecture-most particularly where storage is concerned. If I hear of another poor production site that is head-saturated on a $500,000 storage array when driving a measly 15 or so 15K RPM drives, I’ll BAARF. Please see the following post for what I’m talking about:

Hard Drives Are Arcane Technology. So Why Can’t I Realize Their Full Bandwidth Potential?

In my post called Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!, I linked to a paper Oracle maintains that explains how to use an NFS export from a small Unix/Linux server as storage for a third voting disk in a stretch RAC cluster. I pointed out that the paper instructs on how to use the noac mount option for Linux RAC clusters in spite of the many resources that suggest actimeo=0 will do. The authors of the document are standing fast that if you are building a Linux-based RAC stretch cluster and are using an NFS mount as a third voting device you do indeed need to mount that particular NFS filesystem with noac. That nugget of truth contradicts so many different documents that I don’t care to list. Instead, I’ll list a resource from Metalink that helps clarify the issue. In fact, I would say that no matter what the sundry Installation Guides or Release Notes say, refer to Metalink 359515.1 when the topic of Oracle Database 10g on NFS filesystems comes up.

Datafiles or CRS Files? The Mount Options Differ.
Metalink 359515.1 is a really helpful note. It spells out the RAC-related mount options for 10gR2 on Solaris, AIX, HP-UX and Linux. Most importantly, it spells out the options for the datafiles and the CRS files in two separate columns. Lo and behold, Metalink 359515.1 clearly spells out that noac is needed for CRS files, but not for datafiles.

In the comment section of Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!, a reader points out that the Third Party Vote on NFS paper has a dead end URL (http://www.oracle.com/technology/deploy/availability/htdocs/vendors_nfs.html) in the section that aimed to point out the fact that you cannot just use some Unix/Linux server NFS exports for any other purpose than this unique third voting disk setup in a stretch cluster scenario. He is right, that URL is a dead end, but I’d rather point folks to Linux RTCM (RAC Technology Certification Matrix) or the Unix RTCM, both of which clearly spell out a list of supported NFS File servers. Missing from the list is, of course, some plain old Linux or Unix server dishing out NFS exports-because the only supported application of simple, Unix/Linux NFS exports is the third voting disk scenario in a stretch cluster.

The reader also added this comment:

Many people say Oracle doesn’t support NFS, we need to verify. Searching oracle.com, “We did not find any search results for: vendors_nfs.html” and the references from google all seem to point at that one mysteriously missing doc.

Gee whiz. Where to start? Yes, for the eleventeenth time, NFS filesystems are supported for Oracle Database (including RAC). Let’s not get so easily confused; NFS is a protocol and the storage is NAS. Let’s all enter the following formula in our decoder rings before reading Oracle documents:

(Some Stupid Little Linux/Unix Server Exporting Filesystems via NFS) != NAS

The only supported application of non-NAS NFS is described in the following paper: Using NFS for a Third CRS Voting Device

Now, for some light reading about Oracle on NFS with 11g, I submit:

Oracle Database 11g on NFS filesystems:

Using Network Attached Storage or NFS File Systems Installation Guide for Microsoft Windows

Using Network Attached Storage or NFS File Systems Installation Guide for HP-UX

Using Network Attached Storage or NFS File Systems Installation Guide for AIX 5L Based Systems (64-Bit)

Using Network Attached Storage or NFS File Systems Installation Guide for Solaris Operating System

Using Network Attached Storage or NFS File Systems Installation Guide for Linux

And, of course: Configuring Direct NFS Storage for Datafiles

But, let’s not forget:

• Manly Men Only Deploy Oracle with Fibre Channel Storage – Part I. Oracle Over NFS is Weird.
• Manly Men Only Deploy Oracle with Fibre Channel – Part II. What’s So Simple and Inexpensive About NFS for Oracle?
• Manly Men Only Deploy Oracle with Fibre Channel – Part III. Did I Hear EMC Say NAS?
• Manly Men Only Deploy Oracle with Fibre Channel – Part IV. SANs are Simple, RAC is Difficult!
• Manly Men Only Deploy Oracle with Fibre Channel – Part V. What About Oracle9i on RHAS 2.1? Yippie!
• Manly Men Only Deploy Oracle with Fibre Channel – Part VI. Introducing Oracle11g Direct NFS!
• Manly Men Only Deploy Oracle with Fibre Channel – Part VII. A Very Helpful Step-by-step Install Guide for RAC on NFS.
• Manly Men Only Deploy Oracle With Fibre Channel – Part VIII. After All, Oracle Doesn’t Support Asynchronous I/O On NFS!

In my recent rant about Oracle database installation difficulties, I provided a link to a video in which fellow OakTable Network members Morten Egan and Mogens Norgaard captured how difficult the task really is.

Well, they’re at it again. You’ll see Morten “The Nose” Egan start out this new video taped Oracle installation by configuring a SAN with what looks like the HP Array Configuration Utility, but then my eyes are getting as bad as my blogging frequency. I couldn’t miss the Windows Disk Manager though-not even on fast forward.

I think we should start calling him Morten “The Hair” Egan. The link to the video follows:

Unconventional Oracle Installation Part II

I thought a comment on one of my recent blog entries deserved handling in a blog entry. A reader posted:

Have you done any comparisons of the HP DL585 with an HP DL580? Is the DL580 a NUMA machine? Which one would you by today for a RAC cluster?

I’ll answer these out of order. The DL580 is not a NUMA system. Although it stands to reason that if HP continues the DL580 product line into the future to the point where they bake in the CSI interconnect then at that time the DL580 would be a NUMA system. So, the short answer to whether or not a DL580 is a NUMA system is no, it is not. I think long answers are more fun.

In my series of posts about Oracle on NUMA, I think I must have said it about umpteen times, but I’ll say it again concisely in this post. I’m talking about what “NUMA-aware” software means. I routinely hear that Oracle is NUMA-aware. It is, and it isn’t. The reason I say this is because there are widely varying degrees of NUMA-awareness that varies between hardware platforms and Oracle ports. I made the point in my recent post about Oracle Database 10g 10.2.0.4 that 10.2.0.4 contains NUMA-related fixes, and it does. However, that isn’t saying it is the fullness of NUMA-aware, because it isn’t. However, the only question that matters is whether it is sufficiently NUMA-aware for today’s NUMA systems, and I’d have to say that it is.

I’ll give a hint: No Linux Oracle release can be fully NUMA-aware until processes (e.g., shadow processes, PQO slaves, etc) can quickly and cheaply detect what CPU they are currently executing on and prefer memory resources based on that locality. Way back in 1996 I was in Advanced Oracle Engineering at Sequent and we were in the late stages of producing the first commercial NUMA system. It was my early Oracle work on Sequent’s NUMA that begat the GETENGNO(3SEQ) API, which was an extremely inexpensive call for processes to check what CPU they are executing on.

Let’s fast forward to today. The Linux development folks are considering the Linux corollary for Sequent’s GETENGNO() with the vgetcpu() call. The problem is that the call is very, very slow compared to the 4-6 cycles that Sequent required to inform a process what CPU it was executing on. Nonetheless, the point is that until vgetcpu() works, and Oracle exploits it, the pinnacle of NUMA-awareness has not been met. And while that may not matter given today’s AMD situation, it will certainly matter when Intel system are NUMA (e.g., CSI based). I guess I shouldn’t equate Linux NUMA with AMD since IBM’s x3950 is a building block for large NUMA systems, and there are others as well. But I was focusing on commodity-level NUMA systems which the x3950 most certainly is not.

There are a lot of factors in selecting hardware, but since I’m asked about DL585 vs DL580, I’d say DL580-so long as it is a DL580 G5. I have tested DL585 and DL580 side by side. However, that was a pretty old DL580 G3 (1066 MHz FSB). I see that the DL580 is now fit with the “Penryn” Xeons (e.g., 5460), which have a front-side bus speed of 1333MHz. There are G5’s that are fit with “Tigerton” Xeons which are 1066 MHz FSB. I’ve seen benchmark results that suggest there is some 21% to gain from going with a 5460-based G5 over a 7350-based G5. So, look closely at the specification. Also, I think a shrewd shopper would try to read the crystal ball to see when the DL580 G5 will be fit with the Xeon 5462 which has a 1600 MHz FSB. As always, with Oracle you want big pipes.