Platform, Database and Storage Topics
There are 120 spindles downwind of this….drumroll please….
$ asmcmd ASMCMD> ls SDATA1/ SDATA2_1/ SDATA2_2/ SDATA2_3/ ASMCMD> rm -fr * ASMCMD>
Eek!
In other news, I just learned that Greg Rahn is about to get his hands on a set of HP DL580 G5 (Tigerton goodies). I’d sure like to get Silly Little Benchmark numbers from that sweetheart of a system…wish in one hand, you-know-what in the other I suppose.
Oracle’s acquisition spree has wound up producing odd bedfellows.
I’ve often wondered why companies really want to own their computers. HP has voiced the same rhetorical question by launching AIaaS.
Why Do They Put Those Data Centers There?
But Atlanta? Is power really that cheap in Atlanta? It must be all that cheap hydroelectric power down there.
What’s This Have To Do With Oracle?
This is an Oracle-related blog, but I can’t answer that question. I’m watching just like you all are.
The Irony
I find it very ironic that HP announced this hosting initiative at their Technology@work Conference in Barcelona Spain considering the other hot news about the HP DL785 which is an 8-socket “Barcelona” Socket-F based Server. I am happy to see Proliant 8-socket Servers.
Network Appliance has renamed their company a nickname. That just seems strange to me. Different name, same stuff.
And, yes, NFS works nicely for Oracle Database.
Maybe Oracle should be renamed/rebranded to “scott/tiger?”
Some rambling thoughts…
Oracle Database Administrator or Data Administrator?
Oracle continues to add features/functionality that makes the dutiful Database Administrator look a lot more like a Data Administrator. Some would argue it started with ASM since volume management was placed into the hands of the DBA. I’d argue it started with RMAN since that feature was one of the most significant, early features that took functionality away from the OS (and 3rd Party software) and put control in the hands of the DBA. After all, Oracle backups before RMAN had very little to do with Oracle (version 7 and earlier). I’d argue further that features like SecureFiles also make the Database Administrator more of a Data Administrator-or at least it seems it should make a DBA start to think like a “Data Administrator.” I think that trend will continue. I think that increases the value proposition of the Oracle DBA in the datacenter. I think it is a good thing.
DBAs might even find themselves performing lower-level storage operations such as creating OS-level logical drives. One such example would be HP Smart Array logical drives and the associated CLI is the hpacucli command on Linux.
I’ve been Meaning to Blog This One…
Quite some time ago I was fiddling with hpacucli. Yes, it was late at night. Yes, I was tired and crabby. Why would what’s happened in the following screen output make me crabby?
# hpacucli Array Configuration Utility CLI 7.15.19.0 Detecting Controllers...Done. Type "help" for a list of supported commands. Type "exit" to close the console. > ctrl all show status Error: Expecting "controller" in target. Type "help" for more information. > ctrl all show status Smart Array P400 in Slot 1 Controller Status: OK Cache Status: Not Configured Battery Status: OK
… the CLI doesn’t parse out leading whitespaces.
Maybe somebody, someday will be battling the hpacucli CLI and google their way to this post.
P.S., For goodness sake, please don’t go running to your local HP rep to tell him I’m bashing HP. Yes, HP Smart Array is great technology. I just don’t think whitespaces are as significant as hpacucli does 🙂
Yes, I admit, sometimes little things make me crabby. Or is it that being crabby makes me easily irritated at small things? I think it’s the former rather than the latter.
Angry Email Makes Me Crabby
I am surprised at how many emails I’ve received from angry Storage Administrators about my position against Fibre Channel. I generally just ask them to not take it so personally. After all, there is more to storage than whatever storage networking protocol any particular server is plumbed with. Anyone who reads my Manly Man series about Fibre Channel will see that I’m just not a big fan of the current misapplication of Fibre Channel technology. As I point out in installment number two in the Manly Man series, Fibre Channel was originally intended to solve the difficulties associated with connecting hundreds (or thousands) of low capacity drives to a small number of large servers. These days it is being used to provide storage connectivity for hundreds of servers to a couple of huge FC storage arrays. And these hundreds of servers have both Ethernet and FC connectivity-overkill, plain and simple.
A lot is changing in the storage world where Oracle is concerned. I hope to come out of the foxhole to blog more about it over the next few months. Well, maybe I should say a few months from now. In the meantime I’ll be making some datacenter visits. I always learn something interesting when I get a chance to do that.
Little Changes Make Me Crabby
Like, for instance, the following example of the change to iostat -x output from RHEL4 to RHEL5. Notice how the rkB/s and wkB/s columns are missing? Yes, it is simple math to divide [rw]sec/s by 2 to get the same data, but there must be at least 42 of you folks that are feeding iostat -x output through some text processing, right? Time to adjust your scripts. Or is it? I wonder how many folks are really diving into OEL5/RHEL5? It seems a conservative IT shop with a functional RHEL4 configuration would most likely stay put. Any readers care to voice their experience with OEL5/RHEL5 adoption? Is it happening? Are folks waiting for their 11g adoption with plans to team that up with a move to OEL5/RHEL5?
# uname -r
2.6.18-53.ELsmp
# iostat -x 1 1
Linux 2.6.18-53.el5 (host1) 03/07/2008
avg-cpu: %user %nice %system %iowait %steal %idle
0.08 0.00 0.04 0.03 0.00 99.85
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.16 10.38 0.33 0.95 10.55 90.64 79.47 0.02 14.12 0.86 0.11
sda 0.00 0.00 0.00 0.00 0.01 0.00 21.05 0.00 3.18 2.87 0.00
dm-0 0.00 0.00 0.45 11.33 10.47 90.64 8.58 0.39 32.67 0.09 0.11
dm-1 0.00 0.00 0.00 0.00 0.01 0.00 8.00 0.00 5.05 0.51 0.00
cciss/c0d1 0.00 0.00 0.00 0.00 0.01 0.00 10.12 0.00 4.56 2.63 0.00
cciss/c0d2 0.00 0.00 0.00 0.00 0.01 0.00 10.38 0.00 4.30 2.52 0.00
cciss/c0d3 0.00 0.00 0.00 0.00 0.01 0.00 10.38 0.00 6.17 3.35 0.00
cciss/c0d4 0.00 0.00 0.00 0.00 0.01 0.00 10.38 0.00 5.50 3.22 0.00
cciss/c0d5 0.00 0.00 0.00 0.00 0.01 0.00 10.64 0.00 3.38 2.59 0.00
cciss/c0d6 0.00 0.00 0.00 0.00 0.01 0.00 10.64 0.00 4.05 3.31 0.00
cciss/c0d7 0.00 0.00 0.00 0.00 0.01 0.00 10.12 0.00 4.44 2.29 0.00
$ uname -r
2.6.9-34.ELsmp
$ iostat -x 1 1
Linux 2.6.9-34.ELsmp (host2) 03/07/2008
avg-cpu: %user %nice %sys %iowait %idle
1.79 0.00 1.28 0.30 96.63
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.06 11.83 0.14 3.21 23.07 120.39 11.53 60.19 42.77 0.68 201.33 4.60 1.54
Jonah H Harris has provided a good list of external data unloading tools that people use with Oracle. It is worth a gander at:
I first heard about Jonah’s blog via OakTable Network email from Tanel Poder. I’ve been reading Jonah’s blog and it looks like good stuff so far.
…just a quick blog entry to plug Daniel Morgan’s (of Puget Sound Oracle User Group) very useful collection of scripts and other goodies:
I’ve been reading a lot of blogosphere content about Data Warehousing these days. I’ve taken a lot of interest in such technology as Netezza, GreenPlum, DATAllegro and others and blog reading proves to be an interesting way to augment one’s knowledge. Who’d have thought I’d learn so much about OLTP through this reading.
Memory is Faster than Disk, So Let’s Do a Complete Rewrite
Why, just today I found out that it is time for a total rewrite of commercial RDBMS products. Uh huh. More interestingly, though, I learned:
I found these pearls of wisdom while reading a Stonebraker paper referred to on this blog post. Yes, I know that blog is basically a store-front for Vertica, but I like to learn about different things that are going on in database technology. Unfortunately this time I was wasting my time. The URL in that blog post points to the VLDB front page, but a little sleuthing found the paper posted here: The End of an Architectural Era (It’s Time for a Complete Rewrite).
Recite after me:
If you get two orders of magnitude performance gain, you are either not doing it or you’ve moved it closer to the processor.
Dang, and I ain’t even got no too pretty good pedigree. Pshaw, I dasn’t fidget ‘mungst the quality!
Central versus De-centralized versus Shared-Nothing
No, it isn’t time for a re-write, especially one that requires a complete shared-nothing database approach. Now don’t get me wrong, I’m all for de-coupling and grid architecture-most particularly where storage is concerned. If I hear of another poor production site that is head-saturated on a $500,000 storage array when driving a measly 15 or so 15K RPM drives, I’ll BAARF. Please see the following post for what I’m talking about:
In my post called Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!, I linked to a paper Oracle maintains that explains how to use an NFS export from a small Unix/Linux server as storage for a third voting disk in a stretch RAC cluster. I pointed out that the paper instructs on how to use the noac mount option for Linux RAC clusters in spite of the many resources that suggest actimeo=0 will do. The authors of the document are standing fast that if you are building a Linux-based RAC stretch cluster and are using an NFS mount as a third voting device you do indeed need to mount that particular NFS filesystem with noac. That nugget of truth contradicts so many different documents that I don’t care to list. Instead, I’ll list a resource from Metalink that helps clarify the issue. In fact, I would say that no matter what the sundry Installation Guides or Release Notes say, refer to Metalink 359515.1 when the topic of Oracle Database 10g on NFS filesystems comes up.
Datafiles or CRS Files? The Mount Options Differ.
Metalink 359515.1 is a really helpful note. It spells out the RAC-related mount options for 10gR2 on Solaris, AIX, HP-UX and Linux. Most importantly, it spells out the options for the datafiles and the CRS files in two separate columns. Lo and behold, Metalink 359515.1 clearly spells out that noac is needed for CRS files, but not for datafiles.
In the comment section of Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!, a reader points out that the Third Party Vote on NFS paper has a dead end URL (http://www.oracle.com/technology/deploy/availability/htdocs/vendors_nfs.html) in the section that aimed to point out the fact that you cannot just use some Unix/Linux server NFS exports for any other purpose than this unique third voting disk setup in a stretch cluster scenario. He is right, that URL is a dead end, but I’d rather point folks to Linux RTCM (RAC Technology Certification Matrix) or the Unix RTCM, both of which clearly spell out a list of supported NFS File servers. Missing from the list is, of course, some plain old Linux or Unix server dishing out NFS exports-because the only supported application of simple, Unix/Linux NFS exports is the third voting disk scenario in a stretch cluster.
The reader also added this comment:
Many people say Oracle doesn’t support NFS, we need to verify. Searching oracle.com, “We did not find any search results for: vendors_nfs.html” and the references from google all seem to point at that one mysteriously missing doc.
Gee whiz. Where to start? Yes, for the eleventeenth time, NFS filesystems are supported for Oracle Database (including RAC). Let’s not get so easily confused; NFS is a protocol and the storage is NAS. Let’s all enter the following formula in our decoder rings before reading Oracle documents:
(Some Stupid Little Linux/Unix Server Exporting Filesystems via NFS) != NAS
The only supported application of non-NAS NFS is described in the following paper: Using NFS for a Third CRS Voting Device
Now, for some light reading about Oracle on NFS with 11g, I submit:
Oracle Database 11g on NFS filesystems:
Using Network Attached Storage or NFS File Systems Installation Guide for Microsoft Windows
Using Network Attached Storage or NFS File Systems Installation Guide for HP-UX
Using Network Attached Storage or NFS File Systems Installation Guide for AIX 5L Based Systems (64-Bit)
Using Network Attached Storage or NFS File Systems Installation Guide for Solaris Operating System
Using Network Attached Storage or NFS File Systems Installation Guide for Linux
And, of course: Configuring Direct NFS Storage for Datafiles
But, let’s not forget:
The topic of “stretch clusters” has been interesting to a lot of folks for quite some time. A stretch cluster is one where one or more cluster nodes, one or more portions of the SAN or both are geographically remote. Geographically remote could be within eye-sight (1-2km) or a long distance away. YottaYotta (Robin Harris of StorageMojo.com will notice that name) reached out to me (with hardware to offer) several years ago to set up a 3500km stretch cluster with three 10gR2 RAC nodes. Two of the RAC nodes were co-located and the third was put at 3500km distance using communications hardware that simulates the latency imposed by such great distance. And, yes, it is a valid simulation. It was an interesting exercise and with the YottaYotta distributed block server, the PolyServe (HP) and RAC were totally oblivious to the topology. It was a cool project, but that technology has had a difficult time catching on. In the interim, mainstream vendors have stepped up to offer stretch clustering technology and in the name of business continuity, folks are considering these sorts of solutions-but they are expensive. To that end, most shops would tend to buy, at most, a two-legged SAN. Therein lies the problem. Such a configuration could suffer a disaster on the leg of the SAN that has the majority of the CRS voting disks resulting in a total outage of the solution.
The remedy for this problem is to implement a third leg of storage for more voting disks to ensure an n+1 majority are available, but at what cost? The solution is to implement an inexpensive NFS share in which to host these additional voting disks. And, yes, you can use a simple low end Unix/Linux host as the NFS server for this purpose-so long as the host is running Solaris, AIX or HP-UX, or Linux. The following is a link to a paper that covers Oracle’s recommended/supported approach to this solution with Oracle Database 10g Release 2.
Using NFS for a Third CRS Voting Device
The paper is clear about the fact that using some plain Unix/Linux server to host NFS shares for Oracle files is limited to this specific purpose:
Oracle does NOT support standard NFS for any files, with the one specific exception documented in this white paper.
The paper appears to have a small contradiction about mount options-specifically stating that the noac option is required for Linux (see Figure 1) servers which seems to contradict Metalink 279393.1. I’ve sent an email to the authors about that. We’ll see if it changes.
In my recent rant about Oracle database installation difficulties, I provided a link to a video in which fellow OakTable Network members Morten Egan and Mogens Norgaard captured how difficult the task really is.
Well, they’re at it again. You’ll see Morten “The Nose” Egan start out this new video taped Oracle installation by configuring a SAN with what looks like the HP Array Configuration Utility, but then my eyes are getting as bad as my blogging frequency. I couldn’t miss the Windows Disk Manager though-not even on fast forward.
I think we should start calling him Morten “The Hair” Egan. The link to the video follows:
I thought a comment on one of my recent blog entries deserved handling in a blog entry. A reader posted:
Have you done any comparisons of the HP DL585 with an HP DL580? Is the DL580 a NUMA machine? Which one would you by today for a RAC cluster?
I’ll answer these out of order. The DL580 is not a NUMA system. Although it stands to reason that if HP continues the DL580 product line into the future to the point where they bake in the CSI interconnect then at that time the DL580 would be a NUMA system. So, the short answer to whether or not a DL580 is a NUMA system is no, it is not. I think long answers are more fun.
In my series of posts about Oracle on NUMA, I think I must have said it about umpteen times, but I’ll say it again concisely in this post. I’m talking about what “NUMA-aware” software means. I routinely hear that Oracle is NUMA-aware. It is, and it isn’t. The reason I say this is because there are widely varying degrees of NUMA-awareness that varies between hardware platforms and Oracle ports. I made the point in my recent post about Oracle Database 10g 10.2.0.4 that 10.2.0.4 contains NUMA-related fixes, and it does. However, that isn’t saying it is the fullness of NUMA-aware, because it isn’t. However, the only question that matters is whether it is sufficiently NUMA-aware for today’s NUMA systems, and I’d have to say that it is.
I’ll give a hint: No Linux Oracle release can be fully NUMA-aware until processes (e.g., shadow processes, PQO slaves, etc) can quickly and cheaply detect what CPU they are currently executing on and prefer memory resources based on that locality. Way back in 1996 I was in Advanced Oracle Engineering at Sequent and we were in the late stages of producing the first commercial NUMA system. It was my early Oracle work on Sequent’s NUMA that begat the GETENGNO(3SEQ) API, which was an extremely inexpensive call for processes to check what CPU they are executing on.
Let’s fast forward to today. The Linux development folks are considering the Linux corollary for Sequent’s GETENGNO() with the vgetcpu() call. The problem is that the call is very, very slow compared to the 4-6 cycles that Sequent required to inform a process what CPU it was executing on. Nonetheless, the point is that until vgetcpu() works, and Oracle exploits it, the pinnacle of NUMA-awareness has not been met. And while that may not matter given today’s AMD situation, it will certainly matter when Intel system are NUMA (e.g., CSI based). I guess I shouldn’t equate Linux NUMA with AMD since IBM’s x3950 is a building block for large NUMA systems, and there are others as well. But I was focusing on commodity-level NUMA systems which the x3950 most certainly is not.
There are a lot of factors in selecting hardware, but since I’m asked about DL585 vs DL580, I’d say DL580-so long as it is a DL580 G5. I have tested DL585 and DL580 side by side. However, that was a pretty old DL580 G3 (1066 MHz FSB). I see that the DL580 is now fit with the “Penryn” Xeons (e.g., 5460), which have a front-side bus speed of 1333MHz. There are G5’s that are fit with “Tigerton” Xeons which are 1066 MHz FSB. I’ve seen benchmark results that suggest there is some 21% to gain from going with a 5460-based G5 over a 7350-based G5. So, look closely at the specification. Also, I think a shrewd shopper would try to read the crystal ball to see when the DL580 G5 will be fit with the Xeon 5462 which has a 1600 MHz FSB. As always, with Oracle you want big pipes.
I find myself in the most interesting conversations. You know us propeller heads; we’re always talking about speeds and feeds. I find myself surprised quite often how many Oracle-minded folks give any credence to CPU frequency. I go beyond surprised and fall off my chair any time the same folks give any credence to Linux bogomips values.
They’re called bogomips for a reason.
Here’s a secret formula when running x86 instructions:
Bogomips = cpu_MHz * 2.0012069226
Yes, the formula for PowerPC and Itanium is likely different, but don’t point PPC/Itanium Linux users out in a crowded theater because it might have the same effect as yelling, “fire.” Ok, all kidding aside, I don’t have any PowerPC stuff. I wish I did, but I don’t.
That could be the end of this blog post. I’ve said enough. Bogomips is 2 times the clock rate. What a boring blog entry.
…and nothing else matters. Quite some time back-when I had an occasional 5 minutes to make a blog entry-I had a series of blog posts about Oracle on NUMA. In that series I offered up a microbenchmark called the Silly Little Benchmark. That post contains a URL to the kit where it sits on an OakTable Network server. I’m using a slightly modified version at this point now. I need to update the version that is there, but since it has only been downloaded 440 times it might not be of interest. Nonetheless, I’m tying it in to this thread.
The point is that a workload like SLB will show you a great deal more about what Oracle will behave like on a system that does anything related to CPU frequency, and that includes bogomips for certain.
This is the first in a short series aimed at pointing out just how bogo bogomips are. In the meantime, I recommend taking a peek at the output of dmidecode(8) on your system and make sure you have symmetrical memory populating the DIMM slots on your system. An oddball configuration can cause performance degradation.
I’m at the end of my ropes with how difficult it is to install Oracle and I’m just now getting brave enough to discuss it. You see, I assumed that at some point Microsoft and IBM would have been able to make a solid case for the treachery that is installing Oracle. It is so bad that I’m surprised I haven’t wound up in a straight jacket after dealing with this insanity as much as I have.
Finally, fellow OakTable Network members at MiracleAS have taken the time to document this fact in a video. If you’ve experienced the frustration, or are perhaps a Technical Marketing type from Microsoft or IBM, I encourage you to view the video at the following link:
Click It
Kevin Closson's Blog: Platforms, Databases and Storage 
Recent Comments