I thought a comment on one of my recent blog entries deserved handling in a blog entry. A reader posted:
Have you done any comparisons of the HP DL585 with an HP DL580? Is the DL580 a NUMA machine? Which one would you by today for a RAC cluster?
I’ll answer these out of order. The DL580 is not a NUMA system. Although it stands to reason that if HP continues the DL580 product line into the future to the point where they bake in the CSI interconnect then at that time the DL580 would be a NUMA system. So, the short answer to whether or not a DL580 is a NUMA system is no, it is not. I think long answers are more fun.
In my series of posts about Oracle on NUMA, I think I must have said it about umpteen times, but I’ll say it again concisely in this post. I’m talking about what “NUMA-aware” software means. I routinely hear that Oracle is NUMA-aware. It is, and it isn’t. The reason I say this is because there are widely varying degrees of NUMA-awareness that varies between hardware platforms and Oracle ports. I made the point in my recent post about Oracle Database 10g 10.2.0.4 that 10.2.0.4 contains NUMA-related fixes, and it does. However, that isn’t saying it is the fullness of NUMA-aware, because it isn’t. However, the only question that matters is whether it is sufficiently NUMA-aware for today’s NUMA systems, and I’d have to say that it is.
I’ll give a hint: No Linux Oracle release can be fully NUMA-aware until processes (e.g., shadow processes, PQO slaves, etc) can quickly and cheaply detect what CPU they are currently executing on and prefer memory resources based on that locality. Way back in 1996 I was in Advanced Oracle Engineering at Sequent and we were in the late stages of producing the first commercial NUMA system. It was my early Oracle work on Sequent’s NUMA that begat the GETENGNO(3SEQ) API, which was an extremely inexpensive call for processes to check what CPU they are executing on.
Let’s fast forward to today. The Linux development folks are considering the Linux corollary for Sequent’s GETENGNO() with the vgetcpu() call. The problem is that the call is very, very slow compared to the 4-6 cycles that Sequent required to inform a process what CPU it was executing on. Nonetheless, the point is that until vgetcpu() works, and Oracle exploits it, the pinnacle of NUMA-awareness has not been met. And while that may not matter given today’s AMD situation, it will certainly matter when Intel system are NUMA (e.g., CSI based). I guess I shouldn’t equate Linux NUMA with AMD since IBM’s x3950 is a building block for large NUMA systems, and there are others as well. But I was focusing on commodity-level NUMA systems which the x3950 most certainly is not.
There are a lot of factors in selecting hardware, but since I’m asked about DL585 vs DL580, I’d say DL580-so long as it is a DL580 G5. I have tested DL585 and DL580 side by side. However, that was a pretty old DL580 G3 (1066 MHz FSB). I see that the DL580 is now fit with the “Penryn” Xeons (e.g., 5460), which have a front-side bus speed of 1333MHz. There are G5’s that are fit with “Tigerton” Xeons which are 1066 MHz FSB. I’ve seen benchmark results that suggest there is some 21% to gain from going with a 5460-based G5 over a 7350-based G5. So, look closely at the specification. Also, I think a shrewd shopper would try to read the crystal ball to see when the DL580 G5 will be fit with the Xeon 5462 which has a 1600 MHz FSB. As always, with Oracle you want big pipes.
Kevin Closson's Blog: Platforms, Databases and Storage 
Hi Kevin,
I’ve written the most comprehensive document about CSI (that is accessible without an NDA):
http://realworldtech.com/page.cfm?ArticleID=RWT082807020032
One of the things that I would point out is that not only does “NUMA support” vary in terms of ability to exploit locality at the application level, but the “NUMA-ness” of systems varies quite substantially.
For instance, AMD’s 2 socket systems are only moderately NUMA. Yes, there are resources which are non-local, but all non-local resources cost the same (1 hop over ccHT).
AMD’s current 4 socket systems are significantly worse, since the worst case is 2 hops, which means that a full memory transfer could take 4+ hops versus 2.
In contrast, Intel’s 4S CSI systems (which won’t come out till 2009 anyway) are fully connected, so everything is at most one hop away.
In future systems, AMD will also go to a fully connected architecture.
Note that this analysis is strictly for memory related transactions. I/O may still be slightly asymmetrical.
In other words, there is room for software to improve, and there is also room for the hardware to improve.
David Kanter
Real World Tech
The DL580 G5s are only available with the quad-core processors and from the stand point of performance per box they make a lot of sense,but the quad-core processors will double the oracle license cost. Do they really have double the speed of a DL585 G2? Which box do you think will provide the best performance per license?
Also the DL585 only has four sockets. The sun x4600 has eight sockets and, if you only get four sockets populated, are very close in price to the DL585s. For a RAC cluster that starts with 2 or 3 nodes is it better to get machines that will scale vertically before you have to add another node or to go ahead and plan to scale horizontally by adding additional nodes?
David,
Your points on NUMA-ness are spot-on. You’d see you are preaching to the choir if you read my NUMA posts. By the way, you mentioned 1-hop 2 socket and multi-hop 4 socket, but you left out that sweet spot that nobody seems to talk about: the IBM x3755 which is a 3 socket 1-hop with the pass-through module…very sweet packaging.
Reece,
I’ve never heard of anyone, nor seen any suggestion that the Sun 4600 scales with Oracle. I’d but good money it doesn’t.
You questions about DL585 G2 versus DL580 G5 vis a vis Oracle licensing would be a very odd place for me to go since I work for Oracle. HP are the folks with the product overlap…I’d talk to them.
Kevin
Off topic for this post, but what is your take on “Oracle Storage Guy” and this post in particular: http://oraclestorageguy.typepad.com/oraclestorageguy/2008/01/dnfs-performanc.html?
Regards Nigel
Nigel,
You’re right, it is off topic. Nonetheless, the URL you point to slides into a bucket of rubbish. That is my technical opinion.
Guys – very interesting. We are trying to move up from HP DL760G2 32-bit Xeon 3.0 servers to HP RX8640 (NUMA) 64-bit Itanium servers. We run Oracle on Win-2003.
The 32-bit scream and the 64-bit are twice as slow on the writes. We are on 10.2.0.3 anxiously awaiting 10.2.0.4.
Is this NUMA related? words of wisdom?
Move to Sun’s 8-socket Opteron x4600 server or their SPARC platform. Itanium sucks wind