Question: How to Choose From the Last of the Non-NUMA Xeon-based Servers

I thought a comment on one of my recent blog entries deserved handling in a blog entry. A reader posted:

Have you done any comparisons of the HP DL585 with an HP DL580? Is the DL580 a NUMA machine? Which one would you by today for a RAC cluster?

I’ll answer these out of order. The DL580 is not a NUMA system. Although it stands to reason that if HP continues the DL580 product line into the future to the point where they bake in the CSI interconnect then at that time the DL580 would be a NUMA system. So, the short answer to whether or not a DL580 is a NUMA system is no, it is not. I think long answers are more fun.

In my series of posts about Oracle on NUMA, I think I must have said it about umpteen times, but I’ll say it again concisely in this post. I’m talking about what “NUMA-aware” software means. I routinely hear that Oracle is NUMA-aware. It is, and it isn’t. The reason I say this is because there are widely varying degrees of NUMA-awareness that varies between hardware platforms and Oracle ports. I made the point in my recent post about Oracle Database 10g 10.2.0.4 that 10.2.0.4 contains NUMA-related fixes, and it does. However, that isn’t saying it is the fullness of NUMA-aware, because it isn’t. However, the only question that matters is whether it is sufficiently NUMA-aware for today’s NUMA systems, and I’d have to say that it is.

I’ll give a hint: No Linux Oracle release can be fully NUMA-aware until processes (e.g., shadow processes, PQO slaves, etc) can quickly and cheaply detect what CPU they are currently executing on and prefer memory resources based on that locality. Way back in 1996 I was in Advanced Oracle Engineering at Sequent and we were in the late stages of producing the first commercial NUMA system. It was my early Oracle work on Sequent’s NUMA that begat the GETENGNO(3SEQ) API, which was an extremely inexpensive call for processes to check what CPU they are executing on.

Let’s fast forward to today. The Linux development folks are considering the Linux corollary for Sequent’s GETENGNO() with the vgetcpu() call. The problem is that the call is very, very slow compared to the 4-6 cycles that Sequent required to inform a process what CPU it was executing on. Nonetheless, the point is that until vgetcpu() works, and Oracle exploits it, the pinnacle of NUMA-awareness has not been met. And while that may not matter given today’s AMD situation, it will certainly matter when Intel system are NUMA (e.g., CSI based). I guess I shouldn’t equate Linux NUMA with AMD since IBM’s x3950 is a building block for large NUMA systems, and there are others as well. But I was focusing on commodity-level NUMA systems which the x3950 most certainly is not.

There are a lot of factors in selecting hardware, but since I’m asked about DL585 vs DL580, I’d say DL580-so long as it is a DL580 G5. I have tested DL585 and DL580 side by side. However, that was a pretty old DL580 G3 (1066 MHz FSB). I see that the DL580 is now fit with the “Penryn” Xeons (e.g., 5460), which have a front-side bus speed of 1333MHz. There are G5’s that are fit with “Tigerton” Xeons which are 1066 MHz FSB. I’ve seen benchmark results that suggest there is some 21% to gain from going with a 5460-based G5 over a 7350-based G5. So, look closely at the specification. Also, I think a shrewd shopper would try to read the crystal ball to see when the DL580 G5 will be fit with the Xeon 5462 which has a 1600 MHz FSB. As always, with Oracle you want big pipes.

9 Responses to “Question: How to Choose From the Last of the Non-NUMA Xeon-based Servers”


  1. 1 David Kanter January 10, 2008 at 9:41 pm

    Hi Kevin,

    I’ve written the most comprehensive document about CSI (that is accessible without an NDA):

    http://realworldtech.com/page.cfm?ArticleID=RWT082807020032

    One of the things that I would point out is that not only does “NUMA support” vary in terms of ability to exploit locality at the application level, but the “NUMA-ness” of systems varies quite substantially.

    For instance, AMD’s 2 socket systems are only moderately NUMA. Yes, there are resources which are non-local, but all non-local resources cost the same (1 hop over ccHT).

    AMD’s current 4 socket systems are significantly worse, since the worst case is 2 hops, which means that a full memory transfer could take 4+ hops versus 2.

    In contrast, Intel’s 4S CSI systems (which won’t come out till 2009 anyway) are fully connected, so everything is at most one hop away.

    In future systems, AMD will also go to a fully connected architecture.

    Note that this analysis is strictly for memory related transactions. I/O may still be slightly asymmetrical.

    In other words, there is room for software to improve, and there is also room for the hardware to improve.

    David Kanter
    Real World Tech

  2. 2 Reece Dike January 10, 2008 at 10:55 pm

    The DL580 G5s are only available with the quad-core processors and from the stand point of performance per box they make a lot of sense,but the quad-core processors will double the oracle license cost. Do they really have double the speed of a DL585 G2? Which box do you think will provide the best performance per license?

    Also the DL585 only has four sockets. The sun x4600 has eight sockets and, if you only get four sockets populated, are very close in price to the DL585s. For a RAC cluster that starts with 2 or 3 nodes is it better to get machines that will scale vertically before you have to add another node or to go ahead and plan to scale horizontally by adding additional nodes?

  3. 3 kevinclosson January 11, 2008 at 12:32 am

    David,

    Your points on NUMA-ness are spot-on. You’d see you are preaching to the choir if you read my NUMA posts. By the way, you mentioned 1-hop 2 socket and multi-hop 4 socket, but you left out that sweet spot that nobody seems to talk about: the IBM x3755 which is a 3 socket 1-hop with the pass-through module…very sweet packaging.

  4. 4 kevinclosson January 11, 2008 at 12:36 am

    Reece,

    I’ve never heard of anyone, nor seen any suggestion that the Sun 4600 scales with Oracle. I’d but good money it doesn’t.

    You questions about DL585 G2 versus DL580 G5 vis a vis Oracle licensing would be a very odd place for me to go since I work for Oracle. HP are the folks with the product overlap…I’d talk to them.

  5. 5 Nigel Thomas January 15, 2008 at 7:56 pm

    Kevin

    Off topic for this post, but what is your take on “Oracle Storage Guy” and this post in particular: http://oraclestorageguy.typepad.com/oraclestorageguy/2008/01/dnfs-performanc.html?

    Regards Nigel

  6. 6 kevinclosson January 21, 2008 at 7:59 pm

    Nigel,

    You’re right, it is off topic. Nonetheless, the URL you point to slides into a bucket of rubbish. That is my technical opinion.

  7. 7 Udi Karni February 21, 2008 at 6:54 pm

    Guys – very interesting. We are trying to move up from HP DL760G2 32-bit Xeon 3.0 servers to HP RX8640 (NUMA) 64-bit Itanium servers. We run Oracle on Win-2003.

    The 32-bit scream and the 64-bit are twice as slow on the writes. We are on 10.2.0.3 anxiously awaiting 10.2.0.4.

    Is this NUMA related? words of wisdom?

  8. 8 Phil February 22, 2008 at 8:02 am

    Move to Sun’s 8-socket Opteron x4600 server or their SPARC platform. Itanium sucks wind


  1. 1 8 Useful Technical Posts … - Sinoptix Trackback on February 2, 2008 at 6:49 am

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




EMC Employee Disclaimer

The opinions and interests expressed on EMC employee blogs are the employees' own and do not necessarily represent EMC's positions, strategies or views. EMC makes no representation or warranties about employee blogs or the accuracy or reliability of such blogs. When you access employee blogs, even though they may contain the EMC logo and content regarding EMC products and services, employee blogs are independent of EMC and EMC does not control their content or operation. In addition, a link to a blog does not mean that EMC endorses that blog or has responsibility for its content or use.

This disclaimer was put into place on March 23, 2011.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,058 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2013. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

Follow

Get every new post delivered to your Inbox.

Join 2,058 other followers

%d bloggers like this: