By The Way, How Many NUMA Nodes Is Your AMD Opteron 6100-Based Server?
In my on-going series about Oracle Database 11g configuration for NUMA systems I’ve spoken of the enabling parameter and how it changed from _enable_NUMA_optimization (11.1) to _enable_NUMA_support (11.2). For convenience sake I’ll point to the other two posts in the series for folks that care to catch up.
- You Buy a NUMA System, Oracle Says Disable NUMA! What Gives? Part II.
- You Buy a NUMA System, Oracle Says Disable NUMA! What Gives? Part I.
What does AMD Opteron 6100 (Magny-Cours) have to do with my on-going series on enabling/disabling NUMA features in Oracle Database? That’s a good question. However, wouldn’t it be premature to just presume each of these 12-core processors is a NUMA node?
The AMD Opteron 6100 is a Multi-Chip Module (MCM). The “package” is two hex-core processors essentially “glued” together and placed into a socket. Each die has its own memory controller (hint, hint). I wonder what the Operating System sees in the case of a 4-socket server? Let’s take a peek.
The following is output from the numactl(8) command on a 4s48c Opteron 6100 (G34)-based server:
# numactl --hardware available: 8 nodes (0-7) node 0 size: 8060 MB node 0 free: 7152 MB node 1 size: 16160 MB node 1 free: 16007 MB node 2 size: 8080 MB node 2 free: 8052 MB node 3 size: 16160 MB node 3 free: 15512 MB node 4 size: 8080 MB node 4 free: 8063 MB node 5 size: 16160 MB node 5 free: 15974 MB node 6 size: 8080 MB node 6 free: 8051 MB node 7 size: 16160 MB node 7 free: 15519 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 16 16 22 16 22 16 22 1: 16 10 22 16 16 22 22 16 2: 16 22 10 16 16 16 16 16 3: 22 16 16 10 16 16 22 22 4: 16 16 16 16 10 16 16 22 5: 22 22 16 16 16 10 22 16 6: 16 22 16 22 16 22 10 16 7: 22 16 16 22 22 16 16 10
Heft
It wasn’t that long ago that an 8-node NUMA system was so large that a fork lift was necessary to move it about (think Sequent, SGI, DG, DEC etc). Even much more recent 8-socket (thus 8 NUMA nodes) servers were a 2-man lift and quite large (e.g., 7U HP Proliant DL785). These days, however, an 8-node NUMA system like the AMD Opteron 6100 (G34) comes in a 2U package!
Is it time yet to stop thinking that NUMA is niche technology?
I’ll blog soon about booting Oracle to test NUMA optimizations on these 8-node servers.
Kevin Closson's Blog: Platforms, Databases and Storage 
0 Responses to “You Buy a NUMA System, Oracle Says Disable NUMA! What Gives? Part III.”