I’ve read through the full disclosure report from Oracle’s January 2012 TPC-C. I’ve found that the result was obtained without using any NUMA init.ora parameters (e.g., enable_NUMA_support). The storage was a collection of Sun x64 servers running COMSTAR to serve up F5100 flash storage. The storage connectivity was 8GFC fibre channel. This was a non-RAC result with 8s80c160t Xeon E7. The only things that stand out to me are:
- The settings of disk_async_io=TRUE. This was ASM on raw disk so I should think ASYNC would be the default. Interesting.
- Overriding the default number of DBWR processes by setting db_writer_processes. The default number of DBWR processes would be 20 so the benchmark team increased that 60%. Since sockets are NUMA “nodes” on this architecture the default of 20 would render 2.5 DBWR per “node.” In my experience it is beneficial to have DBWR processes an equal multiple of the number of sockets (NUMA nodes) so if the benchmark team was thinking the way I think they went with 4x socket count.
For more information about the missing enable_NUMA_support parameter see: Meet _enable_NUMA_support: The if-then-else Oracle Database 11g Release 2 Initialization Parameter.
For a lot more about NUMA as it pertains to Oracle, please visit: QPI-Based Systems Related Topics (e.g., Nehalem EP/EX, Westmere EP, etc)
On the topic of increasing DBWR processes I’d like to point out that doing so isn’t one of those “some is good so more must be better” situations. For more reading on that matter I recommend:
Got A Big NUMA Box For Running Oracle? Take Care To Get Interrupt Handling Spread Across The Sockets Evenly
Page 310 of the FDR shows the following script used to arrange good affinity between the FC HBA device drivers and the sockets. I had to do the same sort of thing with the x4800 (aka Exadata X2-8) back before I left Oracle’s Exadata development organization. This sort of thing is standard but I wanted to bring the concept to your attention:
#!/bin/bash service irqbalance stop last_node=-1 declare -i count=0 declare -i cpu cpu1 cpu2 cpu3 cpu4 for dir in /sys/bus/pci/drivers/qla2xxx/0000* do node=`cat $dir/numa_node` irqs=`cat $dir/msi_irqs` if [ "`echo $irqs | wc -w`" != "2" ] ; then echo >&2 "script expects 2 interrupts per device" exit 1 fi first_cpu=`sed 's/-.*//' < $dir/local_cpulist` echo $node $irqs $first_cpu $dir done | sort | while read node irq1 irq2 cpu1 dir do cpu2=$cpu1+10 cpu3=$cpu1+80 cpu4=$cpu1+90 if [ "$node" != "$last_node" ] then count=1 cpu=$cpu1 else count=$count+1 case $count in 2) cpu=$cpu2;; 3) cpu=$cpu3;; 4) cpu=$cpu4;; *) echo "more devices than expected on node $node" count=1 cpu=$cpu1;; esac fi last_node=$node echo "#$dir" echo "echo $cpu > /proc/irq/$irq1/smp_affinity_list" echo "echo $cpu > /proc/irq/$irq2/smp_affinity_list" echo echo $cpu > /proc/irq/$irq1/smp_affinity_list echo $cpu > /proc/irq/$irq2/smp_affinity_list done