Fun With Intel Xeon 5500 Nehalem and Linux cpuspeed(8) Part I.

Intel Xeon 5500 (Nehalem) CPUs–Fast, Slow, Fast, Slow. CPU Throttling Is A Good Thing. Really, It Is!
I’m still blogging about Xeon 5500 (Nehalem) SMT (Symmetric Multi-Threading), but this installment is a slight variation from Part I and Part II. This installment has to do with cpuspeed(8). Please be aware that this post is part one in a series.

One of the systems I’m working on enables cpu throttling via cpuspeed:


# chkconfig --list cpuspeed
cpuspeed        0:off   1:on    2:on    3:on    4:on    5:on    6:off

So I hacked out this quick and dirty script to give me single-line output showing each CPU (see this post about Linux OS CPU to CPU thread mapping with Xeon 5500) and its current clock rate. The script is called howfast.sh and its listing follows:

#!/bin/bash
egrep 'processor|MHz' /proc/cpuinfo | sed 's/^.*: //g' | xargs echo

The following is an example of the output. It shows that currently all 16 processor threads are clocked at 1600 MHz. That’s ok with me because nothing is executing that requires “the heat.”

# ./howfast.sh
0 1600.000 1 1600.000 2 1600.000 3 1600.000 4 1600.000 5 1600.000 6 1600.000 7 1600.000 8 1600.000 9 1600.000 10 1600.000 11 1600.000 12 1600.000 13 1600.000 14 1600.000 15 1600.000

So the question becomes just what does it take to heat these processors up? Let’s take a peek…

Earth To CPU: Hello, Time To Get a Move On
The following script, called busy.sh, is simple. It runs a sub-shell on named processors looping the shell “:” built-in. But don’t confuse “:” with “#.” I’ve seen people use “:” as a comment marker…bad, bad (or at least it use to be when people cared about systems). Anyway, back to the train of thought. Here is the busy.sh script:

#!/bin/bash

function busy() {
local SECS=$1
local WHICH_CPU=$2
local brickwall=0
local x=0

taskset -pc $WHICH_CPU $$ > /dev/null 2>&1
x=$SECONDS
(( brickwall = $x + $SECS ))

until [ $SECONDS -ge $brickwall ]
do
    :
done
}
#--------------
SECS=$1
CPU_STRING="$2"
#(mpstat -P ALL $SECS 1 > mpstat.out 2>&1 )&

for CPU in `echo $CPU_STRING`
do
    ( busy $SECS "$CPU" ) &
done
wait

Let’s see what happens when I execute busy.sh to hammer all 16 processor threads. I’ll first use howfast.sh to get a current reading. I’ll then set busy.sh in motion to whack on all 16 processors after which I immediately check what howfast.sh has to say about them.

#  howfast.sh;sh ./busy.sh 30 '0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15';howfast.sh
0 1600.000 1 1600.000 2 1600.000 3 1600.000 4 1600.000 5 1600.000 6 1600.000 7 1600.000 8 1600.000 9 1600.000 10 1600.000 11 1600.000 12 1600.000 13 1600.000 14 1600.000 15 1600.000
0 2934.000 1 2934.000 2 2934.000 3 2934.000 4 2934.000 5 2934.000 6 2934.000 7 2934.000 8 2934.000 9 2934.000 10 2934.000 11 2934.000 12 2934.000 13 2934.000 14 2934.000 15 2934.000

Boring Your Readers To Death For Fun, Not Profit
Wow, this is such an interesting blog post isn’t it? You’re wondering why I’ve wasted your time, right?

Let’s allow the processors to cool down again and take a slightly different look. In fact, perhaps I should run a multiple command sequence where I start with 120 seconds of sleep followed by howfast.sh and then busy.sh. But, instead of busy.sh targeting all processors, I’ll run it only on OS processor 0. I’ll follow that up immediately with a check of the clock rates using howfast.sh:

# sleep 120; howfast.sh;sh ./busy.sh 30 0;howfast.sh
0 1600.000 1 1600.000 2 1600.000 3 1600.000 4 1600.000 5 1600.000 6 1600.000 7 1600.000 8 1600.000 9 1600.000 10 1600.000 11 1600.000 12 1600.000 13 1600.000 14 1600.000 15 1600.000
0 1600.000 1 2934.000 2 1600.000 3 2934.000 4 1600.000 5 2934.000 6 1600.000 7 2934.000 8 1600.000 9 2934.000 10 1600.000 11 2934.000 12 1600.000 13 2934.000 14 1600.000 15 2934.000

Huh? I stress processor 0 but processors 1,3,5,7,9,11,13 and 15 heat up? That’s weird, and I don’t understand it, but I’ll be investigating.  I wonder what other interesting findings lurk? What happens if I stress processor 1? I think I should start putting date commands in there too. Let’s see what happens:

# ./howfast.sh;date;./busy.sh 30 1;date;./howfast.sh
0 1600.000 1 1600.000 2 1600.000 3 1600.000 4 1600.000 5 1600.000 6 1600.000 7 1600.000 8 1600.000 9 1600.000 10 1600.000 11 1600.000 12 1600.000 13 1600.000 14 1600.000 15 1600.000
Fri May  1 14:30:57 PDT 2009
Fri May  1 14:31:27 PDT 2009
0 1600.000 1 2934.000 2 1600.000 3 2934.000 4 1600.000 5 2934.000 6 1600.000 7 2934.000 8 1600.000 9 2934.000 10 1600.000 11 2934.000 12 1600.000 13 2934.000 14 1600.000 15 2934.000
#

Ok, that too is odd. I stress thread 0 in either core 0 or 1 of socket 0 and I get OS processors 1,3,5,7,9,11,13 and 15 heated up. I wonder what would happen if I hammer on the primary thread of all the cores of socket 0? Let’s see:

# ./howfast.sh;date;./busy.sh 30 '0 1 2 3';date;./howfast.sh
0 1600.000 1 1600.000 2 1600.000 3 1600.000 4 1600.000 5 1600.000 6 1600.000 7 1600.000 8 1600.000 9 1600.000 10 1600.000 11 1600.000 12 1600.000 13 1600.000 14 1600.000 15 1600.000
Fri May  1 14:40:51 PDT 2009
Fri May  1 14:41:21 PDT 2009
0 2934.000 1 2934.000 2 2934.000 3 2934.000 4 2934.000 5 2934.000 6 2934.000 7 2934.000 8 2934.000 9 2934.000 10 2934.000 11 2934.000 12 2934.000 13 2934.000 14 2934.000 15 2934.000

Hmmm. Hurting cores 0 and 1 wasn’t enough to unleash the dogs but hammering all the cores in that socket proved sufficient. Of course it seems odd to me that it would heat up all threads in all cores on both sockets. But this is a blog entry of observations only at this point. I’ll post more about this soon.

Would it surprise anyone if I got the same result from beating on the primary thread of all 4 cores in socket 1? It shouldn’t:

# sleep 120;./howfast.sh;date;./busy.sh 30 '4 5 6 7';date;./howfast.sh
0 1600.000 1 1600.000 2 1600.000 3 1600.000 4 1600.000 5 1600.000 6 1600.000 7 1600.000 8 1600.000 9 1600.000 10 1600.000 11 1600.000 12 1600.000 13 1600.000 14 1600.000 15 1600.000
Fri May  1 14:44:33 PDT 2009
Fri May  1 14:45:03 PDT 2009
0 2934.000 1 2934.000 2 2934.000 3 2934.000 4 2934.000 5 2934.000 6 2934.000 7 2934.000 8 2934.000 9 2934.000 10 2934.000 11 2934.000 12 2934.000 13 2934.000 14 2934.000 15 2934.000

Consider this installment number one in this series…

And, before I forget, I nearly made it through this post without mentioning NUMA. These tests were run with NUMA disabled. Can anyone guess why that matters?

1 Response to “Fun With Intel Xeon 5500 Nehalem and Linux cpuspeed(8) Part I.”


  1. 1 wd September 24, 2009 at 7:30 pm

    i’ve just bought a nehalem, im interested which cpu type should i choose in the kernel config to get the best performace.
    thanks
    wd


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




DISCLAIMER

I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,987 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: