Using Linux Processor Affinity To Beat The Oracle Licensing Police.

In my recent blog entry about LGWR processing I set up a proof case that included establishing hard processor affinity for LGWR and other processors in order to prove a point. I’ve received a few emails about what tool I used to affinity running processes to specific CPUs. The answer is not what you might presume. I did not use the taskset(1) command that ships with 2.6 kernel distributions because I’ve seen some irregularities with that tool in the past. I prefer to use a bit of my own code that uses sched_setaffinity(2). I’d provide that code as a reference but it is a mitt of a mess and does a lot of other stuff that doesn’t pertain to affinity.

The recent Oracle single socket Xeon “Cloverdale” TPC-C used hard affinity for various purposes. The FDR pointed to by that URL has a hard affinity program (affinity.c) listing that is very succinct. If you want to give hard affinity a play I recommend you cut and paste from that FDR, compile and give it a go. For instance, presume you have compiled affinity.c and wish to round up all the processes for a specific $ORACLE_SID and bind them to CPU 0 you could do something like the following. Note, unless the oracle user has CAP_SYS_NICE capability, you’ll have to make the affinity program setuid root in order to play with sched_setaffinity(2). Such as:

$ cc -o aff affinity.c
$ sudo chmod 6755 aff
$ TARGET_SID=PROD
$ ps -ef | grep ${TARGET_SID} | grep -v grep | awk '{ print $2 }' | while read PID
do
	./aff $PID 1
done

Oracle Licensing Again
I like nixCraft as a resource, but not when it comes to giving advice about Oracle licensing. What’s that got to do with hard affinity? I aimed to offer you readers more information about Linux hard affinity (note to self, blog sometime about the difference between hard and soft affinity if a reader ever asks) so I found this nixCraft article. It does offer more information about hard affinity scheduling, but oh so much more as well. In that page, the author is quoting an unspecified source as follows (quoted herein verbatim):

The scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications. For example, application such as Oracle (ERP apps) use # of cpus per instance licensed. You can bound Oracle to specific CPU to avoid license problem. This is a really useful on large server having 4 or 8 CPUS

This quote is talking about how modern *nix schedulers implement soft-affinity (aka cache affinity) and thus handcrafting hard affinity is something better left to advanced situations. That’s fine and dandy, but pardon me for a moment because I’m choking on a hairball over that bit about ERP licensing vis a vis CPU count. That is a complete farce. Please do not try to negotiate a 2 CPU license on an 8 core system where you plan to pin Oracle E-Business Suite processes to the first 2 cores of the system using sched_setaffinity(2). I know I’ve provided a quote of a quote, but that idea is about as brain-dead as they come. It would work, but in a license audit it would not fly.

More About Hard Affinity
Be aware that if you run 100% of a given workload under hard affinity to CPU 0, for instance, that is not the same as running on a single CPU box. The other processors are still online and servicing device interrupts. For instance, if LGWR is running on CPU 0, the hardware interrupt for any given log file parallel write can be serviced by any of the online CPUs in the box. If, for instance, LGWR is pinned to CPU 0 and a LGWR flush I/O interrupt is handled by, say, CPU 3, you are getting both a blessing and a curse. The blessing is that LGWR’s CPU didn’t have to handle the noise of the interrupt. The curse is that your workload is getting “help” from CPUs you didn’t really want in the mix-that is if you are hoping for total isolation of the workload to CPU 0. This is not the same as sophisticated OS partitioning offerings out there that corral event he hardware interrupts to the CPUs in the partition. That is an entirely different topic.

Summary
No, binding a workload to CPUs with sched_setaffinity(2) is not the same as running on the same number of discrete processors. Whoever authored the words “You can bound Oracle to specific CPU to avoid license problem” is likely neither an English professor nor a contract lawyer. I’d ignore that “advice.” I wonder what “problem” the author was referring to? Is legal use of a software product a “problem?” All that aside, hard affinity can play an important role in affording ample processor bandwidth to key Oracle background processes. There were musings in the comment thread on my LGWR processing post about dedicating an entire CPU to LGWR although I have not followed up, that is certainly not the approach. I’ll need to blog about a possible best practice in this concept area soon.

13 Responses to “Using Linux Processor Affinity To Beat The Oracle Licensing Police.”


  1. 1 Jeremy Schneider August 9, 2007 at 7:27 pm

    “note to self, blog sometime about the difference between hard and soft affinity if a reader ever asks”

    Any chance you’d be up for explaining the difference between hard and soft affinity? I for one would be very interested. 🙂

  2. 2 kevinclosson August 9, 2007 at 7:35 pm

    Touche, Jeremy 🙂 Looks like note to self is in pen now.

  3. 3 Alex August 10, 2007 at 6:38 am

    This is an interesting topic.

    If you run Oracle in a Solaris Zone, limited to (say) 2 CPUs then this DOES count as 2 CPUs as far as Oracle licensing is concerned – despite as you point out that you may well have other CPUs (particularly in the case of Solaris Zones) doing useful work eg servicing IO, network, etc.

    This is confirmed in this Sun paper here,

    http://www.sun.com/third-party/global/oracle/consolidation/Containers_Oracle_041305.pdf

    It’d be interesting to see if you can do the same thing in some way on Linux, because this would be potentially very useful.

    Thanks,

    Alex

  4. 4 Niall Litchfield August 10, 2007 at 9:03 am

    The ERP licensing quote is even more off than you might imagine since in common with nearly all ERP vendors you purchase Oracle Apps licenses on a per-user basis and not a per processor basis.

  5. 5 kevinclosson August 10, 2007 at 4:55 pm

    Alex,

    Let’s see if my old buddy Glenn Fawcett can help fill us in…I just sent him email with a request…

  6. 6 kevinclosson August 10, 2007 at 5:32 pm

    Alex,

    Thanks for that URL. While we wait for Glenn to fill in (if he wishes), I’ll point out that I was aware that hard partitioning works as a basis for Oracle licensing. I’ve used hard partitions on HP-UX and AIX before. That aside, Linux sched_setaffinity(2) doesn’t qualify as soft partitioning nor hard partitioning.

    Oracle’s document on the matter (http://www.oracle.com/corporate/pricing/partitioning.pdf) states:

    “Hard partitioning physically segments a server, by taking a single large server and separating it into distinct smaller systems. Each separated system acts as a physically independent, self-contained server, typically with its own CPUs, operating system, separate boot area, memory, input/output subsystem and network resources.
    Examples of such partitioning type include: Dynamic System Domains (DSD)”

    It states that their definition of hard partitions “typically” have their own I/O subsystem which Solaris Zones do not have. They do have their own network interfaces though. I’ll bet (my guess here) Oracle’s ultimate criteria is that as long as a hard partition can be rebooted without affecting the larger host system then it is in…Solaris Zones can be rebooted without disruption to the larger host so it makes a lot of sense that such technology qualifies as a hard partition.

    As an aside, IBM’s approach to hard partitions (LPAR) peg an (minimum) I/O adapter to the partition so that would seem like even more of a system-within-a-system.

  7. 7 Glenn Fawcett August 10, 2007 at 6:18 pm

    As you point out, using sched_affinity(), pbind, … are not going to work for a technology that limits a workload to a set of cpus. These methods are kind of a heath kit of scripting that will allow you go cobble together something that works but will not fly for Oracle licensing purposes.

    Certainly, DSDs fit the hard partitioning definition and upcoming LDOMs will as well. Zones does cut a little close to the bone, but we have been successful working with Oracle to use Zones as a method to limit the number of CPUs for licensing.

    Zones can be rebooted, they have separate CPUs, memory (physical and virtual), scheduler, NICs, and *IO* resources can be dedicated as well to zones.

    http://www.opensolaris.org/os/community/zones/faq

    Hope this helps,
    Glenn

  8. 8 kevinclosson August 10, 2007 at 6:32 pm

    Thanks Glenn… let me guess, to get CPU and I/O zoned together exclusively, you can’t be running Solaris on an AMD box, right?

  9. 9 Glenn Fawcett August 10, 2007 at 7:57 pm

    Zones are part of Solaris and are not limited by the HW stack. Sun LDOMs on the other hand, are limited to use on the UltraSparc T1/T2 servers since they rely on primitives in the chip to support virtualization. For LDOM like functionality on x86/x64 we use VMware.

    Sun virtualization overview:
    http://www.sun.com/2006-1017/feature/
    http://www.sun.com/virtualization/

    LDOM guide:
    http://www.sun.com/blueprints/0207/820-0832.pdf

    take care,
    Glenn

  10. 10 Frits Hoogalnd August 12, 2007 at 12:50 pm

    Kevin, others,

    Oracle corp. describes the partitioning of a system as ‘hard partitioning’ and ‘soft partitioning’. I have got an oracle paper describing both partitioning options, along with implications for licensing. I can mail it if you want.

    (I’ve got the paper from a pre-sales manager at the company I currently work for, who has gone to quite some extent in order to get it.)

  11. 11 kevinclosson August 12, 2007 at 4:30 pm

    Fritz,

    It shouldn’t have been that difficult of a document to get because it is right here:
    http://www.oracle.com/corporate/pricing/partitioning.pdf

    This is not a spooky topic at all.

  12. 12 Frits Hoogland August 13, 2007 at 8:32 am

    That’s the document I mentioned.

    That seems to be the problem with much of the licensing stuff around the oracle products: Clarity.

    In our case, that person I mentioned informed about licensing and virtualisation at oracle. They managed to come up with a description about hardware partitioning and software partitioning, whilst nobody could describe what it really meant. It took about a day to get the linked document.

  13. 13 Mark September 4, 2007 at 1:23 am

    “This is not a spooky topic at all.”

    Not now, maybe. 🙂

    Last time I checked, the documents from Oracle cite some examples of what constitutes “hard” or “soft” partitioning, but then go on to say that the question of whether or not a particular partitioning technology is “hard” or “soft” would be decided on a case-by-case basis.

    Sadly I don’t recall any of these documents telling me who is the “authority” for such decisions, or how to contact them. 😦

    Happily, the document you cited does. “pricinginquiry@oracle.com”. Hmmm… I’ve wrestled with licensing questions like this one (and worse) for years, and don’t recall tripping over that particular address. Yet here it is in a 5-year-old PDF! On the other hand, perhaps I had been aware of this address from past investigations, and just forced myself to forget it! 🙂

    Anyway, on the “hard” / “soft” partitioning issue, things definitely are much clearer than they used to be. Although there are still problems. For example, where precisely does the determination of hard-ness or soft-ness of *my* partitioning scheme appear in the OLSA?

    Oh well. At least the debate over licensing for hyper-threaded CPUs is over. Right?

    Nice blog, by the way. I especially enjoyed the post on LGWR processing, although this is hardly the first time I have “dropped by”.


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




DISCLAIMER

I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,988 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: