Kevin Closson's Blog: Platforms, Databases and Storage

I participate in the Oracle-L mail list managed by fellow OakTable network member Steve Adams. There are a lot of good folks over there. I want to hijack a thread from that list and weave it in to a post I’ve wanted to do about Oracle porting.

Oracle Port-level Optimizations
It is a little known fact that major performance ramifications are the result of port-level implementation decisions. Oracle maintains a “bridge layer” of code between the Oracle Kernel and the lower level routines that interface with the Operating System. This layer is called the Virtual Operating System layer, or VOS. Oracle was brilliant for implementing this layer of the server. Without it, there would be routines at widely varying levels of the Oracle Kernel interfacing with the Operating System—chaos! Under the VOS is where the nitty-gritty happens.

So, what does this have to do with the thread on Oracle-L?

There was a post where a list member stated:

I saw quite a few cas latch waits today, Oracle 9.2.0.4 on HP-UX 11.11 PA-RiSC. Do these CPUs support CAS instructions?

This post sparked a series of follow-ups about Oracle’s usage of shared latches. For the non-Oracle minded reader, the term shared latches means reader/writer locks. You see, for many years, all critical sections in Oracle were protected by complete mutual exclusion—a real waste for mostly-read objects. The post is referring to an optimization where shared latches are implemented using Compare and Swap primitives. Whether or not your port of Oracle has this optimization is a decision made at the port level—either the OS supports it or it doesn’t. If it doesn’t, there are tough choices to make at the porting level. But the topic is bigger than that. When Oracle uses generic terms like “CAS” and “Scattered Read”, a lot is lost in translation. That is, when the VOS calls an underlying “Scattered Read”, is it a simulation? Is it really a single system call that takes a multiblock disk read and populates the SGA buffers with DMA? Or is it more like the age old Berkeley readv(2) which actually just looped with singleton reads in library context? On the other hand, when Oracle executes CAS code, is it really getting a CAS instruction or a set of atomic instructions (with looping)? The latter is generally the case.

Another installment on that particular Oracle-L thread took it to the port level:

Correct, HP doesn’t do CAS. There are some shared read latch operations that Oracle therefore implements through the CAS latch…

Right, HP, or more precisely the PA-RISC instruction set does not have a Compare and Swap instruction—it seems HP took the “reduced” in reduced instruction set computing to the extreme! However, neither does PowerPC or Alpha for that matter. In fact, neither does x86 or IA64. Oh hold it; I’m playing a word game. Both x86 and IA64 do have CAS, but it is called cmpxchg (compare and exchange). But honestly, PowerPC and Alpha do not. So how do these platforms offer CAS?

The porting teams for these various platforms have to make decisions. In the case of offering CAS to the VOS layer, they either have to construct a CAS using an atomic block of instructions or punt and use the reference code which is a spinlock. In the latter case, the wait event CAS latch can pop up. You see, a CAS latch can wait, whereas a real CAS will only stall the CPU. That is, if the port implements a CAS latch where other ports go with a CAS atomic set or single instruction, the former can sleep on a miss and the latter cannot. The processor is going to do that CAS, and nothing else. A contended memory location being hammered by CAS will stall processors, because once the CPU enters that block of (or singleton) instruction(s), it stays there until the work is done. I’m not talking about pathology, just implementation subtleties. So, what does CAS really look like?

Sparc64, x86, IA64, S/370, and get this, Motorola 68020 all offer a CAS instruction. There are others for sure. On the other hand, PPC and Alpha require an atomic set of instructions built off of LL/SC (Load-Link/Store Conditional) which on Power is the famed “lorks and storks” (ldarx/stwcx) and Alpha with their lxd_l/stx_c. Finally, what about PA-RISC? Well, you can’t do Oracle on any CPU that totally lacks atomic instructions. In the case of PA-RISC there is a Load and Clear Word (ldcw) instruction.

The point is that the VOS can be given a CAS of one sort or the other, but not all architectures handle the contention that CAS can cause. For whatever reason, it seems the HP porting group punted on 9i and went with latches where other ports use a real, or constructed, CAS. Be aware that just following the masses and implementing a CAS atomic set is not always the right answer. These pieces of code can do really weird things when the words being modified by the CAS straddle a cacheline and other such issues. Hmmm, trivial pursuit?

How Subtle are These Subtleties?

They can be really really NOT subtle!

I was on a team of folks that implemented an atomic set CAS for Oracle’s System Change Number in Oracle8, which is actually a multi-word structure with the SCN word and another word representing the wrap value. The SCN value always increases, albeit not serially. It just gets “larger”. We were able to pull the latch that protected the incrementing of these values and replace it with a small block of atomic assembly that incremented it without any locks. I doubt we were the first to do that. The result was a 25% performance increase in TPC-C. Why? The SCN used to be a really big problem. Back then, propeller-heads like me used to collect bus traces on workloads like TPC-C and map the physical addresses back to SGA addresses. It so happened that in older versions of Oracle, 27% of all addresses referenced on the bus (64 CPU system) was the cacheline that held the SCN structure! Granted, that was for the sum of load, store and coherency ops (e.g., invalidate, cache to cache transfers).