Kevin Closson's Blog: Platforms, Databases and Storage

I’m working out my 2013 high-level plans and something dawned on me. I’m not sure I know what the term knowledge-sharing means any more.

Large IT vendors treat their top-100 customers with kid gloves. I know this, and I think it is just fine. Kid-gloves meetings occur frequently. These visits often include “knowledge-sharing” sessions where the IT vendor pitches their latest solutions and, perhaps, roadmaps for future products. IT vendors come back from these meetings with “top 10” product deficiency lists and sometimes items from those lists do make it into internal roadmaps or even PRD (product requirement documents).  That’s good. However, I wonder how much of that is thinking inside the box—or better yet, list items to make one’s life inside the box a little better.  Or rather, how much time is spent discussing Y2K-era hangover problems? Trust me, I could convince you that a lot of what is heralded as state-of-the-art IT kit being pushed today is actually just stuff that addresses Y2K-era problems that largely do not exist with today’s hardware. When you read a current datasheet from any IT vendor ask yourself whether someone just plopped a quarter in the juke box to spin Party Like It’s 1999.

Who Is talking to You?

I feel we in the IT community are facing some serious changes. Indeed, I’ll quote myself from the interview I did last year for the Northern California User’s Group Journal (here):

Everything we know in IT has a shelf life

Ok, yes, I just quoted myself. I’m sorry, but it helps me get my point across.

I’m thinking about visiting some customers this year for real knowledge-sharing. I emphasize sharing because I want to learn from you. I want to know about where you need to go so I can help figure out how to get you there.  But I need to get the right audience. Audiences that are, for instance, not convinced we need to solve problems that no longer exists (thus my quote from that article).

So, for example, let’s say I showed up with the type of material I presented in this 2010 Hotsos Symposium presentation .  Does your IT shop even have the type of IT personnel that would find this sort of material interesting? Would they attend? If they did attend, would they then let me pick their brains about where they need to take their IT shop so I can help build the right products?

How about a poll?

This is just a quick blog post to direct readers to treasure trove of public information offered by Luca Canali (of CERN).

Let there be no mistake. These are not just ordinary presentations. Many of them have detailed how-to information in the speaker notes to amplify the information in the slides. They fall into the class of must-read.

Enjoy, and thanks, to Luca!

The list:

Compressing very large data sets in Oracle, UKOUG Conference 2009, Birmingham, Dec 2009.

Upcoming: Active Data Guard at CERN, UKOUG 2012 conference, Birmingham, December 4th 2012

Testing Storage for Oracle RAC 11g with NAS, ASM, and SSD Flash Cache, UKOUG 2011 conference, Birmingham, December 6th, 2011

CERN IT-DB Deployment, Status, Outlook – ESA-GAIA DB Workshop, ISDC, March 2011

ACFS under scrutiny – UKOUG Conference 2010, Birmingham, Nov 2010

Data Lifecycle Management Challenges and Techniques, a user’s experience – UKOUG Conference 2010, Birmingham, Nov 2010

CERN DB Services for Physics – 2010 Report, Distributed Database Workshop, CERN, Nov 2010

Overview of the CERN DB Services for Physics, Orcan Swedish Oracle Users group Conference, Stockholm, May 2010

Evaluating and testing storage performance for Oracle DBs, UKOUG Conference 2009, Birmingham, Dec 2009.

ASM configuration review, Distributed Database Operations Workshop, CERN, November 2009.

Storage for data management, ‘after C5’ CERN-IT presentation, CERN, May 2009.

Data Lifecycle Review and Outlook, Distributed Database Operations Workshop, Barcelona, April 2009

Database Operations Security Overview, Distributed Database Operations Workshop, Barcelona, April 2009.

Overview of Storage and IT-DM lessons learned, IT-DM technical group meeting, CERN, March 2009.

Implementing ASM Without HW RAID, A User’s Experience, presentation at UKOUG Conference 2008, Birmingham Dec 2008.

Datalifecycle_WLCG_DB_workshop_LC: Data lifecycle management ideas, WLCG Database Workshop, CERN Nov 2008.

Workhops_8_Jul_case_PVSS_LC: Oracle performance tuning case study, PVSS archiver, Database Developers Workshop, CERN 8-7-2008.

Workhops_8_Jul_perf_for_developers_LC: Oracle performance tuning ideas for developers, Database Developers Workshop, CERN 8-7-2008.

Oracle Storage Performance Studies: Scalability tests of VLDBs with RAC and ASM for Physics DB Services, WLCG Workshop, CERN, Apr 2008.

A closer look inside Oracle ASM, talk on Oracle ASM internals and performance measurements at UKOUG 2007, Birmingham, December 2007.

WLCG_Oracle_perf_for_admin: Oracle Performance for Administrators, WLCG Reliability Workshop, CERN November 2007.

Oracle_CERN_Service_Architecture, CERN DB workshop, January 2007.

UKOUG_RACSig_Oct06, UKOUG RAC SIG meeting, UKOUG RACSIG meeting, London, October 2006.

DB services at CERN HEPIX 2006: “Database Services for Physics at CERN with Oracle 10g RAC”, HEPIX conference, Rome, April 2006.

DB_Serv_Meeting_ASM_Perf: “ASM-based storage to scale out Database Services for Physics”, April 2006

3D_RAL_Mar_06: “Oracle 10gR2 configuration”, 3D DB Workshop at RAL (UK), March 2006.

T2_tutorials_Jun06_OracleRAC: Oracle and RAC for Physics DB June 2006

For more information visit Luca Canali’s page here: http://canali.web.cern.ch/canali/main.htm

Oracle Exadata X3 Database In-Memory Machine – An Introduction
On October 1, 2012 Oracle issued a press release announcing the Oracle Exadata X3 Database In-Memory Machine. Well-chosen words, Oracle marketing, surgical indeed.

Words matter.
Games Are Games–Including Word Games
Oracle didn’t issue a press release about Exadata “In-Memory Database.” No, not “In-Memory Database” but “Database In-Memory” and the distinction is quite important. I gave some thought to that press release and then searched Google for what is known about Oracle and “in-memory” database technology. Here is what Google offered me:

Note: a right-click on the following photos will enlarge them.

With the exception of the paid search result about voltdb, all of the links Google offered takes one to information about Oracle’s Times Ten In-Memory Database which is a true “in-memory” database. But this isn’t a blog post about semantics. No, not at all. Please read on.

Seemingly Silly Swashbuckling Centered on Semantics?
Since words matter I decided to change the search term to “database in-memory”  and was offered the following:

So, yes, a little way down the page I see a link to webpage about Oracle’s freshly-minted term: “Database In-Memory”.

I’ve read several articles covering the X3 hardware refresh of Oracle Exadata Database Machine and each has argued that Oracle Real Application Clusters (RAC), on a large-ish capacity server, does not qualify as “in-memory database.” I honestly don’t see the point in that argument. Oracle is keenly aware that the software that executes on Exadata is not “in-memory database” technology. That’s why they wordsmith the nickname of the product as they do.  But that doesn’t mean I accept the marketing insinuation nor the hype. Indeed, I shouldn’t accept the marketing message–and neither should you–because having the sacred words “in-memory” in the nickname of this Exadata product update is ridiculous.

I suspect, however, most folks have no idea just how ridiculous. I aim to change that with this post.

Pictures Save A Thousand Words
I’m horrible at drawing, but I need to help readers visualize Oracle Exadata X3 Database In-Memory Machine at a component level. But first, a few words about how silly it is to confuse Exadata Smart Flash Cache with “memory” in the context of database technology.

But A Bunch Of PCI Flash Cards Qualify As Memory, Right?
…only out of necessity when playing word games. I surmise Oracle needed some words to help compete against HANA–so why not come up with a fallacious nickname to confuse the fact? Well, no matter how often a fallacy is repeated it remains a fallacy. But, I suppose I’m just being pedantic. I trust the rest of this article will convince readers otherwise.

If PCI Flash Is Memory Why Access It Like It’s a Disk?
…because the flash devices in Exadata Smart Flash Cache are, in fact, block devices. Allow me to explain.

The PCI flash cards in Exadata storage servers (aka cells) are presented as persistent Linux block devices to, and managed by, the user-mode Exadata Storage Server software (aka cellsrv).

In Exadata, both the spinning media and the PCI flash storage are presented by the Linux Kernel via the same SCSI block internals. Exadata Smart Flash Cache is just a collection of SCSI block devices accessed as SCSI block devices–albeit with lower latency than spinning media. How much lower latency? About 25-fold faster than a high-performance spinning disk (~6ms vs ~250us).  Oh, but wait, this is a blog post about “Database In-Memory” so how does an 8KB read access from Exadata Smart Flash Cache compare to real memory (DRAM)? To answer that question one must think in terms relevant to CPU technology.

In CPU terms, “reading” an 8KB block of data from DRAM consists of loading 128 cachelines into processor cache.  Now, if a compiler were to serialize the load instructions it would take about 6 microseconds (on a modern CPU) to load 8KB into processor cache–about 1/40th the published (datasheet) access times for an Exadata Smart Flash Cache physical read (of 8KB). But, serializing the memory load operations would be absurd because modern processors support numerous concurrent load/store operations and the compilers are optimized for this fact. So, no, not 6us–but splitting hairs when dissecting an absurdity is overkill. Note: the Sun F40 datasheet does not specify how the published 251us 8KB read service time is measured (e.g., end-to-end SCSI block read by the application or other lower-level measurement) but that matters little. In short, Oracle’s “Database In-Memory” data accesses are well beyond 40-fold slower than DRAM. But it’s not so much about device service times. It’s about overhead. Please read on…

High Level Component Diagram
The following drawing shows a high-level component breakout of Exadata X3-2. There are two “grids” of servers each having DRAM and Infiniband network connectivity for communications between database hosts and to/from the “storage grid.” Each storage server (cell) in the storage grid has spinning media and (4 each) PCI flash cards. The cells run Oracle Linux.

Exactly When Did Oracle Get So Confused Over The Difference Between PCI Flash Block Devices And DRAM?
In late 2009 Oracle added F20 PCI flash cards as a read-only cache to the second generation Exadata known as V2–or perhaps more widely known as The First Database Machine for OLTP[sic]. Fast forward to the present.

Oracle has upgraded the PCI flash to 400GB capacity with the F40 flash card.  Thus, with the X3 model there is 22.4TB of aggregate PCI flash block device capacity in a full-rack. How does that compare to the amount of real memory (DRAM) in a full-rack? The amount of real memory matters because this is a post about “Database In-Memory.”

The X3-2 model tops out at 2TB of aggregate DRAM (8 hosts each with 256GB) that can be used (minus overhead) for database caching (up from 1152 GB in the X2 generation). The X3-8, on the other hand, offers an aggregate of 4TB DRAM for database caching (up from the X2 limit of 2TB). To that end the ratio of flash block device capacity to host DRAM is 11:1 (X3-2) or 5.5:1 (X3-8).

This ratio matters because host DRAM is where database processing occurs. More on that later.

Tout “In-Memory” But Offer Worst-of-Breed DRAM Capacity
Long before Oracle became confused over the difference between PCI flash block devices and DRAM, there was Oracle’s first Database In-Memory Machine. Allow me to explain.

In the summer of 2009, Oracle built a 64-node cluster of HP blades with 512 Harpertown Xeons cores and paired it to 2TB of DRAM (aggregate). Yes, over three years before Oracle announced the “Database In-Memory Machine” the technology existed and delivered world-record TPC-H query performance results–because that particular 1TB scale benchmark was executed entirely in DRAM! And whether “database in-memory” or “in-memory database”, DRAM capacity matters. After all, if your “Database In-Memory” doesn’t actually fit in memory it’s not quite “database in-memory” technology.

Oracle Database has offered the In-Memory Parallel Query feature since the release of Oracle Database 11g R2.

So if Database In-Memory technology was already proven (over 3 years ago) why not build really large memory configurations and have a true “Database In-Memory? Well, you can’t with Exadata because it is DRAM-limited–most particularly the X3-2 model which is limited to 256GB per host. On the contrary, every tier-one x64 vendor I have investigated (IBM,HP,Dell,Cisco,Fujitsu) offers 768GB 2-socket E5-2600 based servers. Indeed, even Oracle’s own Sun Server X3-2 supports up to 512GB RAM–but only, of course, when deployed independently of Exadata. Note: this contrast in maximum supported memory spans 1U and 2U servers.

So what’s the point in bringing up tier 1 vendors? Well, Oracle is making an “in-memory” product that doesn’t ship with the maximum memory available for the components used to build the solution. Moreover, the Database In-Memory technology Oracle speaks of is generic database technology as proven by Oracle’s only benchmark result using the technology (i.e., that 1TB scale 512-core TPC-H).

OK, OK, It’s Obviously Not The “Oracle Exadata X3 Database In-DRAM Machine” So What Is It?

So if not DRAM, then what is the “memory” being referred to in Exadata X3 In-Memory Database Machine?  I’ll answer that question with the following drawing and if it doesn’t satisfy your quest for knowledge on this matter please read what follows:

Presto-magic just isn’t going to suffice. The following drawing clearly points out where Exadata DRAM cache exists:

That’s right. There has never been DRAM cache in the storage grid of Exadata Database Machine.

The storage grid servers’ (cells) DRAM is used for purposes of buffering and the metadata managing such things as Storage Index and mapping what HDD disk blocks are present (copies of HDD blocks) in the PCI flash (Exadata Smart Flash Cache).

Since the solution is not “Database In-Memory” the way reasonable technologists would naturally expect–DRAM–then where is the “memory?” Simple, see the next drawing:

Yes, the aggregate (22TB) Exadata Smart Flash Cache is now known as “Database In-Memory” technology. Please don’t be confused. That’s not PCI flash accessed with memory semantics. No, that’s PCI flash accessed as a SCSI device. There’s more to say on the matter.

Please examine the following drawing. Label #1 shows where Oracle Database data manipulation occurs. Sure, Exadata does filter data in storage when tables and indexes are being accessed with full scans in the direct path, but cells do not return a result set to a SQL cursor nor do cells modify rows in Oracle database blocks. All such computation can only occur where label #1 is pointing–the Oracle Instance.

The Oracle Instance is a collection of processes and the DRAM they use (some DRAM for shared cache e.g., SGA and some private e.g., PGA).

No SQL DML statement execution can complete without data flowing from storage into the Oracle Instance. That “data flow” can either be a) a complete Oracle database disk blocks (into the SGA) or b) filtered data (into the PGA).  Please note, I’m quite aware Exadata Smart Scan can rummage through vast amounts of data only to discover there are no rows matching the SQL selection criteria (predicates). I’m also aware of how Storage Indexes behave. However, this is a post about “Database In-Memory” and Smart Scan searching for non-existent needles in mountainous haystacks via physical I/O from PCI block devices and spinning rust (HDD) is not going to make it into the conversation. Put quite succinctly, only the Oracle Database  instance can return a result set to a SQL cursor–storage cells merely hasten the flow of tuples (Smart Scan is essentially a row source) into the Oracle Instance–but not to be cached/shared because Smart Scan data flows through the process private heap (PGA).

Exadata X3: Mass Memory Hierarchy With Automatic What? Sounds Like Tiering.
I’m a fan of auto-tiering storage (e.g., EMC FAST). From Oracle’s marketing of the “Database In-Memory Machine” one might think there is some sort of elegance in the solution along those lines. Don’t be fooled.  I’ll quote Oracle’s press release :

the Oracle Exadata X3 Database In-Memory Machine implements a mass memory hierarchy that automatically moves all active data into Flash and RAM memory, while keeping less active data on low-cost disks.

Sigh. Words Matter! Oracle Database does not draw disk blocks into DRAM (the SGA or PGA) unless the data is requested by SQL statement processing. The words that matter in that quote are: 1) “automatically moves” and 2) “keeping.” Folks, please consider the truth on this matter. Oracle doesn’t “move” or “keep.” It copies. The following are the facts:

1. All Oracle Database blocks are stored persistently on hard disk (spinning) drives. “Active data” is not re-homed onto flash as Oracle press release insinuates. The word “move” is utterly misleading in that context. The accurate word would be “copy.”
2. A block of data read from disk can be automatically copied into flash when read for the first time.
3. If a block of data is not being accessed by an Exadata Smart Scan the entire block is cached in the host DRAM (SGA). Remember the ratios I cited above. It’s a needles-eye situation.
4. If a block of data is being accessed by an Exadata Smart Scan only the relevant data flows through host DRAM (PGA) but is not cached.

In other words, “Copies” != “moves” and “keeping” insinuates tiering. Exadata does neither, it copies. Copies, copies, copies and only sometimes caches in DRAM (SGA). Remember the ratios I cited above.

Where Oh Where Have All My Copies Of Disk Blocks Gone?
The following drawing is a block diagram that depicts the three different memory/storage device hierarchies that hold a copy of a block (colored red in the drawing) that has flowed from disk into the Oracle Instance. The block is also copied into PCI flash (down in the storage grid) for future accelerated access.

Upon first access to this block of data there are 3 identical copies of the data. Notice how there is no cached copy of this block in the storage grid DRAM. As I already mentioned, there is no DRAM block cache in storage.

If, for example, the entirety of this database happened to fit into this one red color-coded block, the drawing would indeed represent a “Database In-Memory.” For Exadata X3 deployments serving OLTP/ERP use cases, this drawing accurately depicts what Oracle means when they say “Database In-Memory.”  That is, if it happens to fit in the aggregate Oracle Instance DRAM (the SGA) then you have a “Database In-Memory” solution. But, what if the database either a) doesn’t fit entirely into host DRAM cache and/or b) the OLTP/ERP workload occasionally triggers an Exadata Smart Scan (Exadata offload processing)? The answer to “a” is simple: LRU. The answer to “b” deserves another drawing. I’ll get to that, but first consider the drawing depicting Oracle’s implied “automatic” “tiering”:

I’ve Heard Smart Scan Complements “Database In-Memory”
Yeah, I’ve heard that one too.

The following drawing shows the flow of data from a Smart Scan. The blocks are being scanned from both HDD and flash block devices (a.k.a Smart Flash Cache) because Exadata scans Flash and HDD concurrently as long as there are relevant copies of blocks in flash. Important to note, however, is that the data flowing from this operation funnels through the non-shared Oracle memory known as PGA on the database host . The PGA is not shared. The PGA is not a cache. The effort expended by the “Database In-Memory Machine” in this example churns through copies of disk blocks present in flash block devices and HDD to produce a stream of data that is, essentially, “chewed and spat out” by the Oracle Instance process on the host. That is, the filtered data is temporal and session-specific and does not benefit another session’s query. To be fair, however, at least the blocks copied from HDD into flash will stand the chance of benefiting the physical disk access time of a future query.

The fundamental attributes of what the industry knows as “in-memory” technology is to have the data in a re-usable form in the fastest storage medium possible (DRAM).

We see neither of these attributes with Oracle Exadata X3 Database In-Memory Machine:

It Takes a Synopsis To Visualize The New “In-Memory” World Order
I’m not nit-picking the semantic differences between “in-memory database” and “database in-memory” because as I pointed out already Oracle has been doing “Database In-Memory” for years. I am, however, going to offer a synopsis of what is involved when Oracle Exadata X3 Database In-Memory Machine processes a block in an OLTP/ERP-style transaction. After all, Exadata is the World’s First Database Machine for OLTP and now that it is also Database In-Memory technology it might interest you to see how that all works out.

Consider the following SQL DML statement:

UPDATE CN_COMMISSION_HEADERS_ALL set DISCOUNT_PERCENTAGE = 99 WHERE TRX_BATCH_ID = 42;

This SQL is a typical OLTP/ERP rowid-based operation. In the following synopsis you’ll notice how an Exadata X3 Database In-Memory Machine transaction requires a wee-bit more than a memory access. Nah, I’ll put the sarcasm aside. This operations involves an utterly mind-boggling amount of overhead when viewed from the rosy lenses of “in-memory” technology. The overhead includes such ilk as cross-server IPC, wire time, SCSI block I/O, OS scheduling time, CPU-intensive mutex code and so forth. And, remember, this is a schematic of accessing a single block of data in the Exadata X3 Database In-Memory Machine:

• On the Host
  • Oracle foreground process (session) suffers SGA cache lookup/miss
    • <spinlocks, SGA metadata block lookup (cache buffers chains, etc)>
  • Oracle foreground LRUs a SGA buffer
    • <spinlocks cache buffers LRU, etc>
  • Oracle foreground performs an Exadata cell single block read
    • Oracle foreground messages across iDB (Infiniband) with request for block
      • <reliable datagram sockets IPC>
    • Oracle foreground process goes to sleep on a single block read wait event
    • <interrupt>
    • <request message DMA to Infiniband HCA “onto the wire”>
    • <wire time>
• On Storage Server
  • <interrupt>
  • <request message DMA from Infiniband HCA into Cellsrv>
  • <Cellsrv thread scheduling delay>
  • Cellsrv gets request and evaluates e.g., specific block to read, IORM metadata, etc
  • Cellsrv allocates a buffer (to read the block into, also serves as send buffer)
  • Cellsrv block lookup in directory (to see if it is in Flash Cache, for this example the block is in the Flash Cache)
  • Cellsrv reads the block from flash Linux block device(Exadata Smart Flash Cache devices are accessed with the same library API calls as spinning disk.)
    • Libaio or LibC read (non-blocking either way since Cellsrv is threaded)
      • <kernel mode>
      • <block layer code>
      • <driver code>
    • <interrupt>
    • < DMA block from PCI Flash block device into send buffer/read buffer> THIS STEP ALONE IS 251us AS PER ORACLE DATASHEET
      
    • <interrupt>
      • <kernel mode>
      • I/O marked complete
      • <Delay until Cellrv “reaps” I/O completion >
  • Cellsrv processes I/O completion (library code)
  • Cellsrv validates buffer contents (DRAM access)
  • Cellsrv sends block (send buffer) via iDB (Infiniband)
    • <interrupt>
    • <IB HCA DMA buffer onto the wire>
    • <wire time>
• Back on Host
  • <interrupt>
  • <IB HCA DMA block from wire into SGA buffer)
  • Oracle foreground process marked in runnable state by Kernel
  • <process scheduling time>
  • Oracle foreground process back on CPU
  • Oracle foreground validates SGA buffer
  • Oracle foreground chains the buffer in (for others to share)
    • <spinlocks, cache buffers chains, etc>
  • Oracle foreground locates data in the buffer (a.k.a “row walk”)

Wow, that’s a relief! We can now properly visualize what Oracle means when they speak of how “Database In-Memory Machine” technology is able to better the “World’s First Database Machine for OLTP.”

Summary
We are not confused over the difference between “database in-memory” and “in-memory database.”

Oracle have given yet another nickname to the Exadata Database Machine. The nickname represents technology that has been proven outside the scope of Exadata even 3 years ago. Oracle’s In-Memory Parallel Query feature is not Exadata-specific and that is a good thing. After all, if you want to deploy a legitimate Oracle database with the “database in-memory” approach you are going to need products from the best-of-breed server vendors since they’ve all stepped up to produce large-memory servers based on the hottest, current Intel technology–Sandy Bridge.

This is a lengthy blog entry and Oracle Exadata X3 Database In-Memory Machine is still just a nickname. You can license Oracle database and do a lot better. You just have know the difference between DRAM and SCSI block devices and choose best of breed platform partners. After all those of us who are involved with bringing best of breed technology to market are not confused at all by nicknames.

Sources:

1. 25 years of Oracle and platforms experience. I didn’t just fall off the turnip truck.
2. Exadata uses RDS (Reliable Datagram Sockets) with is OFED Open Source. The Synopsis of an Oracle-flavored “In-Memory” operation is centered on my understanding of RDS. You too can be an expert at RDS because it is Open source.
3. docs.oracle.com
4. Exadata data sheets (oracle.com)
5. Google
6. Expert Oracle Exadata – Apress
7. http://www.oracle.com/technetwork/database/exadata-smart-flash-cache-twp-v5-1-128560.pdf
8. http://www.oracle.com/technetwork/server-storage/engineered-systems/exadata/exadata-smart-flash-cache-366203.pdf
9. http://www.aioug.org/sangam12/Presentations/20205.pdf

In other words, this is all public domain knowledge assembled for your convenience.

I’ve had countless emails from readers asking for a technical analysis of what Oracle announced at Openworld 2012 pertaining to the X3 refresh of Exadata Database Machine. I attended the show, fell ill and subsequently had a a lot of work backlog to clear. I will get to this next week and, not surprising to readers of this blog, I’ll take aim on  the following words: “Database In-Memory Machine” as they appear in the new marketing nickname for Exadata Database Machine.

Yes, I will blog the matter but would first like to recommend the following excellent blog posts by @flashdba as they relate to “Database In-Memory Machine”:

In Memory Databases Part I

In Memory Databases Part II

Note: Part II has one tiny bit of errata as discussed in the comment section of the post. The post speaks of the cache hierarchy of X3 and includes Exadata Storage Server DRAM in the aggregate. I need to point out that we know from reading the myriads of public information on the matter (Oracle’s whitepapers, employee blogs and Expert Oracle Exadata (Apress)) that the DRAM in Exadata Storage Server cells is not used for cache. DRAM in the storage servers is used for management (metadata) of Exadata Smart Flash Cache contents, Storage Indexes metadata and buffering (IB dend/receive buffers, HCC decompression output, etc). The cache hierarchy of X3 is quite succinctly host DRAM (SGA buffers and Results Cache) and Exadata Smart Flash Cache (the PCI flash devices accessed via SCSI disk driver through the Linux block I/O layer in the cells).

The Enkitec Extreme Exadata Expo is well underway having just brought day 1 to a close with a keynote from Oracle’s Senior Vice President of Database Server Technology, Andy Mendelsohn.

Mr. Mendelson ended his presentation with a slide of futures for Exadata. Frits Hoogland tweeted the slide here.

I too was listening in on the presentation as a virtual attendee. I heard Mr. Mendelsohn state that the features fall into the “within 12 months” category. I suppose that means coinciding with Oracle Database 12.2 (note, dot-2). I could certainly be wrong on that matter though. Perhaps 12.1. We’ll see.

Of the items on the list I’d say the most interesting was the “Flash for all writes” item. The bullet point is enhanced with a pledge of 10x more writes for write-heavy OLTP. I knew of this feature and its eventual release, but now the information is public.

As my many posts on the matter attest, I have been critical of Oracle referring to Exadata as the “World’s First OLTP Machine” simply because it has read-cache. OLTP requires the scaling of writes along with reads. However, according to Oracle’s Exadata datasheets, the “World’s First OLTP Machine” is currently capable of 1.5 million read IOPS in a full-rack X2 configuration but only 50,000 gross random writes. With normal ASM redundancy the gross is reduced to a net WIOPS capacity of approximately 25,000 or a read:write ratio of 60:1. Many users feel compelled to use ASM high redundancy for reasons outside the scope of this post.

An increase in 10x would be 500,000 WIOPS (gross) which, being a write-back cache, will have to be de-staged back to spinning media at some point.

It will be interesting to see how this feature plays out. On first glance it would appear as though the goal is to support a read:write ratio of 6:1 which is drastically better than the World’s First OLTP Machine. If the World’s Second OLTP Machine can handle the de-staging from cache back to spinning media I’ll say kudos. Until we know, we can only guess.

Personally, I’m putting my bets on completely non-mechanical approaches for solving write-intensive OLTP problems. That’s another way of saying total flash storage.  However, I’d also stack up auto-tiering (e.g., EMC FAST) against this approach because the de-staging requirement is reduced as blocks are promoted to Enterprise Flash Drives. I have not performed any benchmark to back that viewpoint…but then I suspect a feature intended to see light of day “within 12 months” is getting much benchmark action within Oracle development either. There again, I could be wrong because I no longer work there.

What Are People Buying This Stuff For?
Since the vast majority of Exadata sales are quarter-rack configurations deployed for applications other than Data Warehousing I think Oracle is focusing on the correct weaknesses.

If you’re interested, I spoke of closely related topics in my recent interview in the August edition of the Northern California Oracle User Groups journal.

Finally, if you haven’t seen my video presentations on the fundamental architectural reasons Exadata is inferior to other solutions in the marketplace for the DW/BI/Analytics use case, I recommend you give them a viewing. Doing so might help you understand why Oracle sells more Exadata for use cases other than DW/BI and, in turn, why they focus on features that have nothing to do with that use case–like write-back flash cache.

This is just a quick blog entry to invite you to get a copy of the latest quarterly journal of Northern California Oracle Users Group. The Editor, Iggy Fernandez, interviewed me on a wide range of topics. The article begins on page 4.

Please click on the following link:

http://www.nocoug.org/Journal/NoCOUG_Journal_201208.pdf

The following is a screenshot of the magazine cover and following that is a list of the questions posed to me in the interview.

Interview questions:

Is hardware the new software? Why bother with indexing,
clustering, partitioning, and sharding (not to mention application
design) if hardware keeps getting bigger and faster
and cheaper every day. Can we just “load and go?”

You were an architect in Oracle’s Exadata development organization.
With all the excitement about Exadata in the Oracle
community, what motivated you to leave?

You’ve been referring to it as “Exaggerdata.” Why so harsh?
You’ve got to admit that there’s plenty of secret sauce under
the covers.

“Do-it-yourself Exadata-level performance?” Really? We’re
all ears.

Are TPC benchmarks relevant today?

Where do you stand in the “Battle Against Any Raid Five
(BAARF)?”

Can NAS match the performance of SAN? Can SAN match
the performance of locally attached storage

As a long-time NetApp fan, I like to stay in my comfort zone
but perhaps I’m biased. Does NetApp still offer anything that
the other NAS vendors (such as EMC) don’t?

Do we need yet another “silly little Oracle benchmark”?

Do you recommend that I use ASM? I worry that a sleepy
system administrator will unplug my disks someday because
“there were no files on them.” I do like to have my files where
I can count them every day!

Do you have an opinion on the NoSQL movement? Will the
relational mothership ever catch up with the speedboats?

My management is pushing me to take my databases virtual.
I can appreciate the advantages (such as elasticity) of virtualization,
but I’m not so sure about performance. Also, I cannot
rely on guest O/S statistics. Is it just the Luddite in me? Can
you help me push back on my management?

I like the Oracle Database Appliance (ODA) because RAC
raises the bar higher than most IT organizations can handle,
and ODA lowers it. What would you improve about the ODA
if you had a wizard wand?

Is NoCOUG a dinosaur, a venerable relic of the pre-information
age that has outlasted its usefulness?

Parts I an II of this series can be found here and here.

My previous installments in this series delved into how Orion does not interact with the operating system in any way similar to a database instance. This is a quick blog entry about CALIBRATE in that same vein.
On a system with 32 logical CPUs (e.g., 2s8c16t) we see CALIBRATE spawn off 32 background processes called ora_cs* such as the following ps(1) output screenshot:

I picked one of the processes and straced it to reveal the fact that CALIBRATE uses several processes each performing large-batch asynchronous I/O to do its job:

This should be a revelation to the purists out there. In my assessment I think CALIBRATE may even deviate worse from the goal of mimicking instance I/O than does Orion!
Nobody really cares about platforms though. So this is one of those “for what it’s worth” posts

Kerry Osborne has a short post to point out that the last book I worked on has been translated into Chinese.  The photo of the book cover looks pretty cool…if only I could read anything except for our names and a few stray English words 🙂

This book is a must-read for anyone that wants to cut beyond the hype and actually learn Exadata.

When file systems run out of space bad things happen. We like to investigate what those “bad things” are but to do so we have to create artificially small installation directories and run CPU-intensive programs to deplete the remaining space. There is a better way on modern Linux systems.

If you should find yourself performing Linux platform fault-injection testing you might care to add spurious space free failures. The fallocate() routine immediately allocates the specified amount of file system space to an open file.  It might be interesting to inject random space depletion in such areas as Oracle Clusterware (Grid Infrastructure) installation directories or application logging directories. Could a node ejection occur if all file system space immediately disappeared? What would that look like on the survivors? What happens if large swaths of space disappear and reappear? Be creative with your destructive tendencies and find out!

#include <asm/unistd.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>


int main(int argc, char *argv[])
{
long int sz;
char *fname;
int ret,fd;

if (argc != 3)
{
fprintf(stderr, "usage: %s file new-size-in-gigabytes\n", argv[0]);
return(-1);
}

fname = argv[1];
sz   = atol(argv[2]);

if ((ret = (fd = open(fname, O_RDWR | O_CREAT | O_EXCL, 0666)))  == -1 ) {
perror("open");
return(ret);

}
if ( (ret = fallocate( fd, 0, (loff_t)0, (loff_t)sz * 1024 * 1024 * 1024 )) != 0 ){
perror ("fallocate");
unlink( fname );
}

close(fd);
return ret;
}

#
# cc fast_alloc.c
#
# ./a.out
usage: ./a.out file new-size-in-gigabytes
#
# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/sdc 2.7T 1.6T 1.2T 57% /data1
#
# time ./a.out bigfile 512
real 0m1.875s
user 0m0.000s
sys 0m0.730s
# du -h bigfile
513G bigfile
# rm -f bigfile
#
# ./a.out bigfile 512
# ls -l bigfile
-rw-r--r-- 1 root root 549755813888 Jul 1 09:48 bigfile

This is just a short blog entry here to refer folks interested in SLOB to the following links:

• A great example of using SLOB to learn platform performance characteristics–specifically ASM mirroring.
• A SLOB Setup Cheat Sheet by Karl Arao.

About SLOB:  Introducing SLOB – The Silly Little Oracle Benchmark

This is just a very short blog entry to inform folks that there is an open discussion group over at LinkedIn for SLOB topics of interest.

The group can be accessed through the following link:  SLOB LinkedIn Group.

This is just a quick blog entry to show the very simple init.ora parameter file I use to stress simple read IOPS testing with SLOB.  On 2s16c32t E5-2600 servers attached to very fast storage this init.ora parameter delivers on the order of 275,000 physical IOPS with 64 SLOB sessions.

I’ll post an init.ora that I use for the REDO model and DBWR testing as soon as possible.

Thanks to Yury for the recommended hidden init.ora parameters to boost the ratio of db file sequential reads.

Additional information can be found here: README file.

Here is the init.ora:

UNDO_MANAGEMENT=AUTO
db_block_size = 8192
db_files = 20000
processes = 500
shared_pool_size = 5000M
db_cache_size=128M
filesystemio_options=setall
parallel_max_servers=0
_db_block_prefetch_limit=0
_db_block_prefetch_quota=0
_db_file_noncontig_mblock_read_count=0
log_buffer=134217728
pga_aggregate_target=1G

In Part I of this series I discussed the concept of “Mindless I/O” and contrasted Orion I/O testing to SLOB.

Some time has passed since that post and a lot of folks have been studying their systems using SLOB. Others have been motivated to prove that the differences between SLOB and Orion are moot. I’m OK with that because I use both of them and a large array of other I/O tools (fio, bonnie, etc,etc).

Test Single Block Random Reads With SLOB And Call It A Day
There is something I’d like to quickly point out about SLOB.

SLOB has 4 models for testing. Since it is a real SGA it can be used for testing the platform’s ability to handle SGA-cached workloads. Think about it. If your platform can’t scale up cached reads you’ve got a problem. This is what a lot of us SLOB users refer to as Logical I/O testing where the metric is LIOPS (logical I/O per second). Another model is, of course, the simple random read model. There are also two models that stress genuine DBWR and LGWR functionality and those models cannot be mimicked accurately, or otherwise, with Orion. Orion is mindless I/O. That fact matters to me and perhaps will matter to some of you as well.

Click here for more on the different SLOB models.

Cool Idea: Let’s use Asynchronous I/O Libraries To Determine Platform Fitness For High-Rate Synchronous I/O
Another thing to be aware of with Orion (as I pointed out 6 years ago) is the fact that Orion actually uses the wrong library for performing random single block reads (db file sequential read).  Yes, it uses VOS routines (skgf*), but the wrong ones–at least for the random single block read tests. The following screen shot shows Orion attempting to measure platform suitability for Oracle single block random reads (the db file sequential read model). Oracle instances do not use io_submit()/io_getevents() for db file sequential read:

A real Oracle process (such as a SLOB session) uses synchronous I/O (specifically libC pread()) to service db file sequential read. Yes, I know this varies by file storage type, dNFS, Exadata and other variations but I’m on blisteringly fast storage accessed via direct I/O through an XFS file system–a perfectly supported storage provisioning approach. Nonetheless, in all cases Oracle instances perform single-block I/O when performing single-block blocking I/O–not some FrankenSynchronousIO(tm) such as a single-block request through the asynchronous API followed by a group-reaping of outstanding requests as Orion is doing in the above screenshot. The screen shot shows a single Linux process (Orion) reaping 128 previously-submitted asynchronous I/Os whilst aiming to simulate db file sequential read.

So, if you happen to be testing modern, non-mechanical storage at a rate of, say, 20,000 IOPS/core you are missing out on the joy of analyzing the associated cost of 20,000 context switches/sec. Most people think a system call is a context switch–it isn’t. A context switch is the stopping of a process, saving its state, executing the scheduler code and switching to the context of another process. When a scheduler gets to make 20,000 affinity (NUMA, L2 cache, etc) decisions per second you get the opportunity of learning whether you like the choices it is making. I prefer to know as opposed to blind faith in the OS. A platform is more than the sum of all it’s components. It needs to all come together.

Another problem with using libaio in place of synchronous pread() is the fact that I/Os submitted in the asynchronous path are flat-out handled differently–and different is not the same as same. This is the case on all operating systems I know of. When requests are submitted through the asynchronous interfaces there are opportunities for the OS to potentially optimize the strategy for servicing the I/O. After all, the I/Os have been submitted from a single process context whereas the same number of I/O requests funneling through the synchronous interface means associating the request with a potentially vast numbers of discrete processes. It’s just different, not “slightly less-same”, but different. And different matters–at least to me.

It’s Just Different–And That’s All That Matters
There are some folks in the blogosphere putting in good SLOB testing. Some good folks at Pythian are doing the heavy lifting of proving to you that SLOB and Orion are on par or even, perhaps, Orion is somehow better for testing random reads. I should point out that random reads testing is just a miniscule portion of platform performance analysis. The fact that SLOB and Orion are different is all that matters to me. I need more tools than just a hammer. And when I need a hammer I don’t what a screwdriver that feels serendipitously hammer-like.

Once again I want to thank Yury for testing SLOB. Even if I don’t agree with his conclusions I very much appreciate his work with the kit. Healthy disagreements are healthy. We cultivate these threads and we all learn (I presume).

Different In So Many Ways
So if you are reading this, and are about to begin SLOB testing please be mindful of the following “feature overview” vis a vis SLOB:

• SLOB is an Oracle instance. It doesn’t toil along trying to behave like an instance nor challenge your faith in accepting the differences as unimportant.
• SLOB is not mindless I/O. You get the chance to learn more about your platform when the CPUs are actually doing something. I/O is not no longer a probelm with modern platforms. It’s all about CPU–always has been but obscured by artificial storage plumbing problems.
• SLOB uses the right routines. If you want to study the fitness for a given platform to handle high-rate db file sequential read you might as well do db file sequential read–or at least use a tool that calls the proper system calls. Libaio and LibC share letters of the alphabet but they are really different–and different in this specific regard matters. Actaully, different always matter to me.
• SLOB will allow you to test / study DBWR and LGWR. If you can get Orion to aid you in the study of log file sync or DBWR flushing capacity please drop me a note  because I missed the memo 🙂 .

About Part III
I discussed “Mindless I/O” in Part I. I needed to inject this installment to point out the little bits about Orion using libaio when you want to test synchronous reads. When I get to Part III I’ll get back to the Mindless I/O topic.

I just checked to find out that there has been 3,000 downloads of SLOB – The Silly Little Benchmark. People seem to be putting it to good use. That’s good.

Before I get very far in this post I’d like to take us back in time–back before the smashing popularity of the Orion I/O testing tool.

When Orion first appeared on the scene there was a general reluctance to adopt it. I suspect some of the reluctance stemmed from the fact that folks had built up their reliance on other tools like bonnie, LMbench, vxbench and other such generic I/O generators. Back in the 2006 (or so) time frame I routinely pointed out that no tool other than Orion used the VOS layer Oracle I/O routines and libraries. It’s important to test as much of the real thing as possible.

Who wants to rely on an unrealistic platform performance measurement tool after all?

My “List”
Over time I built a list of reasons I could no longer accept Orion as sufficient for platform I/O testing. Please note, I just wrote “platform I/O testing” not “I/O subsystem testing.”  I think the rest of this post will make the distinction between these two quoted phrases quite clear. The following is a short version of the list:

• Orion does not simulate Oracle processing in any way, shape or form. More on that as this blog series matures.
• Orion is what I refer to as mindless I/O. More on that as this blog series matures.
• Orion is useless in assessing a platform’s capability to handle modify-intensive DML (thus REDO processing, LGWR and DBWR, etc). More on that as this blog series matures.

My present-tense views on Orion sometimes surface on twitter where I am occasionally met with vigorous disagreement–most notably from my friend Alex Gorbachev. Alex is a friend, co-member of the Oaktable Network, CTO of Pythian (I love those Pythian folks), and someone who generally disagrees with most everything I say.

I respect Alex, because he has vast knowledge and valuable skills. His arguments make me think. That’s a good thing. I’m not sure, however, our respective spheres of expertise overlap.

So how do these disagreements regarding SLOB get started? Recently I tweeted:

The difference between SLOB and Orion is akin to Elliptical trainer versus skiing on the side of a mountain.

Alex replied with:

I could just as well argue that SLOB is useless because that’s not real workload anyway and you should test with your app

This quick exchange of ideas set into motion some Pythian testing by Yury. As it turns out I think the goal of that test was to prove parity between SLOB and Orion for random reads–and perhaps not much more.  If only I had published “My List” above before then.

Yury’s tests were good, albeit, exceedingly small in scope. His blog post suggests more testing on the way. That is good. If you read the comment thread on his blog entry you’ll see where I thank Yury for a good tweak to the SLOB kit that eliminates the db file parallel reads associated with the index range scans incurred by SLOB reader processes. Come to think of it though, Matt from Violin Memory pointed that one out to me some time back. Hmm, oh well, I digress. The modifications Yury detailed (init.ora parameters) will be included in the next drop of the SLOB kit. Again, thanks Yury for the testing and the init.ora parameter change recommendations!

Feel free to see Yury’s findings. They are simple: SLOB and Orion do the same thing. Really, SLOB and Orion do the same thing? Well, that may be the case so long as a) you compare SLOB to Orion only for simple random read testing and/or b) your testing is limited to a little, itsy-bitsy, teeny, tiny, teensy, minute, miniscule, meager, puny, Lilliputian-grade undersized I/O subsystem incapable of producing reasonable, modern-scale IOPS.  Yury’s experiment topped out at roughly 4,500 random read IOPS.  I’ll try to convince you that there is more to it than that (hint, modern servers are fit for IOPS in the 20,000/core range). But first, I have two quotable quotes to offer at this point:

When assessing the validity of an I/O testing tool, do so on a system that isn’t badly bottlenecked on storage.

If your application (e.g., Oracle Database)  is “mindless” use a “mindless” I/O generator–if not, don’t.

Mindless I/O
So what do I mean when I say “mindless I/O?”  The answer to that is simple. If the code performs an I/O into a memory buffer, without any application concurrency overhead, and no processes even so much as peeks at a single byte of that buffer populated through DMA from the I/O adapter device driver–it’s mindless. That is exactly how Orion does what it does. That’s what every other synthetic I/O generator I know of does as well.

So what does mindless I/O look like and why does it show up on my personal radar as a problem? Let’s take a look–but first let me just say one thing–I analyze I/O characteristics on extremely I/O capable platforms. Extremely capable.

The following screen shot shows a dd(1) command performing mindless I/O by copying an Oracle OMF datafile from an XFS file system to /dev/null using direct I/O. After that another dd(1) was used to show the difference between “mindless” and meaningful I/O. The second dd(1) was meaningful because after each 1 MB read the buffer is scanned looking for lower case ASCII chars to convert to their upper-case counterpart. That is, the second dd(1) did data processing–not just a mindless tickling of the I/O subsystem.

The mindless I/O was 2.5 GB/s but the meaningful case fell to about 1/6th that at 399 MB/s. See, CPU matters. It matters in I/O testing. CPU throttles I/O–unless you are interested in mindless I/O. What does this have to do with Orion and SLOB? A moment ago I mentioned that I test very formidable I/O subsystems commensurate with modern platforms–so hold on to your hat while I tie these trains of thought together.

Building on my dd(1) example of mindless I/O, I’ll offer the following screen shot which shows Orion accessing the same OMF SLOB datafile (also via direct I/O validated with strace). Notice how I force all the threads of Orion (it’s threaded with libpthreads) to OS CPU 0 using numactl(8) on this 2s12c24t Xeon 5600 server?  What you are about to see is the single-core capacity of Orion to perform “mindless I/O”:

Unrealistic Platform Performance Measurement Tools
This is only Part I in this series.  I’ll be going through a lot of proof points to solidify backing for my Orion-related assertions in the list above, but please humor me for a moment. I’d like to know just how realistic are platform performance measurements from an I/O tool that demonstrates capacity for 144,339 physical 8K random IOPS while pinned to a single core of a Xeon 5600 processor?

We are interested in database platform IOPS capacity, right?

Through this blog series I aim to help you conclude that any tool demonstrating such an unrealistic platform performance measurement is, well, an unrealistic platform performance measurement tool.

Do you feel comfortable relying on an unrealistic platform performance measurement tool? Before I crafted SLOB I too accepted test results from unrealistic platform performance measurement tools but I learned that I needed to include the rest of the platform (e.g., CPU, bus, etc) when I’m studying platform performance so I left behind unrealistic platform performance measurement tools.

Until recently I didn’t spend any time discussing measurements taken from unrealistic platform performance measurement tools. However, since friends and others in social media are pitting unrealistic platform performance measurement tools against SLOB (not an unrealistic platform performance measurement tool) such comparisons are blog-worthy. Hence, I’ll trudge forward blogging about how unrealistic certain unrealistic platform performance measurement tools are. And, if you stay with me on the series, you might discover some things you don’t know because, perhaps, you’ve been relying on unrealistic platform performance measurement tools.

As this series evolves, I’ll be sharing several similar unrealistic platform performance measurement tool results as I go though the list above. That is, of course, what motivated me to leave behind unrealistic platform performance measurement tools.

Final Words For This Installment
In Yury’s post he quoted me as having said:

It’s VERY easy to get huge Orion nums

His assessment of that quote was, “kind of FALSE on this occasion.”  Having now shown what I mean by “VERY easy” (e.g., even a single core can drive massive Orion IOPS) and “huge Orion” numbers  (e.g., 144K IOPS), I wonder whether Yury will be convinced about my assertions regarding unrealistic platform performance measurement tools? If not yet, perhaps he, and other readers will eventually. After all, this is only Part I. If not, Yury, I still want to say, “thanks for testing with SLOB and please keep the feedback coming.”

Oh, by the way folks, if all you have is Orion, use it. It is better than wild guesses–at least a little better.

Link to Part II of this series.