Recent SPARC T4-4 TPC-H Results Prove Oracle Can Do Better Than…Oracle! Part II.

My recent post entitled Recent SPARC T4-4 TPC-H Benchmark Results. Proving Bandwidth! But What Storage? provoked the following comment/question  from a reader:

Does this summarize your point(s)?

TPC-H produces a number which is a reflection of (hourly?!?) system throughput.

System throughput may not be indicative of system “performance” to its uses b/c users are typically most intersted in response time. Thus, TPC-H is a easily mis-used benchmark for comparing real world performance.

Oracle is using our misunderstanding of throughput as performnace to Market systems which are excellent throughput machines as excellent performance machines, when in fact their performance may be less then desirable.

I hope the reader took the time to read yesterday’s post entitled Recent SPARC T4-4 TPC-H Results Prove Oracle Can Do Better Than…Oracle as I think it goes a long way to address his comment/question. However, I do think the reader’s question deserves proper handling and thus I’m making this blog entry. So, dear reader, the following is my response to your comment/questions, but first I need to clear the air as it were.

There Is No Evil Lurking In This Thread
Let me first state categorically that Oracle is not “using our misunderstanding […] to Market systems […]” They are not doing anything under-handed with these TPC-H results.  They are, however, conveniently failing to compare their results to their own prior results. I only brought up the HP Proliant DL980 SQL Server results because Oracle did so in their press release.

Comparisons
I really do not like to compare TPC-H results across database vendor lines. The benchmark is too tricked out, it is a 3rd normal form schema and many other things about it make it just a goofy benchmark—if you have data warehousing in mind.  Nevertheless, comparisons between a given database vendor are useful for many purposes—such as suiting my ulterior motive which is to suggest that Oracle runs better on platforms other than their very own (recently acquired) SPARC processors.

Before I continue I’d like to interject a proclamation. In fact, I’ll quote myself if you’ll suffer me to do so:

Lack of published TPC-H results does not in any way disqualify any technology offering in the data warehousing space. There are no Oracle Exadata Database Machine  TPC-H results and that does not amount to a hill of beans. There are also no Teradata, EMC Greenplum  nor IBM Netezza results either and none of those beans form a hill.

— Kevin Closson

The point truly is that TPC-H does not reflect DW/BI/Big Data Analytics reality. However,  if a vendor like Oracle chooses to publish results then by all means I’m going to use those results to make my point—but only comparing Oracle’s own results. That’s precisely what I did I in my post entitled Recent SPARC T4-4 TPC-H Results Prove Oracle Can Do Better Than…Oracle.

Now, on to address the readers’ questions.

Throughput is a performance metric and a valid one indeed. However, throughput is generally derived by a concurrent workload of individual units of work that are individually measurable.  Consider disk throughput. If I tell you I have a storage configuration that satisfies, say, 500,000 I/O operations per second (IOPS) but don’t tell you the average service times I’m leaving out a critical piece of information.

How is the IOPS metric calculated? One samples I/O completions for a given period of time and then divides by the number of seconds sampled. It’s only measuring completions. If I have a tremendous number of I/O operations in flight concurrently, and sustained, I can get 500,000 IOPS even if the average completion time is 1 second.  They overlap.  The same goes for query workloads.

If you submit a continual, large stream of a variety of long running queries you get throughput.  Simply run such a hypothetical workload for a long time, sum up the completions and divide by sample period (time) and you get queries per unit of time.  Simple.

For example, if I have 10,000 concurrent queries requiring, on average, 61 minutes monitored for 2 hours I’ll get 10,000 completions or 5,000 queries per hour. So long as that meets my service requirement I’m fine.  However, if even one of my users mandates a 20 minute completion time I’m not going to impress with hand-waving over the great 5,000 QpH throughput I’m pushing through the system. Users really don’t care about how much work the system is doing on behalf of others. Do they?

So, to continue in this three-part series I’ll have to refer once again to the TPH-H disclosures (cited below).

I’ll refer again to the SPARC T4-4 result. If you glance at the report you’ll see that when submitted serially the geometric mean of query completion times is about 20 seconds on the SPARC T4. On the other hand, when we look at the HP BladeSystem result of over 3 years ago (still with Oracle Database 11g) we see that the geometric mean of serially submitted queries is nearly indiscernible…a mere blip. Of course the astute reader will point out that these comparisons—while both Oracle Database 11g—are that of in-memory versus disk-based (since the HP BladeSystem result was an In-Memory Parallel Query result). To that I would reply that it is foremost an old, tired Harpertown Xeon (5400) result with front-side bus technology compared to a state of the art, modern CPU (SPARC T4). And let’s not forget that the SPARC T4 server was connected to solid state storage!

It’s Not Fair Comparing Oracle In-Memory Parallel Query To Flash Storage
Really? Even considering how primitive a Harpertown Xeon was compared to a modern processor like SPARC T4? OK, fine. We can also harken back further to nearly 5 years to a result achieved by the now-defunct systems vendor called PANTA Systems. The PANTA System configuration, at the same 1TB scale, carried the following baggage:

  • Oracle Database 10g (with Real Application Clusters). So, old software.
  • Really, really old AMD Opteron 8000’s (very, very slow by today’s standards).
  • DDR400 DIMMs.

In spite of this aged bio, the configuration produced a geometric mean of 49 seconds for the serially submitted query stream compared to the 20 second result for the SPARC T4.

That’s a vintage 5 year old system, 10g versus 11g, AMD 8000 versus SPARC T4, DDR400 (not even DDR2) versus DDR3 memory and, lest I forget, the PANTA System memory controller was located across a front-side bus compared to the on-die SPARC memory controller. Tally up all of those contrasting system attributes and the resultant benefit to SPARC T4-4 is about 2.5-fold improvement in the geometric mean of query response times (serial). And, yes, time and technology did bring a a 7x increase in the throughput metric…but…once again, I encourage you to look at the disclosures I link to below and see how the completion times stack up in the throughput tests. If  you do so then we will have come full circle.

No, Oracle is not misleading anyone with these recent SPARC T4 results.

http://tpc.org/results/individual_results/Oracle/Oracle_T4-4_1TB_TPCH_ES_092611.pdf

http://tpc.org/results/individual_results/HP/HP_BladeSystem128P_090603_TPCH_ES_v2.pdf

http://tpc.org/results/individual_results/PANTA/PANTAmatrix_tpch_1TB_061019_es.pdf

4 Responses to “Recent SPARC T4-4 TPC-H Results Prove Oracle Can Do Better Than…Oracle! Part II.”


  1. 1 ZF January 16, 2012 at 8:36 am

    Hello Kevin,

    I keep checking-in and hoping to read about your thoughts on the EMC / Greenplum database appliance, given your work there over the past year?

    Is this blog solely for Oracle software and systems? If so, apologies and would you mind directing me to a blog somewhere which you’d recommend as being informative and knowledgeable for EMC / Greenplum?

    Thanks,

    ZF

    • 2 kevinclosson January 16, 2012 at 5:00 pm

      Hi ZF, I will be blogging such content here but the timing has not been right and to be honest I’m not a Greenplum expert yet! That takes a while. Now, that aside, I have sufficient expertise to know for a fact the ways in which Greenplum is vastly superior to other products (including Oracle Exadata) but haven’t taken the time to churn out much content in that regard.

  2. 3 ZF January 18, 2012 at 7:57 am

    Thanks. Look forward to the updates. I suspect the Greenplum information may be more eagerly anticipated and considered more impartial if there were to be less of a preoccupation with Oracle in the meantime!

    Are there any EMC whizzes out there blogging about Greenplum? I’d really like to read thoughts of an expert 😉

    • 4 kevinclosson January 18, 2012 at 8:15 am

      Thanks for your advice on how to run my blog, ZF. Not sure I can muster a reply to your query for a blog reference since I’m too preoccupied with Oracle-related matters. 🙂 Surely the preoccupation is understandable considering Oracle has been my occupation for nearly 25 years. Greenplum has had my attention for less than a year and at EMC I’m expected to not lose much ground with skills related to Oracle.


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




DISCLAIMER

I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,976 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: