Facebook Adopts Oracle Exalogic Elastic Cloud, Replaces Tens Of Thousands Of Servers. Memorable Conference Keynote Addresses.

BLOG UPDATE 2011.08.17: In the original I misstated the Facebook page view rate to be 260,000,000 per day when it is in fact 260,000,000,000 per month. I used the correct per-second rate anyway so this update does not change the original in that regard.

BLOG UPDATE 2011.08.16: The blog title is a come-on.

Conference Keynotes: One Year Ago versus Eleven Years Ago
I spent some time rummaging through my scrapbook over the weekend. I stumbled on a program for the IBM Partnerworld 2000 conference. It might be difficult to see, but I scanned it in to show that on the left hand side there is an announcement of the keynote session with (former) U.S. President George H.W. Bush and on the bottom of the right hand page is an announcement of the session I was offering the same day. That was a really cool conference! The audience (certainly me included) thoroughly enjoyed the keynote. That was eleven years ago but I recall that my session was a bunch of blah-blah-blah compared the keynote.

There was another keynote, only one year ago, that is burned much more clearly into my memory. The conference was Oracle Openworld 2010 and, of course, the keynote speaker was Larry Ellison. Unlike the IBM conference of long ago I had no speaking session. I certainly would have had a session, due to the overwhelming number of votes racked up by my proposed “Oracle Mix Suggest-a-Session” (105 votes), but the Oracle Mix/Conference folks decided, but didn’t share the fact until afterwards, that Oracle employees weren’t eligible to participate in the Suggest-A-Session program. Boo hoo. Attendees missed a lot because the session topic was Do-It-yourself Exadata-level Performance.

Do It Yourself  Exadata-Level Performance
A DIY Exadata-level performance configuration is not what I’m blogging about. However, it seems I just stepped in it.  Quote me on this:

 If you configure a lot of I/O bandwidth and a lot of CPU you can match Exadata performance with Real Application Clusters and conventional hardware.

…but I’m not blogging about that.

Big Sounding Claims
So, back to the comparison between the circa-2000 IBM Partnerworld keynote and last year’s Openworld keynote. The majority of fanfare last year was Exalogic. I remember this Larry Ellison sound bite :

A single setup is capable of handling 1 million HTTP requests per second; two running side-by-side could handle Facebook’s HTTP request workload

For those who don’t know, the following is true:

That’s roughly 100,000 per second (260 * 10^9)/(30*24*60*60). So, the claims of a two-rack Exalogic configuration handling Facebook’s “HTTP request workload” is actually quite true. So why am I blogging about it then? The numbers are indeed stunning! There is a huge difference between the “HTTP request workload” and serving meaningful content. But I’m not blogging about that.

Those Facebook Engineers are Real Dummies
Well, they must be, if they were using  “thousands and thousands” of servers to do what they were doing even back in 2008:

Because we have thousands and thousands of computers, each running a hundred or more Apache processes […]

Silly Facebook engineers. They could save so much space, power and cooling if they only listened to certain keynote wisdom.

Your Choice: A Two-Rack Exalogic Configuration Or 25 Itsy-Bitsy Xeon 5600 Cores
If A two-rack Exalogic configuration can supplant all the processing power of Facebook’s web serving farm, I have to do the math. I won’t argue that Exalogic can perform “1 millions HTTP requests per second.” Because it surely can. I will, however, focus on just how much Westmere-EP CPU it takes to do the 100,000 HTTP requests/sec it would take to handle just the request portion of Facebook’s workload.

I think I’ll fire up httpd on one of my Linux servers and crank up ab(8) (ApacheBench) to see how much CPU is takes to handle a given rate of requests. The workload is pulling about 4K for each request. The server is a 2-socket Westmere-EP (Xeon 5600) system running Linux 2.6:

# ab -n 1000000 -c 24 http://172.28.8.250/
 This is ApacheBench, Version 2.0.40-dev apache-2.0
 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking 172.28.8.250 (be patient)
 Completed 100000 requests
 Completed 200000 requests
 Completed 300000 requests
 Completed 400000 requests
 Completed 500000 requests
 Completed 600000 requests
 Completed 700000 requests
 Completed 800000 requests
 Completed 900000 requests
 Finished 1000000 requests
Server Software: Apache/2.2.3
 Server Hostname: 172.28.8.250
 Server Port: 80
Document Path: /
 Document Length: 3985 bytes
Concurrency Level: 24
 Time taken for tests: 62.656311 seconds
 Complete requests: 1000000
 Failed requests: 0
 Write errors: 0
 Non-2xx responses: 1000002
 Total transferred: 4183008366 bytes
 HTML transferred: 3985007970 bytes
 Requests per second: 15960.08 [#/sec] (mean)
 Time per request: 1.504 [ms] (mean)
 Time per request: 0.063 [ms] (mean, across all concurrent requests)
 Transfer rate: 65196.45 [Kbytes/sec] received
 Connection Times (ms)
 min mean[+/-sd] median max
 Connect: 0 0 0.0 0 1
 Processing: 0 1 4.4 1 885
 Waiting: 0 0 4.5 0 884
 Total: 0 1 4.4 1 885
Percentage of the requests served within a certain time (ms)
 50% 1
 66% 1
 75% 1
 80% 1
 90% 2
 95% 3
 98% 4
 99% 5
 100% 885 (longest request)

No tuning, out of the box gives me 15960.08 HTTP requests/second. And the CPU? Well, it peaked at 66% idle, or 34% utilized.  Of course this is SMT (hypterthreaded) so 34% is ~8 processor threads or ~4 cores:

# vmstat 3
 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 2 0 127816 259408 230712 47307252 0 0 92 2 0 0 0 0 100 0 0
 0 0 127816 259400 230712 47307352 0 0 0 0 1105 432 0 0 100 0 0
 0 0 127816 259408 230712 47307352 1 0 1 12 1074 330 0 0 100 0 0
 0 0 127816 259392 230712 47307352 0 0 0 0 1114 441 0 0 100 0 0
 5 0 127816 259044 230712 47308164 0 0 0 8 7439 23103 4 4 92 0 0
 10 0 127816 237848 230728 47318736 1 0 1 4963 7509 94804 17 14 70 0 0
 2 0 127816 244196 230736 47331552 0 0 0 48 6944 69394 16 12 72 0 0
 6 0 127816 231328 230744 47330476 0 0 0 5927 10704 62483 13 10 78 0 0
 14 0 127816 227280 230756 47307412 1 0 1 0 5756 85341 15 13 72 0 0
 18 0 127816 239084 230764 47306240 0 0 0 6467 6003 83381 15 13 72 0 0
 10 0 127816 237880 230780 47306440 1 0 1 6036 6257 82766 15 12 73 0 0
 1 0 127816 268676 230784 47302892 0 0 0 0 7258 42912 9 7 85 0 0
 9 0 127816 259512 230788 47307268 0 0 0 3169 19715 31181 8 6 85 0 0
 6 0 127816 236700 230804 47319992 1 0 1 3 5636 106346 19 15 66 0 0
 3 0 127816 240576 230816 47330944 0 0 1 7223 12153 49646 12 9 79 0 0
 2 0 127816 229308 230828 47329748 0 0 0 5669 8747 87512 15 12 72 0 0
 3 0 127816 241880 230840 47332748 1 0 1 1 6669 77569 16 13 71 0 0
 7 0 127816 229984 230848 47327988 0 0 0 5672 13008 56876 13 10 77 0 0
 8 0 127816 235400 230864 47330736 0 0 0 51 5411 101470 18 15 67 0 0
 2 0 127816 246684 230868 47328748 1 0 1 7179 12156 34923 9 7 84 0 0
 8 0 127816 234116 230880 47325008 0 0 0 3977 13480 55217 12 9 79 0 0
 10 0 127816 229848 230888 47301592 1 0 1 0 5712 86969 15 13 72 0 0
 9 0 127816 242936 230900 47302292 0 0 0 6248 5907 82740 15 13 72 0 0
 7 0 127816 234464 230908 47312100 0 0 0 0 6211 81525 14 12 74 0 0
 3 0 127816 260600 230916 47310580 1 0 1 6019 10948 49947 10 8 81 0 0
 0 0 127816 252816 230920 47314964 0 0 0 2847 11985 19545 5 4 90 0 0
 0 0 127816 252840 230920 47316000 0 0 0 0 1078 358 0 0 100 0 0
 0 0 127816 252952 230920 47316000 11 0 11 15 1108 426 0 0 100 0 0

With 3,990 HTTP requests per second/core I only need 25 Westmere-EP cores to “handle Facebook’s HTTP request workload.”  That’s only  13 2U servers.

It’s both absurd, and insulting to Facebook, to suggest a couple of Exalogic systems can do anything worthwhile under Facebook’s insanely huge web workload. Since Facebook opted not to replace all their servers with a few tiles of Exalogic goodies, what did they do? Well, they built a behemoth data center in my part of the world. These are panorama shots so click and drag right to left:

Thanks to the Scobleizer for that. I’ve seen the outside of this building while driving by. If you read about the state of the art environmental engineering these folks put into this facility you’ll know why I’d love to get in there to see it—and you’ll certainly know why the odds are, um, rather slim that Exalogic could replace even a partial fragment of a fraction of a percentage of one-thenth of one side of a single aisle of this datacenter.

So, no, Facebook didn’t really replace hundreds of metric tonnes of hardware with two 42U racks worth of Exalogic Elastic Cloud. I meant to say Facebook apparently didn’t get the memo.

Summary
Keynotes are interesting. Some are memorable.

6 Responses to “Facebook Adopts Oracle Exalogic Elastic Cloud, Replaces Tens Of Thousands Of Servers. Memorable Conference Keynote Addresses.”


  1. 1 Luis Moreno Campos August 16, 2011 at 11:48 pm

    Always provocative and that’s why the likes of me read your posts until the end ;-)
    But let me add that Keynote Speeches also live of images and I’ve listen to quite a few in my life and that one you were referring to has also engraved in my mind the same image that did in yours: Facebook running on two Exalogic boxes. You took it literally and that originated a great blog post, but most of the times we shouldn’t. Just leave the image be, there, in the right side of the brain.

    • 2 kevinclosson August 17, 2011 at 1:33 am

      Ah Luis, my friend. I accept your feedback.

      I may not remember much of that conference in San Diego (with President George H.W. Bush), but I sure remember our conference in Berlin! I pumped up too many of you guys with solid Exadata knowledge! :-)

  2. 3 Jan August 17, 2011 at 9:50 am

    Hi Kevin,

    260,000,000 page loads per day / 24 / 60 / 60 = 3,009 per second.
    Not sure where yours 100,000 per second comes from.

    Otherwise I like your summary.

    Jan

    • 4 kevinclosson August 17, 2011 at 2:23 pm

      Hi Jan,

      I got email on that too…I need to fix the post a bit. The number is actually 260,000 million per month ( 260 * 10^9 as per my link to that CNET article). So I had the 100,000/second right but misstated the period. I’ll clean up the post.

      mygpdb=# select (260 * 10^ 9) / (30 * 24 * 60 * 60) ;
      ?column?
      ——————
      100308.641975309
      (1 row)

  3. 5 matt@hogstrom.org September 3, 2011 at 4:40 am

    Simply serving up static content is probably not the best way to characterize the Facebook workload. Apache Bench ain’t a Facebook user.

    Larry is always good for telling a story that is compelling, if not always entirely accurate technically.


  1. 1 Log Buffer #234, A Carnival of the Vanities for DBAs | The Pythian Blog Trackback on August 19, 2011 at 1:54 pm

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




EMC Employee Disclaimer

The opinions and interests expressed on EMC employee blogs are the employees' own and do not necessarily represent EMC's positions, strategies or views. EMC makes no representation or warranties about employee blogs or the accuracy or reliability of such blogs. When you access employee blogs, even though they may contain the EMC logo and content regarding EMC products and services, employee blogs are independent of EMC and EMC does not control their content or operation. In addition, a link to a blog does not mean that EMC endorses that blog or has responsibility for its content or use.

This disclaimer was put into place on March 23, 2011.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,058 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2013. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

Follow

Get every new post delivered to your Inbox.

Join 2,058 other followers

%d bloggers like this: