BLOG UPDATE 2011.08.17: In the original I misstated the Facebook page view rate to be 260,000,000 per day when it is in fact 260,000,000,000 per month. I used the correct per-second rate anyway so this update does not change the original in that regard.
BLOG UPDATE 2011.08.16: The blog title is a come-on.
Conference Keynotes: One Year Ago versus Eleven Years Ago
I spent some time rummaging through my scrapbook over the weekend. I stumbled on a program for the IBM Partnerworld 2000 conference. It might be difficult to see, but I scanned it in to show that on the left hand side there is an announcement of the keynote session with (former) U.S. President George H.W. Bush and on the bottom of the right hand page is an announcement of the session I was offering the same day. That was a really cool conference! The audience (certainly me included) thoroughly enjoyed the keynote. That was eleven years ago but I recall that my session was a bunch of blah-blah-blah compared the keynote.
There was another keynote, only one year ago, that is burned much more clearly into my memory. The conference was Oracle Openworld 2010 and, of course, the keynote speaker was Larry Ellison. Unlike the IBM conference of long ago I had no speaking session. I certainly would have had a session, due to the overwhelming number of votes racked up by my proposed “Oracle Mix Suggest-a-Session” (105 votes), but the Oracle Mix/Conference folks decided, but didn’t share the fact until afterwards, that Oracle employees weren’t eligible to participate in the Suggest-A-Session program. Boo hoo. Attendees missed a lot because the session topic was Do-It-yourself Exadata-level Performance.
Do It Yourself Exadata-Level Performance
A DIY Exadata-level performance configuration is not what I’m blogging about. However, it seems I just stepped in it. Quote me on this:
If you configure a lot of I/O bandwidth and a lot of CPU you can match Exadata performance with Real Application Clusters and conventional hardware.
…but I’m not blogging about that.
Big Sounding Claims
So, back to the comparison between the circa-2000 IBM Partnerworld keynote and last year’s Openworld keynote. The majority of fanfare last year was Exalogic. I remember this Larry Ellison sound bite :
A single setup is capable of handling 1 million HTTP requests per second; two running side-by-side could handle Facebook’s HTTP request workload
For those who don’t know, the following is true:
- A 2-rack Exalogic configuration has 720 Westmere EP (Xeon 5600) cores.
- Over 18 months ago, Facebook was sustaining 260,000,000,000 page loads per month.
That’s roughly 100,000 per second (260 * 10^9)/(30*24*60*60). So, the claims of a two-rack Exalogic configuration handling Facebook’s “HTTP request workload” is actually quite true. So why am I blogging about it then? The numbers are indeed stunning! There is a huge difference between the “HTTP request workload” and serving meaningful content. But I’m not blogging about that.
Those Facebook Engineers are Real Dummies
Well, they must be, if they were using “thousands and thousands” of servers to do what they were doing even back in 2008:
Because we have thousands and thousands of computers, each running a hundred or more Apache processes […]
Silly Facebook engineers. They could save so much space, power and cooling if they only listened to certain keynote wisdom.
Your Choice: A Two-Rack Exalogic Configuration Or 25 Itsy-Bitsy Xeon 5600 Cores
If A two-rack Exalogic configuration can supplant all the processing power of Facebook’s web serving farm, I have to do the math. I won’t argue that Exalogic can perform “1 millions HTTP requests per second.” Because it surely can. I will, however, focus on just how much Westmere-EP CPU it takes to do the 100,000 HTTP requests/sec it would take to handle just the request portion of Facebook’s workload.
I think I’ll fire up httpd on one of my Linux servers and crank up ab(8) (ApacheBench) to see how much CPU is takes to handle a given rate of requests. The workload is pulling about 4K for each request. The server is a 2-socket Westmere-EP (Xeon 5600) system running Linux 2.6:
# ab -n 1000000 -c 24 http://172.28.8.250/ This is ApacheBench, Version 2.0.40-dev apache-2.0 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright 2006 The Apache Software Foundation, http://www.apache.org/ Benchmarking 172.28.8.250 (be patient) Completed 100000 requests Completed 200000 requests Completed 300000 requests Completed 400000 requests Completed 500000 requests Completed 600000 requests Completed 700000 requests Completed 800000 requests Completed 900000 requests Finished 1000000 requests Server Software: Apache/2.2.3 Server Hostname: 172.28.8.250 Server Port: 80 Document Path: / Document Length: 3985 bytes Concurrency Level: 24 Time taken for tests: 62.656311 seconds Complete requests: 1000000 Failed requests: 0 Write errors: 0 Non-2xx responses: 1000002 Total transferred: 4183008366 bytes HTML transferred: 3985007970 bytes Requests per second: 15960.08 [#/sec] (mean) Time per request: 1.504 [ms] (mean) Time per request: 0.063 [ms] (mean, across all concurrent requests) Transfer rate: 65196.45 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 1 Processing: 0 1 4.4 1 885 Waiting: 0 0 4.5 0 884 Total: 0 1 4.4 1 885 Percentage of the requests served within a certain time (ms) 50% 1 66% 1 75% 1 80% 1 90% 2 95% 3 98% 4 99% 5 100% 885 (longest request)
No tuning, out of the box gives me 15960.08 HTTP requests/second. And the CPU? Well, it peaked at 66% idle, or 34% utilized. Of course this is SMT (hypterthreaded) so 34% is ~8 processor threads or ~4 cores:
# vmstat 3 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 127816 259408 230712 47307252 0 0 92 2 0 0 0 0 100 0 0 0 0 127816 259400 230712 47307352 0 0 0 0 1105 432 0 0 100 0 0 0 0 127816 259408 230712 47307352 1 0 1 12 1074 330 0 0 100 0 0 0 0 127816 259392 230712 47307352 0 0 0 0 1114 441 0 0 100 0 0 5 0 127816 259044 230712 47308164 0 0 0 8 7439 23103 4 4 92 0 0 10 0 127816 237848 230728 47318736 1 0 1 4963 7509 94804 17 14 70 0 0 2 0 127816 244196 230736 47331552 0 0 0 48 6944 69394 16 12 72 0 0 6 0 127816 231328 230744 47330476 0 0 0 5927 10704 62483 13 10 78 0 0 14 0 127816 227280 230756 47307412 1 0 1 0 5756 85341 15 13 72 0 0 18 0 127816 239084 230764 47306240 0 0 0 6467 6003 83381 15 13 72 0 0 10 0 127816 237880 230780 47306440 1 0 1 6036 6257 82766 15 12 73 0 0 1 0 127816 268676 230784 47302892 0 0 0 0 7258 42912 9 7 85 0 0 9 0 127816 259512 230788 47307268 0 0 0 3169 19715 31181 8 6 85 0 0 6 0 127816 236700 230804 47319992 1 0 1 3 5636 106346 19 15 66 0 0 3 0 127816 240576 230816 47330944 0 0 1 7223 12153 49646 12 9 79 0 0 2 0 127816 229308 230828 47329748 0 0 0 5669 8747 87512 15 12 72 0 0 3 0 127816 241880 230840 47332748 1 0 1 1 6669 77569 16 13 71 0 0 7 0 127816 229984 230848 47327988 0 0 0 5672 13008 56876 13 10 77 0 0 8 0 127816 235400 230864 47330736 0 0 0 51 5411 101470 18 15 67 0 0 2 0 127816 246684 230868 47328748 1 0 1 7179 12156 34923 9 7 84 0 0 8 0 127816 234116 230880 47325008 0 0 0 3977 13480 55217 12 9 79 0 0 10 0 127816 229848 230888 47301592 1 0 1 0 5712 86969 15 13 72 0 0 9 0 127816 242936 230900 47302292 0 0 0 6248 5907 82740 15 13 72 0 0 7 0 127816 234464 230908 47312100 0 0 0 0 6211 81525 14 12 74 0 0 3 0 127816 260600 230916 47310580 1 0 1 6019 10948 49947 10 8 81 0 0 0 0 127816 252816 230920 47314964 0 0 0 2847 11985 19545 5 4 90 0 0 0 0 127816 252840 230920 47316000 0 0 0 0 1078 358 0 0 100 0 0 0 0 127816 252952 230920 47316000 11 0 11 15 1108 426 0 0 100 0 0
With 3,990 HTTP requests per second/core I only need 25 Westmere-EP cores to “handle Facebook’s HTTP request workload.” That’s only 13 2U servers.
It’s both absurd, and insulting to Facebook, to suggest a couple of Exalogic systems can do anything worthwhile under Facebook’s insanely huge web workload. Since Facebook opted not to replace all their servers with a few tiles of Exalogic goodies, what did they do? Well, they built a behemoth data center in my part of the world. These are panorama shots so click and drag right to left:
- Peer down a single row of servers in the Facebook Prineville Datacenter.
- Row, after row, after row of tall racks of servers — in one section of the Facebook Prineville Datacenter.
Thanks to the Scobleizer for that. I’ve seen the outside of this building while driving by. If you read about the state of the art environmental engineering these folks put into this facility you’ll know why I’d love to get in there to see it—and you’ll certainly know why the odds are, um, rather slim that Exalogic could replace even a partial fragment of a fraction of a percentage of one-thenth of one side of a single aisle of this datacenter.
So, no, Facebook didn’t really replace hundreds of metric tonnes of hardware with two 42U racks worth of Exalogic Elastic Cloud. I meant to say Facebook apparently didn’t get the memo.
Summary
Keynotes are interesting. Some are memorable.
Kevin Closson's Blog: Platforms, Databases and Storage 
Always provocative and that’s why the likes of me read your posts until the end 😉
But let me add that Keynote Speeches also live of images and I’ve listen to quite a few in my life and that one you were referring to has also engraved in my mind the same image that did in yours: Facebook running on two Exalogic boxes. You took it literally and that originated a great blog post, but most of the times we shouldn’t. Just leave the image be, there, in the right side of the brain.
Ah Luis, my friend. I accept your feedback.
I may not remember much of that conference in San Diego (with President George H.W. Bush), but I sure remember our conference in Berlin! I pumped up too many of you guys with solid Exadata knowledge! 🙂
Hi Kevin,
260,000,000 page loads per day / 24 / 60 / 60 = 3,009 per second.
Not sure where yours 100,000 per second comes from.
Otherwise I like your summary.
Jan
Hi Jan,
I got email on that too…I need to fix the post a bit. The number is actually 260,000 million per month ( 260 * 10^9 as per my link to that CNET article). So I had the 100,000/second right but misstated the period. I’ll clean up the post.
mygpdb=# select (260 * 10^ 9) / (30 * 24 * 60 * 60) ;
?column?
——————
100308.641975309
(1 row)
Simply serving up static content is probably not the best way to characterize the Facebook workload. Apache Bench ain’t a Facebook user.
Larry is always good for telling a story that is compelling, if not always entirely accurate technically.