If you are a xenophobe or an ostrich with your head in the sand I encourage you not to watch this video.
Archive Page 29
The young’n has caught the blogging itch. Fellow OakTable Network member Tanel Poder has started blogging at:
I’m looking forward to good stuff there.
Note, the “young’n” bit is an inside joke. Tanel is brilliant and his blog should be a good read.
A Good Blog Post About Monitoring Oracle Over NFS
Published June 14, 2007 NFS Performance , Oracle NAS , Oracle NFS , Oracle on NFS , Oracle over NFS 3 CommentsI’d like to give a shout out to a very good blog post about monitoring Oracle on NFS by Jeremy Schneider.
I won’t blog at all about the actual array they are doing this to because I have never tested one. In the video a bullet is fired through the SAN array and it is portrayed to continue operations. The part of the video I like is the disclaimer at the end.
Whew, close call for the goldfish. Can you imagine the anxiety it felt not knowing it was to be scooped off to safety? No worries, there is treatment. Perhaps preventive treatment would be more effective?
1.2 Transactions Per Second! Enterprise Software is Infinitely Partitionable
Published June 11, 2007 General Oracle Topics , MySQL 6 CommentsI read a post on blogs.zdnet.com about MySQL that I think was interesting. In the post, Dana Blankenhorn is posing that MySQL is “enterprise class” using the Booking.com deployment as case-in-point.
What is “Enterprise Class?”
The post got me thinking. What is “Enterprise Class” anyway? Is it any software used in any enterprise datacenter? I tend to think of an enterprise class database server as one that can vertically scale to exploit the largest servers in support of a single, large application. Using those criteria leaves MySQL out I should think. Or am I behind the times on that? Are there any single MySQL databases running on a 64CPU Superdome for instance? It appears as though MySQL is supported on Itanium HP-UX for 2-processor systems.
Enterprise MySQL
In this computerworlduk.com article, it looks as though Booking.com uses something like 20 MySQL database servers to handle “tens of thousands” of bookings for 30,000 hotels spanning some 8,000 destinations. Let’s say for the sake of argument that it is 20 database servers and “tens of thousands” is 100,000. I admit I don’t know anything about the richness of this application, but I don’t see anything too brutal here. These sorts of applications lend themselves to partitioning naturally. It wouldn’t surprise any of us Oracle types to find out that they partition based upon hotel. That seems like a natural line to partition on. If that is the case, I get 1,500 hotels per database server handling their fair share of about 1.2 transactions per second (100,000/86,400 seconds in a day). I know these things are not that simple, but folks, we are talking about 20 database servers. Even if they are 2-socket/dual core systems you’ve got some 80 cores to work with! At first glance it just doesn’t seem as though these systems would be working that hard. And MySQL? Well, it doesn’t have to work that hard at all since the workload is partitionable. Who knows, maybe all workloads are partitionable and we Oracle-types are just missing the ball. Anyway, I can’t seem to find what storage engine is being used at Booking.com. And speaking of MySQL storage engines…
A 3-legged Pink Elephant
If you’re interested in 3-legged pink elephants, I’ve got one for you since we are on the topic of MySQL. In computerworlduk.com article we find that MySQL announced support for MySQL on IBM System I (yes, OS-400) with DB2 as the storage engine. Wow, that would be weird. Or it seems so at least.
What’s this Really Have to do with Oracle?
Oracle Database can do everything MySQL can do. The opposite is not true. ‘Nuff said. Oh, did I mention that Oracle Corporation is not a “Database Company” anymore. They’ve got the database now they are getting everything else.
SAN Admins: Please Give Me As Much Capacity From As Few Spindles As Possible!
Published June 10, 2007 I/O Topics , Jim Grey , NAND Flash SSD , oracle , Oracle I/O Performance 13 CommentsI was catching up on my mojo reading when I caught a little snippet I’d like to blog about. Oh, by the way, have I mentioned recently that StorageMojo is one of my favorite blogs?
In Robin Harris’ latest installment about ZFS action at Apple, he let out a glimpse of one of his other apparent morbid curiosities—flash. Just joking, I don’t think ZFS on Mac or flash technology are morbid, it just sounded catchy. Anyway, he says:
I’ve been delving deep into flash disks. Can you say “weird”? My take now is that flash drives are to disk drives what quantum mechanics is to Newtonian physics. I’m planning to have something out next week.
I look forward to what he has to say. I too have a great interest in flash.
Now, folks, just because we are Oracle-types and Jim Grey was/is a Microsoft researcher, you cannot overlook what sorts of things Jim was/is interesting in. Jim’s work has had a huge impact on technology over the years and it turns out that Jim took/takes an interest in flash technology with servers in mind. Just the abstract of that paper makes it a natural must-read for Oracle performance minded individuals. Why? Because it states (with emphasis added by me):
Executive summary: Future flash-based disks could provide breakthroughs in IOps, power, reliability, and volumetric capacity when compared to conventional disks.
Yes, IOps! Nothing else really matter where Oracle database is concerned. How can I say that? Folks, round-brown spinning things do sequential I/O just fine—naturally. What they don’t do is random I/O. To make it worse, most SAN array controllers (you know, that late 1990’s technology) pile on overhead that further choke off random I/O performance. Combine all that with the standard IT blunder of allocating space for Oracle on a pure capacity basis and you get the classic OakTable Network response:
Attention DBAs, it’s time for some déjà vu. I’ll state with belligerent repetition, redundantly, over and over, monotonously reiterating this one very important recurrent bit of advice: Do everything you can to get spindles from your storage group—not just capacity.
Flash
Yes that’s right, it wont be long (in relative terms) until you see flash memory storage fit for Oracle databases. The aspect of this likely future trend that I can’t predict, however, is what impact such technology would have on the entrenched SAN array providers. Will it make it more difficult to keep the margins at the levels they demand, or will flash be the final straw that commoditizes enterprise storage? Then again, and Jim Grey points out in that paper, flash density isn’t even being driven by the PC—and most certainly not enterprise storage—ecosystem. The density is being driven by consumer and mobile applications. Hey, I want my MTV. Um, like all of it, crammed into my credit-card sized mpeg player too.
When?
When it gets cheaper and higher capacity of course. Well, its not exactly that simple. I went spelunking for that Samsung 1.8” 32GB SSD and found two providers with street price of roughly USD $700.00 for 32GB here and here. In fact, upon further investigation, Ritek may soon offer a 32GB device at some $8 per GB. But let’s stick with current product for the moment. At $22 per GB, we’re not exactly talking SATA which runs more on the order of $.35 per GB. But then we are talking enterprise applications here, so a better comparison would be to Fibre drives which go for about $3-$4 per GB.
Now that is interesting since Jim Grey pointed out that in-spite of some industry predictions setting the stage for NAND to double every year, NAND had in fact gained 16 fold in 4 years–off by year. If that pace continues, could we really expect 512GB 1.8″ SSD devices in the next 4 years? And would the price stay relatively constant yielding a cost of something like $1.35 per GB? Remember, even the current state of the art (e.g., the Samsung 1.8″ 32GB SSD) delivers on the order of 130,000 random single-sector IOps–that’s approximately 7usec latency for a random I/O. At least that is what Samsung’s literature claims. Jim’s paper, on the other hand reports grim current art performance when measured with DskSpd.exe:
The story for random IOs is more complex – and disappointing. For the typical 4-deep 8KB random request, read performance is a spectacular 2,800 requests/second but write performance is a disappointing 27 requests/second.
The technology is young and technically superior, but there is work to do in getting the most out of NSSD as the paper reports. Jim suspects that short term quick fixes could be made to bring the random I/O performance for 8KB transfers on today’s NSSD technology up to about 1,700 IOps split evenly between read and write. Consider, however, that real world applications seldom exhibit a read:write ratio of 50:50. Jim generalized on the TPC-C workload as a case in point. It seems with “some re-engineering” (Jim’s words) even today’s SSD would be a great replacement for hard drives for typical Oracle OLTP workloads since you’ll see more 70:30 read:write ratios in the real world. And what about sequential writes? Well, there again, even today’s technology can handle some 35MB/s of sequential writes so direct path writes (e.g., sort spills) and redo log writes would be well taken care of. But alas, the $$/GB is still off. Time will fix that problem and when it does, NSSD will be a great fit for databases.
Don’t think for a moment Oracle Corporation is going to pass up on enabling customers to exploit that sort of performance–with or without the major SAN vendors.
But flash burns out, right? Well, yes and no. The thing that matters is how long the device lasts-the sum of its parts. MTBF numbers are crude, but Samsung sticks a 1,000,000hr MTBF on this little jewel-how cool.
Well, I’ve got the cart well ahead of the horse here for sure because it is still too expensive, but put it on the back burner, because we aren’t using Betamax now and I expect we’ll be using fewer round-brown spinning things in the span of our careers.
More Linux Distributions Please! Balkanization Improves Oracle IT!
Published June 7, 2007 Linux balkanization 8 CommentsSometimes small problems really upset me. I have a test harness that nests shell scripts deeply. Why not—that is why fork() and exec() exist after all. I was getting odd failures deep down in the bowels of the harness that were a bit tricky to track down.
This particular test harness was failing with Segmentation fault on SuSE SLES 9 and Memory fault on Red Hat Enterprise Linux RHEL4. Same problem just different error text. When I found the problem I started to look into it on another test system that happened to be Fedora Core 5. I was frustrated to find that the problem did not happen on Fedora, but did on 32 and 64 bit releases of both RHEL4 and SLES9. Ugh.
The problem rested in the first 2 lines of a simple shell script. What was the offending code? Get this:
#!/bin/bash
time
Wow, frightening stuff! In the first line of some stupid shell script I happened to execute the shell built-in time without timing anything, which was a goof on my part. But really, bash seg faults because the time built-in doesn’t have an arg and only when that is the first line of the script? As long as time is the first command in the script it causes bash to freak out—regardless of what follows. Notice in the following screen shot how simply putting the shell built-in : as the first line of code—after forcing bash—alleviates the problem. Notice also that the problem didn’t happen on Fedora.
Balkanization
Oh how I hate problems like this—simple, stupid little shell scripts behaving differently on one Linux distro than on another. Yes, I know it isn’t the shell script that is technically behaving differently—it’s the implementation of bash. And if there were symbols in /bin/bash (it’s stripped) I’d tunnel-rat my way through gdb to see why it is freaking out, however, with limited information I can only determine the routine, which is called execute_command_internal().
I also know this is not technically a Linux issue since bash is just GNU stuff. To that end, some googling reveals that this bug has been reported in the Solaris camp in Open Solaris Bug 6328339 which was opened back in 2005. Even though this is technically a GNU issue I still get varying behavior from different Linux distributions which I believe is a by-product of balkanization.
I thought Linux was going to save us from all the balkanization we supposedly suffered under Unix rule. This is just a simple example of balkanization. We endured much worse side-affects of Linux balkanization with the pre-2.6 Kernel virtual memory fiasco. Some my recall the ill-fated clash of the virtual memory titans featuring the Archangelites opposing the Van Rielians—a war with a large number of customer casualties. Going back further, I recall plenty of workloads (most notably The Tens) that would completely crush one Linux distribution but function just fine with the other. Indeed, during that project we tested both RH2.1 and SLES7 and stayed with the one that could hold up under pressure—and if you read the paper you can envision the pressure!
How many Linux distributions are there now? Does Linux balkanization make life better for the average Oracle IT shop?
Every Release of Oracle Database is “The Best Ever”, Right? Enter Oracle11g!
Published June 6, 2007 Oracle11g 10 Comments
I see Eddie Awad has hit the press about the July 11 launch of Oracle11g. From everything I’ve seen in this release there should be no technical reasons holding back customers’ adoption. There are certain folks out there that say every release of Oracle “is the best I’ve seen yet.” A good way to sell books for sure, but I’m not that way. I most certainly didn’t say that about Oracle 7.2! Anyway, I have been thoroughly impressed with my testing—most particularly in the area of stability. And, as I keep hinting, there are features that neither Rich Niemiec, Mike Ault nor Don Burleson have been discussing that I think will be very attractive—especially in the commodity computing space. Unfortunately I have to remind myself of the real world and how long it takes to get applications qualified for a release of the server. Let’s hope E-Biz gets there as soon as possible.
Blogging or Bashing Oracle for Fun, not Profit. Got Cheezburger?
Published June 6, 2007 Oracle blogging , Oracle Web 2.0 16 CommentsI stood on the sidelines of this thread too long. Today, Justin Kestelyn made another post about the Oracle blogging community. This thread goes back to these original posts where Justin posed that although there is a lot of blogging activity around Oracle, there doesn’t seem to be the same Web 2.0 buzz that someone like Robert Scoble would take notice of. Fellow OakTable Network member Doug Burns stepped in with this post. So what’s my take?
Aristocracy or Meritocracy?
I started this blog in October 2006 and at one point found a reference to my blog on blogs.oracle.com. Not that it generated any traffic, but I thought that was interesting because I didn’t ask to get a reference there. But the fact that it generated no appreciable traffic to my site is what I think Scoble is talking about. When I think about it, it seems my blog more than deserves at least a link from blogs.oracle.com, the bigger question is what criteria goes into that blogroll? Is it aristocracy, or meritocracy?
These days when I find myself sitting with Vice Presidents or members of the technical staff in Oracle Server Technologies Division (ST) or doing something like writing a jointly produced whitepaper with ST (as I am right now on a cool Oracle11g feature), I wonder why there are those small circles of relative late-comers to this Oracle stuff that mistake me for being an “Oracle-basher.” Folks, I spent an entire decade as a member of a small team of platform engineers optimizing Oracle at the port level for improved SMP, and later, NUMA. I also participated in the most important benchmarks that made Oracle money in the 1990s—customer-defined benchmarks where bake-offs between, say, Informix PDQ and IBM SP2 or Teradata or Sybase were at stake. I spent so much time in building 400 (Server Technologies) of Oracle’s HQ that I maintained a fully furnished apartment right down the street on Marine Parkway. I’m an Oracle basher? No.
Let’s say I was to state matter-of-factly that Miscrosoft Windows 3.1 was a complete pile of garbage. Does that actually make me a Microsoft basher? No, it simply means that there was an offering from Microsoft that I didn’t like. Big deal. I wasn’t much of a fan of Oracle SQL*Calc either—and neither was anyone else so Oracle discontinued it. I bet there will be no more than about 42 readers of this blog that even remember SQL*Calc.
Automatic Storage Management. Bashing?
I have taken a position that in its current form, Automatic Storage Management (ASM) is often times over-positioned. Let me be clear about this. I have never taken a stand against any Oracle revenue-generating product. It turns out that ASM is optional software that eases storage management pains most common to SAN environments that are also devoid of any optional software such as clustered filesystems. Indeed, install Oracle10g sometime and pay close attention to the fact that the default DBCA placement for a Real Application Clusters database is in fact cluster filesystem. You have to cursor down to select ASM. What is my point? My point is that either way the customer using DBCA with RAC has already paid Oracle the same amount of money regardless of where they put their database—whether in the default locale of cluster filesystem or ASM.
ASM is routinely referred to as a “replacement for filesystems and volume managers.” That is incorrect. You still have to install Oracle and do things like imp/exp, SQL*Loader, BFILE/UTIL_FILE, logging, trace, scripts, etc, etc, etc. Until such time as ASM is a part of a fully-baked general purpose filesystem—which anyone skilled in the reading of tea leaves should easily be able to foresee—I prefer NFS. And, get this, Oracle makes the same amount of money when you deploy on NAS (NFS) as they do if you choose iSCSI or FCP with CFS or ASM. These choices don’t affect Oracle’s bottom line. I make my points about my preference for NFS over block protocols (and therefore ASM) in this set of postings. No, I am not an Oracle basher. But do my views fit in the aristocracy that blogs.oracle.com seems to be? It doesn’t seem so.
More on ASM
I am excited about where ASM will make a showing in the future. It is a component of bigger and better things that I cannot discuss openly. In those future technology offerings, ASM will perform wonderfully and provide vital functionality—and I’m not just talking about some passé short-term market vision like displacing Veritas VxVM from Oracle implementations. There are much bigger and better things ahead for ASM…’nuff said. In the meantime, choose success, go with NFS. Ok, I’m off that soapbox. Back to this Web 2.0 mystery.
The Popularity Contest
So if Justin is honestly concerned about building a vibrant Web 2.0 community, it seems blogs.oracle.com should be more in tune with the readership. Let’s consult Technorati.
In defense of Oracle’s Web 2.0 community, I saw Justin mention blogs such as Steve Chan’s blog and Doug Burns’ blog. I think I’ll walk through Technorati for those and, of course, the ueberblog (where’s my umlaut?): Jonathan Lewis’ blog. To spice things up, how about comparing to one of my favorite blogs, StorageMojo.com. Of course I’ll have to include Scobleizer to show Web 2.0 weight. Be aware that of the following blogs, Jonathan Lewis’ and mine are the youngest blogs—by a long shot.
Proper Perspective
Now, just to put things into perspective, consider a true Web 2.0 phenomenon: I CAN HAS CHEEZBURGER. This site—which has a Technorati authority of 5,025 and rank of 98—is proof positive that the Internet and social networking are as mainstream as Pet Trusts(for real), designer pet supplies, fluffy with a sniffle, pets with stress and of course Barbi with a scooper.
Oracle on Opteron with Linux-The NUMA Angle (Part VII).
Published June 5, 2007 AMD Memory Latency , AMD Memory Throughput , AMD Quad-core , NUMA Oracle , Opteron NUMA , Opteron Oracle , Oracle Barcelona 5 CommentsThis installment in my series about Oracle on Linux with NUMA hardware is very, very late. I started this series at the end of last year and it just kept getting put off—mostly because the hardware I needed to use was being used for other projects (my own projects). This is the seventh in the series and it’s time to show some Oracle numbers. Previously, I laid groundwork about such topics as SUMA/NUMA, NUMA API and so forth. To make those points I relied on microbenchmarks such as the Silly Little Benchmark. The previous installments can be found here.
To bring home the point that Oracle should be run on AMD boxes in NUMA mode (as opposed to SUMA), I decided to pick an Oracle workload that is very easy to understand as well as processor intensive. After all, the difference between SUMA and NUMA is higher memory latency so testing at any level below processor saturation actually provides the same throughput-albeit the SUMA result would come at a higher processor cost. To that end, measuring SUMA and NUMA at processor saturation is the best way to see the difference.
The workload I’ll use for this testing is what my friend Anjo Kolk refers to as the Jonathan Lewis Oracle Computing Index workload. The workload comes in script form and is very straightforward. The important thing about the workload is that it hammers memory which, of course, is the best way to see the NUMA effect. Jonathan Lewis needs no introduction of course.
The test was set up to execute 4, 8 16 and 32 concurrent invocations of the JL Comp script. The only difference in the test setup was that in one case I booted the server in SUMA mode and in another I booted in NUMA mode and allocated hugepages. As I point out in this post about SUMA, hugepages are allocated in a NUMA fashion and booting an SGA into this memory offers at least crude fairness placement of the SGA pages—certainly much better than a Cyclops. In short, what is being tested here one case where memory is allocated at boot time in a completely round-robin fashion versus the SGA being quasi-round robin yet page tables, kernel-side process-related structures and heap are all NUMA-optimized. Remember, this is no more difficult than a system boot option. Let’s get to the numbers.
I have also rolled up all the statspack reports into a word document (as required by WordPress). The document is numa-statspack.doc and it consist of 8 statspacks each prefaced by the name of what the specific test was. If you pattern search for REPORT NAME you will see each entry. Since this is a simple memory latency improvement, you might not be surprised how uninteresting the stats are-except of course the vast improvement in the number of logical reads per second the NUMA tests were able to push through the system.
SUMA or NUMA
A picture speaks a thousand words. This simple test combined with this simple graph covers it all pretty well. The job complete time ranged from about 12 to 15 percent better with NUMA at each of the concurrent session counts. While 12 to 15% isn’t astounding, remember this workload is completely processor bound. How do you usually recuperate 12-15% from a totally processor-bound workload without changing even a single line of code? Besides, this is only one workload and the fact remains that the more your particular workload does outside the SGA (e.g., sorting, etc) the more likely you are to see improvement. But by all means, do not run Oracle with Cyclops memory.
The Moral of the Story
Processors are going to get more cores and slower clock rates and memory topologies will look a lot more NUMA than SUMA as time progresses. I think it is important to understand NUMA.
What is Oracle Doing About It?
Well, I’ve blogged about the fact that the Linux ports of 10g do not integrate with libnuma. That means it is not NUMA-aware. What I’ve tried to show in this series is that the world of NUMA is not binary. There is more to it than SUMA or NUMA-aware. In the middle is booting the server and database in a fashion that at least allows benefit from the OS-side NUMA-awareness. The next step is Oracle NUMA-awareness.
Just recently I was sitting in a developer’s office in bldg 400 of Oracle HQ talking about NUMA. It was a good conversation. He stated that Oracle actually has NUMA awareness in it and I said, “I know.” I don’t think Sequent was on his mind and I can’t blame him—that was a long time ago. The vestiges of NUMA awareness in Oracle 10g trace back to the high-end proprietary NUMA implementations of the 1990s. So if “it’s in there” what’s missing? We both said vgetcpu() at the same time. You see, you can’t have Oracle making runtime decisions about local versus remote memory if a process doesn’t know what CPU it is currently executing on (detection with less than a handful of instructions). Things like vgetcpu() seem to be coming along. That means once these APIs are fully baked, I think we’ll see Oracle resurrect intrinsic NUMA awareness in the Linux port of Oracle Database akin to those wildcat ports of the late 90s…and that should be a good thing.
Oracle Over NFS. I Need More Monitoring Tools? A Bonded NIC Roller Coaster.
Published June 5, 2007 NFS CFS ASM , Oracle I/O Performance , Oracle over NFS , Oracle Storage Related Problems 10 CommentsAs you can tell by my NFS-related blog entries, I am an advocate of Oracle over NFS. Forget those expensive FC switches and HBAs in every Oracle Database server. That is just a waste. Oracle11g will make that point even more clearly soon enough. I’ll start sharing how and why as soon as I am permitted. In the meantime…
Oracle over NFS requires bonded NICs for redundant data paths and performance. That is an unfortunate requirement that Oracle10g is saddled with. And, no, I’m not going to blog about such terms as IEEE 802.3ad, PAgP, LACP, balance-tlb, balance-alb or even balance-rr. The days are numbered for those terms-at least in the Oracle database world. I’m not going to hint any futher about that though.
Monitoring Oracle over NFS
If you are using Oracle over NFS, there are a few network monitoring tools out there. I don’t like any of them. Let’s see, there’s akk@da and Andrisoft WanGuard. But don’t forget Anue and Aurora, Aware, BasicState, CommandCenter NOC, David, Dummmynet, GFI LANguard, Gomez, GroundWork, Hyperic HQ, IMMonitor, Jiploo, Monolith, moods, Network Weathermap, OidView, Pandetix, Pingdom, Pingwy, skipole-monitor, SMARTHawk, Smarts, WAPT, WFilter, XRate1, arping, Axence NetVision, BBMonitor, Cacti, CSchmidt collection, Cymphonix Network Composer, Darkstat, Etherape, EZ-NOC, Eye-on Bandwidth, Gigamon University, IPTraf, Jnettop, LITHIUM, mrtg-ping-probe, NetMRG, NetworkActiv Scanner, NimTech, NPAD, Nsauditor, Nuttcp, OpenSMART, Pandora FMS, PIAFCTM, Plab, PolyMon, PSentry, Rider, RSP, Pktstat, SecureMyCompany, SftpDrive, SNM, SpeedTest, SpiceWorks, Sysmon, TruePath, Unbrowse, Unsniff, WatchMouse, Webalizer, Web Server Stress Tool, Zenoss, Advanced HostMonitor, Alvias, Airwave, AppMonitor, BitTorrent, bulk, BWCTL, Caligare Flow Inspector, Cittio, ClearSight, Distinct Network Monitor, EM7, EZMgt, GigaMon, Host Grapher II, HPN-SSH, Javvin Packet Analyzer, Just-ping, LinkRank, MoSSHe, mturoute, N-able OnDemand, Netcool, netdisco, Netflow Monitor, NetQoS, Pathneck, OWAMP, PingER, RANCID, Scamper, SCAMPI, Simple Infrastructure Capacity Monitor, Spirent, SiteMonitor, STC, SwitchMonitor, SysUpTime, TansuTCP, thrulay, Torrus, Tstat, VSS Monitoring, WebWatchBot, WildPackets, WWW Resources for Communications & Networking Technologies, ZoneRanger, ABwE, ActivXpets, AdventNet Web NMS, Analyse It, Argus, Big Sister, CyberGauge, eGInnovations, Internet Detective, Intellipool Network Monitor, JFF Network Management System, LANsurveyor, LANWatch, LoriotPro, MonitorIT, Nagios, NetIntercept, NetMon, NetStatus, Network Diagnostic Tool, Network Performance Advisor, NimBUS, NPS, Network Probe, NetworksA-OK, NetStat Live, Open NerveCenter, OPENXTRA, Packeteer, PacketStorm, Packetyzer, PathChirp, Integrien, Sniff’em, Spong, StableNet PME, TBIT, Tcptraceroute, Tping, Trafd, Trafshow, TrapBlaster, Traceroute-nanog, Ultra Network Sniffer, Vivere Networks, ANL Web100 Network Configuration Tester, Anritsu, aslookup, AlertCenter, Alertra, AlertSite, Analyse-it, bbcp, BestFit, Bro, Chariot, CommView, Crypto-PAn, elkMonitor, DotCom, Easy Service Monitor, Etherpeek, Fidelia, Finisar, Fpinger, GDChart, HipLinkXS, ipMonitor, LANExplorer, LinkFerret, LogisoftAR, MGEN, Netarx, NetCrunch, NetDetector, NetGeo, NEPM, NetReality, NIST Net, NLANR AAD, NMIS, OpenNMS PageREnterprise, PastMon, Pathprobe, remstats, RIPmon, RFT, ROMmon, RUDE, Silverback, SmokePing, Snuffle, SysOrb, Telchemy, TCPTune, TCPurify, UDPmon, WebAttack, Zabbix, AdventNet SNMP API, Alchemy Network Monitor, Anasil analyzer, Argent, Autobuf, Bing, Clink, DSLReports, Firehose, GeoBoy, PacketBoy, Internet Control Portal, Internet Periscope, ISDNwatch, Metrica/NPR, Mon, NetPredict, NetTest, Nettimer, Net-One-1, Pathrate, RouteView, sFlow, Shunra, Third Watch, Traceping, Trellian, HighTower, WCAT, What’s Up Gold, WS_FTP, Zinger, Analyzer, bbftp, Big Brother, Bronc, Cricket, EdgeScape, Ethereal (now renamed Wireshark), gen_send/gen_recv, GSIFTP, Gtrace, Holistix, InMon, NcFTP, Natas, NetAlly, NetScout, Network Simulator, Ntop, PingGraph, PingPlotter, Pipechar, RRD, Sniffer, Snoop, StatScope, Synack, View2000, VisualPulse, WinPcap, WU-FTPD, WWW performance monitoring, Xplot, Cheops, Ganymede, hping2, Iperf, JetMon, MeasureNet, MatLab, MTR, NeoTrace, Netflow, NetLogger, Network health, NextPoint, Nmap, Pchar, Qcheck, SAA, SafeTP, Sniffit, SNMP from UCSD, Sting, ResponseNetworks, Tcpshow, Tcptrace WinTDS, INS Net Perf Mgmt survey, tcpspray, Mapnet, Keynote, prtraceroute clflowd flstats, fping, tcpdpriv, NetMedic Pathchar, CAIDA Measurement Tool Taxonomy, bprobe & cprobe, mrtg, NetNow, NetraMet, Network Probe Daemon, InterMapper, Lachesis, Optimal Networks and last but not least, Digex.
Simplicity Please
The networking aspect of Oracle over NFS is the simplest type of networking imaginable. The database server issues I/O to NFS filesystems being exported over simple, age old Class C private networks (192.168.N.N). We have Oracle statspack to monitor what Oracle is asking of the filesystem. However, if the NFS traffic is being sent over a bonded NIC, monitoring the flow of data is important as well. That is also a simple feat on Linux since /proc tracks all that on a per-NIC basis.
I hacked out a very simple little script to monitor eth2 and eth3 on my system. It isn’t anything special, but it shows some interesting behavior with bonded NICS. The following screen shot shows the simple script executing. A few seconds after starting the script, I executed an Oracle full table scan with Parallel Query in another window. Notice how /proc data shows that the throughput has peaks and valleys on a per-second basis. The values being reported are Megabytes so it is apparent that the bursts of I/O are achieving full bandwidth of the GbE network storage paths, but what’s up with the pulsating action? Is that Oracle, or the network? I can’t tell you just yet. Here is the screen shot nonetheless:
For what it is worth, here is a listing of that silly little script. Its accuracy compensates for its lack of elegance. Cut and paste this on a Linux server and tailor eth[23] to whatever you happen to have.
$ cat ntput.sh
#!/bin/bash
function get_data() {
cat /proc/net/dev | egrep “${token1}|${token2}” \
| sed ‘s/^.*://g’ | awk ‘{ print $1 }’ | xargs echo
}
token1=eth2
token2=eth3
INTVL=1
while true
do
set – `get_data`
b_if1=$1
b_if2=$2
sleep $INTVL
set – `get_data`
a_if1=$1
a_if2=$2
echo $a_if1 $b_if1 | awk -v intvl=$INTVL ‘{
printf(“%7.3f\t”, (($1 – $2) / 1048576) / intvl) }’
echo $a_if2 $b_if2 | awk -v intvl=$INTVL ‘{
printf(“%7.3f\n”, (($1 – $2) / 1048576) / intvl) }’
done
Nobody’s Profect.
Published June 5, 2007 Bush Internets , Gore invented the Internet Leave a CommentHere is a chuckle from our local rag. I wonder if the editor learned about the internets from President Bush or Al Gore—the supposed self-proclaimed inventor of the internet—a claim mistakingly attributed to Gore.
When I first started blogging I was told that it is important to blog regularly-so I did. I’ve made over 170 posts that WordPress recons have been viewed some 200,000 times. But then my former company, PolyServe, got bought by HP. My blogging rate has plummeted, but the search engines seem to still be piling in the readers.
I am not blogging as frequently because, honestly, I can’t figure out the balance between political correctness and interesting content. That is, I’m a bit overwhelmed trying to assimilate into HP culture. I don’t know what I can safely blog about without getting someone upset in the eleventeen layers of management between me and Mark Hurd (HP CEO). I remember feeling this way back in 1999 when Sequent (where I had done port-level Oracle work for some 10 years) got bought by IBM.
The HP folks that maintain their blogroll invited me to complete a 30 minute training session on blogging at HP so they can list my blog there. I haven’t gotten around to that yet.
Old Topics
I still have the final installment on my series on NUMA in the bull-pen. I think I’ll pound that one out today or tomorrow. In the meantime, how about some fluff.
Geek-free Topic
Over the Memorial Day weekend, the family and I joined up with my age-old friend Tom for some camping on his family’s ranch. With 2005 acres to stretch out in, camping on his ranch is what I call peace and quiet. Wildlife viewing is unbelievable up there. We saw well over 200 elk, dozens of deer, troops of wild turkeys, coyotes and a few rattlesnakes. The water was a little low for fishing, but we picked up some trout anyway. I’ll drop some photos in here.
Oracle RDBMS Technology is Old. It Must Be Obsolete! Disk Drives Forever!
Published May 18, 2007 Oracle I/O Performance , Oracle RDBMS , SAN , Storage Area Network 11 CommentsI ran across an interesting quote on www.perl.com that challenged my appreciation of relational database technology—and most particularly Oracle. The article states:
Relational databases started to get to be a big deal in the 1970’s, and they’re still a big deal today, which is a little peculiar, because they’re a 1960’s technology.
Forget the “started to get to be” bit for a moment. Uh, yes, RDBMS technology became important in the 70s (ISI, PRTV and Honeywell MRDS). However, since E.F. Codd didn’t write the defining paper until 1970 it is a stretch to call it “1960’s technology.” Oh well, that perl.com article was written in 1999 after all.
What a Great Idea: I Want to Force Some New, “Cool” Technology into My IT Shop
The bit in the quote that got me thinking was how astonished the author was that 1960’s—well, 1970’s actually—technology was “still a big deal” way back in 1999. You see, I think one thing that actually hurts IT shops is the nearly absurd rate of new technology injection. It seems to me that the datacenters with the highest level of success are those that take new technology as slowly as possible. Am I talking about newer versions of, say, the Oracle database server? No, absolutely not. Adopting a newer revision of Oracle is not radical. Those of us who revere the code rest soundly at night knowing that deep down in kernel are bits and pieces that haven’t really changed much in the last 20+ years—perhaps even 30 years given the KCB role in multi-block read consistency (if in fact MBRC was there in the first version of Oracle).
Why is there a mindset in IT that all old technology must go? Folks, we still use little round, brown spinning things (hard drives). Now there’s a bit of information technology that has been around longer than relational databases and ripe for a fond farewell. DBAs are asking for “disk space” from their storage administrators and that is exactly what they are getting. Forget for a moment that roughly 30 drives worth of large, sequential read throughput can saturate the bandwidth of most high-end FC SAN array controllers. Yes, there are exceptions, but I’m trying to make a point. The point is, here we are 27 years after the introduction of the ST506 5.25” and we are not getting full utilization of our drives—at least not when they are provisioned the way most space is provisioned these days. That is, you ask for 2TB of “space” for your Oracle database and you get it—allocated from something like 6 disks in the same enclosure as dozens of other disks. You are getting space, not bandwidth.
What’s This Have to Do with Oracle?
Someone should ask the author of that perl.com article if old technology deserves the rubbish heap simply because it is old. Ask him if he has tires on his car (powered with an internal-combustion engine no less). Yep, pneumatic tires for cars date back to P. Strauss circa 1911. No, really, what does this have to do with Oracle? You see, it is software from companies like Oracle—with their “old” RDBMS front and center—that will first help us tackle this problem we have with untapped hard-drive bandwidth and eventually move us along to whatever replaces those little, round brown spinning things in the future. Yes I am hinting, but I’m not saying anything more.
That’s right, that old crusty Oracle RDBMS technology—created to efficiently manage data stored on hard drives—will outlive hard drives and, quite likely, whatever replaces hard drives. That isn’t so hard to accept. After all, P. Strauss’ pneumatic tire will certainly be getting us to and fro long after we move beyond the internal combustion engine.
Dry, Techno-Geek Humor
The perl.com article also contained some humor. The following definition was given for what an RDBMS is:
A relational database is a bunch of rectangular tables. Each row of a table is a record about one person or thing; the record contains several pieces of information called fields.
Couldn’t a table be square? Somebody please tell that guy that tables have columns (attributes). Files, being hierarchical, have fields.
Recent Comments