I received a few pieces of (not)fan-mail about my latest post in the Crabby Series. One reader took offense at the fact that I bother to blog about hugepages because, in his words:
…you insult the intelligence of your readers. You know full well everyone uses hugepages
Is that why Metalink Note 749851.1 goes to the trouble of advising DBAs that the default database setup from Database Configuration Assistant (DBCA) configures Automatic Memory Management which does not use hugepages?
I assure you, not everyone uses hugepages and part of that is because it can be difficult to set it up if you have several databases—especially if your databases have a mix of heavy PGA usage and heavy SGA usages. Also, if your calculations are off and there are insufficient hugepages to cover the SGA, Oracle will go ahead and allocate with a shmget() that doesn’t pass in SHM_HUGETLB. The effect of that little twist is you’ll be “missing” the memory that was carved out for hugepages and the SGA will reside in other non-hugepages memory. So, for instance, if you calculate your SGA to be 1GB and you allocate 513 (1GB + 1 page for wiggle room) but your SGA turns out to be 1073758208 (1GB + 16KB), you’ll get a non-hugepages SGA and eventually there will be roughly 2GB tied up. I think it is an important topic.
Metalink 401749.1
Oracle Support offers a script to assist DBAs in calculating hugepages requirement. With all your instances up, run the script and it will calculate a setting for you. The note is entitled Shell Script to Calculate Values Recommended HugePages / HugeTLB Configuration.
There is a small nit regarding this note ( the procedure it involves actually). In order for the script to give you a recommendation, you have to revert from AMM first, then do a boot of your instances with MMM so it can peek what SysV IPC segments are being allocated for the instances. So, it’s a multi-step process. I suppose with a lot of extra thought the same thing could be calculated by tallying up all the “granule files” found in /dev/shm under AMM, but no matter. This is fairly simple.
Let’s look at my system. Here’s what we’ll see:
First, we’ll see how large the SGA really is.
Next, we’ll see how large of an IPC segment the instance called for. In my case it is about 37MB larger than the actual SGA. That’s fine.
Finally we’ll see the output of the hugepages_settings.sh script to see what it advises.
SQL*Plus: Release 11.X.0.X.0 Production on Mon Jul 27 13:35:41 2009 Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.X.0.X.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> show sga Total System Global Area 8351150080 bytes Fixed Size 2214808 bytes Variable Size 1543505000 bytes Database Buffers 6777995264 bytes Redo Buffers 27435008 bytes SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.X.0.X.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options $ ipcs -m ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x522d5fd4 327681 oracle 660 8390705152 53 $ sh ./hugepages_setting.sh Recommended setting: vm.nr_hugepages = 4003 $ grep Huge /proc/meminfo HugePages_Total: 5000 HugePages_Free: 999 HugePages_Rsvd: 0 Hugepagesize: 2048 kB
So it looks like the script is accurate and even allows a little wiggle room. That’s good. I think this script (being helpful) combined with a healthy fear of the nastiness in large SGA+large dedicated connection deployments (without hugepages) should get us all one step closer to insisting on hugepages backing for our SGAs.
Kevin Closson's Blog: Platforms, Databases and Storage 
Hi Kevin,
Having 11+ DBA experience I found your last posts very useful (as all others posts).
It might be that I am not that intelligence (according to the other reader :).
Please keep a very good job sharing your experience.
Thank you,
Yury
Thanks, Jurjis, I appreciate that.
I love it when people come up with the “everyone knows” nonsense!
I’d bet anything that less than 25% of *n*x Oracle installations use anything other than default pagesize and memory allocations.
And now with 11g and that AMM thing, that is even more likely!
Let’s hope 11gr2 uses large/hugepages with AMM, otherwise the blessed thing will be yet another useless “advance”.
Excellent post, Kevin. Thanks heaps for the pointers on those scripts and Metalink notes, damn useful stuff!
Thanks Noons…always appreciate it when you stop by…
I can’t say anything about 11gR2 (Oracle employee gag orders)… I can say that there is a reason I’m blogging about hugepages as of late though…
“Oracle employee gag orders”
So that’s the meaning of “Release 11.X.0.X.0” in the output 🙂
Good eye.
Kevin, this is an excellent post, but I must say that huge pages are a bit overrated. I have set up an instance with huge pages and without huge pages. I was unable to detect any real difference in performance. The OS was RH 4.3, 64 bit. Having in mind that this is a 64 bit port, I can address all the RAM I need, even without the huge pages. Is anybody still running production servers on 32 bit platforms? On the other hand, the 10g version of AMM tends to produce over-expanded shared pool and shrinks everything else to the bare minimum. It wouldn’t be half as bad, if the shared pool wasn’t mostly free. This happens when developers do not use bind variables. The only way to eliminate that behavior is to set cursor_sharing to force. In other words, I don’t see AMM as such a beautiful thing. Being able to use AMM with huge pages sounds interesting, but not much more than that.
I find hubpages a huge pain in the butt and I know what your going through 🙂
I do think it is the most oddball way of implementing shared-page-table, locked IPC shared memory… It was done better on Dynix PTX over 15 years ago for heaven’s sake. That being said, it is a necessary evil and therefore worth the pain.
Ironically, consider, Linus Torvald’s impression of implementing direct I/O via the O_DIRECT open flag (you know, the way every Unix implementation other than Solaris did it)…this is directly from the Linux manpage for open:
Fact is stranger than fiction…
Just to say that I on the same (not-huge)page as Jurijs Velikanovs. Thanks for providing information for those of us who are eager to learn more about something we do not yet use.
Finally ! I found people that are not huge fans of hugepages !
Hemant K Chitale