Little Things Doth Crabby Make – Part X. Posts About Linux Hugepages Makes Some Crabby It Seems. Also, Words About Sizing Hugepages.

I received a few pieces of (not)fan-mail about my latest post in the Crabby Series. One reader took offense at the fact that I bother to blog about hugepages because, in his words:

…you insult the intelligence of your readers. You know full well everyone uses hugepages

Is that why Metalink Note 749851.1 goes to the trouble of advising DBAs that the default database setup from Database Configuration Assistant (DBCA) configures Automatic Memory Management which does not use hugepages?

I assure you, not everyone uses hugepages and part of that is because it can be difficult to set it up if you have several databases—especially if your databases have a mix of heavy PGA usage and heavy SGA usages. Also, if your calculations are off and there are insufficient hugepages to cover the SGA, Oracle will go ahead and allocate with a shmget() that doesn’t pass in SHM_HUGETLB. The effect of that little twist is you’ll be “missing” the memory that was carved out for hugepages and the SGA will reside in other non-hugepages memory. So, for instance, if you calculate your SGA to be 1GB and you allocate 513 (1GB + 1 page for wiggle room) but your SGA turns out to be 1073758208 (1GB + 16KB), you’ll get a non-hugepages SGA and eventually there will be roughly 2GB tied up. I think it is an important topic.

Metalink 401749.1
Oracle Support offers a script to assist DBAs in calculating hugepages requirement. With all your instances up, run the script and it will calculate a setting for you. The note is entitled Shell Script to Calculate Values Recommended HugePages / HugeTLB Configuration.

There is a small nit regarding this note ( the procedure it involves actually).  In order for the script to give you a recommendation, you have to revert from AMM first, then do a boot of your instances with MMM so it can peek what SysV IPC segments are being allocated for the instances. So, it’s a multi-step process. I suppose with a lot of extra thought the same thing could be calculated by tallying up all the “granule files” found in /dev/shm under AMM, but no matter. This is fairly simple.

Let’s look at my system. Here’s what we’ll see:

First, we’ll see how large the SGA really is.

Next, we’ll see how large of an IPC segment the instance called for. In my case it is about 37MB larger than the actual SGA. That’s fine.

Finally we’ll see the output of the script to see what it advises.

SQL*Plus: Release 11.X.0.X.0 Production on Mon Jul 27 13:35:41 2009

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.X.0.X.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show sga

Total System Global Area 8351150080 bytes
Fixed Size                  2214808 bytes
Variable Size            1543505000 bytes
Database Buffers         6777995264 bytes
Redo Buffers               27435008 bytes
SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.X.0.X.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
$ ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x522d5fd4 327681     oracle    660        8390705152 53                      

$ sh ./
Recommended setting: vm.nr_hugepages = 4003
$ grep Huge /proc/meminfo
HugePages_Total:  5000
HugePages_Free:    999
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

So it looks like the script is accurate and even allows a little wiggle room. That’s good. I think this script (being helpful) combined with a healthy fear of the nastiness in large SGA+large dedicated connection deployments (without hugepages) should get us all one step closer to insisting on hugepages backing for our SGAs.

12 Responses to “Little Things Doth Crabby Make – Part X. Posts About Linux Hugepages Makes Some Crabby It Seems. Also, Words About Sizing Hugepages.”

  1. 1 Jurijs Velikanovs July 28, 2009 at 12:02 am

    Hi Kevin,

    Having 11+ DBA experience I found your last posts very useful (as all others posts).
    It might be that I am not that intelligence (according to the other reader :).
    Please keep a very good job sharing your experience.

    Thank you,

  2. 3 Noons July 28, 2009 at 12:47 am

    I love it when people come up with the “everyone knows” nonsense!
    I’d bet anything that less than 25% of *n*x Oracle installations use anything other than default pagesize and memory allocations.

    And now with 11g and that AMM thing, that is even more likely!
    Let’s hope 11gr2 uses large/hugepages with AMM, otherwise the blessed thing will be yet another useless “advance”.

    Excellent post, Kevin. Thanks heaps for the pointers on those scripts and Metalink notes, damn useful stuff!

  3. 7 Mladen Gogala July 28, 2009 at 9:21 am

    Kevin, this is an excellent post, but I must say that huge pages are a bit overrated. I have set up an instance with huge pages and without huge pages. I was unable to detect any real difference in performance. The OS was RH 4.3, 64 bit. Having in mind that this is a 64 bit port, I can address all the RAM I need, even without the huge pages. Is anybody still running production servers on 32 bit platforms? On the other hand, the 10g version of AMM tends to produce over-expanded shared pool and shrinks everything else to the bare minimum. It wouldn’t be half as bad, if the shared pool wasn’t mostly free. This happens when developers do not use bind variables. The only way to eliminate that behavior is to set cursor_sharing to force. In other words, I don’t see AMM as such a beautiful thing. Being able to use AMM with huge pages sounds interesting, but not much more than that.

  4. 8 dsi console July 30, 2009 at 12:02 am

    I find hubpages a huge pain in the butt and I know what your going through 🙂

    • 9 kevinclosson July 30, 2009 at 12:26 am

      I do think it is the most oddball way of implementing shared-page-table, locked IPC shared memory… It was done better on Dynix PTX over 15 years ago for heaven’s sake. That being said, it is a necessary evil and therefore worth the pain.

      Ironically, consider, Linus Torvald’s impression of implementing direct I/O via the O_DIRECT open flag (you know, the way every Unix implementation other than Solaris did it)…this is directly from the Linux manpage for open:

      “The thing that has always disturbed me about O_DIRECT is that the
      whole interface is just stupid, and was probably designed by a deranged
      monkey on some serious mind-controlling substances.” ? Linus

      Fact is stranger than fiction…

  5. 10 Charles Schultz August 6, 2009 at 4:23 pm

    Just to say that I on the same (not-huge)page as Jurijs Velikanovs. Thanks for providing information for those of us who are eager to learn more about something we do not yet use.

  6. 11 Hemant K Chitale August 5, 2010 at 9:48 am

    Finally ! I found people that are not huge fans of hugepages !

    Hemant K Chitale

  1. 1 Blogroll Report 24/07/2009 – 31/07/2009 « Coskan’s Approach to Oracle Trackback on August 12, 2009 at 2:34 pm

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.


I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 744 other subscribers
Oracle ACE Program Status

Click It

website metrics

Fond Memories


All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: