Quantifying Hugepages Memory Savings with Oracle Database 11g

In my recent post about physical memory consumed by page tables when hugepages are not in use I showed an example of 500 dedicated connections to an Oracle Database 11g instance with 8000M SGA consuming roughly 7 gigabytes of physical memory just for page tables. A reader emailed me to point out that it would be informational to show page table consumption with hugepages employed. How right.

The following script was running while I invoked sqlplus 500 times to connect to the same instance discussed in this post. As the script shows, page table cost peaked at roughly 265 MB compared to the 7 GB lost to page tables in the non-hugepages case.

I’m scratching my head and thinking of what else could possibly be said on the matter…

$ while true
> do
> ps -ef | grep oracletest | wc -l
> grep PageTables /proc/meminfo
> sleep 30
> done
PageTables:      23484 kB
PageTables:     264432 kB
PageTables:     264900 kB
PageTables:     265400 kB
PageTables:     265944 kB
PageTables:      83112 kB
PageTables:      78672 kB
PageTables:      74188 kB
PageTables:      69712 kB
PageTables:      65264 kB
PageTables:      60804 kB
PageTables:      56332 kB
PageTables:      51872 kB
PageTables:      47376 kB
PageTables:      42848 kB
PageTables:      37904 kB
PageTables:      32976 kB
PageTables:      28024 kB
PageTables:      23532 kB
PageTables:      23528 kB

14 Responses to “Quantifying Hugepages Memory Savings with Oracle Database 11g”

  1. 1 Bernd Eckenfels July 29, 2009 at 2:36 am

    So it does not keep the page table memory allocated after the process has quit.

    Oh BTW, I pointed to your numbers here:


  2. 2 yafang gao July 29, 2009 at 8:05 am

    In our db machine,
    everytime when the db startup ,the HugePages_Rsvd may increased.
    Now it is 5963,and can not to release and cant not used by oracle.
    Only todo is reboot machine to release it.
    Are their other method to decrease it except reboot ?

    thanks a bunch!

    [root@dgpubdb3 ~]# cat /proc/meminfo
    MemTotal: 32959992 kB
    MemFree: 10257644 kB
    Buffers: 137896 kB
    Cached: 9728592 kB
    SwapCached: 159328 kB
    Active: 8431064 kB
    Inactive: 1781244 kB
    HighTotal: 0 kB
    HighFree: 0 kB
    LowTotal: 32959992 kB
    LowFree: 10257644 kB
    SwapTotal: 16386292 kB
    SwapFree: 13313700 kB
    Dirty: 96 kB
    Writeback: 0 kB
    AnonPages: 186480 kB
    Mapped: 2391716 kB
    Slab: 134276 kB
    PageTables: 80524 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    CommitLimit: 26760176 kB
    Committed_AS: 16327048 kB
    VmallocTotal: 34359738367 kB
    VmallocUsed: 271316 kB
    VmallocChunk: 34359466727 kB
    HugePages_Total: 5963
    HugePages_Free: 5963
    HugePages_Rsvd: 5963
    Hugepagesize: 2048 kB

    • 3 kevinclosson July 29, 2009 at 2:07 pm


      What does the output of ipcs -m look like? Also, I presume there are no other apps on the box calling shmget with the SHM_HUGETLB (or no usage of hugetlbfs/libhugetlbfs for that matter). The pages are only in reserved status until processes attached to the segment start to fault the pages in. If you set the init.ora parameter pre_page_sga to TRUE you should notice that the pages come out of reserved and are deducted from HugePages_Free as soon as the instance boot process is complete.

      If the hugepages are not being used by Oracle it is probably because it is an insufficient amount (as I blogged about). So, what does show sga report? You might care to run the sizing script.

  3. 4 yafang gao July 30, 2009 at 6:08 am

    this machine only run oracle database.
    I has not set parameter “pre_page_sga”

    The problem is HugePages_Rsvd always increase when ths db startup.
    when you shutdown db,HugePages_Rsvd is not retrun to zero.
    when it increase to a number,the Hugepage that can allocated for sga less than SGA,then the SGA can not use Hugepage at all and the HugePages_Rsvd can reach a very high value ,as 5963 in example.
    This memory(5963*2MB) can not release to system.
    Only reboot can free the reserved Hugepage and use Hugepage normally.
    But if you shutdown db and restart,maybe can not use Hugepage and HugePages_Rsvd*2MB amount memory is wasted ,can not use by any application,and can not relesed to system until reboot.

    Thanks for your response.
    I like your blog.

  4. 6 Steve August 19, 2009 at 7:22 pm

    thanks for the information Kevin. I have been testing huge pages and noticed the page table size differences as well, on a 2 node NUMA configured IBM3950 with 256GB memory, without huge pages the page table can really expand under load. Overall I am seeing 10% or greater performance gains on several platforms for SGA sizes 6G and 10G for my test databases.

  5. 8 Christo Kutrovsky September 21, 2009 at 6:20 pm

    Hi Kevin,

    I’ve given a presentation on Oracle and memory a few times. I talk a lot about hugepages there.

    Slides with synced audio can be found here:


  6. 9 Narendra Prabhudesai December 20, 2013 at 5:13 am

    Hello Kevin,

    First, thanks for this series of posts on Hugepages with detail explanations and test cases.
    However, I have a doubt about the test case used above. In your test case, you are only creating DB sessions (in dedicated server mode) but those DB sessions do nothing (and in particular do not “use” SGA). In that case, how does using/not using HugePgaes help? Are Hugepages used by SGA as well as PGA or only SGA?


    • 10 kevinclosson December 20, 2013 at 12:25 pm

      “you are only creating DB sessions (in dedicated server mode) but those DB sessions do nothing (and in particular do not “use” SGA).”

      Hello Narenda,

      If you get a session via a dedicated connection your foreground process maps the entire SGA–immediately. That is sufficient use to see the cost in page tables as I show via meminfo in the post.

      • 11 Narendra Prabhudesai December 22, 2013 at 5:55 am

        Hello Kevin,

        Thanks for your response.
        Apologies if this sounds like a naive question but what exactly does it mean by “foreground process maps the entire SGA-immediately.” ? If process “mapping” memory does not mean process “using” memory, why would page tables be need to contain details about it?

  1. 1 Oracle Exadata Performance series – Part 1: Should I use Hugepages on Linux Database Nodes? | Tanel Poder's blog: IT & Mobile for Geeks and Pros Trackback on March 13, 2011 at 6:55 pm
  2. 2 HugePages Overhead « OraStory Trackback on May 30, 2012 at 12:08 pm

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.


I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 744 other subscribers
Oracle ACE Program Status

Click It

website metrics

Fond Memories


All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: