Kevin Closson's Blog: Platforms, Databases and Storage

14 Responses to “Quantifying Hugepages Memory Savings with Oracle Database 11g”

1. 1 Bernd Eckenfels July 29, 2009 at 2:36 am

   So it does not keep the page table memory allocated after the process has quit.

   Oh BTW, I pointed to your numbers here:
   http://stackoverflow.com/questions/853736/how-to-find-or-calculate-a-linux-processs-page-table-size-and-other-kernel-accou/1182989

   Bernd

   Reply
2. 2 yafang gao July 29, 2009 at 8:05 am

   In our db machine,
   everytime when the db startup ,the HugePages_Rsvd may increased.
   Now it is 5963,and can not to release and cant not used by oracle.
   Only todo is reboot machine to release it.
   Are their other method to decrease it except reboot ?

   thanks a bunch!

   [root@dgpubdb3 ~]# cat /proc/meminfo
   MemTotal: 32959992 kB
   MemFree: 10257644 kB
   Buffers: 137896 kB
   Cached: 9728592 kB
   SwapCached: 159328 kB
   Active: 8431064 kB
   Inactive: 1781244 kB
   HighTotal: 0 kB
   HighFree: 0 kB
   LowTotal: 32959992 kB
   LowFree: 10257644 kB
   SwapTotal: 16386292 kB
   SwapFree: 13313700 kB
   Dirty: 96 kB
   Writeback: 0 kB
   AnonPages: 186480 kB
   Mapped: 2391716 kB
   Slab: 134276 kB
   PageTables: 80524 kB
   NFS_Unstable: 0 kB
   Bounce: 0 kB
   CommitLimit: 26760176 kB
   Committed_AS: 16327048 kB
   VmallocTotal: 34359738367 kB
   VmallocUsed: 271316 kB
   VmallocChunk: 34359466727 kB
   HugePages_Total: 5963
   HugePages_Free: 5963
   HugePages_Rsvd: 5963
   Hugepagesize: 2048 kB

   Reply
   • 3 kevinclosson July 29, 2009 at 2:07 pm

     Yafang,

     What does the output of ipcs -m look like? Also, I presume there are no other apps on the box calling shmget with the SHM_HUGETLB (or no usage of hugetlbfs/libhugetlbfs for that matter). The pages are only in reserved status until processes attached to the segment start to fault the pages in. If you set the init.ora parameter pre_page_sga to TRUE you should notice that the pages come out of reserved and are deducted from HugePages_Free as soon as the instance boot process is complete.

     If the hugepages are not being used by Oracle it is probably because it is an insufficient amount (as I blogged about). So, what does show sga report? You might care to run the sizing script.

     Reply
3. 4 yafang gao July 30, 2009 at 6:08 am

   this machine only run oracle database.
   I has not set parameter “pre_page_sga”

   The problem is HugePages_Rsvd always increase when ths db startup.
   when you shutdown db,HugePages_Rsvd is not retrun to zero.
   when it increase to a number,the Hugepage that can allocated for sga less than SGA,then the SGA can not use Hugepage at all and the HugePages_Rsvd can reach a very high value ,as 5963 in example.
   This memory(5963*2MB) can not release to system.
   Only reboot can free the reserved Hugepage and use Hugepage normally.
   But if you shutdown db and restart,maybe can not use Hugepage and HugePages_Rsvd*2MB amount memory is wasted ,can not use by any application,and can not relesed to system until reboot.

   Thanks for your response.
   I like your blog.

   Reply
   • 5 kevinclosson July 31, 2009 at 7:47 pm

     if you have no ipc segs (ipcs -m) and you echo 0 > /proc/sys/vm/nr_hugepages it should certainly clear

     Reply
4. 6 Steve August 19, 2009 at 7:22 pm

   thanks for the information Kevin. I have been testing huge pages and noticed the page table size differences as well, on a 2 node NUMA configured IBM3950 with 256GB memory, without huge pages the page table can really expand under load. Overall I am seeing 10% or greater performance gains on several platforms for SGA sizes 6G and 10G for my test databases.

   Reply
   • 7 kevinclosson August 19, 2009 at 10:07 pm

     …thanks for stopping by, Steve

     Reply
5. 8 Christo Kutrovsky September 21, 2009 at 6:20 pm

   Hi Kevin,

   I’ve given a presentation on Oracle and memory a few times. I talk a lot about hugepages there.

   Slides with synced audio can be found here:

   http://www.pythian.com/news/741/pythian-goodies-free-memory-swap-oracle-and-everything

   Reply
6. 9 Narendra Prabhudesai December 20, 2013 at 5:13 am

   Hello Kevin,

   First, thanks for this series of posts on Hugepages with detail explanations and test cases.
   However, I have a doubt about the test case used above. In your test case, you are only creating DB sessions (in dedicated server mode) but those DB sessions do nothing (and in particular do not “use” SGA). In that case, how does using/not using HugePgaes help? Are Hugepages used by SGA as well as PGA or only SGA?

   Thanks

   Reply
   • 10 kevinclosson December 20, 2013 at 12:25 pm

     “you are only creating DB sessions (in dedicated server mode) but those DB sessions do nothing (and in particular do not “use” SGA).”

     Hello Narenda,

     If you get a session via a dedicated connection your foreground process maps the entire SGA–immediately. That is sufficient use to see the cost in page tables as I show via meminfo in the post.

     Reply
     • 11 Narendra Prabhudesai December 22, 2013 at 5:55 am

       Hello Kevin,

       Thanks for your response.
       Apologies if this sounds like a naive question but what exactly does it mean by “foreground process maps the entire SGA-immediately.” ? If process “mapping” memory does not mean process “using” memory, why would page tables be need to contain details about it?

       Reply
       • 12 kevinclosson December 26, 2013 at 2:39 pm

         Mapping memory is using memory. You get page table entries immediately and that has cost.

         Reply

1. 1 Oracle Exadata Performance series – Part 1: Should I use Hugepages on Linux Database Nodes? | Tanel Poder's blog: IT & Mobile for Geeks and Pros Trackback on March 13, 2011 at 6:55 pm
2. 2 HugePages Overhead « OraStory Trackback on May 30, 2012 at 12:08 pm