After my recent blog entry entitled Configuring Linux Hugepages for Oracle Is Just Too Difficult! Isnt It? Part I, I engaged in a couple of email threads and a thread on oracle-l about when to employ hugepages. In those exchanges I was amazed to find that it is still a borderline issue for folks. I feel it is very cut and dried and thus I prepared the following guidelines that more or less spell it out.
- Reasons for Using Hugepages
- Use hugepages if OLTP or ERP. Full stop.
- Use hugepages if DW/BI with large numbers of dedicated connections or a large SGA. Full stop.
- Use hugepages if you don’t like the amount of memory page tables are costing you (/proc/meminfo). Full stop.
- SGA Memory Management Models
- AMM does not support hugepages. Full stop.
- ASMM supports hugepages.
- Instance Type
- ASM uses AMM by default. ASM instances do not need hugepages. Full stop.
- All non-ASM instances should be considered candidate for hugepages. See 1.1->1.3 above.
- Configuration
- Limits (multiple layers)
- /etc/security/limits.conf establishes limits for hugepages for processes. Note, setting these values does not pre-allocate any resources.
- Ulimit also establishes hugepages limits for processes.
- Limits (multiple layers)
- Allocation
- /etc/sysctl.conf vm.nr_hugepages allocates memory to the hugepages pool.
- Sizing
- Read MOS 401749.1 for information on tools available to aid in the configuration of vm/nr_hugepages
To make the point of how urgently Oracle DBAs need to qualify their situation against list items 1.1 through 1.3 above, please consider the following quote from an internal email I received. The email is real and the screen output came from a real customer system. Yes, 120+ gigabytes of memory wasted in page tables. Fact is often stranger than fiction!
And here is an example of kernel pagetables usage, with a 24GB SGA, and 6000+ connections .. with no hugepages in use .
# grep PageT /proc/meminfo
PageTables: 123731372 kB
Link to Part III in this series:
Configuring Linux Hugepages for Oracle Database is Just Too Difficult! Isn’t It? Part – III.
Kevin Closson's Blog: Platforms, Databases and Storage 
Hi Kevin,
I think there is a lot of confusion around this issue. I was reviewing an AMM versus Hugepages decision and the advice (from everywhere) was less than clearcut.
I just had a conversation last week when I was asking how the decision came about that the standard 11g builds within my client were AMM not Hugepages – which I can understand from a generic systems build perspective. But in fact it turns out that none of their 11g systems use hugepages.
In fact, one of the things quoted back to me about the decision in favour of AMM was your article:
https://kevinclosson.wordpress.com/2007/08/23/oracle11g-automatic-memory-management-and-linux-hugepages-support/
The conclusion from the engineering team team was that for 10g, hugepages is better for large SGA + large session count but for 11g, AMM was better.
At the end of the day, people should do their own tests, come to their own conclusions for their own circumstances.
However, just to be more cut and dried, is your advice above the same regardless of 10g, 11gR1 or 11gR2?
Thanks.
Cheers,
Dominic
Dom,
First, I need to apologize because I think your comment got trapped up in SPAM protection. Anyway, I think I like the following quote:
“At the end of the day, people should do their own tests, come to their own conclusions for their own circumstances.”
Here, here!
>>However, just to be more cut and dried, is your advice above the same regardless of 10g, 11gR1 or 11gR2?
The answer to that would be a resounding yes.
Nice advise. I also don’t recommend to configure Linux Hugepages;
Full stop.
Maclean,
Do you own a DIMM foundry or something? 🙂
Actually, to be even more cut and dried, do you have guideline definitions for “large numbers of dedicated connections” and “large SGA”?
Cheers,
Dominic
dombrooks,
I should revisit that Aug 2007 post because it does sound like an endorsement of AMM. That was really old 11g software and the topic deserves another visit. Perhaps on X2-8 so we get a good sprinkling of NUMA in the mix as well.
Here is how I’d like to put this:
1. If 1.3 (in the list above) is the most important criteria and the performance you see with AMM is otherwise acceptable then stay with AMM. Indeed, ASM instances do just fine with AMM.
2. If AMM is sufficient in your deployment then you either have a small SGA and few connections or a large SGA and fewer connections.
dombrooks,
“Large number of dedicated connections” and “large SGA” in the context of hugepages can only be quantified by the amount of memory wasted in page tables and whether the administrator is satisfied with that cost.
Hi.
I was just going to ask about your recommendation of hugepages when your previous article on AMM implied it’s acceptable compared to hugepages. Now you’ve gone an answered my question before I’ve asked it I can’t think of anything intelligent to say. 🙂
Cheers
Tim…
Hi Tim,
Thanks for stopping by. I’m about to blog a fresh piece about AMM + hugepages. In short, however, the whole topic boils down to how much RAM people are willing to spend on overhead.
Goofy link?
https://kevinclosson.wordpress.com/2010/09/28/configuring-linux-hugepages-for-oracle-database-is-just-too-difficult-part-i/
is probably better than
https://kevinclosson.wordpress.com/2010/10/21/configuring-linux-hugepages-for-oracle-database-is-just-too-difficult-isn%e2%80%99t-it-part-%e2%80%93-ii/2006_10_17
Hi Kevin,
Do your recommendations hold good for Linux or for Unix flavors also? Just asking the question since I have been working at multiple customer locations running AIX and have not seen any of them use “Large Pages” as such. Most of them do have heavily used OLTPs running with 100+ GB SGAs also.
thanks,
Anindya
Anindya:
Most Unix offerings (count them on one hand these days) have large page support and fewer (if any) administrative difficulties involved. But, yes, the answer to your question is large page table entries for shared memory between a lot of processes is generally a good thing.
Kevin
I am glad to see this post from you.
It is surprising to see how many performance problems / server hang situations can be resolved by configuring HugePages in Linux. I blogged about one such issue in this blog entry:
http://orainternals.wordpress.com/2008/11/13/performance-tuning-hugepages-and-linux/
As you pointed out, it should be a rule: If you use Oracle on Linux and care about performance or scalability, use HugePages.
Surprisingly there is hesitation from Linux Admins and DBAs to configure HugePages. Your post should alleviate some of that hesitation.
I have a question though: In a fully virtualized environment (such as Linux clients running on IBM Z/OS host), do you see any issues setting up HugePages?
Thanks,
Cheers
Riyaj
Hello orainternals,
I do see an issue using hugepages on virtualized linux clients in a z/OS environment. That issue, however, is simply that I don’t know anything about virtualized linux clients running on z/OS 🙂 Sorry. Thanks for stopping by!
We are running hugepages … and also using ASM.
Does ASM use/allocate hugepages by default or not? My ASM instances are pretty vanilla no customization of init parameters.
It sounds like it would be bad if ASM instance did use hugepages under default config.
Currently on 11.1.0.7.5 on OEL 5.5 64 bit.
Thanks in advance.
John,
There is really no need to modify ASM. It is an AMM instance and can stay that way.
So the Oracle ASM software is smart enough to know when running in an AMM configuration to not use hugepages? Sorry that’s the part I as apparently missing.
How about a regular database instance? If you try to run it using AMM ( mine are all configured fixed sizing ) will it ignore completely the hugepages?
Guess I should do the testing eh?
Hi John,
Please google: “closson hugepages”
I’ve written extensively about the relationship between AMM and hugepages.
Kevin.
Can you comment if future versions of Oracle will support hugepages and AMM?
Scott,
I can only comment with the bug I filed:
Bug 10165985 – AMM MMAP CALLS SHOULD PASS IN THE MAP_HUGETLB ON MODERN LINUX
Hey Kevin
Have you had a look at transparent largepages?
http://lwn.net/Articles/359158/
Looks like you’d basically get huge/large pages automatically?
Hi Kevin, just a maddening observation in Centos 6.2 (I assume this didn’t happen earlier): setting memlock in limits.conf results then starting Oracle in a “full crash report” being sent to root. This is actually just a warning stating that “Using mlock ulimits for SHM_HUGETLB deprecated”.
This is listed as a known issue in Oracle Linux (http://oss.oracle.com/el6/docs/RELEASE-NOTES-GA-en.html) with the workaround being 1) configure the application to use CAP_IPC_LOCK or 2) add the process to hugetlb_shm_group. This is really out of my element here, but it appears I got it working properly by granting CAP_IPC_LOCK to $ORACLE_HOME/bin/oracle thusly:
[root@server bin]# setcap CAP_IPC_LOCK+ep oracle
Oracle starts, Hugepages are used, and no “crash report” appears. I’m not positive that it’s all “proper” however. It’ll be fun when we go into production with this machine.
@beitinsw : Thanks for the information. That might certainly help some wayward googler of those terms at some point!