Jan
17
2012

Oracle, RedHat, vm.swappiness and Huge Pages

Update

The night after I wrote this, Apache crashed on all of our nodes. So that thing about not worrying about the web server nodes, scratch that! ;-) We added another node, now running 4 user facing nodes. I think that is what you call jinxing you self!

Original Post

Here are some details on our Blackboard Learn install. I put it together from a request from a colleague from another university. Hopefully others will find it useful.

Especially useful and not Blackboard specific, are the Oracle tuning parameters for RedHat. These seem very important and will be for Oracle and other memory intensive applications.

We are running 3 user facing nodes all with the same config. We also have an admin node that we keep out of the LB, maybe still scared from the old Vista days. It seems all jobs get distributed across all the nodes regardless from where they are started. I see it really as a node ready for service now if we see we need it, but have not had the need yet nor do I see it needed in the near future.

We have been fighting more with our Oracle DB then anything else. The nodes seem to handle the traffic just fine, but Oracle has been working VERY hard. We had to make some changes to our Oracle server due to swapping that occurred. I learned more about Kernel tweaking in the last few weeks then I ever cared to know. Even found my favorite Kernel parameter, vm.swappiness. The end result was to have Oracle use Huge Pages to GUARANTEE that swapping NEVER happens. This server has 32GB of memory, we were only allocating like 12GB to Learn and it was still swapping. The way the linux kernel is optimized is not geared for having 32GB of memory and a bunch of filesystem cache was pushing out our Oracle processes and killing everything. I have also added those tuning parameters. I also attached a few PDF’s that lead us to these parameters. The Oracle 10g Recommendations v1-_2 pdf from RedHat is the best of them I think.

I also added the extra.tomcat option as there are some thread things that we changed to increase GC performance on these large memory footprints.

Hardware

App Nodes are all Xen VM’s with 8 VCPU’s and 18GB Memory running RedHat 5.6
Oracle 11.2.0.2 server is a Bare Metal Dell r710 w/ 32GB of memory with sga_target set to 8GB and pga_aggregate_target set to 4GB. You can’t use Automatic Memory Management with Huge Pages.

bbconfig.properties

bbconfig.min.heapsize.tomcat=8192m
bbconfig.max.heapsize.tomcat=8192m
bbconfig.max.permsize.tomcat=512m
bbconfig.max.stacksize.tomcat=320k
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:SurvivorRatio=4 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseTLAB -XX:ParallelCMSThreads=2 -XX:ParallelGCThreads=4 -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/usr/local/blackboard/logs/tomcat/gc.log -Dcom.sun.management.snmp.port=1162 -Dcom.sun.management.snmp.acl.file=/home/bbuser/snmp.acl -Dsun.net.inetaddr.ttl=86400

bbconfig.appserver.maxthreads=200

bbconfig.database.instance.maxpoolsize=200

bbconfig.unix.httpd.maxclients=200

Oracle Parameters


processes integer 1800
sessions integer 2736
transactions integer 3009

Oracle OS Kernel Tuning

Made Kernel changes so dirty pages in memory are written out more frequently


vm.dirty_background_ratio = 3
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100

Changed the kernel parameter that determines how aggressive swapping is done. Set to 0 so swapping is only done when needed.


vm.swappiness = 0

Configured hugepages for Oracle which makes the memory pagesize larger for Oracle processes (this memory is not swappable)


vm.nr_hugepages = 6656
vm.hugetlb_shm_group = 101

ulimit settings for Oracle:

oracle soft memlock 13631488
oracle hard hemlock 13631488

scroft_240_turning_rhel_for_oracle

RHELTuningandOptimizationforOracleV11

summit_jbw_2010_presentation

Oracle-10-g-recommendations-v1_2

VN:F [1.9.22_1171]
Rating: 9.5/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)
Oracle, RedHat, vm.swappiness and Huge Pages, 9.5 out of 10 based on 2 ratings
Written by admin in: Internet,Technobabble |

No Comments »

RSS feed for comments on this post. TrackBack URL

Leave a comment

You must be logged in to post a comment.

Powered by WordPress | Aeros Theme | TheBuckmaker.com WordPress Themes