The night after I wrote this, Apache crashed on all of our nodes. So that thing about not worrying about the web server nodes, scratch that! We added another node, now running 4 user facing nodes. I think that is what you call jinxing you self!
Here are some details on our Blackboard Learn install. I put it together from a request from a colleague from another university. Hopefully others will find it useful.
Especially useful and not Blackboard specific, are the Oracle tuning parameters for RedHat. These seem very important and will be for Oracle and other memory intensive applications.
We are running 3 user facing nodes all with the same config. We also have an admin node that we keep out of the LB, maybe still scared from the old Vista days. It seems all jobs get distributed across all the nodes regardless from where they are started. I see it really as a node ready for service now if we see we need it, but have not had the need yet nor do I see it needed in the near future.
We have been fighting more with our Oracle DB then anything else. The nodes seem to handle the traffic just fine, but Oracle has been working VERY hard. We had to make some changes to our Oracle server due to swapping that occurred. I learned more about Kernel tweaking in the last few weeks then I ever cared to know. Even found my favorite Kernel parameter, vm.swappiness. The end result was to have Oracle use Huge Pages to GUARANTEE that swapping NEVER happens. This server has 32GB of memory, we were only allocating like 12GB to Learn and it was still swapping. The way the linux kernel is optimized is not geared for having 32GB of memory and a bunch of filesystem cache was pushing out our Oracle processes and killing everything. I have also added those tuning parameters. I also attached a few PDF’s that lead us to these parameters. The Oracle 10g Recommendations v1-_2 pdf from RedHat is the best of them I think.
I also added the extra.tomcat option as there are some thread things that we changed to increase GC performance on these large memory footprints.
App Nodes are all Xen VM’s with 8 VCPU’s and 18GB Memory running RedHat 5.6
Oracle 220.127.116.11 server is a Bare Metal Dell r710 w/ 32GB of memory with sga_target set to 8GB and pga_aggregate_target set to 4GB. You can’t use Automatic Memory Management with Huge Pages.
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:SurvivorRatio=4 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseTLAB -XX:ParallelCMSThreads=2 -XX:ParallelGCThreads=4 -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/usr/local/blackboard/logs/tomcat/gc.log -Dcom.sun.management.snmp.port=1162 -Dcom.sun.management.snmp.acl.file=/home/bbuser/snmp.acl -Dsun.net.inetaddr.ttl=86400
processes integer 1800
sessions integer 2736
transactions integer 3009
Oracle OS Kernel Tuning
Made Kernel changes so dirty pages in memory are written out more frequently
vm.dirty_background_ratio = 3
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100
Changed the kernel parameter that determines how aggressive swapping is done. Set to 0 so swapping is only done when needed.
vm.swappiness = 0
Configured hugepages for Oracle which makes the memory pagesize larger for Oracle processes (this memory is not swappable)
vm.nr_hugepages = 6656
vm.hugetlb_shm_group = 101
ulimit settings for Oracle:
oracle soft memlock 13631488
oracle hard hemlock 13631488
No Comments »
Leave a comment
You must be logged in to post a comment.