The night after I wrote this, Apache crashed on all of our nodes. So that thing about not worrying about the web server nodes, scratch that! We added another node, now running 4 user facing nodes. I think that is what you call jinxing you self!

Original Post

Here are some details on our Blackboard Learn install. I put it together from a request from a colleague from another university. Hopefully others will find it useful.

Especially useful and not Blackboard specific, are the Oracle tuning parameters for RedHat. These seem very important and will be for Oracle and other memory intensive applications.

We are running 3 user facing nodes all with the same config. We also have an admin node that we keep out of the LB, maybe still scared from the old Vista days. It seems all jobs get distributed across all the nodes regardless from where they are started. I see it really as a node ready for service now if we see we need it, but have not had the need yet nor do I see it needed in the near future.

We have been fighting more with our Oracle DB then anything else. The nodes seem to handle the traffic just fine, but Oracle has been working VERY hard. We had to make some changes to our Oracle server due to swapping that occurred. I learned more about Kernel tweaking in the last few weeks then I ever cared to know. Even found my favorite Kernel parameter, vm.swappiness. The end result was to have Oracle use Huge Pages to GUARANTEE that swapping NEVER happens. This server has 32GB of memory, we were only allocating like 12GB to Learn and it was still swapping. The way the linux kernel is optimized is not geared for having 32GB of memory and a bunch of filesystem cache was pushing out our Oracle processes and killing everything. I have also added those tuning parameters. I also attached a few PDF’s that lead us to these parameters. The Oracle 10g Recommendations v1-_2 pdf from RedHat is the best of them I think.

I also added the extra.tomcat option as there are some thread things that we changed to increase GC performance on these large memory footprints.

Hardware

App Nodes are all Xen VM’s with 8 VCPU’s and 18GB Memory running RedHat 5.6
Oracle 11.2.0.2 server is a Bare Metal Dell r710 w/ 32GB of memory with sga_target set to 8GB and pga_aggregate_target set to 4GB. You can’t use Automatic Memory Management with Huge Pages.

bbconfig.properties

bbconfig.min.heapsize.tomcat=8192m
bbconfig.max.heapsize.tomcat=8192m
bbconfig.max.permsize.tomcat=512m
bbconfig.max.stacksize.tomcat=320k
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:SurvivorRatio=4 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseTLAB -XX:ParallelCMSThreads=2 -XX:ParallelGCThreads=4 -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/usr/local/blackboard/logs/tomcat/gc.log -Dcom.sun.management.snmp.port=1162 -Dcom.sun.management.snmp.acl.file=/home/bbuser/snmp.acl -Dsun.net.inetaddr.ttl=86400

bbconfig.appserver.maxthreads=200

bbconfig.database.instance.maxpoolsize=200

bbconfig.unix.httpd.maxclients=200

Oracle Parameters


processes integer 1800
sessions integer 2736
transactions integer 3009


Oracle OS Kernel Tuning

Made Kernel changes so dirty pages in memory are written out more frequently


vm.dirty_background_ratio = 3
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100


Changed the kernel parameter that determines how aggressive swapping is done. Set to 0 so swapping is only done when needed.


vm.swappiness = 0


Configured hugepages for Oracle which makes the memory pagesize larger for Oracle processes (this memory is not swappable)


vm.nr_hugepages = 6656
vm.hugetlb_shm_group = 101

ulimit settings for Oracle:

oracle soft memlock 13631488
oracle hard hemlock 13631488

scroft_240_turning_rhel_for_oracle

RHELTuningandOptimizationforOracleV11

summit_jbw_2010_presentation

Oracle-10-g-recommendations-v1_2

Integration Between Blackboard Learn™, Release 9.1, Vista 8, Others and PeopleSoft – with a Catch

Scan with your favorite QR scanner or the Daqri app

UPDATE:

OK… Got it… Augmented Reality! Check out my GreenO hovering over my desk! You can even look around it. Made a quick O using Google SketchUp and added an Augmented Reality widget to my Daqri. Kinda neat, not sure what else I can do with it, but if I ever create that fancy new invention this would be a GREAT way to show it off!

Tried to make it Green but could not get the texture to take

This is my first WS-Security via Python so I am not sure if there is some liberty taken with this protocol by Blackboard or not. The Blackboard looks like it uses the Rampart module for Axis, so I would hope it is standard. Below is our Signature Handler that is working GREAT!

Also below is a sample of how its used for Blackboard Learn. Learn uses a sessionId in the “password” field for WS-Security. This session is then authenticated using web services calls. The info below connects the Context web services for the first time and requests a session. You can then reuse the sessionId it creates and just reuse the Signature Handler created for the other web services calls.

Complete Signature Handler

import time
import random
import hashlib
import binascii
import logging
'''
Created on Oct 27, 2010
 
@author: cdg2
'''
 
class SignatureHandler(object):
    '''
    classdocs
    '''
    OASIS_PREFIX = "http://docs.oasis-open.org/wss/2004/01/oasis-200401"
    SEC_NS = OASIS_PREFIX + "-wss-wssecurity-secext-1.0.xsd"
    UTIL_NS = OASIS_PREFIX + "-wss-wssecurity-utility-1.0.xsd"
    PASSWORD_DIGEST_TYPE = OASIS_PREFIX + "-wss-username-token-profile-1.0#PasswordDigest"
    PASSWORD_PLAIN_TYPE = OASIS_PREFIX + "-wss-username-token-profile-1.0#PasswordText"
    log=logging.getLogger("SignatureHandler")
    def __init__(self,user,password,useDigest=False):
        '''
        Constructor
        '''
        self._user=user
        self._created=time.strftime('%Y-%m-%dT%H:%M:%SZ',time.gmtime(time.time()))
        self._nonce=hashlib.new("sha",str(random.random())).digest()
        if(useDigest):
            self._passwordType=self.PASSWORD_DIGEST_TYPE
            digest=hashlib.new("sha",self._nonce+self._created+password).digest()
            self._password=binascii.b2a_base64(digest)[:-1]
        else:
            self._passwordType=self.PASSWORD_PLAIN_TYPE
            self._password=password
 
    def sign(self,soapWriter):
        # create element
        securityElem = soapWriter._header.createAppendElement("", "wsse:Security")
        securityElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        securityElem.node.setAttribute("SOAP-ENV:mustunderstand", "true")
 
        # create element
        timestampElem = securityElem.createAppendElement("", "wsse:Timestamp")
        timestampElem.node.setAttribute("xmlns:wsse", self.UTIL_NS)
 
        # create element
        createdElem = timestampElem.createAppendElement("", "wsse:Created")
        createdElem.node.setAttribute("xmlns:wsse", self.UTIL_NS)
 
        createdElem.createAppendTextNode(self._created)
 
        # create element
        usernameTokenElem = securityElem.createAppendElement("", "wsse:UsernameToken")
        usernameTokenElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        usernameTokenElem.node.setAttribute("xmlns:wsu", self.UTIL_NS)
 
        # create element
        usernameElem = usernameTokenElem.createAppendElement("", "wsse:Username")
        usernameElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
 
        # create element
        passwordElem = usernameTokenElem.createAppendElement("", "wsse:Password")
        passwordElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        passwordElem.node.setAttribute("Type", self._passwordType)
 
        # create element
        #nonceElem = usernameTokenElem.createAppendElement("", "wsse:Nonce")
        #nonceElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
 
 
 
        # put values in elements
        usernameElem.createAppendTextNode(self._user)
        passwordElem.createAppendTextNode(self._password)
        # binascii.b2a_base64 adds a newline at the end
        #nonceElem. createAppendTextNode(binascii.b2a_base64(self._nonce)[:-1])
 
        self.log.debug(soapWriter)
 
    def verify(self,soapWriter):
        self

Out of context snippet for connection

        self.baseUrl=baseUrl
        locator = Context_WSLocator()
        self.sigHandler = SignatureHandler("session","nosession",False)
        self.port=locator.getContext_WSPortType(baseUrl)
        self.port.binding.sig_handler=self.sigHandler
 
        request = getServerVersionRequest()
        response = self.port.getServerVersion(request)
 
        ret = response._return
        self.log.info("Connecting to ContextWS version: %s" % ret._version)
 
        request = initializeVersion2Request()
        response = self.port.initializeVersion2(request)
        sessionId = response._return
 
        self.sigHandler = SignatureHandler("session",sessionId,False)
        self.port.binding.sig_handler=self.sigHandler
        self.log.info("Received sessionId:%s from initializeVersion2 and set as password" %sessionId)
        self.log.info("Should be all set to use other functions!")
        self.loginTool("venderId","programId","sharedSecret")
.....
def loginTool(self,vendorId,programId,sharedSecret):
        self.log.debug("loginTool")
        request = loginToolRequest()
        request._password = sharedSecret
        request._clientVendorId = vendorId
        request._clientProgramId = programId
 
        response = self.port.loginTool(request)
        return response._return

“As Cloud Backups are done over the public Internet, their performance is highly
dependent on Internet network throughput – typically less than 1 MB/Sec per
connection. Additionally, Cloud vendors may also throttle sessions to prevent
individual users from consuming disproportionate amounts of resources.
According to internal tests conducted at Oracle, Amazon S3 limits an individual
session’s read/write throughput to around 2-3 MB/Sec. However by using the right
Oracle Database Backup in the Cloud Page 9
combination of parallelism and compression, backup speeds of up to 40-50
MB/Sec were attained. ”

Plenty fast for a great offsite backup solution! Can get a little pricy for terabytes of data, but a cheap for 100′s of gigs. May be a good solution for people in need of off site backups.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import java.util.Date;
class st{
	public static void main(String args[]){
		// Test the + operator. We know this one is bad.
		// Its good to know its still bad in Java6
		long t = new Date().getTime();
		String x="";
		for(int i=0;i<100000;i++)
			x = x + " ";
		long e = new Date().getTime();
		System.out.println("Time to use the + operations on a string: " + (e-t));
 
		// Test the String.concat method. 
		// Is it any faster?
		t=new Date().getTime();
		x="";
		for(int i=0;i<100000;i++)
			x = x.concat(" ");
		e=new Date().getTime();
		System.out.println("Time to use the concat method on a string: " + (e-t));
 
		// Finaly out baseline. We know StringBuilder is(was)
		// the fastest way to build Strings.
		t=new Date().getTime();
		StringBuilder sb = new StringBuilder();
		for(int i=0;i<100000;i++)
			sb.append(" ");
		String xx=sb.toString();
		e=new Date().getTime();
		System.out.println("Time to use a StringBuilder object: " + (e-t));
	}
}

And the results are!

abc:~ def$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode)
abc:~ def$ javac st.java
abc:~ def$ java st
Time to use the + operations on a string: 29464
Time to use the concat method on a string: 7075
Time to use a StringBuilder object: 2
abc:~ def$

So, what did we learn?

1. Download the cx_Oracle library.
wget http://prdownloads.sourceforge.net/cx-oracle/cx_Oracle-4.3.1-10g-py23-1.i386.rpm?download

2. Install cx_Oracle
rpm -i cx_Oracle-4.3.1-10g-py23-1.i386.rpm

3. Download Oracle Instant Client from the URL below. Must be logged in so do this from a browser.

http://download.oracle.com/otn/linux/instantclient/10204/oracle-instantclient-basiclite-10.2.0.4-1.i386.rpm

4. Install Oracle Instant Client
rpm -i oracle-instantclient-basic-10.2.0.4-1.i386.rpm

5. Add the path to the instant client libs to LD_LIBRARY_PATH.
echo “export LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.4/client/lib/:\$LD_LIBRARY_PATH” >> /etc/profile

Thats it! Want to be sure?

[xyz@pdq ~]$ python2
Python 2.3.4 (#1, Jul 16 2009, 07:01:37)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> oraConn=cx_Oracle.Connection("username/password@server/SqlNetAlias")
>>> curs = oraConn.cursor()
>>> curs.execute("select 1 from dual")
[]
>>> rows = curs.fetchmany(curs.arraysize)
>>> for row in rows:
... print row[0]
...
1
>>> oraConn.close()
>>>

The more I learn about cx_Oracle and Python I will try to update this space. All I know right now is that it is FAST!

Below is my new GPG key for Chris.Greenough@nau.edu. All other keys for this email have expired. All other GPG/PGP keys should be ignored. While I was at it I also added a GPG key for Chris@ChrisGreenough.com.

Chris.Greenough@nau.edu

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.2 (MingW32)
Chris.Greenough@nau.edu GPG Public Key ID 41F1608F
mQGiBEoMm5IRBADwzWXZ8DCX9TuGK/i2Njqn1Sv12vWehAQ8g8i0B1G8TEkHnaK+
9Fwhz3z9GQLGcJXxpjL1XcnAO7JwwYeCPkOvJx9dFCCCjuHopB9kVpuv620cuivI
j72MyAu+LQgCL5j5fNiEsV/FR+pRgAYr/Dc2L1w541i1xd2kTjK8rwGBHwCgupnz
5+hvxe5ZQuUQU8dHPy7qVvED/iN3M2HXQSCBARrtdl4CICZDPsBjAzU9wNxh097R
P7SlDy3HhMGuPG1ZqmO4Zkj+ewTmMVQExUgdUJOP4A2HYsZZQ7Zg95MTHeOVjakV
WfQsH1ySbmJfE9Fh6q+AW3/2U4ovTDNJp++E4c9Twf4HX8LZXjAOx7jhziF6Fw26
BqWJBADJoN5qBOq+ousFfH70IhUoukXVjRp0Ye/TV61mSdxeiF+TJhRmlOJPwKCS
R0WreFfVIrGRZnaJIytg3JhtzLimSwfvc9p7AB09eo+NZKGHvetYDmzwK84pJ2OJ
ngz0Kdh3LujlB2YSCKfcvEm7AMENhNzv48jwjdb0eRp0NwDo/rQpQ2hyaXMgR3Jl
ZW5vdWdoIDxDaHJpcy5HcmVlbm91Z2hAbmF1LmVkdT6IYAQTEQIAIAUCSgybkgIb
IwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEIq93FJB8WCPgZwAmwbR+nEuJ3QL
FWAvJOn0w0yukHefAJ4mlJKPqjAw5bv7s0E7rOphQEeDl7kBDQRKDJuSEAQAwAqK
hwiAuoY+NsafknsOU7oJf1dJO6FQp8xdYXLSWSM1X6/t4wfNxNfiEyluONeGgaqf
FtuOlXnppIYosvUAud431ZfuF5/3MZNnordxLVKM4orzY1Z0db0uvZLHxYPtL+Oo
lhK+x3ioDufdMbpOd+1BOFRimJsFpfyeHMdBQk8AAwUD/3pHuMCICSDVkh+2lSa8
wKo/O9J/GX7plLmFsOQrvTGzJJHD0GqGnGNEBwXhDUkjaAjGzto2pVQ6RAYNjzhA
MjJ8J+g0TMXUqiAUytiV7ZbvgoZSqhdGxZDU0CpYlz+WOKJB0VnhDU9HVz+a2a7f
/SnHrYKQapXGlfXrhcZmkvX0iEkEGBECAAkFAkoMm5ICGwwACgkQir3cUkHxYI9o
rgCfdum8mSlgi0sjXLTyLBhbqFLyT38An2dJBYfLTPcNEKDUNnTVRRsEnPbT
=+AlU
-----END PGP PUBLIC KEY BLOCK-----

Chris.Greenough@nau.edu GPG Public Key ID 41F1608F

Chris@ChrisGreenough.com

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.9 (MingW32)

mQGiBEoMoQ0RBACjV30Xl9Is9K68yluALhlwnGQUD7ZlWBxHOFdQB3vd5Bn0UDrb
QJYbpTSQOvalsXUy5FshgG7g5Ktzz92hp44zugsxPP529sX9VB7gfJ5Dvc3DChZ6
H2k/WWtHqMNAc6MV47Fco4lKpooH9f77Vx2Od917Qd5y3haUdAaT+7DFiwCg7tyT
kxSNgVs129E2Bu0+us+VnusD/3vE3b2l9cdaS0ae1zdiKMNEawYymKemHLvq+JCe
ZI6X+wM3VDv9fxVU3ivF8YTIc55gNH637lqLhOpC+P8cT4DfqR2weFQCzkoahI0L
ANstAumgcPSDF0wqqNEpykoEF9XbNHemQdr6P2xnmFmDMt5rkebTk4BtYkK8WCcE
9QQnA/wOaJKne+pj89uWMRP6TcFn2R3UiarD8C64y3LqhsmR1Vqt0JF4BNu8dLZe
MF+mvwy77oELN4iMjFkZeBevL0o853BHZdNIlwNHY1DGIy3u0ecmexYe/114XGMm
HPudRMXwxmElvwXXabERAwjzxz2Nf3+G0nc98IyYBWCrLvEwarQ1Q2hyaXMgR3Jl
ZW5vdWdoIChQZXJzb25hbCkgPENocmlzQENocmlzR3JlZW5vdWdoLmNvbT6IYAQT
EQIAIAUCSgyhDQIbIwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEJ0RvbslwH6w
408AoImSW7DfW7v1Gb3XsNumTLLZrstmAKCFPZfoalnwLG/dfEMp3dwH6pzAi7kB
DQRKDKENEAQAms2edG3Ob50gZ7/bldyXgrv8HVQJ2WYz0z+UegIX6eao9ftjQoec
bibtnoYkoUbRK9Pg8mf4rDpcivFANP3hTm3XIxOqJ4fh2anb2HMVF+vRKoTduWzf
svovx164xTT22R3O2rt45CIaF4GWejK8OPkiQSPQZvMrW61pI0I2mDcAAwUD/jMR
4ot1ef7Cry10aNv6n69iYUCm9goSuFlwZ7muSeVrdAa07IyedxGEZOVakK4HzTMs
6fRhwhzrn6vceluOaJEb0YEXMl8AmTrUS+DII7QOjUFG1TqalHDKzVrY14BSSv7w
iHJfJfELgo9+5fyAr5Z0P/3H7izeO6XcTUOGbU6jiEkEGBECAAkFAkoMoQ0CGwwA
CgkQnRG9uyXAfrBDQQCg5WsPVH78/oAoBDCMOGej683eE0AAoMwQOv+0XG4ywT3v
qoH/ZB5SQIFC
=To1g
-----END PGP PUBLIC KEY BLOCK-----

Chris@ChrisGreenough.com Public GPG Key ID 25C07EB0

1. Click the Bluetooth icon in the task bar.
2. Go to add device
3. If you device is discoverable you should see it listed, right click and go to properties
4. Put a check in the box next to Mouse, keyboard drivers (Something like that).
5. Continue to add as normal.

Hope that help some other people! It drove me crazy!