The night after I wrote this, Apache crashed on all of our nodes. So that thing about not worrying about the web server nodes, scratch that! We added another node, now running 4 user facing nodes. I think that is what you call jinxing you self!

Original Post

Here are some details on our Blackboard Learn install. I put it together from a request from a colleague from another university. Hopefully others will find it useful.

Especially useful and not Blackboard specific, are the Oracle tuning parameters for RedHat. These seem very important and will be for Oracle and other memory intensive applications.

We are running 3 user facing nodes all with the same config. We also have an admin node that we keep out of the LB, maybe still scared from the old Vista days. It seems all jobs get distributed across all the nodes regardless from where they are started. I see it really as a node ready for service now if we see we need it, but have not had the need yet nor do I see it needed in the near future.

We have been fighting more with our Oracle DB then anything else. The nodes seem to handle the traffic just fine, but Oracle has been working VERY hard. We had to make some changes to our Oracle server due to swapping that occurred. I learned more about Kernel tweaking in the last few weeks then I ever cared to know. Even found my favorite Kernel parameter, vm.swappiness. The end result was to have Oracle use Huge Pages to GUARANTEE that swapping NEVER happens. This server has 32GB of memory, we were only allocating like 12GB to Learn and it was still swapping. The way the linux kernel is optimized is not geared for having 32GB of memory and a bunch of filesystem cache was pushing out our Oracle processes and killing everything. I have also added those tuning parameters. I also attached a few PDF’s that lead us to these parameters. The Oracle 10g Recommendations v1-_2 pdf from RedHat is the best of them I think.

I also added the extra.tomcat option as there are some thread things that we changed to increase GC performance on these large memory footprints.

Hardware

App Nodes are all Xen VM’s with 8 VCPU’s and 18GB Memory running RedHat 5.6
Oracle 11.2.0.2 server is a Bare Metal Dell r710 w/ 32GB of memory with sga_target set to 8GB and pga_aggregate_target set to 4GB. You can’t use Automatic Memory Management with Huge Pages.

bbconfig.properties

bbconfig.min.heapsize.tomcat=8192m
bbconfig.max.heapsize.tomcat=8192m
bbconfig.max.permsize.tomcat=512m
bbconfig.max.stacksize.tomcat=320k
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:SurvivorRatio=4 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseTLAB -XX:ParallelCMSThreads=2 -XX:ParallelGCThreads=4 -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/usr/local/blackboard/logs/tomcat/gc.log -Dcom.sun.management.snmp.port=1162 -Dcom.sun.management.snmp.acl.file=/home/bbuser/snmp.acl -Dsun.net.inetaddr.ttl=86400

bbconfig.appserver.maxthreads=200

bbconfig.database.instance.maxpoolsize=200

bbconfig.unix.httpd.maxclients=200

Oracle Parameters


processes integer 1800
sessions integer 2736
transactions integer 3009


Oracle OS Kernel Tuning

Made Kernel changes so dirty pages in memory are written out more frequently


vm.dirty_background_ratio = 3
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100


Changed the kernel parameter that determines how aggressive swapping is done. Set to 0 so swapping is only done when needed.


vm.swappiness = 0


Configured hugepages for Oracle which makes the memory pagesize larger for Oracle processes (this memory is not swappable)


vm.nr_hugepages = 6656
vm.hugetlb_shm_group = 101

ulimit settings for Oracle:

oracle soft memlock 13631488
oracle hard hemlock 13631488

scroft_240_turning_rhel_for_oracle

RHELTuningandOptimizationforOracleV11

summit_jbw_2010_presentation

Oracle-10-g-recommendations-v1_2

Integration Between Blackboard Learn™, Release 9.1, Vista 8, Others and PeopleSoft – with a Catch

Scan with your favorite QR scanner or the Daqri app

UPDATE:

OK… Got it… Augmented Reality! Check out my GreenO hovering over my desk! You can even look around it. Made a quick O using Google SketchUp and added an Augmented Reality widget to my Daqri. Kinda neat, not sure what else I can do with it, but if I ever create that fancy new invention this would be a GREAT way to show it off!

Tried to make it Green but could not get the texture to take

This is my first WS-Security via Python so I am not sure if there is some liberty taken with this protocol by Blackboard or not. The Blackboard looks like it uses the Rampart module for Axis, so I would hope it is standard. Below is our Signature Handler that is working GREAT!

Also below is a sample of how its used for Blackboard Learn. Learn uses a sessionId in the “password” field for WS-Security. This session is then authenticated using web services calls. The info below connects the Context web services for the first time and requests a session. You can then reuse the sessionId it creates and just reuse the Signature Handler created for the other web services calls.

Complete Signature Handler

import time
import random
import hashlib
import binascii
import logging
'''
Created on Oct 27, 2010
 
@author: cdg2
'''
 
class SignatureHandler(object):
    '''
    classdocs
    '''
    OASIS_PREFIX = "http://docs.oasis-open.org/wss/2004/01/oasis-200401"
    SEC_NS = OASIS_PREFIX + "-wss-wssecurity-secext-1.0.xsd"
    UTIL_NS = OASIS_PREFIX + "-wss-wssecurity-utility-1.0.xsd"
    PASSWORD_DIGEST_TYPE = OASIS_PREFIX + "-wss-username-token-profile-1.0#PasswordDigest"
    PASSWORD_PLAIN_TYPE = OASIS_PREFIX + "-wss-username-token-profile-1.0#PasswordText"
    log=logging.getLogger("SignatureHandler")
    def __init__(self,user,password,useDigest=False):
        '''
        Constructor
        '''
        self._user=user
        self._created=time.strftime('%Y-%m-%dT%H:%M:%SZ',time.gmtime(time.time()))
        self._nonce=hashlib.new("sha",str(random.random())).digest()
        if(useDigest):
            self._passwordType=self.PASSWORD_DIGEST_TYPE
            digest=hashlib.new("sha",self._nonce+self._created+password).digest()
            self._password=binascii.b2a_base64(digest)[:-1]
        else:
            self._passwordType=self.PASSWORD_PLAIN_TYPE
            self._password=password
 
    def sign(self,soapWriter):
        # create element
        securityElem = soapWriter._header.createAppendElement("", "wsse:Security")
        securityElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        securityElem.node.setAttribute("SOAP-ENV:mustunderstand", "true")
 
        # create element
        timestampElem = securityElem.createAppendElement("", "wsse:Timestamp")
        timestampElem.node.setAttribute("xmlns:wsse", self.UTIL_NS)
 
        # create element
        createdElem = timestampElem.createAppendElement("", "wsse:Created")
        createdElem.node.setAttribute("xmlns:wsse", self.UTIL_NS)
 
        createdElem.createAppendTextNode(self._created)
 
        # create element
        usernameTokenElem = securityElem.createAppendElement("", "wsse:UsernameToken")
        usernameTokenElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        usernameTokenElem.node.setAttribute("xmlns:wsu", self.UTIL_NS)
 
        # create element
        usernameElem = usernameTokenElem.createAppendElement("", "wsse:Username")
        usernameElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
 
        # create element
        passwordElem = usernameTokenElem.createAppendElement("", "wsse:Password")
        passwordElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        passwordElem.node.setAttribute("Type", self._passwordType)
 
        # create element
        #nonceElem = usernameTokenElem.createAppendElement("", "wsse:Nonce")
        #nonceElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
 
 
 
        # put values in elements
        usernameElem.createAppendTextNode(self._user)
        passwordElem.createAppendTextNode(self._password)
        # binascii.b2a_base64 adds a newline at the end
        #nonceElem. createAppendTextNode(binascii.b2a_base64(self._nonce)[:-1])
 
        self.log.debug(soapWriter)
 
    def verify(self,soapWriter):
        self

Out of context snippet for connection

        self.baseUrl=baseUrl
        locator = Context_WSLocator()
        self.sigHandler = SignatureHandler("session","nosession",False)
        self.port=locator.getContext_WSPortType(baseUrl)
        self.port.binding.sig_handler=self.sigHandler
 
        request = getServerVersionRequest()
        response = self.port.getServerVersion(request)
 
        ret = response._return
        self.log.info("Connecting to ContextWS version: %s" % ret._version)
 
        request = initializeVersion2Request()
        response = self.port.initializeVersion2(request)
        sessionId = response._return
 
        self.sigHandler = SignatureHandler("session",sessionId,False)
        self.port.binding.sig_handler=self.sigHandler
        self.log.info("Received sessionId:%s from initializeVersion2 and set as password" %sessionId)
        self.log.info("Should be all set to use other functions!")
        self.loginTool("venderId","programId","sharedSecret")
.....
def loginTool(self,vendorId,programId,sharedSecret):
        self.log.debug("loginTool")
        request = loginToolRequest()
        request._password = sharedSecret
        request._clientVendorId = vendorId
        request._clientProgramId = programId
 
        response = self.port.loginTool(request)
        return response._return

We have been on a mad sprint to our first Production semester on Blackboard Learn. During this process we have been creating many different projects to solve problems and integrate BB9.1 on our campus. I will keep this post updated we release more projects to the wild.

Blackboard CAS Fallback Authentication Module
This module allows the use of CAS, LDAP and internal Bb authentication. Get more info at the URL below.
http://code.google.com/p/blackboard-cas-fallback/

Ubuntu

root@***:/var/log# uname -a
Linux ***2.6.31-19-generic-pae #56-Ubuntu SMP Thu Jan 28 02:29:51 UTC 2010 i686 GNU/Linux
root@***:/var/log# ssh -V
OpenSSH_5.1p1 Debian-6ubuntu2, OpenSSL 0.9.8g 19 Oct 2007

Redhat

[root@*** root]# uname -a
Linux *** 2.4.21-63.ELsmp #1 SMP Wed Oct 28 23:15:46 EDT 2009 i686 i686 i386 GNU/Linux
[root@*** root]# ssh -V
OpenSSH_3.6.1p2, SSH protocols 1.5/2.0, OpenSSL 0x0090701f

sshProbe.tcl
SSH_PROBE_SCRIPT.tcl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

#!name = SSH_PROBE_SCRIPT
########################################################################################
# 
# Description :
#    Script connects sends a connect request to a SSH server to see if its alive.
#
# ACE version :
#   1.0+
#    
# Author :
#   Chris Greenough (Chris.Greenough@nau.edu)
#
# Parameters :
#   [debugFlag]
#     debug  - default 0. Do NOT turn on while multiple probes are configured
#
# Example config :
#       probe sshProbe script
#           script SSH_PROBE [0]
# 
########################################################################################
#-------------------------------------------
# debug procedure
# set the EXIT_MSG environment varaible to help debug
# also print the debug message when debug flag is on
#-------------------------------------------
proc ace_debug { msg } { 
    global debug ip port EXIT_MSG
 
    set EXIT_MSG $msg
    if { [ info exists ip ] && [ info exists port ] } { 
	set EXIT_MSG "[ info script ]:$ip:$port: $EXIT_MSG "
    }  
    if { [ info exists debug ] && $debug } { 
	puts $EXIT_MSG
    } 
} 
 
#-------------------------------------------
# main 
#-------------------------------------------
 
ace_debug "initializing varaible"
set EXIT_MSG "Error config:  script SSH_PROBE  <debug(0|1)>"
 
set ip $scriptprobe_env(realIP) 
set port $scriptprobe_env(realPort)
 
## if port is 0 , use default port 22
if { $port == 0 } {
    set port 22
} 
set debug [ lindex $argv 0 ]
if { $debug == "" } {
    set debug 0
} 
# open connection 
ace_debug "opening socket"
set sock [ socket $ip $port ]
 
ace_debug "setting fconfigure to binary"
fconfigure $sock -translation binary 
 
ace_debug "sending ssh client hand-shake message"
# Sending SSH-2.0-SecureCRT_6.5.0 (build 335) SecureCRT??
set ssh_hello "5353482d322e302d5365637572654352545f362e352e3020286275696c642033333529205365637572654352540d0a"
 
set ssh_length  [ string length $ssh_hello ]
puts -nonewline $sock [ binary format "H${ssh_length}" $ssh_hello ]
flush $sock
 
#  read frist 100 bytes from server
ace_debug "receiving response"
set lines [ read $sock 100 ]
 
#  close connection
ace_debug "closing socket"
close $sock
 
#  parsing the 1st 3 bytes from the ssh headers 
#  if it is not a ssh hand shake successful message. failed the probe with exit 30002
# If the server responds with its SSH info then it must be alive
set ssh_header "SSH"
 
ace_debug $lines
 
if { ![ binary scan $lines "@0a3" res ] } { 
    ace_debug "probe fail : ssh server response parsing failure"
    exit 30002
} 
ace_debug $res
if { $res != $ssh_header } { 
    ace_debug "probe fail : ssh hand shake failure with $res !"
    exit 30002
} 
 
#  Everything went fine. probe exit with success exit_code 30001
ace_debug "probe success"
exit 30001

“As Cloud Backups are done over the public Internet, their performance is highly
dependent on Internet network throughput – typically less than 1 MB/Sec per
connection. Additionally, Cloud vendors may also throttle sessions to prevent
individual users from consuming disproportionate amounts of resources.
According to internal tests conducted at Oracle, Amazon S3 limits an individual
session’s read/write throughput to around 2-3 MB/Sec. However by using the right
Oracle Database Backup in the Cloud Page 9
combination of parallelism and compression, backup speeds of up to 40-50
MB/Sec were attained. ”

Plenty fast for a great offsite backup solution! Can get a little pricy for terabytes of data, but a cheap for 100′s of gigs. May be a good solution for people in need of off site backups.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import java.util.Date;
class st{
	public static void main(String args[]){
		// Test the + operator. We know this one is bad.
		// Its good to know its still bad in Java6
		long t = new Date().getTime();
		String x="";
		for(int i=0;i<100000;i++)
			x = x + " ";
		long e = new Date().getTime();
		System.out.println("Time to use the + operations on a string: " + (e-t));
 
		// Test the String.concat method. 
		// Is it any faster?
		t=new Date().getTime();
		x="";
		for(int i=0;i<100000;i++)
			x = x.concat(" ");
		e=new Date().getTime();
		System.out.println("Time to use the concat method on a string: " + (e-t));
 
		// Finaly out baseline. We know StringBuilder is(was)
		// the fastest way to build Strings.
		t=new Date().getTime();
		StringBuilder sb = new StringBuilder();
		for(int i=0;i<100000;i++)
			sb.append(" ");
		String xx=sb.toString();
		e=new Date().getTime();
		System.out.println("Time to use a StringBuilder object: " + (e-t));
	}
}

And the results are!

abc:~ def$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode)
abc:~ def$ javac st.java
abc:~ def$ java st
Time to use the + operations on a string: 29464
Time to use the concat method on a string: 7075
Time to use a StringBuilder object: 2
abc:~ def$

So, what did we learn?

1. Download the cx_Oracle library.
wget http://prdownloads.sourceforge.net/cx-oracle/cx_Oracle-4.3.1-10g-py23-1.i386.rpm?download

2. Install cx_Oracle
rpm -i cx_Oracle-4.3.1-10g-py23-1.i386.rpm

3. Download Oracle Instant Client from the URL below. Must be logged in so do this from a browser.

http://download.oracle.com/otn/linux/instantclient/10204/oracle-instantclient-basiclite-10.2.0.4-1.i386.rpm

4. Install Oracle Instant Client
rpm -i oracle-instantclient-basic-10.2.0.4-1.i386.rpm

5. Add the path to the instant client libs to LD_LIBRARY_PATH.
echo “export LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.4/client/lib/:\$LD_LIBRARY_PATH” >> /etc/profile

Thats it! Want to be sure?

[xyz@pdq ~]$ python2
Python 2.3.4 (#1, Jul 16 2009, 07:01:37)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> oraConn=cx_Oracle.Connection("username/password@server/SqlNetAlias")
>>> curs = oraConn.cursor()
>>> curs.execute("select 1 from dual")
[]
>>> rows = curs.fetchmany(curs.arraysize)
>>> for row in rows:
... print row[0]
...
1
>>> oraConn.close()
>>>

The more I learn about cx_Oracle and Python I will try to update this space. All I know right now is that it is FAST!