Update

The night after I wrote this, Apache crashed on all of our nodes. So that thing about not worrying about the web server nodes, scratch that! We added another node, now running 4 user facing nodes. I think that is what you call jinxing you self!

Original Post

Here are some details on our Blackboard Learn install. I put it together from a request from a colleague from another university. Hopefully others will find it useful.

Especially useful and not Blackboard specific, are the Oracle tuning parameters for RedHat. These seem very important and will be for Oracle and other memory intensive applications.

We are running 3 user facing nodes all with the same config. We also have an admin node that we keep out of the LB, maybe still scared from the old Vista days. It seems all jobs get distributed across all the nodes regardless from where they are started. I see it really as a node ready for service now if we see we need it, but have not had the need yet nor do I see it needed in the near future.

We have been fighting more with our Oracle DB then anything else. The nodes seem to handle the traffic just fine, but Oracle has been working VERY hard. We had to make some changes to our Oracle server due to swapping that occurred. I learned more about Kernel tweaking in the last few weeks then I ever cared to know. Even found my favorite Kernel parameter, vm.swappiness. The end result was to have Oracle use Huge Pages to GUARANTEE that swapping NEVER happens. This server has 32GB of memory, we were only allocating like 12GB to Learn and it was still swapping. The way the linux kernel is optimized is not geared for having 32GB of memory and a bunch of filesystem cache was pushing out our Oracle processes and killing everything. I have also added those tuning parameters. I also attached a few PDF’s that lead us to these parameters. The Oracle 10g Recommendations v1-_2 pdf from RedHat is the best of them I think.

I also added the extra.tomcat option as there are some thread things that we changed to increase GC performance on these large memory footprints.

Hardware

App Nodes are all Xen VM’s with 8 VCPU’s and 18GB Memory running RedHat 5.6
Oracle 11.2.0.2 server is a Bare Metal Dell r710 w/ 32GB of memory with sga_target set to 8GB and pga_aggregate_target set to 4GB. You can’t use Automatic Memory Management with Huge Pages.

bbconfig.properties

bbconfig.min.heapsize.tomcat=8192m
bbconfig.max.heapsize.tomcat=8192m
bbconfig.max.permsize.tomcat=512m
bbconfig.max.stacksize.tomcat=320k
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:SurvivorRatio=4 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseTLAB -XX:ParallelCMSThreads=2 -XX:ParallelGCThreads=4 -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/usr/local/blackboard/logs/tomcat/gc.log -Dcom.sun.management.snmp.port=1162 -Dcom.sun.management.snmp.acl.file=/home/bbuser/snmp.acl -Dsun.net.inetaddr.ttl=86400

bbconfig.appserver.maxthreads=200

bbconfig.database.instance.maxpoolsize=200

bbconfig.unix.httpd.maxclients=200

Oracle Parameters


processes integer 1800
sessions integer 2736
transactions integer 3009


Oracle OS Kernel Tuning

Made Kernel changes so dirty pages in memory are written out more frequently


vm.dirty_background_ratio = 3
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100


Changed the kernel parameter that determines how aggressive swapping is done. Set to 0 so swapping is only done when needed.


vm.swappiness = 0


Configured hugepages for Oracle which makes the memory pagesize larger for Oracle processes (this memory is not swappable)


vm.nr_hugepages = 6656
vm.hugetlb_shm_group = 101

ulimit settings for Oracle:

oracle soft memlock 13631488
oracle hard hemlock 13631488

scroft_240_turning_rhel_for_oracle

RHELTuningandOptimizationforOracleV11

summit_jbw_2010_presentation

Oracle-10-g-recommendations-v1_2

Here is a presentation I gave at BlackBoard World 2011 Developers Conference. Below is the slide deck.

Integration Between Blackboard Learn™, Release 9.1, Vista 8, Others and PeopleSoft – with a Catch

Not exactly sure what Daqri is, maybe About.me but heavy on QR? About.me is much better looking, but I think this may have legs.

Scan with your favorite QR scanner or the Daqri app

UPDATE:

OK… Got it… Augmented Reality! Check out my GreenO hovering over my desk! You can even look around it. Made a quick O using Google SketchUp and added an Augmented Reality widget to my Daqri. Kinda neat, not sure what else I can do with it, but if I ever create that fancy new invention this would be a GREAT way to show it off!

Tried to make it Green but could not get the texture to take

We have a current integration with Blackboard Vista and as we start to deploy Blackboard Learn we wanted to use this same integration with this new end point. The feed is written in Python and the Learn web services use WS-Security for authentication/authorization. I could find very little information on WS-Security and Python. We decided to go with ZSI for our web services layer in Python. After digging for a long time we found that ZSI allows a hook to a Signature Handler that signs each soap packet before its was sent! PERFECT… Now what!

This is my first WS-Security via Python so I am not sure if there is some liberty taken with this protocol by Blackboard or not. The Blackboard looks like it uses the Rampart module for Axis, so I would hope it is standard. Below is our Signature Handler that is working GREAT!

Also below is a sample of how its used for Blackboard Learn. Learn uses a sessionId in the “password” field for WS-Security. This session is then authenticated using web services calls. The info below connects the Context web services for the first time and requests a session. You can then reuse the sessionId it creates and just reuse the Signature Handler created for the other web services calls.

Complete Signature Handler

import time
import random
import hashlib
import binascii
import logging
'''
Created on Oct 27, 2010
 
@author: cdg2
'''
 
class SignatureHandler(object):
    '''
    classdocs
    '''
    OASIS_PREFIX = "http://docs.oasis-open.org/wss/2004/01/oasis-200401"
    SEC_NS = OASIS_PREFIX + "-wss-wssecurity-secext-1.0.xsd"
    UTIL_NS = OASIS_PREFIX + "-wss-wssecurity-utility-1.0.xsd"
    PASSWORD_DIGEST_TYPE = OASIS_PREFIX + "-wss-username-token-profile-1.0#PasswordDigest"
    PASSWORD_PLAIN_TYPE = OASIS_PREFIX + "-wss-username-token-profile-1.0#PasswordText"
    log=logging.getLogger("SignatureHandler")
    def __init__(self,user,password,useDigest=False):
        '''
        Constructor
        '''
        self._user=user
        self._created=time.strftime('%Y-%m-%dT%H:%M:%SZ',time.gmtime(time.time()))
        self._nonce=hashlib.new("sha",str(random.random())).digest()
        if(useDigest):
            self._passwordType=self.PASSWORD_DIGEST_TYPE
            digest=hashlib.new("sha",self._nonce+self._created+password).digest()
            self._password=binascii.b2a_base64(digest)[:-1]
        else:
            self._passwordType=self.PASSWORD_PLAIN_TYPE
            self._password=password
 
    def sign(self,soapWriter):
        # create element
        securityElem = soapWriter._header.createAppendElement("", "wsse:Security")
        securityElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        securityElem.node.setAttribute("SOAP-ENV:mustunderstand", "true")
 
        # create element
        timestampElem = securityElem.createAppendElement("", "wsse:Timestamp")
        timestampElem.node.setAttribute("xmlns:wsse", self.UTIL_NS)
 
        # create element
        createdElem = timestampElem.createAppendElement("", "wsse:Created")
        createdElem.node.setAttribute("xmlns:wsse", self.UTIL_NS)
 
        createdElem.createAppendTextNode(self._created)
 
        # create element
        usernameTokenElem = securityElem.createAppendElement("", "wsse:UsernameToken")
        usernameTokenElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        usernameTokenElem.node.setAttribute("xmlns:wsu", self.UTIL_NS)
 
        # create element
        usernameElem = usernameTokenElem.createAppendElement("", "wsse:Username")
        usernameElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
 
        # create element
        passwordElem = usernameTokenElem.createAppendElement("", "wsse:Password")
        passwordElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
        passwordElem.node.setAttribute("Type", self._passwordType)
 
        # create element
        #nonceElem = usernameTokenElem.createAppendElement("", "wsse:Nonce")
        #nonceElem.node.setAttribute("xmlns:wsse", self.SEC_NS)
 
 
 
        # put values in elements
        usernameElem.createAppendTextNode(self._user)
        passwordElem.createAppendTextNode(self._password)
        # binascii.b2a_base64 adds a newline at the end
        #nonceElem. createAppendTextNode(binascii.b2a_base64(self._nonce)[:-1])
 
        self.log.debug(soapWriter)
 
    def verify(self,soapWriter):
        self

Out of context snippet for connection

        self.baseUrl=baseUrl
        locator = Context_WSLocator()
        self.sigHandler = SignatureHandler("session","nosession",False)
        self.port=locator.getContext_WSPortType(baseUrl)
        self.port.binding.sig_handler=self.sigHandler
 
        request = getServerVersionRequest()
        response = self.port.getServerVersion(request)
 
        ret = response._return
        self.log.info("Connecting to ContextWS version: %s" % ret._version)
 
        request = initializeVersion2Request()
        response = self.port.initializeVersion2(request)
        sessionId = response._return
 
        self.sigHandler = SignatureHandler("session",sessionId,False)
        self.port.binding.sig_handler=self.sigHandler
        self.log.info("Received sessionId:%s from initializeVersion2 and set as password" %sessionId)
        self.log.info("Should be all set to use other functions!")
        self.loginTool("venderId","programId","sharedSecret")
.....
def loginTool(self,vendorId,programId,sharedSecret):
        self.log.debug("loginTool")
        request = loginToolRequest()
        request._password = sharedSecret
        request._clientVendorId = vendorId
        request._clientProgramId = programId
 
        response = self.port.loginTool(request)
        return response._return

Just a quick update on Projects.

We have been on a mad sprint to our first Production semester on Blackboard Learn. During this process we have been creating many different projects to solve problems and integrate BB9.1 on our campus. I will keep this post updated we release more projects to the wild.

Blackboard CAS Fallback Authentication Module
This module allows the use of CAS, LDAP and internal Bb authentication. Get more info at the URL below.
http://code.google.com/p/blackboard-cas-fallback/

We were having problems with default TCP probes on port 22 from our Cisco ACE Load Balancer showing up in our logs as bad SSH connections. I modified a Cisco TCL script for checking SSL to send a greeting to a SSH server and check that a SSH greeting is returned. It by no means does a complete login, but it does check to make sure the SSH service is alive. Tested on RedHat and Ubuntu. Let me know if this works on other services.

Ubuntu

root@***:/var/log# uname -a
Linux ***2.6.31-19-generic-pae #56-Ubuntu SMP Thu Jan 28 02:29:51 UTC 2010 i686 GNU/Linux
root@***:/var/log# ssh -V
OpenSSH_5.1p1 Debian-6ubuntu2, OpenSSL 0.9.8g 19 Oct 2007

Redhat

[root@*** root]# uname -a
Linux *** 2.4.21-63.ELsmp #1 SMP Wed Oct 28 23:15:46 EDT 2009 i686 i686 i386 GNU/Linux
[root@*** root]# ssh -V
OpenSSH_3.6.1p2, SSH protocols 1.5/2.0, OpenSSL 0x0090701f

sshProbe.tcl
SSH_PROBE_SCRIPT.tcl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

#!name = SSH_PROBE_SCRIPT
########################################################################################
# 
# Description :
#    Script connects sends a connect request to a SSH server to see if its alive.
#
# ACE version :
#   1.0+
#    
# Author :
#   Chris Greenough (Chris.Greenough@nau.edu)
#
# Parameters :
#   [debugFlag]
#     debug  - default 0. Do NOT turn on while multiple probes are configured
#
# Example config :
#       probe sshProbe script
#           script SSH_PROBE [0]
# 
########################################################################################
#-------------------------------------------
# debug procedure
# set the EXIT_MSG environment varaible to help debug
# also print the debug message when debug flag is on
#-------------------------------------------
proc ace_debug { msg } { 
    global debug ip port EXIT_MSG
 
    set EXIT_MSG $msg
    if { [ info exists ip ] && [ info exists port ] } { 
	set EXIT_MSG "[ info script ]:$ip:$port: $EXIT_MSG "
    }  
    if { [ info exists debug ] && $debug } { 
	puts $EXIT_MSG
    } 
} 
 
#-------------------------------------------
# main 
#-------------------------------------------
 
ace_debug "initializing varaible"
set EXIT_MSG "Error config:  script SSH_PROBE  <debug(0|1)>"
 
set ip $scriptprobe_env(realIP) 
set port $scriptprobe_env(realPort)
 
## if port is 0 , use default port 22
if { $port == 0 } {
    set port 22
} 
set debug [ lindex $argv 0 ]
if { $debug == "" } {
    set debug 0
} 
# open connection 
ace_debug "opening socket"
set sock [ socket $ip $port ]
 
ace_debug "setting fconfigure to binary"
fconfigure $sock -translation binary 
 
ace_debug "sending ssh client hand-shake message"
# Sending SSH-2.0-SecureCRT_6.5.0 (build 335) SecureCRT??
set ssh_hello "5353482d322e302d5365637572654352545f362e352e3020286275696c642033333529205365637572654352540d0a"
 
set ssh_length  [ string length $ssh_hello ]
puts -nonewline $sock [ binary format "H${ssh_length}" $ssh_hello ]
flush $sock
 
#  read frist 100 bytes from server
ace_debug "receiving response"
set lines [ read $sock 100 ]
 
#  close connection
ace_debug "closing socket"
close $sock
 
#  parsing the 1st 3 bytes from the ssh headers 
#  if it is not a ssh hand shake successful message. failed the probe with exit 30002
# If the server responds with its SSH info then it must be alive
set ssh_header "SSH"
 
ace_debug $lines
 
if { ![ binary scan $lines "@0a3" res ] } { 
    ace_debug "probe fail : ssh server response parsing failure"
    exit 30002
} 
ace_debug $res
if { $res != $ssh_header } { 
    ace_debug "probe fail : ssh hand shake failure with $res !"
    exit 30002
} 
 
#  Everything went fine. probe exit with success exit_code 30001
ace_debug "probe success"
exit 30001

Was looking into backup solutions for a big oracle database and ran across a plugin to RMAN to backup directly to s3! http://www.oracle.com/technology/software/tech/cloud/index.html Seems like a very interesting option.

“As Cloud Backups are done over the public Internet, their performance is highly
dependent on Internet network throughput – typically less than 1 MB/Sec per
connection. Additionally, Cloud vendors may also throttle sessions to prevent
individual users from consuming disproportionate amounts of resources.
According to internal tests conducted at Oracle, Amazon S3 limits an individual
session’s read/write throughput to around 2-3 MB/Sec. However by using the right
Oracle Database Backup in the Cloud Page 9
combination of parallelism and compression, backup speeds of up to 40-50
MB/Sec were attained. ”

Plenty fast for a great offsite backup solution! Can get a little pricy for terabytes of data, but a cheap for 100′s of gigs. May be a good solution for people in need of off site backups.

I have know for a long time that the + operation for Strings in Java is horrible for performance. Because Strings are immutable in Java another String object is created for each +, not so good! I did notice the other day that there is a String.concat() operation. I wondered if it would run faster then + or even as fast as a StringBuilder. The following is the test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import java.util.Date;
class st{
	public static void main(String args[]){
		// Test the + operator. We know this one is bad.
		// Its good to know its still bad in Java6
		long t = new Date().getTime();
		String x="";
		for(int i=0;i<100000;i++)
			x = x + " ";
		long e = new Date().getTime();
		System.out.println("Time to use the + operations on a string: " + (e-t));
 
		// Test the String.concat method. 
		// Is it any faster?
		t=new Date().getTime();
		x="";
		for(int i=0;i<100000;i++)
			x = x.concat(" ");
		e=new Date().getTime();
		System.out.println("Time to use the concat method on a string: " + (e-t));
 
		// Finaly out baseline. We know StringBuilder is(was)
		// the fastest way to build Strings.
		t=new Date().getTime();
		StringBuilder sb = new StringBuilder();
		for(int i=0;i<100000;i++)
			sb.append(" ");
		String xx=sb.toString();
		e=new Date().getTime();
		System.out.println("Time to use a StringBuilder object: " + (e-t));
	}
}

And the results are!

abc:~ def$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode)
abc:~ def$ javac st.java
abc:~ def$ java st
Time to use the + operations on a string: 29464
Time to use the concat method on a string: 7075
Time to use a StringBuilder object: 2
abc:~ def$

So, what did we learn?

After a bunch of trial and error, the following packages are needed to Install cx_Oracle for Python and the required Oracle Instant Client for Red Hat Enterprise Linux 4. Its not that this is too difficult, its that RHEL4 is old. Getting the two to place nice took a sec to figure out. This will install cx_Oracle for python that comes with RHEL4, which as of this writing is Python version 2.3.4 for Red Hat Enterprise Linux AS release 4 (Nahant Update 8). I am sure you can use the newer libraries with the a new install of Python, but I wanted to stay as close as possible to the versions supported by Red Hat. For better for worse!

1. Download the cx_Oracle library.
wget http://prdownloads.sourceforge.net/cx-oracle/cx_Oracle-4.3.1-10g-py23-1.i386.rpm?download

2. Install cx_Oracle
rpm -i cx_Oracle-4.3.1-10g-py23-1.i386.rpm

3. Download Oracle Instant Client from the URL below. Must be logged in so do this from a browser.

http://download.oracle.com/otn/linux/instantclient/10204/oracle-instantclient-basiclite-10.2.0.4-1.i386.rpm

4. Install Oracle Instant Client
rpm -i oracle-instantclient-basic-10.2.0.4-1.i386.rpm

5. Add the path to the instant client libs to LD_LIBRARY_PATH.
echo “export LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.4/client/lib/:\$LD_LIBRARY_PATH” >> /etc/profile

Thats it! Want to be sure?

[xyz@pdq ~]$ python2
Python 2.3.4 (#1, Jul 16 2009, 07:01:37)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> oraConn=cx_Oracle.Connection("username/password@server/SqlNetAlias")
>>> curs = oraConn.cursor()
>>> curs.execute("select 1 from dual")
[]
>>> rows = curs.fetchmany(curs.arraysize)
>>> for row in rows:
... print row[0]
...
1
>>> oraConn.close()
>>>

The more I learn about cx_Oracle and Python I will try to update this space. All I know right now is that it is FAST!

We had a great time at Adult Prom 2009 at the Orpheum in Downtown Flagstaff, AZ. Link to the pics.