---

[Top page] [Rants and Raves] [C.V.] [IBMS computers] [Beowulf Cluster] [Beowulf Cluster Queues] [Home computers] [Gateway computer] [S. America photos] [Dynamic mem in Fortran]

---

Hardware and Software data.

Contents

Machine timings for a variety of jobs.

The g98 results marked with '*' are not comparible to those without, with '*' were completed with version a5 the others with a7. I am attempting to rerun these as I can as soon as I can

Notes on the runs.

• The DOCK run involved protein protein docking with about 500 residues being invloved. Dock results are missing for g77 and Intel machines because for the benchmarks the files used are in binary format and not readable. I no longer have the original files I created the binary files from (I never expected this problem). The Portland Compiler can read the files if Dock is compiled with the -byteswapio option.
• The Charmm minimisation run involved 2500 steps of steepest descent performed on a 2000 atoms system (128 residues).
• The Charmm dynamics run involved 1200 steps of heating to 300K on the structure output from the minimisation run. I found to get this to run using either the Portland or g77 compilers the file $CHARMM/source/image/diagq.src could not be compiled using any optimization flags.
• The homology modelling run was building the 128 residue chain from 2 homologs using simulated annealing to create one structure, the program used was Modeller4.
• The Cyrix was running at 2x75Mhz and Linux 2.0.27, code was compiled with the g77 0.5.21 compiler with -O2 -fno-globals -ffast-math -malign-double. Apparently the 6x86 MMX (M2) has a faster fpu unit.
• he Pii/Piii runs used Linux 2.4 and the Portland Group compiler or the Absoft Compiler. The g77 run was with version 0.5.21 and options -O2 -ffast-math -fno-globals -malign-double gives a pretty good showing. We have totally switched to the Pgroup compilers now so although there are faster verisons of g77 we do not use them.
• The Alpha runs were performed using Linux (Redhat 6.0 or 6.1) and the Compaq fortran compilers.
• The G98 runs were BLYP DFT jobs with the 6-31+G* basis set on a phosphate group containing 10 heavy atoms and 10 hydrogens. There were 90 electrons and 215 basis functions overall. The dual Piii results used the same binaries as all the x86 machines but with the %NProc=2 keyword. Runs marked with an '*' were using g98_a5, others were with g98_a7 which appears about twice as fast. Runs newer than the 1800+ AMD's results were done using g98_a11 or g03b2 if marked g03.

Things that stand out are, with a performance compiler Linux and AMD or P4 chips the results are stunning. The SGI O200 was once good but alas has had its day with the IBMs not being very good at all. I've also tried to use the Absoft compiler under Linux but Gaussian only support the Portland one which worked so well we bought the Portland one and are very very happy with it indeed. For a mid range cluster Celeron chips on a cheap fully integrated mainboard such as an Intel or SiS seem to be a really decent purchase.

2nd series of test, much larger ones.

Gaussian is g03d2, same binary for all machines listed with the superscript a, SVWN 6-31+G* job, 31 atoms, 381 basis functions. 

Dynamics is parallel Charmm34, a 680 residue dimer in a truncated octahdron with images, total 60,000 atoms, 10000 steps with 18Angs cutoffs, same binary is used for the runs with superscript a.

^aThese results using pgroup compilers with target -tp x86

^bThese results using pgroup compilers with target -tp k8-64,p7-64,core2-64,barcelona-64, compiled for 512Mb cache system.

There are benchmarks for Gromacs listed at this site.

This site is a huge Beowulf dedicated to computational Biology, it produces great speedups for simulations with Charmm.

Additional software at IBMS.

procheck, protein structure verification.
bbdep, calulation of bbone and side chain angles.
solvate, creates solvent shells.
molmol, molecular graphics.
rasmol, molecular graphics.
pymol, molecular graphics
vim, vi replacement editor.
molden, molecular graphics.
gnuplot, graphing package.
perl, a scripting language.
python, another scripting language.
mpich, message passing libraries
Samba, exports unix filesystems for mounting under Windows.
Cups, common unified print system.
portland compilers, high performance compiler suite (HPF,F90,F77,C++,C) for Linux.
Intel compilers, high performance comipilers for Linux (F95, F77, C++, C).
firefox, web broswer.

System Administration.

While NIS was doing a great job for us with the single site in Taipei and indeed also managing the accounts and hosts and also the automount maps here and at the university over the VPN this years (2003) Chinese New Years project was to get LDAP up and running. After about 5 days of total frustration it all started to come togeather in one evening. Five days of no progress and then suddenly it started to work and within another 8 hours I had all the users added (for both Unix and Samba accounts), the hosts up and also the automount maps. Not only that I had also managed to configure a slave LDAP server at the students site which would keep them synchronised with us. As a bonus feature we are only using LDAP via the tls encryption mechanism for additional security. Overall I'm impressed, the package I used was from openldap, the documentation was pretty poor for a beginner really and I found more help from reading the Nov 2002 Linux Journal and then searching the deja and google archives for examples from other people. It also involved recompiling samba (for RedHat 7.3/8.0) using the .src.rpm files to use LDAP features. LDAP is very very frustrating and confusing at first but after you manage to do one thing with it (for me adding my first user) it then becomes much easier to do anything else as you can just keep modifying the scripts. Samba comes with some excellent scripts for converting from smbpasswd to LDAP, openldap itself comes with excellent scripts for populating the directories (useraccounts etc) its just the documentation that is weak. Perhaps I should document sometime what I did sometime rather than complain! Anyway there is no going back for us now, LDAP is far superior for us to use than NIS.

OpenLDAP servers can also provide data to the Apple Desktop machines with very little needed in the way of configuration. All in all it works and it works well.

Linux Firewalls.

After far too many attacks from crackers which resulted in three machines in our office being cracked (year 1999) and hearing about cracked SGI O2000's from another building we decided to go with a firewall. Instead of our previous 30 IP addresses we moved to just 3. One IP is the firewall ethernet card, the other two handle the communications for the user-mode-linux kernel that acts as the internet server for various things such as smtp, imaps, opensshd etc etc. We are surprised to find that our entire internet connections can be reduced to a single 100Mbit ethernet connection. The firewall is a standard PC running Linux with 2 ethernet cards (hi-quality 3Coms) and the Linux 2.4.21 kernel with iptables and it just works fine. We converted to the private C-class domain in less than one morning and fixed the only problem what came up that afternoon. It is now so much more secure and in fact easier to do things, for instance we can now run a dhcpd server without worring about other people. The firewall config is a Gigabyte board with a Piii-850 cpu and a 3ware 2-Channel 3W-7006-2 raid-1 card. It worked so well we installed another one at our students site and have created a VPN linking the two domains using ipsec from openswan. One thing we did find was that runing NAT and ipsec togeather meant we had to filter the ipsec packets leaving both firewalls so that they were not NAT'd. The freeswan docs had suggestion for that which did not work. After some digging I realised that I could mark the packets entering each firewall based on their destination and then use that mark to tell the netfilter POSTROUTING rules not to SNAT them the lines below do that

iptables -t mangle -I PREROUTING -s 192.168.100.0/24 -d 192.168.101.0/24 -i eth1 -j MARK --set-mark 0x1
iptables -t nat -I POSTROUTING -m mark --mark 0x1 -j ACCEPT

iptables -t mangle -A FORWARD -s 192.168.0.224/255.255.255.240 -j MARK --set-mark 0x3
iptables -t mangle -A FORWARD -d 192.168.0.224/255.255.255.240 -j MARK --set-mark 0x3
iptables -t mangle -A FORWARD -d 192.168.0.224/255.255.255.240 -p tcp -m tcp --sport 80 -j MARK --set-mark 0x2
iptables -t mangle -A FORWARD -s 192.168.0.224/255.255.255.240 -p tcp -m tcp --dport 22 -j MARK --set-mark 0x2

TC=/sbin/tc
 
 $TC qdisc add dev eth0 root handle 11: htb
 $TC class add dev eth0 parent 11:0 classid 11:1 htb rate 5Mbit
 $TC class add dev eth0 parent 11:0 classid 11:2 htb rate 12Kbit
 $TC filter add dev eth0 parent 11:0 protocol ip handle 2 fw flowid 11:1
 $TC filter add dev eth0 parent 11:0 protocol ip handle 3 fw flowid 11:2

 $TC qdisc add dev eth1 root handle 10: htb
 $TC class add dev eth1 parent 10:0 classid 10:1 htb rate 5Mbit
 $TC class add dev eth1 parent 10:0 classid 10:2 htb rate 12Kbit
 $TC filter add dev eth1 parent 10:0 protocol ip handle 2 fw flowid 10:1
 $TC filter add dev eth1 parent 10:0 protocol ip handle 3 fw flowid 10:2

The wireless network.

Since the arrival of Apple notebooks we decided to provide a wireless access point, this is using WPA and also filters on the MAC address. The device is a D-Link DWL-7100AP access point and links right into the ethernet network. We used to firewall it off and go via IPSEC to a Linux machine when we used WEP but the WPA is supposed to be enought to keep us secure and it is also easy for the users to manage.

Roadwarrior for the Macs, i.e. connecting from outside via OpenVPN.

We use OpenVPN, the macs can connect via a VPN using the tunnelblick software to the internal network using . Which basically sets up a host to network transport, the authentation using SSL certificates which where very easy to set up using the openvpn software. Of course the iptables needed a few tweaks to allow the tun connections and we had to push a internal network but by and large this is working well at the moment, it even works if the notebook is behind a firewall on a NAT network which is really really useful.

So the lab to lab VPN runs over OpenSwan and is up and running all the time but for the roadwarriors we have then connect via OpenVPN, using both together on a single firewall is actually less promblematic than trying to do both things with a single package. We can make changes to the roadwarriors without it touching the lab-to-lab VPN and vice-versa. With things being routed over different interfaces we can also limit the connections if needed.

A quick hotwo:

• set up openvpn on the lab firewall,
• generate the ca.crt, server.key and server.crt
• generate a new client.key and client.crt for each connecting clinet.
• copy the client.key. client.crt and ca.crt to each client
• Edit the openvpn.conf file on each client
• Connect

Our raid setup, A brief HOWTO.

Quite a few people have asked me about out raid setups, this is essentially quite easy to explain. The main and backup fileservers have dedicated raid-6 devices connected via U320 SCSI. These devices are stand-alone and have 8 hot-swap SATA disk bays and 750Gb disks. The cluster, firewall and admin machines each have a 2 channel 3Ware card and 2 disks in a hot-swap cabinet to ensure that a disk failure will not taken them off-line. The mail ssh gateway is now running on a QNAP TS-210 so has internal software raid-1. The File-servers use Linux software raid and have 2 disks setup as raid-1, althought we may switch this to 4x 1.5Tb software Raid-5's in the future so that the internal disks can be used to backup an external system if need be.

To monitor all these raids is quite easy, 3ware provide a tool that monitors its raids and can be connected to via a web browser (3dm). This we set up on all 3Ware raid machines on a high port. Each night one machine connects to all the 3Ware raid machines and downloads the 1st page, it then parses the downloaded file looking for a single report line which is then packaged up with all the other single report lines. For the Linux machines things are similar, every hour a perl script runs on each software raid machine that checks the /proc/mdstat file and counts the number of "[UU]" lines it finds, if it finds four the script then writes a small html file with the hostname, date and an OK, if there are not four "[UU]" lines instead of "OK" it says "Degraded". Each software raid machine is running a tiny web server which serves only this file and also on the same port number as the 3Ware machines. The machine that checks the 3Ware raids each night also downloads and parses these files and bundles all the results togeather and then emails the results to me. This way each morning I am greeted with an email saying if we have any disks failed in any of the raids.

File servers.

In the main office at IBMS the two file servers are Linux Servers with Xeon 3440 cpus, these use fully integrated mainboards from Supermicro based on Intel chipsets and 4Gb of ram. Each machine has 2 gigabit network sockets and a third one for a dedicated ipmi controller card. To help with the stability of the machines each has a pair of 250 Gb disks setup as a raid-1 system using Linux software raid. Each machine has an LSI U320 SCSI card linking it to its storage of 2x Janus 4.2Tb raid-6 devices from SA-3340S. Each machine handles this fine. For backups we have another pair of servers in a different place but the disks are changed, the main set has WD disks and the backups uses Seagates. To handle the backups from the students site another machine has a Linux software raid-5 system based on 5x 750 disks. To cover the circumstances that one of the raids may actually break and need to go off for repair we also have a pair of QNAP TS-410s with 4x 1.5Tb disks. These can be attached to the main servers via iSCSI and take the place of one raid (each) if need be. When we upgraded the raids from 500Gb to 750Gb disks we used the QNAPs for a week as the file store for the main servers and they functioned fine if alittle on the slow side.

At the students site we have gone with a smaller solution, we use a Proware 5 disk hardware raid box with 5x 500Gb disks in it. This connects to a single SATA port on the mainboard and appears as a single huge disk. All the disks were mounted in a 5 tray hot-swap disk box which takes up 3 cd-rom sized bays in the server. The capacity is 2Tb with the server itself is another Intel Core2 E6500 with a Supermicro mainboard with 4Gb of ram and another Intel Gigabit card. Their raid-5 is mirrored each week to one we have in the basement at IBMS.

Mac Mini and iMac Office desktop machines.

My work notebook.

Recently I have picked up a new MacBookPro 13", 2.4 core duo. Its a nice machine. The rest of the group has metal MacBooks while my boss has a MacBook Air. Everyone is very happy with the notebooks. I think in future we will probably go from the metal Macbooks/Pros to plastic macbooks because they offer better value. Being a Linux lab using OS X is ideal for us for our notebooks and now that they can be intel with the intel C/Fortran compiler it is even more useful for us.

Compute Nodes.

The compute nodes are organised as a Beowulf Cluster consisting of Quad Xeons, Core2's and Phenoms. In total we have ~100 compute CPU's at the moment, all running Scientific Linux 5.4 More details can be found here.

Students machines.

Windows machines - Actually nowdays we have a single one and the last time it was turned on was about 2 months ago!

Printers

We have a small collection of quite nice printers available all of which support duplex printing, in the main office we have a fast HP lj2430DTN with twin trays and a colour hp lj3800DN and a lj2300DN is in the another office The students are doing okay as well as their office has a fast networked hp lj2200DN and a colour lj2605DN. In case anyone is really interested we use CUPS to manage all our printing. We even use the cups driver under WinXP to send postscript jobs to the cups server and from there to the printers. The integration between CUPS and SAMBA is really really good nowdays. The OS X machines use CUPS as well and so are integrated into the printing network straight from being switched on.

Last update: Wed Jan 19 15:43:25 CST 2011 Comments to: jon _at_ sinica.edu.tw
These pages were created using vim -a very much vi-improved.

Opinions on these pages are generally not Academia Sinica's.