Tuesday, May 18, 2010

Scalability study of StarCCM+ on cluster


Latest version of StarCCM+ starts supporting multi nodes job on POE/IBM/AIX environment. Previously, it was not possible to run StarCCM+ on more than one node in parallel. Therefore, the maximum number of cores is limited to 16 on hydra[1]. But, with StarCCM+ ver 5.02, it became possible to spawn parallel processes on remote compute node through POE parallel environment on IBM cluster. Now, the question is how scalable StarCCM+ is?

Experimental environment

For experiment , lemans_poly_17m.sim file has been tested, which is one of simulation input file frequently used for benchmark. The file size is 6.5GB and it has 17 millions of cells and 95 millions of vertices.

The primary purpose of experiment is to identify how scale StarCCM+ is on hydra, however, the relative performance of two different machines is also evaluated. Each compute node on eos[2] has two quad core Intel X5560 based cluster and each node is connected via Infiniband, whereas, compute node on hydra has 8 dual core Power5+ and HPS is used as a interconnect.


The benchmark results are evaluated from two different perspectives; scalability, relative performance.


The scalability on eos and hydra shows not much difference each other. The performance with 16 cores is used as a basis for speedup comparison.

On eos, speedup is easily doubled as the number of core increase and it stays very scalable up to 64 cores. With 128 cores(8x), it slows down and shows only 6.2x speedup.

On hydra, it shows very similar result and stays very scalable up to 64 cores and starts slow down around 128 cores.

Based on this result, there is no significant difference on the scalability of StarCCM+ on eos and hydra. However, it could be changed with different input files, I/O patterns, and characteristics of data in input file.


The definition of 'performance' here is, simply, who finish simulation fast. With same input file, how long it take on each machine? The differences between eos and hydra varies as the number of cores increase. eos is about 2.51x faster than hydra on 16 cores and 2.41x faster on 128 cores.


Benchmark results show that eos performs better than hydra to run StarCCM+ with given input file. However, the differences in hardware specification between 2 machines such as L3 cache size, interconnect, number of cores per node, can change the result with different characteristics of input file.


1. http://sc.tamu.edu/help/hydra/
2. http://sc.tamu.edu/systems/eos/hardware.php

Thursday, May 13, 2010

Quick install guide for LAMMPS on Linux cluster

This is a quick installation guide for LAMMS 10 May 2010 version for Linux cluster and it can be done within 1 hours if you follow this carefully.


Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
Red Hat Enterprise Linux Server release 5.4 (Tikanga) 2.6.18-164.11.1.el5
Intel compiler and MPI stack
mpiicc for the Intel(R) MPI Library 4.0 for Linux*
Copyright(C) 2003-2010, Intel Corporation. All rights reserved.
Version 11.1


Download these 2 files and untar them, respectively.

- fftw-2.1.5.tar.gz
- lammps.tar.gz (lib for LAMMPS, LAMMPS itself)


In a nutshell, installation procedure consists of 3 steps; (1)FFTW, (2)libs for LAMMPS, (3)LAMMPS itself. And if you use mpiicc and mpiifort, then you don't have to worry about the PATH of MPI package and it will be taken care of automatically.


Even though the latest version of FFTW is 3.2.2, but, unfortunately, LAMMS can not work with it. You better stick to 2.1.5 for now. And I assume you're installing it at /usr/local/fftw-2.1.5 directory.

After uncompressing it,

cd fftw-2.1.5
./configure CC=mpiicc F77=mpiifort --prefix=/usr/local/fftw-2.1.5
make check
make install

(2)libs for LAMMPS

After uncompressing lammps.tar.gz,(actually, libs for LAMMPS is part of lammps.tar.gz)

cd lib/reax
Change ifort to mpiifort in Makefile.ifort
make -f Makefile.ifort

cd lib/meam
Change ifort to mpiifort in Makefile.ifort
make -f Makefile.ifort

cd lib/poems
Change icc to mpiicc in Makefile.icc
make -f Makefile.icc

cd src

Check which packages are included
make package-status

Choose all standard packages to be included
make yes-standard

If you don't have GPU, then exclude gpu package. Otherwise, you will see tons of error message when you compile it.
make no-gpu

Edit MAKE/Makefile.linux

Remove MPI_PATH,MPI_LIB, it will be taken care of by mpiicc, mpiifort
#MPI_LIB = -lmpich -lpthread

FFT_INC = -I/usr/local/fftw-2.1.5/include -DFFT_FFTW
FFT_PATH = -L/usr/local/fftw-2.1.5/lib
FFT_LIB = -lfftw

Build LAMMPS as an executable and library as well

make linux
make makelib
make -f Makefile.lib linux


I assume you're installing lammps at /usr/local/lammps directory.

copy bench/ doc/ examples/ potentials/ README tools/ to /usr/local/lammps
copy all *.a to /usr/local/lammps/lib
copy lmp_linux to /usr/local/lammps/bin


Once you have gotten this far, just let user know lammps is available at /usr/local/lammps, then they know how to play around it.