How to install and run supermatrix in tough places, during hard times!

We start with downloading and unzipping antlr (e.g. antlr_2.7.7.orig.tar.gz).
Configure it, make, then make install to specific folder (e.g. make install prefix=/home/rkurc/usr).

Then morfeusz (morfeusz-linux64-20080205.tar.bz2).
Unzip it, move files with "so" in name to previously used folder's lib subfolder (/home/rkurc/usr/lib) and .h file to include subfolder.

ICU (icu4c-4_2_1-src.tgz).
Unzip, configure (in ./source folder), make, then make install to previous foulder( make install prefix=/home/rkurc/usr).

Add your chosen folder to system paths:

export PATH=/home/rkurc/usr/include:$PATH
export LD_LIBRARY_PATH=/home/rkurc/usr/lib:$LD_LIBRARY_PATH

Boost (boost_1_42_0.tar.gz).
Unzip, run bootstrap.sh --with-icu=/home/rkurc/usr
then ./bjam
and ./bjam install --prefix=/home/rkurc/usr/
to install boost to specified folder.

Linking our projects to boost:
Add
set(BOOST_INCLUDEDIR "/home/rkurc/usr/include")
set(BOOST_LIBRARYDIR "/home/rkurc/usr/lib")
to main CMakeLists.txt files in all projects requiring boost.

Installing takipi:
Using cmake installed by WCSS crew:
/usr/local/cmake-2.8.2/bin/cmake -i
We choose 'yes', to see advanced options.
CMAKE_BUILD_TYPE should be set to "Release"
CMAKE_CXX_FLAGS_RELEASE should use previous values and additionally -I/home/rkurc/usr/include
CMAKE_C_FLAGS_RELEASE should use previous values and additionally -I/home/rkurc/usr/include
CMAKE_EXE_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib
CMAKE_INSTALL_PREFIX should be set to our chosen folder (/home/rkurc/usr)
CMAKE_MODULE_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib
CMAKE_SHARED_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib

Couple ICU paths should be set properly to /home/rkurc/usr.
Then we run 'make' and then 'make install prefix=/home/rkurc/usr'.

Installing poliqarp_reader:

Cmake config:
CMAKE_BUILD_TYPE should be set to "Release"
CMAKE_CXX_FLAGS_RELEASE should use previous values and additionally -I/home/rkurc/usr/include
CMAKE_C_FLAGS_RELEASE should use previous values and additionally -I/home/rkurc/usr/include
CMAKE_EXE_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib
CMAKE_INSTALL_PREFIX should be set to our chosen folder (/home/rkurc/usr)
CMAKE_MODULE_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib
CMAKE_SHARED_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib

Then it's matter of 'make' and 'make install prefix=/home/rkurc/usr'.

Installing supermatrix:
We use WCSS's git: /usr/local/git-1.7.1.1/bin/git to download supermatrix.
We have to comment line add_subdirectory(swig_wrap) in main CMakeLists.txt, because swig is out of date on cluster. It could be installed locally though.
At first run, cmake will not be able to locate MPI, don't worry.
Cmake config:

CMAKE_BUILD_TYPE should be set to "Release"
CMAKE_CXX_FLAGS_RELEASE should use previous values and additionally -I/home/rkurc/usr/include
CMAKE_C_FLAGS_RELEASE should use previous values and additionally -I/home/rkurc/usr/include
CMAKE_EXE_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib
CMAKE_INSTALL_PREFIX should be set to our chosen folder (/home/rkurc/usr)
CMAKE_MODULE_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib
CMAKE_SHARED_LINKER_FLAGS_RELEASE should use previous values and additionally -L/home/rkurc/usr/lib
CORPUS_LIBRARY should be set to it's proper path (/home/rkurc/usr/lib/libcorpus.so)
MPI_COMPILE_FLAGS should be set to empty string! If you can't do it via cmake tool, do it using ccmake and manually delete whole string.
MPI_LIBRARY should be set to /usr/mpi/intel/mvapich-1.0.1/lib/shared/libmpich.so
SHARK_LIBRARY, if shark is installed, should be set to proper path

Afterwards, we have to run cmake again, so it can find MPI paths properly.
Then it's just matter of 'make' and we can enjoy our supermatrix on WCSS' space!


Submitting MPI jobs on WCSS cluster:
Place for bigger files is at /lustre/scratch/rkurc. All huge files should be stored there.
After we prepare our data, we end up with looking like mpirun --hostfile <hostfile>-n <nodes_num> application [arguments]
We scrape the hostfile, WCSS will provide it for us. Change mpirun to mpiexec.
It is neccessary to prepare simple script for job to be properly run on cluster.
It will contain pretty much such lines:
#!/bin/bash
#PBS -S /bin/bash
#PBS -N <job's name>
#PBS -l nodes=<nodes_num>
<our command (e.g. mpiexec -n 30 /home/rkurc/supermatrix/bin/parallel/me_architect_columns/me_architect_parallel --cfg-file /home/rkurc/me_architect/kgr2_ACIN_nowe_kol.cfg)>

It is possible to put more than one command in such script, they will be run sequentially (i think). There are also argument, which allow user to specify time required for computation and memory, but i haven't explored it yet.

With script ready, it should be submitted with command:
qsub -V <script> -q <queue's name>

The queue that should be used is probably queue named 'parallel'. After such command, we'll receive job's ID, e.g. "1539238.nova". We can check status of all our jobs by using command "qstat | grep rkurc". After the job is finished, standard output will be saved to current dir in file named <job's name>.o<job's ID num> and standard error saved in <job's name>.e<job's ID num>. I believe, there are ways to receive output as it goes, but no idea how.
