Perhaps you have a lot of basins that you want to perform some analysis on, or you want to re-run a previous analysis with slightly different paramters, different m/n values, and so on. This section will explain some of the scripting tools, supercomputer applications, and other utilities to help speed up this process.
Scripts (located in the LSD_Visualisation folder):
channel_extraction_driver_gen.py
forced_mn_driver_gen.py
chi_multidriver_gen_qsub_ver.py
You will need a text file containing a list of your basins with their junction index numbers (JI). The text file can take the following format: (note, the header is not actually included in the file.)
| Basin Name | Starting JI | Other Columns as needed |
|---|---|---|
| turnpike | 1468 | soupy_twists |
| reginald | 5167 | spam_and_eggs |
| tubador | 6422 | john_cleese |
| marmaduke | 13994 | antifreeze |
Only the first two columns are required as a minimum. Make sure you do not include a blank line at the end of the file, otherwise the script will return an error about the list index being out of range.
The standard version of this script spawns driver files that include the junction index to hel identfy which driver runs with which input data file/dem.
The qsub_ver version of the script spawns files that have driver names increasing linearly from 1 upwards to the maximum number of basins in your basin list. This is useful if you want to run the analysis on a system that has a job queuing utility such as PBS, SGE, xjobs, GNU Parallel, etc.
The LSD software is written to run completely in serial. (i.e. there is no built in parallelisation). However, if you are running the same analyses on a large number of basins, the problem becomes easy to set up in a parallel-like fashion, if you have access to some form of multiple cpu computing facility. (This is termed embarassingly/trivially parallel, since it isn’t really written in parallel code).
This doesn’t necessarily require a supercomputing cluster; most laptops are at least dual core (as of 2014), and some will have as many as 8 seperate cpu threads to run tasks on. That means you could be running analysis on 8 basins at a time, rather than waiting for each one to finish separately, in serial.
You can find out how many cpus/cores/threads you have to play with by typing:
nproc
at the terminal in linux/cygwin. On a Intel i5 laptop, for example, it would probably return 2 or 4 ‘cpus’ as the result. Without going into the details, a ‘cpu’ in this case is the smallest processing ‘unit’ (or a ‘thread’). Also, by means of some clever electronics, some Intel cpus have twice as many threads as they do physical cores. Hooray!
Now suppose you have 16 basins that you want to do the chi analysis on. Should you run them all one-by-one on a single thread/cpu and let the other 3 threads sit there idle-ing? No! Use a utility such as xjobs or GNU Parallel. This example uses GNU parallel:
Prepare your numbered driver files as required. This example uses 50 driver files named: test1.driver, test2.driver, etc. The job-queuer used is GNU parallel, which is installed by default on many linux distributions:
seq 50 | parallel --gnu -j 4 ../driver_functions ./ 'test{}.driver' &
or:
parallel --gnu -j 4 ../driver_functions/chi_get_profiles.exe ./ test{}.driver ::: {1..50} &
This will queue up you 50 separate jobs to run on 4 threads at a time (the -j 4 argument)
If you have access to a suitable cluster computer or other supercomputer (HPC - high performance computing) service (ARCHER, HECToR, Eddie, or your local department’s cluster) it will probably have some type of job submission service. SGE and PBS are two of the most commonly used ones. You use the job scheduler to submit your by providing a simple script that requests cpu and memory resources from the supercomputer. By running the script, the job(s) are then placed in a queue to be distributed across the nodes and cpus.
It is also possible to run jobs in ‘interactive’ mode, which skips the script writing part and just starts the jobs from the commmand line once you are logged in. This is ok for testing, but not advised when running large, numerous, or memory-intensive jobs as it tends to hog the resources. Consult your HPC service’s documentation for best practice.
PBS (Portable batch system) is the name of the job submission scheduler used on the UK’s ARCHER supercomuting service, housed in Edinburgh.
Here is an example of a batch of channel file extractions. 50 channel files were extracted in a few minutes, compared to several hours on a desktop/laptop computer.
First, set up your driver files using the python script described above. Upload them to the filesystem on the HPC service.
Upload (using scp) and compile the LSD Topo Tools software directly on the HPC system. I.e. do not upload it ready-compiled from your own computer, it will probably not work - unless you have compiled it with static libraries. (This is not the default option)
Upload any raster files required for whichever part of the LSD ToolBox you are using. You would normally do this using the linux scp command. Type man scp at the terminal for help on using this.
Now write the job submission script. The example below is for the PBS job submisison utility, but SGE has a very similar layout. See specific documentation for the one you are using.:
#!/bin/bash --login
#
#PBS -N chan_extrac
#PBS -A n02-ncas
#PBS -l walltime=2:0:0
#PBS -l select=serial=true:ncpus=1
#PBS -J 1-50
#PBS -r y
# Switch to current working directory
cd $HOME/wasatch_chi/channel_extraction_legacy/new_dense_chans/
# Run the serial program
../driver_functions/chi2_write_channel_file.exe ./ wasatch_$PBS_ARRAY_INDEX.driver
This is an array job: the array switch is specified by #PBS -J {RANGE OF VALUES}. The range will increment the variable $PBS_ARRAY_INDEX by 1 each time. In this example, the driver files used will be wasatch_1.driver, wasatch_2.driver, ..., and so on.
Since the job (jobs) are all written in serial code, there is no need to run this on parallel nodes, should they be available to you. This is specified here with #PBS -l select=serial=true:ncpus=1, since we only need one cpu for each individual job in the array. (Altough they will all run concurrently on multiple cpus).
There is no need to specifiy an output pipe. Screen output and any errors/aborts will be written to log files automatically.
Submit the array job using:
qsub my_array_script.pbs
The status of the jobs can be checked using qstat. (See the separate documentation for this, or just type ‘man qstat’)
SGE (Sun grid engine) is used on the Eddie cluster (Edinburgh) and Redqueen (Manchester). It works much the same way as PBS does. Here is an example script used to perform the chi analysis on the 50 channels we extracted in the previous example.:
#!/bin/bash
#$ -S /bin/bash # Inform SGE we are using the bash shell
#$ -cwd # Job will run in the current directory (where you ran qsub)
#$ -V # Inherit current environment (e.g., any loaded modulefiles)
#$ -q atmos-b.q # tell qsub to use the atmosci-b queue
#$ -t 1-50 # TASK GRID ARRAY with 50 jobs
../bin/chi_analysis.exe ../data_dir/ inputfile.${SGE_TASK_ID}.parameters
The ${SGE_TASK_ARRAY} part will increment by 1 across the sequence specified, so make sure your inputfiles conform to this format. e.g.:
inputfile.1.parameters
inputfile.2.parameters
inputfile.3.parameters
{etc...}
The job is submitted using qsub my_batch_job.sh. In similar fashion to the previous example, we are submitting a task grid array job, which is useful for carrying out sensitivity analysis (such as with the chi-analysis tools). The chi software is not written with any parallelisation, but we can simulate the effect here using multiple cpu threads on the cluster computer with the task-grid-array job.
In this example, we specified a particular queue to use: -q atmos-b.q. This is not necessary on some systems, so check the documentation first.
qstat on its own should display a list of all your running jobs by default. To delete a job, use:
qdel -j {JOB_ID_NUMBER}
The job ID number is found from qstat. To delete all your jobs, use:
qdel -u {USERNAME}
To delete specific tasks from an array of jobs (like in the above examples, use:
qdel -j {JOB_ID_NUMBER} -t {TASK_NUMBER(S)}
So if you want to delete the tasks 23 through 64 from job ID 1300. Use:
qdel -j 1300 -t 23-64
On certain HPC services, different modules or libraries need to be loaded or switched to create the right environment to run your software. Normally this is not an issue and nothing needs to be changed. By convention, the LSD Topo Tools software is compiled using the gcc compilers. (A set of free compilers released under the GNU licence). Some supercomputer clusters will come with a range of compilers such as Intel, Cray, etc... One of them will be loaded by default. If gcc is not the default, you may wish to swap it before compiling.:
module list
Will list all the currently loaded modules:
module avail
Will list the available modules. You can narrow down your search by doing:
module avail gcc
for example, to find the gcc-releated modules. To load, type module load gcc. If you are swapping modules, for example the Intel compiler for the gcc one, then do:
module swap intel gcc
Sometimes, you may find that there is a ready-to-use compiler suite/environment set up on your HPC. Such as:
PrgEnv-intel
PrgEnv-gnu
PrgEnv-cray
Loading or swapping one of these is the equivalent of the above, but loads a whole suite of other modules and libraries that match the compiler suite. It is probably best to use these if they are available.
Note: The compiler suite you use has nothing to do with the type of CPU or supercomputer manufacturer. So you don’t have to use the Cray compilers with a Cray supercomputer, for example. You may also find that the LSD Topo Tools compile perfectly fine using the Intel/Cray/whatever compilers, though we have only tested the code using the GNU/gcc ones.
Other modules that may of interest to LSDers include python (though not all HPCs will have an implementation installed.) netCDF (a data format for large data arrays).
Note: This may not be an issue on certain cluster systems. I include it here in case anyone is having similar issues.
Users of certain libraries (some of the fast fourier libraries, very new compiler features, etc.) may find that the libraries they have on their personal systems are not availble on the cluster computer. This can lead to errors when trying to recompile on the cluster service. As you won’t have any admin rights when using the HPC service, you can’t simply install the libraries yourself. (Though you could try asking the admin).
As an example, the gcc version on a the Redqueen (Manchester) cluster computing service is only maintained up to ver. 4.4. (The latest version as of 2014 is 4.8.x!). This rather antiquated version of gcc lacks a lot of the library features of the newer gcc compilers. For example, I had issues with out of date libstdc++ and libc shared libraries. Trying to boilerplate newer versions of the libraries into my home directory and link them to the existing binaries didn’t seem to work (the system would always seem to default to the older libraries).
A workaround for this, if your code is only dependent on a few (but recent versions) of certain libraries, is to compile your program on your desktop workstation with static-linked libraries, using the -static complier tag (for gcc/g++). By default, compilation is done with dynamically-linked libraries, which are stored somewhere on the local machine at compilation time, and so they don’t get transferred with the .exe file if you upload the program to a different machine.
Using static-linked libraries effectively combines any library functions into your .exe binary file, so there is no need to worry about library dependencies in an environment that you don’t have the admin rights to modify. (Such as on the supercomputer) The size of the executable file will be larger, but this should only be an issue if your program is dependent on (numerous) very large library files. Experiment with caution.
In the Makefile:
CC= g++ -static
...rest of makefile
When you are compiling this for the first time, you may get errors about certain libraries not being found. (In my case it was -lstdc++, -lm, and -lc. These libraries will probably be present on your computer, but only the dynamic versions. You will need to download and install the static counterparts of such libraries. In my case this was done on linux with:
yum install lstdc++-static
yum install glib-static
Again, this will vary depending on what flavour of linux you are using (or Cygwin).
Once you have compiled your static library binary, you can check if there are any linked dependencies by doing:
ldd ./<NAME_OF_EXECUTABLE>
It should say something along the lines of "This is not a dynamic executable" or suchlike. Provided the file was compiled in a similar linux distribution and bit-version (32/64-bit), the static executable should run without issues by uploading it to your supercomputer of choice.