[CIG-SHORT] Run on cluster using SLURM

[Brad Aagaard]

Birendra,

If you are getting an import error when importing petscmodule, then it 
is probably an environment issue. Have you sent us the whole error 
message (I might have missed it)? If you do something to setup your 
environment for PyLith after logging in, then you need to build that 
into your shell script too. Alternatively, some batch systems have 
arguments that save the environment settings from when you submit the 
job for use when the job is run.

Regards,
Brad


On 5/26/14, 12:46 PM, Birendra jha wrote:
> Tabrez
>
> Thank you for the tip. However, my problem is to find out the mpirun command, not the script. I have a slurm job script that runs a simple mpi helloworld.c but I still haven't figured out the correct mpirun command to submit a pylith run. I am getting importerror with libpetsc.so. So it's probably a path issue in my mpirun command.
>
> Thanks and regards
> Birendra
> --------------------------------------------
> On Mon, 5/26/14, Tabrez Ali <stali at geology.wisc.edu> wrote:
>
>   Subject: Re: [CIG-SHORT] Run on cluster using SLURM
>   To: cig-short at geodynamics.org
>   Date: Monday, May 26, 2014, 4:38 AM
>
>   Some machines offer the
>   option of running jobs interactively (for
>   testing/debugging etc.). E.g. on our cluster
>   (also uses slurm) we can
>   request 2 nodes,
>   each having 16 cores using the following command
>
>   # salloc -N 2 -n 32 bash
>
>   and then we can just run any
>   MPI program normally, e.g.,
>
>   # mpiexec -n 32 ./a.out
>
>   or alternatively
>
>   # salloc -N 2 -n 32 mpiexec -n 32 ./a.out
>
>   You can investigate if
>   interactive jobs are allowed your cluster.
>
>   T
>
>   On
>   05/25/2014 01:56 PM, Birendra jha wrote:
>   > I tried this as follows but it does not
>   work:
>   >
>   > 1. Get the
>   mpirun command from a cluster that uses PBS (and pylith
>   works there) by
>   > pylith test.cfg
>   --scheduler.dry
>   >
>   > 2.
>   Create a bash script e.g. slurm.sh that has the mpirun
>   command from PBS cluster except the following parts:
>   > -np ${nodes} -machinefile
>   ${PBS_NODEFILE}
>   > --scheduler.dry
>   --nodes=1 --macros.nodes=1 --macros.job.name=
>   --macros.job.id=None
>   >
>   > Here is my slurm.sh:
>   >
>   > #!/bin/bash
>   > #SBATCH --job-name=testjob
>   > #SBATCH --workdir=/home2/bjha/test
>   > #SBATCH --output=test.out
>   > #SBATCH --error=test.err
>   > #SBATCH --nodes=2
>   >
>   #SBATCH --ntasks-per-node=4
>   >
>   > mpirun /home2/bjha/pylith/bin/mpinemesis
>   --pyre-start
>   >   /home2/bjha/pylith/bin:/home2/bjha/pylith/lib/python2.7/site-packages/pythia-0.8.1.16-py2.7.egg:/home2/bjha/pylith/lib/python2.7/site-packages:/home2/bjha/pylith/lib64/python2.7/site-packages:/home2/bjha/test:/usr/lib/python2.7:/usr/lib/python2.7/plat-linux2:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload:/usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages/PIL:/usr/lib/python2.7/dist-packages/gst-0.10:/usr/lib/python2.7/dist-packages/gtk-2.0:/usr/lib/pymodules/python2.7:/usr/lib/python2.7/dist-packages/ubuntu-sso-client:/usr/lib/python2.7/dist-packages/ubuntuone-client:/usr/lib/python2.7/dist-packages/ubuntuone-control-panel:/usr/lib/python2.7/dist-packages/ubuntuone-couch:/usr/lib/python2.7/dist-packages/ubuntuone-installer:/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol:/home2/bjha/pylith/lib64/python2.7/site-packages
>   pythia mpi:mpistart
>   >   pylith.apps.PyLithApp:PyLithApp
>   test.cfg
>   >
>   > 3.
>   Submit
>   > sbatch slurm.sh
>   >
>   > Question: if
>   Pylith/Pyre does not support SLURM--like it does PBS, SGE,
>   LSF---how can I extract the mpirun command that I can submit
>   to a SLURM cluster?
>   >
>   > Thanks and regards
>   >
>   Birendra
>   >
>   >
>   --------------------------------------------
>   > On Sun, 5/25/14, Birendra jha<bjha7333 at yahoo.com>
>   wrote:
>   >
>   >   Subject: Run on cluster using
>   SLURM
>   >   To: cig-short at geodynamics.org
>   >   Date: Sunday, May 25, 2014,
>   10:21 PM
>   >
>   >   Hi
>   >
>   >   Is it possible to submit a
>   pylith run to a
>   >   cluster
>   that uses SLURM, just like in case of PBS or SGE?
>   >
>   >
>   >   Thanks and regards
>   >   Birendra
>   >
>   >
>   _______________________________________________
>   > CIG-SHORT mailing list
>   > CIG-SHORT at geodynamics.org
>   > http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
>
>   --
>   No one trusts a model except the one who wrote
>   it; Everyone trusts an observation except the one who made
>   it- Harlow Shapley
>
>
>   _______________________________________________
>   CIG-SHORT mailing list
>   CIG-SHORT at geodynamics.org
>   http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>