[CIG-MC] PBS batch configurations

[Robert Moucha]

Hi Leif and Eh,

I tried your suggestions, two questions:

1) In order for me to specifically use infiniband nodes, I must add
"ib" to the PBS -l line in the script as follows:

#PBS -l nodes=2:ib:ppn=8

  Is this possible, or should I generate the script, edit it to add ib
and submit it?

2) The number of processors np appears to be incorrectly set for
mpirun, see below:

Thanks Rob

I modified the script example1.cfg to use 16 procs:

-------------------------------------------------
[CitcomS]
steps = 70

[CitcomS.controller]
monitoringFrequency = 10

[CitcomS.solver]
datafile = example1

[CitcomS.solver.mesher]
nprocx =  4
nprocy =  4
nodex  = 33
nodey  = 33
nodez  = 17
-------------------------------------------------

I use the CitcomS.cfg:

-------------------------------------------------
[CitcomS]
scheduler = pbs

[CitcomS.launcher]
command = mpirun -np ${nodes} -hostfile ${PBS_NODEFILE}

[CitcomS.pbs]
ppn = 8  ; processors per node
-------------------------------------------------

The command:

citcoms example1.cfg --job.name=test --job.walltime=1*hour --scheduler.dry

Generates the following output; note that number of nodes for
pycitcoms is the correct 16, but in the comment the mpirun " -np
${nodes} ", is the ${nodes} taken from the PBS line where nodes=2, or
is taken from the pycitcoms line where --nodes=16.
-------------------------------------------------

#!/bin/sh
#PBS -S /bin/sh
#PBS -N test
#PBS -o stdout.txt
#PBS -e stderr.txt
#PBS -l nodes=2:ppn=8,walltime=1:00:00

cd $PBS_O_WORKDIR
/home/rmoucha/convection/CitComS/bin/pycitcoms --pyre-start
/home/rmoucha/convection/CitComS/bin:/home/rmoucha/convection/CitComS/lib/python2.4/site-packages/pythia-0.8.1.10-py2.4.egg:/home/rmoucha/convection/CitComS/lib/python2.4/site-packages/merlin-1.7.egg:/home/rmoucha/convection/CitComS/lib/python2.4/site-packages:/home/rmoucha/convection/CitComS/lib64/python2.4/site-packages:/home/rmoucha/convection/CitComS/share/CitcomS/examples:/usr/lib64/python24.zip:/usr/lib64/python2.4:/usr/lib64/python2.4/plat-linux2:/usr/lib64/python2.4/lib-tk:/usr/lib64/python2.4/lib-dynload:/usr/lib64/python2.4/site-packages:/usr/lib64/python2.4/site-packages/Numeric:/usr/lib64/python2.4/site-packages/PIL:/usr/lib64/python2.4/site-packages/gtk-2.0:/usr/lib/python2.4/site-packages
pythia pyre.schedulers:jobstart
--scheduler-class=pyre.schedulers.SchedulerPBS:SchedulerPBS
CitcomS.SimpleApp:SimpleApp example1.cfg --job.name=test
--job.walltime=1*hour --scheduler.dry --nodes=16 --macros.nodes=16
--macros.job.name=test

# ~~~~ comments ~~~~
# [mpich] command: mpirun -np ${nodes} -hostfile ${PBS_NODEFILE}
/home/rmoucha/convection/CitComS/bin/mpipycitcoms --pyre-start
/home/rmoucha/convection/CitComS/bin:/home/rmoucha/convection/CitComS/lib/python2.4/site-packages/pythia-0.8.1.10-py2.4.egg:/home/rmoucha/convection/CitComS/lib/python2.4/site-packages/merlin-1.7.egg:/home/rmoucha/convection/CitComS/lib/python2.4/site-packages:/home/rmoucha/convection/CitComS/lib64/python2.4/site-packages:/home/rmoucha/convection/CitComS/share/CitcomS/examples:/usr/lib64/python24.zip:/usr/lib64/python2.4:/usr/lib64/python2.4/plat-linux2:/usr/lib64/python2.4/lib-tk:/usr/lib64/python2.4/lib-dynload:/usr/lib64/python2.4/site-packages:/usr/lib64/python2.4/site-packages/Numeric:/usr/lib64/python2.4/site-packages/PIL:/usr/lib64/python2.4/site-packages/gtk-2.0:/usr/lib/python2.4/site-packages
pythia mpi:mpistart CitcomS.SimpleApp:SimpleApp example1.cfg
--job.name=test --job.walltime=1*hour --scheduler.dry --nodes=16
--macros.nodes=16 --macros.job.name=test --macros.job.id=None

# ~~~~ submit command ~~~~
# qsub < [script]


-------------------------------------------------

On Tue, Oct 27, 2009 at 4:44 PM, Leif Strand <leif at geodynamics.org> wrote:
> Hi Rob,
>
> There is an example PBS file on this page:
>
> http://www.geodynamics.org/cig/software/packages/cs/pythia/docs/batch
>
> Based on the information you gave, your CitcomS.cfg file should look like
> the following:
>
> ----------------------------------snip----------------------------------
> [CitcomS]
> scheduler = pbs
>
> [CitcomS.launcher]
> command = mpirun -np ${nodes} -hostfile ${PBS_NODEFILE}
>
> [CitcomS.pbs]
> ppn = 8  ; processors per node
> ----------------------------------snip----------------------------------
>
>
> You can set the job name and walltime from the command line:
>
>    citcoms --job.name=test --job.walltime=1*hour [...]
>
> Or, from a .cfg file:
>
> ----------------------------------snip----------------------------------
> [CitcomS.job]
> walltime = 1*hour
> name = test
> ----------------------------------snip----------------------------------
>
> To debug the PBS settings, use "--scheduler.dry":
>
>    citcoms --scheduler.dry
>
> This will dump the script to the terminal, instead of actually submitting
> it.
>
> --Leif
>
>
> Robert Moucha wrote:
>>
>> Hi All,
>>
>> I'm using a new cluster that uses the MOAB pbs scheduler, our previous
>> cluster did not use a batch system. I am trying to figure out how to
>> set the parameters in CitcomS.cfg, in the manual only lsf settings are
>> shown.  In particular, how do I specify multiple special options for
>> pbs. Is the following cfg file correct for the launch script below?
>> Or, does any one have a sample cfg file for pbs?
>>
>> [CitComS]
>> scheduler = pbs
>>
>> [CitcomS.pbs]
>> pbs-options = [-N test, -l nodes=2:ppn=8,walltime=1:00:00]
>>
>> [Citcoms.launcher]
>> command = mpirun
>>
>> Here is an example of a batch submission script that is used on the
>> cluster:
>>
>> #!/bin/bash
>> # MOAB/Torque submission script for SciNet GPC (ethernet)
>> #
>> #PBS -l nodes=2:ppn=8,walltime=1:00:00
>> #PBS -N test
>>
>> # DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
>> cd $PBS_O_WORKDIR
>>
>> # EXECUTION COMMAND; -np = nodes*ppn
>> mpirun -np 16 -hostfile $PBS_NODEFILE ./a.out
>>
>> Thanks all,
>> Rob
>>
>



-- 
GEOTOP - Département des Sciences de la Terre et de l'Atmosphère
Université du Québec à Montréal
CP 8888, succursale Centre-Ville
Montréall, Québec
Canada  H3C 3P8
Tel:     (1-514) 987-3000, ext 1554#
FAX:     (1-514) 987-3635