[CIG-SHORT] Problem attempting to run on mutiple processors

[Leif Strand]

Oliver,

Thanks for the script. I have attached it to a new issue in our bug 
tracker, in case others may find it useful:

    http://geodynamics.org/roundup/issues/issue117

In the future, we might support SGE directly, as we do for LSF and PBS. 
(On LSF and PBS systems, PyLith can generate the batch script for you.)

I'm not sure, but I don't think any of the "macros" arguments are 
strictly necessary. They are internal machinery that you wouldn't 
ordinarily see. But, you might find them useful. For instance, 
"--macros.job.id=12345" sets the "job.id" macro, which you would 
reference using ${job.id} within your .cfg input file. This is designed 
to make it easy to generate a different set of output files for each run:

[pylith3d]
outputFileRoot = output-${job.id}

For this to work, you'd have to set the macro to a unique ID from within 
your batch script. On LSF and PBS systems, there are environment 
variables containing the job ID (LSB_JOBID and PBS_JOBID, respectively). 
So on an LSF system, PyLith does something like the following:

    --macros.job.id=$LSB_JOBID

Perhaps there is an equivalent environment variable for SGE.

PyLith v0.8 does not support parallel output; each processor writes a 
separate set of output files. For the *.inp files, you first create a 
UCD file for each processor using the 'makeucd' utility. From within 
ParaView, you can combine the output for multiple processors. This is 
detailed in the manual under the "Tutorials" section.

--Leif


Oliver Boyd wrote:
> The --launcher.dry was useful in giving me the command that can be submitted
> to SGE. The script for a particular run of qsub with 2 processors, run as
> 'qsub -pe mpich 2 script', now looks like
>
> #!/bin/bash
> #$ -cwd
> #$ -j y
> #$ -S /bin/bash
> MPI_DIR=/opt/mpich/gnu
> $MPI_DIR/bin/mpirun -np $NSLOTS -machinefile $TMP/machines
> /data/Software/Store/pylith3d-0.8.2/pylith3d/mpipypylith3d --pyre-start
> /data/Software/PyLith-0.8.2:/data/Software/Store/pylith3d-0.8.2/python/pythi
> a-0.8.1.3-py2.4.egg:/data/Softwar
> e/Store/pylith3d-0.8.2/python/Cheetah-2.0rc8-py2.4-linux-x86_64.egg:/data/So
> ftware/Store/pylith3d-0.8.2/python/merlin-1.
> 2.egg PyLith mpi:mpistart pylith3d.PyLithApp:PyLithApp pylith3d-npX.cfg
> --nodes=2 --macros.nodes=2 --macros.job.name= --
> macros.job.id=24652
>
> I have attached qsubPyLith, which will create this script and run pylith
> with sge (not sure of you want it). To use it type 'qsubPyLith np
> cfgfile1.cfg cfgfile2.cfg ...', where np is the number of processors.
>
> Couple more questions:
> 1) What do I do about --macros.job.id? Is it necessary?
> 2) Running the above or pylith3dapp.py pylith3d-np2.cfg where --nodes=2
> produces bmrsnog.0.* and bmrsnog.1.*. What am I supposed to do with these
> two sets of files?
>
> Please let me know if you see any problems with what I've done. Thanks for
> your help.
>
> Oliver
>
>