[CIG-SEISMO] xgenerate_databases error

[Yingzi Ying]

Hi Dimitri,

I am using Stokes at ICHEC cluster, whose each node has 16 processors and 
24G of ram.

Following your suggestions, I changed my complier to ifort and set
-----------------------------------
FLAGS_CHECK="-O3 -assume byterecl -traceback -ftrapuv -ftz -check all 
-traceback"
-----------------------------------
in flags.guess.

The followings are from PBS error snip:
-----------------------------------
forrtl: severe (59): list-directed I/O syntax error, unit -5, file 
Internal List-Directed Read
Image              PC                Routine            Line        Source
xmeshfem3D         000000000063E3BE  Unknown               Unknown  Unknown
xmeshfem3D         000000000063CE56  Unknown               Unknown  Unknown
xmeshfem3D         00000000005F4342  Unknown               Unknown  Unknown
xmeshfem3D         00000000005BE55B  Unknown               Unknown  Unknown
xmeshfem3D         00000000005BDAC2  Unknown               Unknown  Unknown
xmeshfem3D         00000000005D0DCB  Unknown               Unknown  Unknown
xmeshfem3D         00000000005CF5FC  Unknown               Unknown  Unknown
xmeshfem3D         000000000040715D  read_value_intege          42 
read_value_parameters.f90
xmeshfem3D         00000000004230FD  readparfile_mp_re         159 
read_parameter_file.f90
xmeshfem3D         000000000040AFEE  meshfem3d_                332 
meshfem3D.f90
xmeshfem3D         0000000000407000  MAIN__                     32 
program_meshfem3D.f90
xmeshfem3D         0000000000406F7C  Unknown               Unknown  Unknown
libc.so.6          00007F0E6CA76BC6  Unknown               Unknown  Unknown
xmeshfem3D         0000000000406E79  Unknown               Unknown  Unknown
mpiexec_raw: Warning: tasks 0-35 exited with status 59.
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 30) - process 0
...
mpiexec_raw: Warning: tasks 0-35 exited with status 30.
-----------------------------------

Here I just only called 3 nodes in the PBS script and reduced walltime 
to avoid waiting too long time in the queue. It then seems that 
meshfem3d crashed.

Please help me to have a look what goes wrong. Thank you.

Best,
Yingzi

On 12/07/2012 07:04 PM, Dimitri Komatitsch wrote:
>
> Hi,
>
> You are likely running out of memory per core. In some allocate()
> statements we do not check the error code (we should, but we sometimes
> don't) and thus the program may then segfault when trying to use some
> arrays that have not been allocated if there is not enough memory left.
>
> Could you try running with full debugging and checking turned on in your
> compiler? (e.g. adding -check all -traceback to flags.guess and
> rerunning "configure") (-check all -traceback is for Intel ifort but
> there are similar options for other compilers; some of them are already
> listed in flags.guess).
> Then please email me the error message and we will add a check right
> after the corresponding allocate() statement.
>
> Thank you,
> Dimitri.
>
> On 12/07/2012 01:02 PM, Yingzi Ying wrote:
>> Hi all,
>>
>> I am using v2.0.1 specfem3d and meshfem3d. I have no problem in running
>> xgenerate_databases with relatively small mesh size, for example,
>> 144*144*72 with 324 cpus. But when I increase the mesh to 576*576*144
>> with the same cpu number, I always meet the xgenerate_databases crash
>> with following pbs error message
>> ------------------------
>> mpiexec_raw: Warning: tasks
>> 0,17,35-36,53,71-72,89,96,108,125,143-144,161-162,179-180,197,215-216,233-234,251-252,269,287-288,306,323
>>
>> died with signal 9 (Killed).
>> mpiexec_raw: Warning: tasks
>> 1-16,18-34,37-52,54-70,73-88,90-95,97-107,109-124,126-142,145-160,163-178,181-196,198-214,217-232,235-250,253-268,270-286,289-305,307-322
>>
>> died with signal 15 (Terminated).
>> ------------------------
>>
>> I also attached the Mesh_Par_file.
>>
>> Anyone who was in the similar situation and knows how to solve this
>> problem please kindly give me a help.
>>
>> Thanks in advance.
>>
>> Best regards,
>> Yingzi
>>
>>
>>
>> _______________________________________________
>> CIG-SEISMO mailing list
>> CIG-SEISMO at geodynamics.org
>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>