[CIG-SEISMO] xgenerate_databases error
Yingzi Ying
yingzi.ying at me.com
Fri Dec 7 14:50:43 PST 2012
Hi Dimitri,
I am using Stokes at ICHEC cluster, whose each node has 16 processors and
24G of ram.
Following your suggestions, I changed my complier to ifort and set
-----------------------------------
FLAGS_CHECK="-O3 -assume byterecl -traceback -ftrapuv -ftz -check all
-traceback"
-----------------------------------
in flags.guess.
The followings are from PBS error snip:
-----------------------------------
forrtl: severe (59): list-directed I/O syntax error, unit -5, file
Internal List-Directed Read
Image PC Routine Line Source
xmeshfem3D 000000000063E3BE Unknown Unknown Unknown
xmeshfem3D 000000000063CE56 Unknown Unknown Unknown
xmeshfem3D 00000000005F4342 Unknown Unknown Unknown
xmeshfem3D 00000000005BE55B Unknown Unknown Unknown
xmeshfem3D 00000000005BDAC2 Unknown Unknown Unknown
xmeshfem3D 00000000005D0DCB Unknown Unknown Unknown
xmeshfem3D 00000000005CF5FC Unknown Unknown Unknown
xmeshfem3D 000000000040715D read_value_intege 42
read_value_parameters.f90
xmeshfem3D 00000000004230FD readparfile_mp_re 159
read_parameter_file.f90
xmeshfem3D 000000000040AFEE meshfem3d_ 332
meshfem3D.f90
xmeshfem3D 0000000000407000 MAIN__ 32
program_meshfem3D.f90
xmeshfem3D 0000000000406F7C Unknown Unknown Unknown
libc.so.6 00007F0E6CA76BC6 Unknown Unknown Unknown
xmeshfem3D 0000000000406E79 Unknown Unknown Unknown
mpiexec_raw: Warning: tasks 0-35 exited with status 59.
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 30) - process 0
...
mpiexec_raw: Warning: tasks 0-35 exited with status 30.
-----------------------------------
Here I just only called 3 nodes in the PBS script and reduced walltime
to avoid waiting too long time in the queue. It then seems that
meshfem3d crashed.
Please help me to have a look what goes wrong. Thank you.
Best,
Yingzi
On 12/07/2012 07:04 PM, Dimitri Komatitsch wrote:
>
> Hi,
>
> You are likely running out of memory per core. In some allocate()
> statements we do not check the error code (we should, but we sometimes
> don't) and thus the program may then segfault when trying to use some
> arrays that have not been allocated if there is not enough memory left.
>
> Could you try running with full debugging and checking turned on in your
> compiler? (e.g. adding -check all -traceback to flags.guess and
> rerunning "configure") (-check all -traceback is for Intel ifort but
> there are similar options for other compilers; some of them are already
> listed in flags.guess).
> Then please email me the error message and we will add a check right
> after the corresponding allocate() statement.
>
> Thank you,
> Dimitri.
>
> On 12/07/2012 01:02 PM, Yingzi Ying wrote:
>> Hi all,
>>
>> I am using v2.0.1 specfem3d and meshfem3d. I have no problem in running
>> xgenerate_databases with relatively small mesh size, for example,
>> 144*144*72 with 324 cpus. But when I increase the mesh to 576*576*144
>> with the same cpu number, I always meet the xgenerate_databases crash
>> with following pbs error message
>> ------------------------
>> mpiexec_raw: Warning: tasks
>> 0,17,35-36,53,71-72,89,96,108,125,143-144,161-162,179-180,197,215-216,233-234,251-252,269,287-288,306,323
>>
>> died with signal 9 (Killed).
>> mpiexec_raw: Warning: tasks
>> 1-16,18-34,37-52,54-70,73-88,90-95,97-107,109-124,126-142,145-160,163-178,181-196,198-214,217-232,235-250,253-268,270-286,289-305,307-322
>>
>> died with signal 15 (Terminated).
>> ------------------------
>>
>> I also attached the Mesh_Par_file.
>>
>> Anyone who was in the similar situation and knows how to solve this
>> problem please kindly give me a help.
>>
>> Thanks in advance.
>>
>> Best regards,
>> Yingzi
>>
>>
>>
>> _______________________________________________
>> CIG-SEISMO mailing list
>> CIG-SEISMO at geodynamics.org
>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>
More information about the CIG-SEISMO
mailing list