[CIG-SEISMO] BG/P support for SPECFEM3d

Wed Mar 3 10:06:16 PST 2010

Dear all,

Please also note that we will release a much faster,
brand new version (version 5.0.0) in a few days, which
runs up to twice faster on some machines. Therefore I suggest you 
upgrade to that version as soon as it is released
(probably tomorrow or on Friday).

Best regards,
Dimitri.

Dimitri Komatitsch wrote:
> 
> Dear all,
> 
> I agree, the problem comes from the fact that the code is compiled
> on a frontend machine that cannot run the executable code, and we need
> "xcreate_header_file" to be able to run on the machine on which we compile.
> 
> I see two options:
> 
> 1/ if the xlf compiler is also installed on the Blue Gene nodes,
> create a script that runs "make all" on one node instead of on the frontend
> 
> 2/ otherwise, this will work:
> 
> - create two copies of the Makefile:
> 
> mv Makefile Makefile1
> cp Makefile1 Makefile2
> 
> In Makefile1 put this instead of the current values:
> 
> FLAGS_CHECK = -O0
> FLAGS_NO_CHECK = -O0
> 
> and replace
> 
> create_header_file: $O/program_create_header_file.o $(LIBSPECFEM)
>  ${FCCOMPILE_CHECK} -o xcreate_header_file 
> $O/program_create_header_file.o $(LIBSPECFEM)
> 
> with
> 
> create_header_file: $O/program_create_header_file.o $O/exit_mpi.o 
> $(LIBSPECFEM)
>   ${MPIFCCOMPILE_CHECK} -o xcreate_header_file 
> $O/program_create_header_file.o $O/exit_mpi.o $(LIBSPECFEM)
> 
> In Makefile2 comment out these two lines:
> 
> #OUTPUT_FILES/values_from_mesher.h: xcreate_header_file
> # ./xcreate_header_file
> 
> 
> Then:
> 
>  make -f Makefile1 clean
>  make -f Makefile1 create_header_file
>  ./xcreate_header_file
>  make -f Makefile2 clean
>  make -f Makefile2 meshfem3D
>  make -f Makefile2 specfem3D
> 
> should work.
> 
> I am going to cut and paste this in the user manual.
> (or maybe change the "configure" script)
> 
> Best regards,
> Dimitri.
> 
> 
> Peter Danecek wrote:
>> On 27 Feb 2010, at 09:37, Mark Cheeseman wrote:
>>
>>> Sorry, I forgot to include the error messages I receive:
>>>
>>> When I first try to compile with the IBM XLF90 compiler, I first get  
>>> the error message about the xcreate_header_file binary
>>>
>>> 1501-510  Compilation successful for file exit_mpi.f90.
>>> mpixlf90  -O3 -qsave -qstrict -qtune=450 -qarch=450d -qfree=f90 - 
>>> qsuffix=f=f90 -qhalt=e -qsigtrap - qinitauto 
>>> =7Fswap_all,mxm_m1_m2_5points,mxm_m1_m1_5points,mxm_m2_m1_5points -C  
>>> -qddim -qfloat=nans -qfullpath -o xmeshfem3D j/exit_mpi.o obj/ 
>>> libspecfem.a
>>> mkdir -p OUTPUT_FILES
>>> ./xcreate_header_file
>>> make: *** [OUTPUT_FILES/values_from_mesher.h] Illegal instruction
>>>
>>> Running the ./xcreate_header_file seems to work fine with the  
>>> required header file being generated in the OUTPUT_FILES  
>>> subdirectory.  Next, I comment out the call to the  
>>> xcreate_header_file binary in the main Makefile and rerun make.  The  
>>> result is a large number of error messages about zero-sized arrays  
>>> in the following source code files
>>>
>>>     compute_forces_crust_mantle.f90
>>>     compute_forces_inner_core.f90
>>>     compute_forces_outer_core.f90
>>>     write_movie_volume.f90
>>>
>>> with the error messages being
>>> .
>>> .
>>> "./write_movie_volume.f90", line 425.14: 1511-013 (W) The value of  
>>> the DO-loop increment should be negative when initial value is  
>>> greater than the terminal value.
>>> "./write_movie_volume.f90", line 429.12: 1516-152 (S) Zero-sized  
>>> arrays must not be subscripted.
>>> "./write_movie_volume.f90", line 431.18: 1516-152 (S) Zero-sized  
>>> arrays must not be subscripted.
>>> "./write_movie_volume.f90", line 432.28: 1516-152 (S) Zero-sized  
>>> arrays must not be subscripted.
>>> ** write_movie_volume_vector   === End of Compilation 4 ===
>>> "./write_movie_volume.f90", line 495.16: 1511-013 (W) The value of  
>>> the DO-loop increment should be negative when initial value is  
>>> greater than the terminal value.
>>> "./write_movie_volume.f90", line 502.52: 1516-152 (S) Zero-sized  
>>> arrays must not be subscripted.
>>> ** write_movie_volume_divcurl   === End of Compilation 5 ===
>>> 1501-511  Compilation failed for file write_movie_volume.f90.
>>> .
>>> .
>>> .
>>> The actual offending parameters (after some tracking) are those  
>>> defined in the header file
>>>
>>>     ./OUTPUT_FILES/values_from_mesher.h
>>>
>>> and include
>>>
>>>    NSPEC_CRUST_MANTLE, NSPEC_OUTER_CORE, NSPEC_INNER_CORE
>>>
>>> It looks like I am missing an important configuration step or  
>>> datafile.  Any assistance would be greatly appreciated.
>>>
>>
>>
>> Hi Mark,
>>
>> The xcreate_header_file executable is used to create some header file  
>> (values_from_mesher.h) which otherwise would be created by the mesher  
>> exec. But in contrast to the mesher itself, it does not need to be 
>> run  in parallel though the queuing system. Instead 
>> xcreate_header_file  calculates/estimates the size of several arrays 
>> immediately and can be  executed during compilation process.
>>
>>  From what you are writing, i would guess as well that the problem  
>> arrises from the cross compiler environment. The xcreate_header_file  
>> binary is created for the target machine, but executed on the  
>> compiling machine itself. That's why the makefile fails. So you could  
>> try two things:
>>
>> (1) compile the xcreate_header_file exec for the local compiling  
>> machine (not with the cross compiler) to generate the  
>> values_from_mesher.h header file during compilation. You probably 
>> need  to compile some dependences too and it is probably not straight  
>> forward to get the Makefile working properly.
>> NOTE: I am not very sure to which extend architectural differences  
>> will influences the resulting estimates in values_from_mesher.h file.  
>> So be careful with this approach.
>>
>> (2) use the more conservative way: First (cross-)compile the mesher  
>> executable only (make target: meshfem3D) and run it through the queue  
>> system. Than use the generated values_from_mesher.h file to compile  
>> the solver (make target: specfem3D) .
>>
>> Running xcreate_header_file through the queue system (serial job) --  
>> as I understand you have tried -- might also work, though I do not 
>> see  any benefits with respect to approach (2). From the error 
>> messages,  however, it looks like the generated  values_from_mesher.h 
>> is not  valid or incomplete. Check if the content looks reasonable. 
>> (does the  execution really complete correctly, is your Par_file 
>> reasonable)
>>
>> An other problem might arrise from running the ./configure script (on  
>> the compiling machine, i suppose). To my understanding this should  
>> fail for the same reason the xcreate_... binary does, or at best give  
>> results relative to the compiling machine and not relative to the  
>> target machine. You might want to check this.
>>
>> In conclusion I would suggest you:
>> - check if your configure really work fine, and if the results are  
>> relative to the target machine
>> - check your configuration in Par_file
>> - recompile the mesher (make meshfem3D)
>> - run it
>> - check and copy the values_from_mesher.h file
>> - compile the solver
>>
>> Hope this could be helpful ....
>> ~petr
>>
>>
>> ---------------------------------------------------------
>> Peter Danecek, PhD -- Ricercatore
>> Istituto Nazionale di Geofisica e Vulcanologia (INGV)
>> Sezione di Bologna
>> via Donato Creti, 12, I-40128 Bologna
>>
>> tel: (+39) 051 4151-458
>> fax: (+39) 051 4151-499
>> ---------------------------------------------------------
>>
>> _______________________________________________
>> CIG-SEISMO mailing list
>> CIG-SEISMO at geodynamics.org
>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>
> 

-- 
Dimitri Komatitsch - dimitri.komatitsch aT univ-pau.fr
Professor, University of Pau, Institut universitaire de France,
CNRS and INRIA Magique3D, France   http://www.univ-pau.fr/~dkomati1