[CIG-SEISMO] specfem3d_globe with CUDA

Wed Apr 26 03:54:51 PDT 2017

Hi Phil, Hi all,

Thanks! Let me cc Daniel Peter and Peter Messmer, who are very familiar 
with the CUDA version. Daniel, would you know how to commit Phil's fix 
to the official Git version of the code? (and I guess also for 
SPECFEM3D_Cartesian and for SPECFEM2D?).

Thanks,
Best wishes,

Dimitri.

On 04/26/2017 05:12 AM, Phil Cummins wrote:
> Just a quick reply to my own post. I got xpsecfem3d to link by removing
> cuda_device_obj.o from the mpif90 link line, and I also had to add
> -lstdc++ to resolve the missing symbols from the c++ library.
>
> Phil Cummins wrote:
>> Hello again,
>>
>> Still trying to compile specfem3d_globe using CUDA. I have obtaiend
>> advice from our HPC center regarding the appropriate modules to use,
>> which are:
>>         intel-fc/16.0.3.210 intel-cc/16.0.3.210 openmpi/1.10.2 cuda/8.0
>> I hope they're going to be OK. But I just can't make sense of what the
>> Makefiule is trying to do with cuda_device_obj. As an exampe,  let's
>> consider what's happening with the CUDA module check_fields_gpu. First
>> it is compiled into object code:
>> nvcc -c src/gpu/check_fields_gpu.c -o obj/check_fields_gpu.cuda.o
>> --cudart=shared -I/apps/openmpi/1.10.2/include -DUSE_OLDER_CUDA4_GPU
>> -gencode=arch=compute_20,code=\"sm_20,compute_20\" -x cu -I./setup
>> -I./src/gpu/kernels.gen  -DUSE_CUDA
>>
>> That's fine (it does produce a deprecation warning). Then, for some
>> reason that I can't figure out, the makefile is trying to link this in
>> some way to the module cuda_device_obj, for which no source code seems
>> to exist and for which the command appears to have omitted the nvcc
>> compiler (so of course it doesn't work):
>>
>> DUSE_OLDER_CUDA4_GPU
>> -gencode=arch=compute_20,code=\"sm_20,compute_20\" -o
>> obj/cuda_device_obj.o obj/assemble_MPI_scalar_gpu.cuda.o
>> obj/assemble_MPI_vector_gpu.cuda.o obj/check_fields_gpu.cuda.o
>> obj/compute_add_sources_elastic_gpu.cuda.o
>> obj/compute_coupling_gpu.cuda.o
>> obj/compute_forces_crust_mantle_gpu.cuda.o
>> obj/compute_forces_inner_core_gpu.cuda.o
>> obj/compute_forces_outer_core_gpu.cuda.o
>> obj/compute_kernels_gpu.cuda.o obj/compute_stacey_acoustic_gpu.cuda.o
>> obj/compute_stacey_elastic_gpu.cuda.o obj/compute_strain_gpu.cuda.o
>> obj/helper_functions_gpu.cuda.o obj/initialize_gpu.cuda.o
>> obj/noise_tomography_gpu.cuda.o obj/prepare_mesh_constants_gpu.cuda.o
>> obj/transfer_fields_gpu.cuda.o obj/update_displacement_gpu.cuda.o
>> obj/write_seismograms_gpu.cuda.o
>> obj/save_and_compare_cpu_vs_gpu.cuda.o
>> obj/assemble_boundary_accel_on_device.cuda-kernel.o
>> obj/assemble_boundary_potential_on_device.cuda-kernel.o
>> obj/prepare_boundary_potential_on_device.cuda-kernel.o
>> obj/prepare_boundary_accel_on_device.cuda-kernel.o
>> obj/get_maximum_scalar_kernel.cuda-kernel.o
>> obj/get_maximum_vector_kernel.cuda-kernel.o
>> obj/compute_add_sources_adjoint_kernel.cuda-kernel.o
>> obj/compute_add_sources_kernel.cuda-kernel.o
>>
>> Then, the commands issued by the Makefile try to create the main
>> program xpsecfem3d by linking not only with cuda_device_obj.o, but
>> also with check_fields_gpu.cuda.o. So even if I could get the above
>> command to work somehow, the mpif90 command would almost certainly
>> fail because of multiple definitions. In any case, I am struggling a
>> bit because I just don't understand how or why cuda_device_obj.o is
>> meant to be used. Can anyone help?
>>
>> mpif90 -g   -xHost -fpe0 -ftz -assume buffered_io -assume byterecl
>> -align sequence -vec-report0 -std03 -diag-disable 6477 -implicitnone
>> -gen-interfaces -warn all -O3 -check nobounds -DFORCE_VECTORIZATION -o
>> ./bin/xspecfem3D ./obj/assemble_MPI_scalar.solver.o
>> ./obj/assemble_MPI_vector.solver.o ./obj/comp_source_spectrum.solver.o
>> ./obj/compute_adj_source_frechet.solver.o ./obj/convert_time.solver.o
>> ./obj/define_derivation_matrices.solver.o ./obj/file_io_threads.cc.o
>> ./obj/force_ftz.cc.o ./obj/get_backazimuth.solver.o
>> ./obj/get_cmt.solver.o ./obj/get_event_info.solver.o
>> ./obj/make_gravity.solver.o ./obj/netlib_specfun_erf.solver.o
>> ./obj/asdf_data.solverstatic_module.o
>> ./obj/comp_source_time_function.solverstatic.o
>> ./obj/specfem3D_par.solverstatic_module.o
>> ./obj/write_seismograms.solverstatic.o
>> ./obj/check_stability.solverstatic.o
>> ./obj/compute_add_sources.solverstatic.o
>> ./obj/compute_arrays_source.solverstatic.o
>> ./obj/compute_boundary_kernel.solverstatic.o
>> ./obj/compute_coupling.solverstatic.o
>> ./obj/compute_element.solverstatic.o
>> ./obj/compute_element_att_memory.solverstatic.o
>> ./obj/compute_element_strain.solverstatic.o
>> ./obj/compute_forces_acoustic_calling_routine.solverstatic.o
>> ./obj/compute_forces_viscoelastic_calling_routine.solverstatic.o
>> ./obj/compute_forces_crust_mantle_noDev.solverstatic.o
>> ./obj/compute_forces_crust_mantle_Dev.solverstatic.o
>> ./obj/compute_forces_inner_core_noDev.solverstatic.o
>> ./obj/compute_forces_inner_core_Dev.solverstatic.o
>> ./obj/compute_forces_outer_core_noDev.solverstatic.o
>> ./obj/compute_forces_outer_core_Dev.solverstatic.o
>> ./obj/compute_kernels.solverstatic.o
>> ./obj/compute_seismograms.solverstatic.o
>> ./obj/compute_stacey_crust_mantle.solverstatic.o
>> ./obj/compute_stacey_outer_core.solverstatic.o ./obj/finali\
>> ze_simulation.solverstatic.o ./obj/get_attenuation.solverstatic.o
>> ./obj/initialize_simulation.solverstatic.o
>> ./obj/iterate_time.solverstatic.o
>> ./obj/iterate_time_undoatt.solverstatic.o
>> ./obj/locate_receivers.solverstatic.o ./obj/locate_regular_points.s\
>> olverstatic.o ./obj/locate_sources.solverstatic.o
>> ./obj/multiply_arrays_source.solverstatic.o
>> ./obj/noise_tomography.solverstatic.o
>> ./obj/prepare_timerun.solverstatic.o
>> ./obj/read_adjoint_sources.solverstatic.o
>> ./obj/read_arrays_solver.solverstatic.o
>> ./obj/read_forward_arrays.solverstatic.o
>> ./obj/read_mesh_databases.solverstatic.o
>> ./obj/read_topography_bathymetry.solverstatic.o
>> ./obj/save_forward_arrays.solverstatic.o
>> ./obj/save_kernels.solverstatic.o
>> ./obj/save_regular_kernels.solverstatic.o
>> ./obj/setup_GLL_points.solverstatic.o
>> ./obj/setup_sources_receivers.solverstatic.o
>> ./obj/specfem3D.solverstatic.o ./obj/update_displ\
>> acement_LDDRK.solverstatic.o
>> ./obj/update_displacement_Newmark.solverstatic.o
>> ./obj/write_movie_output.solverstatic.o
>> ./obj/write_movie_volume.solverstatic.o
>> ./obj/write_movie_surface.solverstatic.o
>> ./obj/write_output_ASCII.solverstatic.o
>> ./obj/write_output_SAC.solverstatic.o
>> ./obj/assemble_MPI_scalar_gpu.cuda.o
>> ./obj/assemble_MPI_vector_gpu.cuda.o ./obj/check_fields_gpu.cuda.o
>> ./obj/compute_add_sources_elastic_gpu.cuda.o
>> ./obj/compute_coupling_gpu.cuda.o
>> ./obj/compute_forces_crust_mantle_gpu.cuda.o
>> ./obj/compute_forces_inner_core_gpu.cuda.o
>> ./obj/compute_forces_outer_core_gpu.cuda.o
>> ./obj/compute_kernels_gpu.cuda.o
>> ./obj/compute_stacey_acoustic_gpu.cuda.o
>> ./obj/compute_stacey_elastic_gpu.cuda.o
>> ./obj/compute_strain_gpu.cuda.o ./obj/helper_functions_gpu.cuda.o
>> ./obj/initialize_gpu.cuda.o ./obj/noise_tomography_gpu.cuda.o
>> ./obj/prepare_mesh_constants_gpu.cuda.o
>> ./obj/transfer_fields_gpu.cuda.o ./obj/update_displacement_gpu.cuda.o
>> ./obj/write_seismograms_gpu.cuda.o
>> ./obj/save_and_compare_cpu_vs_gpu.cuda.o  ./obj/cuda_device_obj.o
>> ./obj/assemble_boundary_accel_on_device.cuda-kernel.o
>> ./obj/assemble_boundary_potential_on_device.cuda-kernel.o
>> ./obj/prepare_boundary_potential_on_device.cuda-kernel.o
>> ./obj/prepare_boundary_accel_on_device.cuda-kernel.o
>> ./obj/get_maximum_scalar_kernel.cuda-kernel.o
>> ./obj/get_maximum_vector_kernel.cuda-kernel.o
>> ./obj/compute_add_sources_adjoint_kernel.cuda-kernel.o
>> ./obj/compute_add_sources_kernel.cuda-kernel.o
>> ./obj/compute_coupling_fluid_CMB_kernel.cuda-kernel.o
>> ./obj/compute_coupling_fluid_ICB_kernel.cuda-kernel.o
>> ./obj/compute_coupling_CMB_fluid_kernel.cuda-kernel.o
>> ./obj/compute_coupling_ICB_fluid_kernel.cuda-kernel.o
>> ./obj/compute_coupling_ocean_kernel.cuda-kernel.o
>> ./obj/write_seismograms_transfer_from_device_kernel.cuda-kernel.o
>> ./obj/write_seismograms_transfer_strain_from_device_kernel.cuda-kernel.o
>> ./obj/noise_transfer_surface_to_host_kernel.cuda-kernel.o
>> ./obj/noise_add_source_master_rec_kernel.cuda-kernel.o
>> ./obj/noise_add_surface_movie_kernel.cuda-kernel.o
>> ./obj/compute_stacey_acoustic_kernel.cuda-kernel.o
>> ./obj/compute_stacey_acoustic_backward_kernel.cuda-kernel.o \
>> ./obj/compute_stacey_elastic_kernel.cuda-kernel.o
>> ./obj/compute_stacey_elastic_backward_kernel.cuda-kernel.o
>> ./obj/update_disp_veloc_kernel.cuda-kernel.o
>> ./obj/update_potential_kernel.cuda-kernel.o
>> ./obj/update_accel_elastic_kernel.cuda-kernel.o
>> ./obj/update_veloc_elastic_kernel.cuda-kernel.o
>> ./obj/update_accel_acoustic_kernel.cuda-kernel.o
>> ./obj/update_veloc_acoustic_kernel.cuda-kernel.o
>> ./obj/compute_rho_kernel.cuda-kernel.o
>> ./obj/compute_iso_kernel.cuda-kernel.o
>> ./obj/compute_ani_kernel.cuda-kernel.o
>> ./obj/compute_hess_kernel.cuda-kernel.o
>> ./obj/compute_acoustic_kernel.cuda-kernel.o
>> ./obj/compute_strength_noise_kernel.cuda-kernel.o
>> ./obj/compute_ani_undoatt_kernel.cuda-kernel.o
>> ./obj/compute_iso_undoatt_kernel.cuda-kernel.o
>> ./obj/compute_strain_kernel.cuda-kernel.o
>> ./obj/outer_core_impl_kernel_forward.cuda-kernel.o
>> ./obj/outer_core_impl_kernel_adjoint.cudakernel.o
>> ./obj/inner_core_impl_kernel_forward.cuda-kernel.o
>> ./obj/inner_core_impl_kernel_adjoint.cuda-kernel.o
>> ./obj/crust_mantle_impl_kernel_forward.cuda-kernel.o
>> ./obj/crust_mantle_impl_kernel_adjoint.cuda-kernel.o
>> ./obj/visual_vtk_stubs.visualc.o  ./obj/shared_par.shared_module.o
>> ./obj/auto_ner.shared.o ./obj/binary_c_io.cc.o
>> ./obj/broadcast_computed_parameters.shared.o ./obj/calendar.shared.o
>> ./obj/count_elements.shared.o ./obj/count_number_of_sources.shared.o
>> ./obj/count_points.shared.o ./obj/create_name_database.shared.o
>> ./obj/define_all_layers.shared.o ./obj/exit_mpi.shared.o
>> ./obj/flush_system.shared.o ./obj/get_model_parameters.shared.o
>> ./obj/get_timestep_and_layers.shared.o ./obj/gll_library.shared.o
>> ./obj/hex_nodes.shared.o ./obj/intgrl.shared.o
>> ./obj/lagrange_poly.shared.o ./obj/make_ellipticity.shared.o
>> ./obj/model_prem.shared.o ./obj/model_topo_bathy.shared.o
>> ./obj/parallel.sharedmpi.o ./obj/param_reader.cc.o
>> ./obj/read_compute_parameters.shared.o
>> ./obj/read_parameter_file.shared.o
>> ./obj/read_value_parameters.shared.o ./obj/recompute_jacobian.shared.o
>> ./obj/reduce.shared.o ./obj/rthetaphi_xyz.shared.o
>> ./obj/spline_routines.shared.o ./obj/write_VTK_file.shared.o
>> ./obj/adios_method_stubs.cc.o       -lcudart
>>
>>
>> Thanks,
>>
>> - Phil
>>
>> P.S. when i simply remove cuda_dev_obj.o from the above mpif90
>> command, the link fails with:
>> ./obj/assemble_MPI_scalar_gpu.cuda.o:(.eh_frame+0x12): undefined
>> reference to `__gxx_personality_v0'
>> ./obj/assemble_MPI_vector_gpu.cuda.o:(.eh_frame+0x12): undefined
>> reference to `__gxx_personality_v0'
>> ./obj/check_fields_gpu.cuda.o:(.eh_frame+0x12): undefined reference to
>> `__gxx_personality_v0'
>> ./obj/compute_add_sources_elastic_gpu.cuda.o:(.eh_frame+0x12):
>> undefined reference to `__gxx_personality_v0'
>> ./obj/compute_coupling_gpu.cuda.o:(.eh_frame+0x12): undefined
>> reference to `__gxx_personality_v0'
>> ./obj/compute_forces_crust_mantle_gpu.cuda.o:(.eh_frame+0x12): more
>> undefined references to `__gxx_personality_v0' follow
>>
>> Phil Cummins wrote:
>>> Hello,
>>>
>>> I'm would like to use specfem3d_globe with my university's GPU
>>> cluster. I am trying to compile specfem3d_globe-7.0.0 after having
>>> configured it using:
>>> ./configure --with-cuda=cuda6
>>> I am using the ifort fortran compiler version 12.1.9.293, and the
>>> version of cuda I have loaded is 6.5.
>>> The "make all" command does a lot of compilation using ifort and
>>> nvcc, but eventually fails when trying to link xspecfem3D itself with
>>> the error:
>>>
>>> ifort: error #10236: File not found:  './obj/cuda_device_obj.o'
>>> make: *** [bin/xspecfem3D] Error 1
>>>
>>> Delving into what happened during the make (see below), I noticed
>>> that after using nvcc to compile crust_mantle_impl_kernel_adjoint.cu,
>>> it attempts to use the command DUSE_OLDER_CUDA4_GPU to compile
>>> cuda_device_obj, which of course doesn't work because
>>> DUSE_OLDER_CUDA4_GPU is a compiler option not a command. It seems
>>> that make has somehow lost the beginning of this particular
>>> compilation command.
>>> Has anyone else seen or know what to do about it?
>>> Thanks,
>>>
>>> - Phil
>>>
>>> P.S. Some of the hardware on our cluster can be used only with cuda
>>> version 8. Does anyone know if specfem3d will work with cuda8?
>>>
>>> nvcc -c src/gpu/kernels.gen/crust_mantle_impl_kernel_adjoint.cu -o
>>> obj/crust_ma\
>>> ntle_impl_kernel_adjoint.cuda-kernel.o   --cudart=shared
>>> -I/apps/openmpi/1.7.5/\
>>> include -DUSE_OLDER_CUDA4_GPU
>>> -gencode=arch=compute_20,code=\"sm_20,compute_20\\
>>> " -x cu -I./setup -I./src/gpu/kernels.gen  -DUSE_CUDA -include
>>> src/gpu/mesh_con\
>>> stants_gpu.h
>>> DUSE_OLDER_CUDA4_GPU
>>> -gencode=arch=compute_20,code=\"sm_20,compute_20\" -o obj/\
>>> cuda_device_obj.o obj/assemble_MPI_scalar_gpu.cuda.o
>>> obj/assemble_MPI_vector_gp\
>>> u.cuda.o obj/check_fields_gpu.cuda.o
>>> obj/compute_add_sources_elastic_gpu.cuda.o\
>>>  obj/compute_coupling_gpu.cuda.o
>>> obj/compute_forces_crust_mantle_gpu.cuda.o obj\
>>> /compute_forces_inner_core_gpu.cuda.o
>>> obj/compute_forces_outer_core_gpu.cuda.o \
>>> obj/compute_kernels_gpu.cuda.o obj/compute_stacey_acoustic_gpu.cuda.o
>>> obj/compu\
>>> te_stacey_elastic_gpu.cuda.o obj/compute_strain_gpu.cuda.o
>>> obj/helper_functions\
>>> _gpu.cuda.o obj/initialize_gpu.cuda.o obj/noise_tomography_gpu.cuda.o
>>> obj/prepa\
>>> re_mesh_constants_gpu.cuda.o obj/transfer_fields_gpu.cuda.o
>>> obj/update_displace\
>>> ment_gpu.cuda.o obj/write_seismograms_gpu.cuda.o
>>> obj/save_and_compare_cpu_vs_gp\
>>> u.cuda.o obj/assemble_boundary_accel_on_device.cuda-kernel.o
>>> obj/assemble_bound\
>>> ary_potential_on_device.cuda-kernel.o
>>> obj/prepare_boundary_potential_on_device.\
>>> cuda-kernel.o obj/prepare_boundary_accel_on_device.cuda-kernel.o
>>> obj/get_maximu\
>>> m_scalar_kernel.cuda-kernel.o
>>> obj/get_maximum_vector_kernel.cuda-kernel.o obj/c\
>>> ompute_add_sources_adjoint_kernel.cuda-kernel.o
>>> obj/compute_add_sources_kernel.\
>>> cuda-kernel.o
>>> nel.cuda-kernel.o
>>> obj/write_seismograms_transfer_strain_from_device_kernel.cuda\
>>> -kernel.o obj/noise_transfer_surface_to_host_kernel.cuda-kernel.o
>>> obj/noise_add\
>>> _source_master_rec_kernel.cuda-kernel.o
>>> obj/noise_add_surface_movie_kernel.cuda\
>>> -kernel.o obj/compute_stacey_acoustic_kernel.cuda-kernel.o
>>> obj/compute_stacey_a\
>>> coustic_backward_kernel.cuda-kernel.o
>>> obj/compute_stacey_elastic_kernel.cuda-ke\
>>> rnel.o obj/compute_stacey_elastic_backward_kernel.cuda-kernel.o
>>> obj/update_disp\
>>> _veloc_kernel.cuda-kernel.o obj/update_potential_kernel.cuda-kernel.o
>>> obj/updat\
>>> e_accel_elastic_kernel.cuda-kernel.o
>>> obj/update_veloc_elastic_kernel.cuda-kerne\
>>> l.o obj/update_accel_acoustic_kernel.cuda-kernel.o
>>> obj/update_veloc_acoustic_ke\
>>> rnel.cuda-kernel.o obj/compute_rho_kernel.cuda-kernel.o
>>> obj/compute_iso_kernel.\
>>> cuda-kernel.o obj/compute_ani_kernel.cuda-kernel.o
>>> obj/compute_hess_kernel.cuda\
>>> -kernel.o obj/compute_acoustic_kernel.cuda-kernel.o
>>> obj/compute_strength_noise_\
>>> kernel.cuda-kernel.o obj/compute_ani_undoatt_kernel.cuda-kernel.o
>>> obj/compute_i\
>>> so_undoatt_kernel.cuda-kernel.o
>>> obj/compute_strain_kernel.cuda-kernel.o obj/out\
>>> er_core_impl_kernel_forward.cuda-kernel.o
>>> obj/outer_core_impl_kernel_adjoint.cu\
>>> da-kernel.o obj/inner_core_impl_kernel_forward.cuda-kernel.o
>>> obj/inner_core_imp\
>>> l_kernel_adjoint.cuda-kernel.o
>>> obj/crust_mantle_impl_kernel_forward.cuda-kernel\
>>> .o obj/crust_mantle_impl_kernel_adjoint.cuda-kernel.o
>>> make: DUSE_OLDER_CUDA4_GPU: Command not found
>>> make: [obj/cuda_device_obj.o] Error 127 (ignored)
>>>
>>> --
>>> Phil Cummins
>>> Prof. Natural Hazards
>>> Research School of Earth Sciences
>>> Australian National University
>>>
>>> _______________________________________________
>>> CIG-SEISMO mailing list
>>> CIG-SEISMO at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>
>> _______________________________________________
>> CIG-SEISMO mailing list
>> CIG-SEISMO at geodynamics.org
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>
>
>
> _______________________________________________
> CIG-SEISMO mailing list
> CIG-SEISMO at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>

-- 
Dimitri Komatitsch, CNRS Research Director (DR CNRS)
Laboratory of Mechanics and Acoustics, Marseille, France
http://komatitsch.free.fr