[CIG-SEISMO] specfem3d_globe with CUDA
Phil Cummins
phil.cummins at anu.edu.au
Tue Apr 25 20:12:06 PDT 2017
Just a quick reply to my own post. I got xpsecfem3d to link by removing
cuda_device_obj.o from the mpif90 link line, and I also had to add
-lstdc++ to resolve the missing symbols from the c++ library.
Phil Cummins wrote:
> Hello again,
>
> Still trying to compile specfem3d_globe using CUDA. I have obtaiend
> advice from our HPC center regarding the appropriate modules to use,
> which are:
> intel-fc/16.0.3.210 intel-cc/16.0.3.210 openmpi/1.10.2 cuda/8.0
> I hope they're going to be OK. But I just can't make sense of what the
> Makefiule is trying to do with cuda_device_obj. As an exampe, let's
> consider what's happening with the CUDA module check_fields_gpu. First
> it is compiled into object code:
> nvcc -c src/gpu/check_fields_gpu.c -o obj/check_fields_gpu.cuda.o
> --cudart=shared -I/apps/openmpi/1.10.2/include -DUSE_OLDER_CUDA4_GPU
> -gencode=arch=compute_20,code=\"sm_20,compute_20\" -x cu -I./setup
> -I./src/gpu/kernels.gen -DUSE_CUDA
>
> That's fine (it does produce a deprecation warning). Then, for some
> reason that I can't figure out, the makefile is trying to link this in
> some way to the module cuda_device_obj, for which no source code seems
> to exist and for which the command appears to have omitted the nvcc
> compiler (so of course it doesn't work):
>
> DUSE_OLDER_CUDA4_GPU
> -gencode=arch=compute_20,code=\"sm_20,compute_20\" -o
> obj/cuda_device_obj.o obj/assemble_MPI_scalar_gpu.cuda.o
> obj/assemble_MPI_vector_gpu.cuda.o obj/check_fields_gpu.cuda.o
> obj/compute_add_sources_elastic_gpu.cuda.o
> obj/compute_coupling_gpu.cuda.o
> obj/compute_forces_crust_mantle_gpu.cuda.o
> obj/compute_forces_inner_core_gpu.cuda.o
> obj/compute_forces_outer_core_gpu.cuda.o
> obj/compute_kernels_gpu.cuda.o obj/compute_stacey_acoustic_gpu.cuda.o
> obj/compute_stacey_elastic_gpu.cuda.o obj/compute_strain_gpu.cuda.o
> obj/helper_functions_gpu.cuda.o obj/initialize_gpu.cuda.o
> obj/noise_tomography_gpu.cuda.o obj/prepare_mesh_constants_gpu.cuda.o
> obj/transfer_fields_gpu.cuda.o obj/update_displacement_gpu.cuda.o
> obj/write_seismograms_gpu.cuda.o
> obj/save_and_compare_cpu_vs_gpu.cuda.o
> obj/assemble_boundary_accel_on_device.cuda-kernel.o
> obj/assemble_boundary_potential_on_device.cuda-kernel.o
> obj/prepare_boundary_potential_on_device.cuda-kernel.o
> obj/prepare_boundary_accel_on_device.cuda-kernel.o
> obj/get_maximum_scalar_kernel.cuda-kernel.o
> obj/get_maximum_vector_kernel.cuda-kernel.o
> obj/compute_add_sources_adjoint_kernel.cuda-kernel.o
> obj/compute_add_sources_kernel.cuda-kernel.o
>
> Then, the commands issued by the Makefile try to create the main
> program xpsecfem3d by linking not only with cuda_device_obj.o, but
> also with check_fields_gpu.cuda.o. So even if I could get the above
> command to work somehow, the mpif90 command would almost certainly
> fail because of multiple definitions. In any case, I am struggling a
> bit because I just don't understand how or why cuda_device_obj.o is
> meant to be used. Can anyone help?
>
> mpif90 -g -xHost -fpe0 -ftz -assume buffered_io -assume byterecl
> -align sequence -vec-report0 -std03 -diag-disable 6477 -implicitnone
> -gen-interfaces -warn all -O3 -check nobounds -DFORCE_VECTORIZATION -o
> ./bin/xspecfem3D ./obj/assemble_MPI_scalar.solver.o
> ./obj/assemble_MPI_vector.solver.o ./obj/comp_source_spectrum.solver.o
> ./obj/compute_adj_source_frechet.solver.o ./obj/convert_time.solver.o
> ./obj/define_derivation_matrices.solver.o ./obj/file_io_threads.cc.o
> ./obj/force_ftz.cc.o ./obj/get_backazimuth.solver.o
> ./obj/get_cmt.solver.o ./obj/get_event_info.solver.o
> ./obj/make_gravity.solver.o ./obj/netlib_specfun_erf.solver.o
> ./obj/asdf_data.solverstatic_module.o
> ./obj/comp_source_time_function.solverstatic.o
> ./obj/specfem3D_par.solverstatic_module.o
> ./obj/write_seismograms.solverstatic.o
> ./obj/check_stability.solverstatic.o
> ./obj/compute_add_sources.solverstatic.o
> ./obj/compute_arrays_source.solverstatic.o
> ./obj/compute_boundary_kernel.solverstatic.o
> ./obj/compute_coupling.solverstatic.o
> ./obj/compute_element.solverstatic.o
> ./obj/compute_element_att_memory.solverstatic.o
> ./obj/compute_element_strain.solverstatic.o
> ./obj/compute_forces_acoustic_calling_routine.solverstatic.o
> ./obj/compute_forces_viscoelastic_calling_routine.solverstatic.o
> ./obj/compute_forces_crust_mantle_noDev.solverstatic.o
> ./obj/compute_forces_crust_mantle_Dev.solverstatic.o
> ./obj/compute_forces_inner_core_noDev.solverstatic.o
> ./obj/compute_forces_inner_core_Dev.solverstatic.o
> ./obj/compute_forces_outer_core_noDev.solverstatic.o
> ./obj/compute_forces_outer_core_Dev.solverstatic.o
> ./obj/compute_kernels.solverstatic.o
> ./obj/compute_seismograms.solverstatic.o
> ./obj/compute_stacey_crust_mantle.solverstatic.o
> ./obj/compute_stacey_outer_core.solverstatic.o ./obj/finali\
> ze_simulation.solverstatic.o ./obj/get_attenuation.solverstatic.o
> ./obj/initialize_simulation.solverstatic.o
> ./obj/iterate_time.solverstatic.o
> ./obj/iterate_time_undoatt.solverstatic.o
> ./obj/locate_receivers.solverstatic.o ./obj/locate_regular_points.s\
> olverstatic.o ./obj/locate_sources.solverstatic.o
> ./obj/multiply_arrays_source.solverstatic.o
> ./obj/noise_tomography.solverstatic.o
> ./obj/prepare_timerun.solverstatic.o
> ./obj/read_adjoint_sources.solverstatic.o
> ./obj/read_arrays_solver.solverstatic.o
> ./obj/read_forward_arrays.solverstatic.o
> ./obj/read_mesh_databases.solverstatic.o
> ./obj/read_topography_bathymetry.solverstatic.o
> ./obj/save_forward_arrays.solverstatic.o
> ./obj/save_kernels.solverstatic.o
> ./obj/save_regular_kernels.solverstatic.o
> ./obj/setup_GLL_points.solverstatic.o
> ./obj/setup_sources_receivers.solverstatic.o
> ./obj/specfem3D.solverstatic.o ./obj/update_displ\
> acement_LDDRK.solverstatic.o
> ./obj/update_displacement_Newmark.solverstatic.o
> ./obj/write_movie_output.solverstatic.o
> ./obj/write_movie_volume.solverstatic.o
> ./obj/write_movie_surface.solverstatic.o
> ./obj/write_output_ASCII.solverstatic.o
> ./obj/write_output_SAC.solverstatic.o
> ./obj/assemble_MPI_scalar_gpu.cuda.o
> ./obj/assemble_MPI_vector_gpu.cuda.o ./obj/check_fields_gpu.cuda.o
> ./obj/compute_add_sources_elastic_gpu.cuda.o
> ./obj/compute_coupling_gpu.cuda.o
> ./obj/compute_forces_crust_mantle_gpu.cuda.o
> ./obj/compute_forces_inner_core_gpu.cuda.o
> ./obj/compute_forces_outer_core_gpu.cuda.o
> ./obj/compute_kernels_gpu.cuda.o
> ./obj/compute_stacey_acoustic_gpu.cuda.o
> ./obj/compute_stacey_elastic_gpu.cuda.o
> ./obj/compute_strain_gpu.cuda.o ./obj/helper_functions_gpu.cuda.o
> ./obj/initialize_gpu.cuda.o ./obj/noise_tomography_gpu.cuda.o
> ./obj/prepare_mesh_constants_gpu.cuda.o
> ./obj/transfer_fields_gpu.cuda.o ./obj/update_displacement_gpu.cuda.o
> ./obj/write_seismograms_gpu.cuda.o
> ./obj/save_and_compare_cpu_vs_gpu.cuda.o ./obj/cuda_device_obj.o
> ./obj/assemble_boundary_accel_on_device.cuda-kernel.o
> ./obj/assemble_boundary_potential_on_device.cuda-kernel.o
> ./obj/prepare_boundary_potential_on_device.cuda-kernel.o
> ./obj/prepare_boundary_accel_on_device.cuda-kernel.o
> ./obj/get_maximum_scalar_kernel.cuda-kernel.o
> ./obj/get_maximum_vector_kernel.cuda-kernel.o
> ./obj/compute_add_sources_adjoint_kernel.cuda-kernel.o
> ./obj/compute_add_sources_kernel.cuda-kernel.o
> ./obj/compute_coupling_fluid_CMB_kernel.cuda-kernel.o
> ./obj/compute_coupling_fluid_ICB_kernel.cuda-kernel.o
> ./obj/compute_coupling_CMB_fluid_kernel.cuda-kernel.o
> ./obj/compute_coupling_ICB_fluid_kernel.cuda-kernel.o
> ./obj/compute_coupling_ocean_kernel.cuda-kernel.o
> ./obj/write_seismograms_transfer_from_device_kernel.cuda-kernel.o
> ./obj/write_seismograms_transfer_strain_from_device_kernel.cuda-kernel.o
> ./obj/noise_transfer_surface_to_host_kernel.cuda-kernel.o
> ./obj/noise_add_source_master_rec_kernel.cuda-kernel.o
> ./obj/noise_add_surface_movie_kernel.cuda-kernel.o
> ./obj/compute_stacey_acoustic_kernel.cuda-kernel.o
> ./obj/compute_stacey_acoustic_backward_kernel.cuda-kernel.o \
> ./obj/compute_stacey_elastic_kernel.cuda-kernel.o
> ./obj/compute_stacey_elastic_backward_kernel.cuda-kernel.o
> ./obj/update_disp_veloc_kernel.cuda-kernel.o
> ./obj/update_potential_kernel.cuda-kernel.o
> ./obj/update_accel_elastic_kernel.cuda-kernel.o
> ./obj/update_veloc_elastic_kernel.cuda-kernel.o
> ./obj/update_accel_acoustic_kernel.cuda-kernel.o
> ./obj/update_veloc_acoustic_kernel.cuda-kernel.o
> ./obj/compute_rho_kernel.cuda-kernel.o
> ./obj/compute_iso_kernel.cuda-kernel.o
> ./obj/compute_ani_kernel.cuda-kernel.o
> ./obj/compute_hess_kernel.cuda-kernel.o
> ./obj/compute_acoustic_kernel.cuda-kernel.o
> ./obj/compute_strength_noise_kernel.cuda-kernel.o
> ./obj/compute_ani_undoatt_kernel.cuda-kernel.o
> ./obj/compute_iso_undoatt_kernel.cuda-kernel.o
> ./obj/compute_strain_kernel.cuda-kernel.o
> ./obj/outer_core_impl_kernel_forward.cuda-kernel.o
> ./obj/outer_core_impl_kernel_adjoint.cudakernel.o
> ./obj/inner_core_impl_kernel_forward.cuda-kernel.o
> ./obj/inner_core_impl_kernel_adjoint.cuda-kernel.o
> ./obj/crust_mantle_impl_kernel_forward.cuda-kernel.o
> ./obj/crust_mantle_impl_kernel_adjoint.cuda-kernel.o
> ./obj/visual_vtk_stubs.visualc.o ./obj/shared_par.shared_module.o
> ./obj/auto_ner.shared.o ./obj/binary_c_io.cc.o
> ./obj/broadcast_computed_parameters.shared.o ./obj/calendar.shared.o
> ./obj/count_elements.shared.o ./obj/count_number_of_sources.shared.o
> ./obj/count_points.shared.o ./obj/create_name_database.shared.o
> ./obj/define_all_layers.shared.o ./obj/exit_mpi.shared.o
> ./obj/flush_system.shared.o ./obj/get_model_parameters.shared.o
> ./obj/get_timestep_and_layers.shared.o ./obj/gll_library.shared.o
> ./obj/hex_nodes.shared.o ./obj/intgrl.shared.o
> ./obj/lagrange_poly.shared.o ./obj/make_ellipticity.shared.o
> ./obj/model_prem.shared.o ./obj/model_topo_bathy.shared.o
> ./obj/parallel.sharedmpi.o ./obj/param_reader.cc.o
> ./obj/read_compute_parameters.shared.o
> ./obj/read_parameter_file.shared.o
> ./obj/read_value_parameters.shared.o ./obj/recompute_jacobian.shared.o
> ./obj/reduce.shared.o ./obj/rthetaphi_xyz.shared.o
> ./obj/spline_routines.shared.o ./obj/write_VTK_file.shared.o
> ./obj/adios_method_stubs.cc.o -lcudart
>
>
> Thanks,
>
> - Phil
>
> P.S. when i simply remove cuda_dev_obj.o from the above mpif90
> command, the link fails with:
> ./obj/assemble_MPI_scalar_gpu.cuda.o:(.eh_frame+0x12): undefined
> reference to `__gxx_personality_v0'
> ./obj/assemble_MPI_vector_gpu.cuda.o:(.eh_frame+0x12): undefined
> reference to `__gxx_personality_v0'
> ./obj/check_fields_gpu.cuda.o:(.eh_frame+0x12): undefined reference to
> `__gxx_personality_v0'
> ./obj/compute_add_sources_elastic_gpu.cuda.o:(.eh_frame+0x12):
> undefined reference to `__gxx_personality_v0'
> ./obj/compute_coupling_gpu.cuda.o:(.eh_frame+0x12): undefined
> reference to `__gxx_personality_v0'
> ./obj/compute_forces_crust_mantle_gpu.cuda.o:(.eh_frame+0x12): more
> undefined references to `__gxx_personality_v0' follow
>
> Phil Cummins wrote:
>> Hello,
>>
>> I'm would like to use specfem3d_globe with my university's GPU
>> cluster. I am trying to compile specfem3d_globe-7.0.0 after having
>> configured it using:
>> ./configure --with-cuda=cuda6
>> I am using the ifort fortran compiler version 12.1.9.293, and the
>> version of cuda I have loaded is 6.5.
>> The "make all" command does a lot of compilation using ifort and
>> nvcc, but eventually fails when trying to link xspecfem3D itself with
>> the error:
>>
>> ifort: error #10236: File not found: './obj/cuda_device_obj.o'
>> make: *** [bin/xspecfem3D] Error 1
>>
>> Delving into what happened during the make (see below), I noticed
>> that after using nvcc to compile crust_mantle_impl_kernel_adjoint.cu,
>> it attempts to use the command DUSE_OLDER_CUDA4_GPU to compile
>> cuda_device_obj, which of course doesn't work because
>> DUSE_OLDER_CUDA4_GPU is a compiler option not a command. It seems
>> that make has somehow lost the beginning of this particular
>> compilation command.
>> Has anyone else seen or know what to do about it?
>> Thanks,
>>
>> - Phil
>>
>> P.S. Some of the hardware on our cluster can be used only with cuda
>> version 8. Does anyone know if specfem3d will work with cuda8?
>>
>> nvcc -c src/gpu/kernels.gen/crust_mantle_impl_kernel_adjoint.cu -o
>> obj/crust_ma\
>> ntle_impl_kernel_adjoint.cuda-kernel.o --cudart=shared
>> -I/apps/openmpi/1.7.5/\
>> include -DUSE_OLDER_CUDA4_GPU
>> -gencode=arch=compute_20,code=\"sm_20,compute_20\\
>> " -x cu -I./setup -I./src/gpu/kernels.gen -DUSE_CUDA -include
>> src/gpu/mesh_con\
>> stants_gpu.h
>> DUSE_OLDER_CUDA4_GPU
>> -gencode=arch=compute_20,code=\"sm_20,compute_20\" -o obj/\
>> cuda_device_obj.o obj/assemble_MPI_scalar_gpu.cuda.o
>> obj/assemble_MPI_vector_gp\
>> u.cuda.o obj/check_fields_gpu.cuda.o
>> obj/compute_add_sources_elastic_gpu.cuda.o\
>> obj/compute_coupling_gpu.cuda.o
>> obj/compute_forces_crust_mantle_gpu.cuda.o obj\
>> /compute_forces_inner_core_gpu.cuda.o
>> obj/compute_forces_outer_core_gpu.cuda.o \
>> obj/compute_kernels_gpu.cuda.o obj/compute_stacey_acoustic_gpu.cuda.o
>> obj/compu\
>> te_stacey_elastic_gpu.cuda.o obj/compute_strain_gpu.cuda.o
>> obj/helper_functions\
>> _gpu.cuda.o obj/initialize_gpu.cuda.o obj/noise_tomography_gpu.cuda.o
>> obj/prepa\
>> re_mesh_constants_gpu.cuda.o obj/transfer_fields_gpu.cuda.o
>> obj/update_displace\
>> ment_gpu.cuda.o obj/write_seismograms_gpu.cuda.o
>> obj/save_and_compare_cpu_vs_gp\
>> u.cuda.o obj/assemble_boundary_accel_on_device.cuda-kernel.o
>> obj/assemble_bound\
>> ary_potential_on_device.cuda-kernel.o
>> obj/prepare_boundary_potential_on_device.\
>> cuda-kernel.o obj/prepare_boundary_accel_on_device.cuda-kernel.o
>> obj/get_maximu\
>> m_scalar_kernel.cuda-kernel.o
>> obj/get_maximum_vector_kernel.cuda-kernel.o obj/c\
>> ompute_add_sources_adjoint_kernel.cuda-kernel.o
>> obj/compute_add_sources_kernel.\
>> cuda-kernel.o
>> nel.cuda-kernel.o
>> obj/write_seismograms_transfer_strain_from_device_kernel.cuda\
>> -kernel.o obj/noise_transfer_surface_to_host_kernel.cuda-kernel.o
>> obj/noise_add\
>> _source_master_rec_kernel.cuda-kernel.o
>> obj/noise_add_surface_movie_kernel.cuda\
>> -kernel.o obj/compute_stacey_acoustic_kernel.cuda-kernel.o
>> obj/compute_stacey_a\
>> coustic_backward_kernel.cuda-kernel.o
>> obj/compute_stacey_elastic_kernel.cuda-ke\
>> rnel.o obj/compute_stacey_elastic_backward_kernel.cuda-kernel.o
>> obj/update_disp\
>> _veloc_kernel.cuda-kernel.o obj/update_potential_kernel.cuda-kernel.o
>> obj/updat\
>> e_accel_elastic_kernel.cuda-kernel.o
>> obj/update_veloc_elastic_kernel.cuda-kerne\
>> l.o obj/update_accel_acoustic_kernel.cuda-kernel.o
>> obj/update_veloc_acoustic_ke\
>> rnel.cuda-kernel.o obj/compute_rho_kernel.cuda-kernel.o
>> obj/compute_iso_kernel.\
>> cuda-kernel.o obj/compute_ani_kernel.cuda-kernel.o
>> obj/compute_hess_kernel.cuda\
>> -kernel.o obj/compute_acoustic_kernel.cuda-kernel.o
>> obj/compute_strength_noise_\
>> kernel.cuda-kernel.o obj/compute_ani_undoatt_kernel.cuda-kernel.o
>> obj/compute_i\
>> so_undoatt_kernel.cuda-kernel.o
>> obj/compute_strain_kernel.cuda-kernel.o obj/out\
>> er_core_impl_kernel_forward.cuda-kernel.o
>> obj/outer_core_impl_kernel_adjoint.cu\
>> da-kernel.o obj/inner_core_impl_kernel_forward.cuda-kernel.o
>> obj/inner_core_imp\
>> l_kernel_adjoint.cuda-kernel.o
>> obj/crust_mantle_impl_kernel_forward.cuda-kernel\
>> .o obj/crust_mantle_impl_kernel_adjoint.cuda-kernel.o
>> make: DUSE_OLDER_CUDA4_GPU: Command not found
>> make: [obj/cuda_device_obj.o] Error 127 (ignored)
>>
>> --
>> Phil Cummins
>> Prof. Natural Hazards
>> Research School of Earth Sciences
>> Australian National University
>> _______________________________________________
>> CIG-SEISMO mailing list
>> CIG-SEISMO at geodynamics.org
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>
> _______________________________________________
> CIG-SEISMO mailing list
> CIG-SEISMO at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20170426/7b5f69e1/attachment-0001.html>
More information about the CIG-SEISMO
mailing list