[CIG-SEISMO] specfem3d_globe with CUDA

[Phil Cummins]

Hi Dimitri,

Thanks for the kind words, but my approach was just a workaround. I 
think someone smarter than me should try to fix the Makefile or 
configure script. I'm still not sure what was wrong with it.
Good luck!

- Phil


> Dimitri Komatitsch <mailto:komatitsch at lma.cnrs-mrs.fr>
> 26 April 2017 at 8:54 PM
>
> Hi Phil, Hi all,
>
> Thanks! Let me cc Daniel Peter and Peter Messmer, who are very 
> familiar with the CUDA version. Daniel, would you know how to commit 
> Phil's fix to the official Git version of the code? (and I guess also 
> for SPECFEM3D_Cartesian and for SPECFEM2D?).
>
> Thanks,
> Best wishes,
>
> Dimitri.
>
>
>
> Phil Cummins <mailto:phil.cummins at anu.edu.au>
> 23 April 2017 at 11:45 AM
> Hello,
>
> I'm would like to use specfem3d_globe with my university's GPU 
> cluster. I am trying to compile specfem3d_globe-7.0.0 after having 
> configured it using:
> ./configure --with-cuda=cuda6
> I am using the ifort fortran compiler version 12.1.9.293, and the 
> version of cuda I have loaded is 6.5.
> The "make all" command does a lot of compilation using ifort and nvcc, 
> but eventually fails when trying to link xspecfem3D itself with the error:
>
> ifort: error #10236: File not found:  './obj/cuda_device_obj.o'
> make: *** [bin/xspecfem3D] Error 1
>
> Delving into what happened during the make (see below), I noticed that 
> after using nvcc to compile crust_mantle_impl_kernel_adjoint.cu, it 
> attempts to use the command DUSE_OLDER_CUDA4_GPU to compile 
> cuda_device_obj, which of course doesn't work because 
> DUSE_OLDER_CUDA4_GPU is a compiler option not a command. It seems that 
> make has somehow lost the beginning of this particular compilation 
> command.
> Has anyone else seen or know what to do about it?
> Thanks,
>
> - Phil
>
> P.S. Some of the hardware on our cluster can be used only with cuda 
> version 8. Does anyone know if specfem3d will work with cuda8?
>
> nvcc -c src/gpu/kernels.gen/crust_mantle_impl_kernel_adjoint.cu -o 
> obj/crust_ma\
> ntle_impl_kernel_adjoint.cuda-kernel.o   --cudart=shared 
> -I/apps/openmpi/1.7.5/\
> include -DUSE_OLDER_CUDA4_GPU 
> -gencode=arch=compute_20,code=\"sm_20,compute_20\\
> " -x cu -I./setup -I./src/gpu/kernels.gen  -DUSE_CUDA -include 
> src/gpu/mesh_con\
> stants_gpu.h
> DUSE_OLDER_CUDA4_GPU 
> -gencode=arch=compute_20,code=\"sm_20,compute_20\" -o obj/\
> cuda_device_obj.o obj/assemble_MPI_scalar_gpu.cuda.o 
> obj/assemble_MPI_vector_gp\
> u.cuda.o obj/check_fields_gpu.cuda.o 
> obj/compute_add_sources_elastic_gpu.cuda.o\
>  obj/compute_coupling_gpu.cuda.o 
> obj/compute_forces_crust_mantle_gpu.cuda.o obj\
> /compute_forces_inner_core_gpu.cuda.o 
> obj/compute_forces_outer_core_gpu.cuda.o \
> obj/compute_kernels_gpu.cuda.o obj/compute_stacey_acoustic_gpu.cuda.o 
> obj/compu\
> te_stacey_elastic_gpu.cuda.o obj/compute_strain_gpu.cuda.o 
> obj/helper_functions\
> _gpu.cuda.o obj/initialize_gpu.cuda.o obj/noise_tomography_gpu.cuda.o 
> obj/prepa\
> re_mesh_constants_gpu.cuda.o obj/transfer_fields_gpu.cuda.o 
> obj/update_displace\
> ment_gpu.cuda.o obj/write_seismograms_gpu.cuda.o 
> obj/save_and_compare_cpu_vs_gp\
> u.cuda.o obj/assemble_boundary_accel_on_device.cuda-kernel.o 
> obj/assemble_bound\
> ary_potential_on_device.cuda-kernel.o 
> obj/prepare_boundary_potential_on_device.\
> cuda-kernel.o obj/prepare_boundary_accel_on_device.cuda-kernel.o 
> obj/get_maximu\
> m_scalar_kernel.cuda-kernel.o 
> obj/get_maximum_vector_kernel.cuda-kernel.o obj/c\
> ompute_add_sources_adjoint_kernel.cuda-kernel.o 
> obj/compute_add_sources_kernel.\
> cuda-kernel.o
> nel.cuda-kernel.o 
> obj/write_seismograms_transfer_strain_from_device_kernel.cuda\
> -kernel.o obj/noise_transfer_surface_to_host_kernel.cuda-kernel.o 
> obj/noise_add\
> _source_master_rec_kernel.cuda-kernel.o 
> obj/noise_add_surface_movie_kernel.cuda\
> -kernel.o obj/compute_stacey_acoustic_kernel.cuda-kernel.o 
> obj/compute_stacey_a\
> coustic_backward_kernel.cuda-kernel.o 
> obj/compute_stacey_elastic_kernel.cuda-ke\
> rnel.o obj/compute_stacey_elastic_backward_kernel.cuda-kernel.o 
> obj/update_disp\
> _veloc_kernel.cuda-kernel.o obj/update_potential_kernel.cuda-kernel.o 
> obj/updat\
> e_accel_elastic_kernel.cuda-kernel.o 
> obj/update_veloc_elastic_kernel.cuda-kerne\
> l.o obj/update_accel_acoustic_kernel.cuda-kernel.o 
> obj/update_veloc_acoustic_ke\
> rnel.cuda-kernel.o obj/compute_rho_kernel.cuda-kernel.o 
> obj/compute_iso_kernel.\
> cuda-kernel.o obj/compute_ani_kernel.cuda-kernel.o 
> obj/compute_hess_kernel.cuda\
> -kernel.o obj/compute_acoustic_kernel.cuda-kernel.o 
> obj/compute_strength_noise_\
> kernel.cuda-kernel.o obj/compute_ani_undoatt_kernel.cuda-kernel.o 
> obj/compute_i\
> so_undoatt_kernel.cuda-kernel.o 
> obj/compute_strain_kernel.cuda-kernel.o obj/out\
> er_core_impl_kernel_forward.cuda-kernel.o 
> obj/outer_core_impl_kernel_adjoint.cu\
> da-kernel.o obj/inner_core_impl_kernel_forward.cuda-kernel.o 
> obj/inner_core_imp\
> l_kernel_adjoint.cuda-kernel.o 
> obj/crust_mantle_impl_kernel_forward.cuda-kernel\
> .o obj/crust_mantle_impl_kernel_adjoint.cuda-kernel.o
> make: DUSE_OLDER_CUDA4_GPU: Command not found
> make: [obj/cuda_device_obj.o] Error 127 (ignored)
>
> _______________________________________________
> CIG-SEISMO mailing list
> CIG-SEISMO at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

-- 

Phil Cummins
Prof. Natural Hazards
Research School of Earth Sciences
Australian National University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20170427/a846003e/attachment-0001.html>