[CIG-SEISMO] specfem3d_globe with CUDA

Thu Apr 27 02:27:10 PDT 2017

Phil,

could you attach the config.log and original Makefile? for some reason the cuda option is not properly recognized in your configuration step.

@Brad:
for the funny cuda flagging scheme, it is mainly due to historic reasons. we had to distinguish between cuda versions 4 or older (which was default at the time) and the newer version 5 and higher.  so we introduced the cuda5 flag option. then came newer hardware and with each new hardware architecture a new cuda version appeared.

so basically each major cuda version introduced a new type of hardware. i assume that a user doesn't know the architecture number for his card (do you know what the sm_** code is for your P100 card?), but the cudatoolkit version he needs for the compilation (you will need cuda version 8 for the new pascal architecture). also, having a separate flag for each card is not convenient, as the optimization depends on the architecture.  the cuda flag option thus gives you a brief history of GPU time.

we could change it and use the architecture number, maybe. anyway, if you need a specific cuda flag,  you can always modify the Makefile directly.

best,
daniel

On Apr 27, 2017, at 10:12, Piero Lanucara <p.lanucara at cineca.it<mailto:p.lanucara at cineca.it>> wrote:

Hi Peter

I agree but, in principle at least, it should the configure able to point to the system recovered CUDA libs and runtime?

so, something like

--with-cuda

should work fine and should point to the correct version of CUDA runtime?

Thanks

Piero

Il 27/04/2017 00:10, Daniel B. Peter ha scritto:
Hi Phil, hi all,

thanks for pointing out that the default —with-cuda=cuda which leads to CUDA4 compilation is probably obsolete by now and we should change this default. i’m going to make the changes and fix to the configuration script.

in your case however, i’m not sure why the initial
./configure —with-cuda=cuda6
didn’t work.

first, what are the GPU cards you want to run it on? then choose:
-for Tesla K20: ./configure —with-cuda=cuda5
-for Tesla K80: ./configure —with-cuda=cuda6
-for Maxwell K2200: ./configure —with-cuda=cuda7
-for Pascal P100: ./configure —with-cuda=cuda8

we use the cuda*** specification as a short flag to point to the specific architecture and optimization. even if you have the cudatoolkit version 8, but want to compile kernels for a Tesla K20 card, you would choose —with-cuda=cuda5

best wishes,
daniel

On Apr 26, 2017, at 5:26 PM, Piero Lanucara <p.lanucara at cineca.it<mailto:p.lanucara at cineca.it>> wrote:

Hi all

as far as I understand it should work with:

configure.... --with-cuda=cuda5

regards

Piero

Il 26/04/2017 16:04, Phil Cummins ha scritto:
Hi Dimitri,

Thanks for the kind words, but my approach was just a workaround. I think someone smarter than me should try to fix the Makefile or configure script. I'm still not sure what was wrong with it.
Good luck!

- Phil

Dimitri Komatitsch<mailto:komatitsch at lma.cnrs-mrs.fr>
26 April 2017 at 8:54 PM

Hi Phil, Hi all,

Thanks! Let me cc Daniel Peter and Peter Messmer, who are very familiar with the CUDA version. Daniel, would you know how to commit Phil's fix to the official Git version of the code? (and I guess also for SPECFEM3D_Cartesian and for SPECFEM2D?).

Thanks,
Best wishes,

Dimitri.

Phil Cummins<mailto:phil.cummins at anu.edu.au>
23 April 2017 at 11:45 AM
Hello,

I'm would like to use specfem3d_globe with my university's GPU cluster. I am trying to compile specfem3d_globe-7.0.0 after having configured it using:
./configure --with-cuda=cuda6
I am using the ifort fortran compiler version 12.1.9.293, and the version of cuda I have loaded is 6.5.
The "make all" command does a lot of compilation using ifort and nvcc, but eventually fails when trying to link xspecfem3D itself with the error:

ifort: error #10236: File not found:  './obj/cuda_device_obj.o'
make: *** [bin/xspecfem3D] Error 1

Delving into what happened during the make (see below), I noticed that after using nvcc to compile crust_mantle_impl_kernel_adjoint.cu, it attempts to use the command DUSE_OLDER_CUDA4_GPU to compile cuda_device_obj, which of course doesn't work because DUSE_OLDER_CUDA4_GPU is a compiler option not a command. It seems that make has somehow lost the beginning of this particular compilation command.
Has anyone else seen or know what to do about it?
Thanks,

- Phil

P.S. Some of the hardware on our cluster can be used only with cuda version 8. Does anyone know if specfem3d will work with cuda8?

nvcc -c src/gpu/kernels.gen/crust_mantle_impl_kernel_adjoint.cu -o obj/crust_ma\
ntle_impl_kernel_adjoint.cuda-kernel.o   --cudart=shared -I/apps/openmpi/1.7.5/\
include -DUSE_OLDER_CUDA4_GPU -gencode=arch=compute_20,code=\"sm_20,compute_20\\
" -x cu -I./setup -I./src/gpu/kernels.gen  -DUSE_CUDA -include src/gpu/mesh_con\
stants_gpu.h
DUSE_OLDER_CUDA4_GPU -gencode=arch=compute_20,code=\"sm_20,compute_20\" -o obj/\
cuda_device_obj.o obj/assemble_MPI_scalar_gpu.cuda.o obj/assemble_MPI_vector_gp\
u.cuda.o obj/check_fields_gpu.cuda.o obj/compute_add_sources_elastic_gpu.cuda.o\
 obj/compute_coupling_gpu.cuda.o obj/compute_forces_crust_mantle_gpu.cuda.o obj\
/compute_forces_inner_core_gpu.cuda.o obj/compute_forces_outer_core_gpu.cuda.o \
obj/compute_kernels_gpu.cuda.o obj/compute_stacey_acoustic_gpu.cuda.o obj/compu\
te_stacey_elastic_gpu.cuda.o obj/compute_strain_gpu.cuda.o obj/helper_functions\
_gpu.cuda.o obj/initialize_gpu.cuda.o obj/noise_tomography_gpu.cuda.o obj/prepa\
re_mesh_constants_gpu.cuda.o obj/transfer_fields_gpu.cuda.o obj/update_displace\
ment_gpu.cuda.o obj/write_seismograms_gpu.cuda.o obj/save_and_compare_cpu_vs_gp\
u.cuda.o obj/assemble_boundary_accel_on_device.cuda-kernel.o obj/assemble_bound\
ary_potential_on_device.cuda-kernel.o obj/prepare_boundary_potential_on_device.\
cuda-kernel.o obj/prepare_boundary_accel_on_device.cuda-kernel.o obj/get_maximu\
m_scalar_kernel.cuda-kernel.o obj/get_maximum_vector_kernel.cuda-kernel.o obj/c\
ompute_add_sources_adjoint_kernel.cuda-kernel.o obj/compute_add_sources_kernel.\
cuda-kernel.o
nel.cuda-kernel.o obj/write_seismograms_transfer_strain_from_device_kernel.cuda\
-kernel.o obj/noise_transfer_surface_to_host_kernel.cuda-kernel.o obj/noise_add\
_source_master_rec_kernel.cuda-kernel.o obj/noise_add_surface_movie_kernel.cuda\
-kernel.o obj/compute_stacey_acoustic_kernel.cuda-kernel.o obj/compute_stacey_a\
coustic_backward_kernel.cuda-kernel.o obj/compute_stacey_elastic_kernel.cuda-ke\
rnel.o obj/compute_stacey_elastic_backward_kernel.cuda-kernel.o obj/update_disp\
_veloc_kernel.cuda-kernel.o obj/update_potential_kernel.cuda-kernel.o obj/updat\
e_accel_elastic_kernel.cuda-kernel.o obj/update_veloc_elastic_kernel.cuda-kerne\
l.o obj/update_accel_acoustic_kernel.cuda-kernel.o obj/update_veloc_acoustic_ke\
rnel.cuda-kernel.o obj/compute_rho_kernel.cuda-kernel.o obj/compute_iso_kernel.\
cuda-kernel.o obj/compute_ani_kernel.cuda-kernel.o obj/compute_hess_kernel.cuda\
-kernel.o obj/compute_acoustic_kernel.cuda-kernel.o obj/compute_strength_noise_\
kernel.cuda-kernel.o obj/compute_ani_undoatt_kernel.cuda-kernel.o obj/compute_i\
so_undoatt_kernel.cuda-kernel.o obj/compute_strain_kernel.cuda-kernel.o obj/out\
er_core_impl_kernel_forward.cuda-kernel.o obj/outer_core_impl_kernel_adjoint.cu\
da-kernel.o obj/inner_core_impl_kernel_forward.cuda-kernel.o obj/inner_core_imp\
l_kernel_adjoint.cuda-kernel.o obj/crust_mantle_impl_kernel_forward.cuda-kernel\
.o obj/crust_mantle_impl_kernel_adjoint.cuda-kernel.o
make: DUSE_OLDER_CUDA4_GPU: Command not found
make: [obj/cuda_device_obj.o] Error 127 (ignored)

_______________________________________________
CIG-SEISMO mailing list
CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org>
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

--

Phil Cummins
Prof. Natural Hazards
Research School of Earth Sciences
Australian National University

_______________________________________________
CIG-SEISMO mailing list
CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org>
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

--
Piero Lanucara
CINECA
SuperComputing Applications and Innovation Department - SCAI
Phone: +39 06 44486709
Fax: +39 06 4957083
Address: CINECA Sede di Roma Via dei Tizii,6/b - 00185 Roma - Italy
E-mail: p.lanucara at cineca.it<mailto:p.lanucara at cineca.it>
skype: lpeter175
site: http://www.hpc.cineca.it<http://www.hpc.cineca.it/>

_______________________________________________
CIG-SEISMO mailing list
CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org>
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

________________________________
This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

_______________________________________________
CIG-SEISMO mailing list
CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org>
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

--
Piero Lanucara
CINECA
SuperComputing Applications and Innovation Department - SCAI
Phone: +39 06 44486709
Fax: +39 06 4957083
Address: CINECA Sede di Roma Via dei Tizii,6/b - 00185 Roma - Italy
E-mail: p.lanucara at cineca.it<mailto:p.lanucara at cineca.it>
skype: lpeter175
site: http://www.hpc.cineca.it

_______________________________________________
CIG-SEISMO mailing list
CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org>
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20170427/0ff0e2fd/attachment-0001.html>