[CIG-SEISMO] specfem3d_globe with CUDA

Thu Apr 27 01:12:24 PDT 2017

Hi Peter

I agree but, in principle at least, it should the configure able to 
point to the system recovered CUDA libs and runtime?

so, something like

--with-cuda

should work fine and should point to the correct version of CUDA runtime?

Thanks

Piero

Il 27/04/2017 00:10, Daniel B. Peter ha scritto:
> Hi Phil, hi all,
>
> thanks for pointing out that the default —with-cuda=cuda which leads 
> to CUDA4 compilation is probably obsolete by now and we should change 
> this default. i’m going to make the changes and fix to the 
> configuration script.
>
> in your case however, i’m not sure why the initial
> ./configure —with-cuda=cuda6
> didn’t work.
>
> first, what are the GPU cards you want to run it on? then choose:
> -for Tesla K20: ./configure —with-cuda=cuda5
> -for Tesla K80: ./configure —with-cuda=cuda6
> -for Maxwell K2200: ./configure —with-cuda=cuda7
> -for Pascal P100: ./configure —with-cuda=cuda8
>
> we use the cuda*** specification as a short flag to point to the 
> specific architecture and optimization. even if you have the 
> cudatoolkit version 8, but want to compile kernels for a Tesla K20 
> card, you would choose —with-cuda=cuda5
>
> best wishes,
> daniel
>
>
>
>
>
>
>> On Apr 26, 2017, at 5:26 PM, Piero Lanucara <p.lanucara at cineca.it 
>> <mailto:p.lanucara at cineca.it>> wrote:
>>
>> Hi all
>>
>> as far as I understand it should work with:
>>
>> configure.... --with-cuda=cuda5
>>
>> regards
>>
>> Piero
>>
>>
>> Il 26/04/2017 16:04, Phil Cummins ha scritto:
>>> Hi Dimitri,
>>>
>>> Thanks for the kind words, but my approach was just a workaround. I 
>>> think someone smarter than me should try to fix the Makefile or 
>>> configure script. I'm still not sure what was wrong with it.
>>> Good luck!
>>>
>>> - Phil
>>>
>>>
>>>> Dimitri Komatitsch <mailto:komatitsch at lma.cnrs-mrs.fr>
>>>> 26 April 2017 at 8:54 PM
>>>>
>>>> Hi Phil, Hi all,
>>>>
>>>> Thanks! Let me cc Daniel Peter and Peter Messmer, who are very 
>>>> familiar with the CUDA version. Daniel, would you know how to 
>>>> commit Phil's fix to the official Git version of the code? (and I 
>>>> guess also for SPECFEM3D_Cartesian and for SPECFEM2D?).
>>>>
>>>> Thanks,
>>>> Best wishes,
>>>>
>>>> Dimitri.
>>>>
>>>>
>>>>
>>>> Phil Cummins <mailto:phil.cummins at anu.edu.au>
>>>> 23 April 2017 at 11:45 AM
>>>> Hello,
>>>>
>>>> I'm would like to use specfem3d_globe with my university's GPU 
>>>> cluster. I am trying to compile specfem3d_globe-7.0.0 after having 
>>>> configured it using:
>>>> ./configure --with-cuda=cuda6
>>>> I am using the ifort fortran compiler version 12.1.9.293, and the 
>>>> version of cuda I have loaded is 6.5.
>>>> The "make all" command does a lot of compilation using ifort and 
>>>> nvcc, but eventually fails when trying to link xspecfem3D itself 
>>>> with the error:
>>>>
>>>> ifort: error #10236: File not found: './obj/cuda_device_obj.o'
>>>> make: *** [bin/xspecfem3D] Error 1
>>>>
>>>> Delving into what happened during the make (see below), I noticed 
>>>> that after using nvcc to compile 
>>>> crust_mantle_impl_kernel_adjoint.cu, it attempts to use the command 
>>>> DUSE_OLDER_CUDA4_GPU to compile cuda_device_obj, which of course 
>>>> doesn't work because DUSE_OLDER_CUDA4_GPU is a compiler option not 
>>>> a command. It seems that make has somehow lost the beginning of 
>>>> this particular compilation command.
>>>> Has anyone else seen or know what to do about it?
>>>> Thanks,
>>>>
>>>> - Phil
>>>>
>>>> P.S. Some of the hardware on our cluster can be used only with cuda 
>>>> version 8. Does anyone know if specfem3d will work with cuda8?
>>>>
>>>> nvcc -c src/gpu/kernels.gen/crust_mantle_impl_kernel_adjoint.cu -o 
>>>> obj/crust_ma\
>>>> ntle_impl_kernel_adjoint.cuda-kernel.o --cudart=shared 
>>>> -I/apps/openmpi/1.7.5/\
>>>> include -DUSE_OLDER_CUDA4_GPU 
>>>> -gencode=arch=compute_20,code=\"sm_20,compute_20\\
>>>> " -x cu -I./setup -I./src/gpu/kernels.gen -DUSE_CUDA -include 
>>>> src/gpu/mesh_con\
>>>> stants_gpu.h
>>>> DUSE_OLDER_CUDA4_GPU 
>>>> -gencode=arch=compute_20,code=\"sm_20,compute_20\" -o obj/\
>>>> cuda_device_obj.o obj/assemble_MPI_scalar_gpu.cuda.o 
>>>> obj/assemble_MPI_vector_gp\
>>>> u.cuda.o obj/check_fields_gpu.cuda.o 
>>>> obj/compute_add_sources_elastic_gpu.cuda.o\
>>>>  obj/compute_coupling_gpu.cuda.o 
>>>> obj/compute_forces_crust_mantle_gpu.cuda.o obj\
>>>> /compute_forces_inner_core_gpu.cuda.o 
>>>> obj/compute_forces_outer_core_gpu.cuda.o \
>>>> obj/compute_kernels_gpu.cuda.o 
>>>> obj/compute_stacey_acoustic_gpu.cuda.o obj/compu\
>>>> te_stacey_elastic_gpu.cuda.o obj/compute_strain_gpu.cuda.o 
>>>> obj/helper_functions\
>>>> _gpu.cuda.o obj/initialize_gpu.cuda.o 
>>>> obj/noise_tomography_gpu.cuda.o obj/prepa\
>>>> re_mesh_constants_gpu.cuda.o obj/transfer_fields_gpu.cuda.o 
>>>> obj/update_displace\
>>>> ment_gpu.cuda.o obj/write_seismograms_gpu.cuda.o 
>>>> obj/save_and_compare_cpu_vs_gp\
>>>> u.cuda.o obj/assemble_boundary_accel_on_device.cuda-kernel.o 
>>>> obj/assemble_bound\
>>>> ary_potential_on_device.cuda-kernel.o 
>>>> obj/prepare_boundary_potential_on_device.\
>>>> cuda-kernel.o obj/prepare_boundary_accel_on_device.cuda-kernel.o 
>>>> obj/get_maximu\
>>>> m_scalar_kernel.cuda-kernel.o 
>>>> obj/get_maximum_vector_kernel.cuda-kernel.o obj/c\
>>>> ompute_add_sources_adjoint_kernel.cuda-kernel.o 
>>>> obj/compute_add_sources_kernel.\
>>>> cuda-kernel.o
>>>> nel.cuda-kernel.o 
>>>> obj/write_seismograms_transfer_strain_from_device_kernel.cuda\
>>>> -kernel.o obj/noise_transfer_surface_to_host_kernel.cuda-kernel.o 
>>>> obj/noise_add\
>>>> _source_master_rec_kernel.cuda-kernel.o 
>>>> obj/noise_add_surface_movie_kernel.cuda\
>>>> -kernel.o obj/compute_stacey_acoustic_kernel.cuda-kernel.o 
>>>> obj/compute_stacey_a\
>>>> coustic_backward_kernel.cuda-kernel.o 
>>>> obj/compute_stacey_elastic_kernel.cuda-ke\
>>>> rnel.o obj/compute_stacey_elastic_backward_kernel.cuda-kernel.o 
>>>> obj/update_disp\
>>>> _veloc_kernel.cuda-kernel.o 
>>>> obj/update_potential_kernel.cuda-kernel.o obj/updat\
>>>> e_accel_elastic_kernel.cuda-kernel.o 
>>>> obj/update_veloc_elastic_kernel.cuda-kerne\
>>>> l.o obj/update_accel_acoustic_kernel.cuda-kernel.o 
>>>> obj/update_veloc_acoustic_ke\
>>>> rnel.cuda-kernel.o obj/compute_rho_kernel.cuda-kernel.o 
>>>> obj/compute_iso_kernel.\
>>>> cuda-kernel.o obj/compute_ani_kernel.cuda-kernel.o 
>>>> obj/compute_hess_kernel.cuda\
>>>> -kernel.o obj/compute_acoustic_kernel.cuda-kernel.o 
>>>> obj/compute_strength_noise_\
>>>> kernel.cuda-kernel.o obj/compute_ani_undoatt_kernel.cuda-kernel.o 
>>>> obj/compute_i\
>>>> so_undoatt_kernel.cuda-kernel.o 
>>>> obj/compute_strain_kernel.cuda-kernel.o obj/out\
>>>> er_core_impl_kernel_forward.cuda-kernel.o 
>>>> obj/outer_core_impl_kernel_adjoint.cu\
>>>> da-kernel.o obj/inner_core_impl_kernel_forward.cuda-kernel.o 
>>>> obj/inner_core_imp\
>>>> l_kernel_adjoint.cuda-kernel.o 
>>>> obj/crust_mantle_impl_kernel_forward.cuda-kernel\
>>>> .o obj/crust_mantle_impl_kernel_adjoint.cuda-kernel.o
>>>> make: DUSE_OLDER_CUDA4_GPU: Command not found
>>>> make: [obj/cuda_device_obj.o] Error 127 (ignored)
>>>>
>>>> _______________________________________________
>>>> CIG-SEISMO mailing list
>>>> CIG-SEISMO at geodynamics.org
>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>>
>>> -- 
>>> Phil Cummins
>>> Prof. Natural Hazards
>>> Research School of Earth Sciences
>>> Australian National University
>>>
>>>
>>> _______________________________________________
>>> CIG-SEISMO mailing list
>>> CIG-SEISMO at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>
>> -- 
>> Piero Lanucara
>> CINECA
>> SuperComputing Applications and Innovation Department - SCAI
>> Phone: +39 06 44486709
>> Fax: +39 06 4957083
>> Address: CINECA Sede di Roma Via dei Tizii,6/b - 00185 Roma - Italy
>> E-mail:p.lanucara at cineca.it
>> skype: lpeter175
>> site:http://www.hpc.cineca.it
>> _______________________________________________
>> CIG-SEISMO mailing list
>> CIG-SEISMO at geodynamics.org <mailto:CIG-SEISMO at geodynamics.org>
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>
>
> ------------------------------------------------------------------------
> This message and its contents including attachments are intended 
> solely for the original recipient. If you are not the intended 
> recipient or have received this message in error, please notify me 
> immediately and delete this message from your computer system. Any 
> unauthorized use or distribution is prohibited. Please consider the 
> environment before printing this email.
>
>
> _______________________________________________
> CIG-SEISMO mailing list
> CIG-SEISMO at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

-- 
Piero Lanucara
CINECA
SuperComputing Applications and Innovation Department - SCAI
Phone: +39 06 44486709
Fax: +39 06 4957083
Address: CINECA Sede di Roma Via dei Tizii,6/b - 00185 Roma - Italy
E-mail: p.lanucara at cineca.it
skype: lpeter175
site: http://www.hpc.cineca.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20170427/4e0f5253/attachment-0001.html>