[CIG-SEISMO] CIG-SEISMO Digest, Vol 120, Issue 5

Mon Feb 12 15:13:36 PST 2018

Hi all,

PS: only works (directly) for forward runs. For FWI / imaging / adjoint 
runs, the adjoint sources are needed at each time step. Interpolating is 
possible, but more involved.

Best,
Dimitri.

On 02/13/2018 12:10 AM, Dimitri Komatitsch wrote:
> 
> Hi Daniel and Moritz,
> 
> Thanks!
> 
> Another option that we frequently use here is to subsample the 
> seismograms, i.e. allocate seismograms arrays that are e.g. 10 times 
> smaller and save one every 10 time steps.
> This is very often OK because the CFL time step is much smaller than the 
> Nyquist frequency of the source.
> 
> In the 2D code we have this flag:
> 
> # subsampling of the seismograms to create smaller files (but less 
> accurately sampled in time)
> subsamp_seismos                 = 1
> 
> Not sure if it is available in 3D as well (if not, very easy to 
> implement; if someone does it, please let us know and/or commit that to 
> Git, it would be very useful).
> 
> thanks
> Best,
> Dimitri.
> 
> On 02/12/2018 11:29 PM, Daniel B. Peter wrote:
>> Hi Moritz,
>>
>> again good point - your pushing the limits :)
>>
>> concerning the mesh, the GPU memory is just fine for that number of 
>> elements. 1M elements will take about 10GB memory for an elastic 
>> simulation. so you could even add more if you like for those two K40s.
>>
>> in your setup, it’s likely a problem with the seismogram allocations 
>> on the GPUs. this changed recently such that the full seismogram 
>> arrays are allocated on the GPU as well to speed up simulations even 
>> further. unfortunately, this comes with a hit on memory consumption.
>>
>> in your setup, you have about 10,000 stations and 111,111 time steps. 
>> depending on the number of receivers in a single slice and how they 
>> are shared between the 2 GPUs, a rough estimate for your case shows up 
>> to about 6.3GB additional memory for a single 3-component seismogram 
>> (e.g. displacement) per GPU (assuming you have an even split with 
>> 5,000 local stations per GPU). given the ~7.5GB from the mesh, this 
>> exceeds a single GPU memory of 12GB.
>>
>> as a quick workaround, you would split up the stations and run several 
>> simulations for different station setups. this can limit the 
>> seismogram memory allocation. the other workarounds involve some 
>> changes in the code: (i) we could add checkpointing (again, the global 
>> version has it, so in principle easy to add to the Cartesian version) 
>> but this means you will have to run multiple simulations as well, (ii) 
>> we reduce the seismogram allocations. well, both would be nice, let me 
>> focus on (ii) first since we already have a parameter in the Par_file, 
>> NTSTEP_BETWEEN_OUTPUT_SEISMOS, which could be used here to limit the 
>> array allocation size.
>>
>> best wishes,
>> daniel
>>
>>
>>
>>
>>> On Feb 12, 2018, at 7:25 PM, Fehr, Moritz <Moritz.Fehr at dmt-group.com> 
>>> wrote:
>>>
>>> Hi Daniel,
>>>
>>> thanks a lot for adjusting the receiver detection routine. First the 
>>> routine seems to work fine, but after finishing the routine the 
>>> calculation immediately breaks up with CUDA memory errors (see error 
>>> files). I have used two Tesla K40m GPUs (2 x 12 gig) and the model 
>>> consists of 1.500.000 elements. I think the GPU memory should be 
>>> enough for that model size. The same problem has occurred with the 
>>> V100 GPUs.
>>> Do you have any idea?
>>>
>>> Thanks
>>> Mo
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: CIG-SEISMO [mailto:cig-seismo-bounces at geodynamics.org] Im 
>>> Auftrag von cig-seismo-request at geodynamics.org
>>> Gesendet: Donnerstag, 25. Januar 2018 11:02
>>> An: cig-seismo at geodynamics.org
>>> Betreff: CIG-SEISMO Digest, Vol 120, Issue 5
>>>
>>> Send CIG-SEISMO mailing list submissions to
>>> cig-seismo at geodynamics.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>> or, via email, send a message with subject or body 'help' to
>>> cig-seismo-request at geodynamics.org
>>>
>>> You can reach the person managing the list at
>>> cig-seismo-owner at geodynamics.org
>>>
>>> When replying, please edit your Subject line so it is more specific 
>>> than "Re: Contents of CIG-SEISMO digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>    1. Re: SPECFEM3D: GPU memory usage limited (Daniel B. Peter)
>>>    2.  SPECFEM3D: GPU memory usage limited (Fehr, Moritz)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Wed, 24 Jan 2018 21:02:53 +0000
>>> From: "Daniel B. Peter" <daniel.peter at kaust.edu.sa>
>>> To: "cig-seismo at geodynamics.org" <cig-seismo at geodynamics.org>
>>> Subject: Re: [CIG-SEISMO] SPECFEM3D: GPU memory usage limited
>>> Message-ID: <5EC00584-31D0-4CFE-8708-4142FEC8D106 at kaust.edu.sa>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> hi Moritz,
>>>
>>> is there also an error output?
>>>
>>> I’m not aware that there should be such an issue with SPECFEM3D on 
>>> this newest GPU hardware. running simulations on Pascal GPUs with 
>>> multiple GB memory usage works just fine. so I would expect that the 
>>> run exits because of another issue, not because of the GPU memory part.
>>>
>>> your output_solver.txt stops at the receiver detection. based on your 
>>> setup which uses about 13,670 stations, i expect it to be a receiver 
>>> detection issue. this routines works fine for a few hundred station, 
>>> but becomes very slow for more than a few thousand on a single 
>>> process. this issue has been addressed in the global version, let me 
>>> see if i can implement it in a similar way in the SPECFEM3D devel 
>>> version.
>>>
>>> many thanks for pointing out,
>>> daniel
>>>
>>>
>>>
>>> On Jan 24, 2018, at 6:35 PM, Fehr, Moritz 
>>> <Moritz.Fehr at dmt-group.com<mailto:Moritz.Fehr at dmt-group.com>> wrote:
>>>
>>> Hallo,
>>>
>>> I have a problem simulating a CUBIT meshed model (1.000.000 elements) 
>>> on a Tesla GPU V100-SXM2 (amazon cloud CPU / GPU cluster). I am using 
>>> SPECEM3D Cartesian (V3.0) and the newest CUDA 9 lib. I want to share 
>>> this issue: It seems that the GPU memory usage is limited to a value 
>>> of 420 MiB although the maximum GPU memory is 16000 Mib.
>>> Do you have any idea about the origin of this limitation?
>>>
>>> Thanks
>>> Mo
>>>
>>> <image001.png>
>>>
>>>
>>>
>>> ___________________________________________________________________________________________________ 
>>>
>>> Sitz der Gesellschaft/Headquarters: DMT GmbH & Co. KG * Am 
>>> Technologiepark 1 * 45307 Essen * Deutschland/Germany 
>>> Registergericht/County Court: Amtsgericht Essen * HRA 9091 * USt-ID 
>>> DE 253275653 Komplementär/Fully Liable Partner: DMT 
>>> Verwaltungsgesellschaft mbH, Essen Registergericht/County Court: 
>>> Amtsgericht Essen * HRB 20420 Geschäftsführer/Board of Directors: 
>>> Prof. Dr. Eiko Räkers (Vorsitzender/CEO), Dr. Maik Tiedemann, Ulrich 
>>> Pröpper, Jens-Peter Lux Vorsitzender des Aufsichtsrates/Chairman of 
>>> the Supervisory Board: Jürgen Himmelsbach TÜV NORD GROUP 
>>> ___________________________________________________________________________________________________ 
>>>
>>> Diese Nachricht enthält vertrauliche Informationen und ist nur für 
>>> den Empfänger bestimmt. Wenn Sie nicht der Empfänger sind, sollten 
>>> Sie die E-Mail nicht verbreiten, verteilen oder diese E-Mail 
>>> kopieren. Benachrichtigen Sie bitte den Absender per E-Mail, wenn Sie 
>>> diese E-Mail irrtümlich erhalten haben und löschen dann diese E-Mail 
>>> von Ihrem System.
>>>
>>> This message contains confidential information and is intended only 
>>> for the recipient. If you are not the recipient you should not 
>>> disseminate, distribute or copy this e-mail. Please notify the sender 
>>> immediately by e-mail if you have received this e-mail by mistake and 
>>> delete this e-mail from your system.
>>>
>>>
>>> <CMTSOLUTION><output_solver.txt><gpu_device_info.txt><Par_file><output_generate_databases.txt><values_from_mesher.h>_______________________________________________ 
>>>
>>> CIG-SEISMO mailing list
>>> CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org>
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>>
>>>
>>> ________________________________
>>> This message and its contents including attachments are intended 
>>> solely for the original recipient. If you are not the intended 
>>> recipient or have received this message in error, please notify me 
>>> immediately and delete this message from your computer system. Any 
>>> unauthorized use or distribution is prohibited. Please consider the 
>>> environment before printing this email.
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: 
>>> <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180124/7c4f0513/attachment-0001.html> 
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Thu, 25 Jan 2018 10:07:18 +0000
>>> From: "Fehr, Moritz" <Moritz.Fehr at dmt-group.com>
>>> To: "cig-seismo at geodynamics.org" <cig-seismo at geodynamics.org>
>>> Subject: [CIG-SEISMO]  SPECFEM3D: GPU memory usage limited
>>> Message-ID:
>>> <b0f6d12197f24424b880f2ae10910a39 at H04EXC13.netz.tuev-nord.de>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> Hi Daniel,
>>>
>>> thanks for your rapid response. Sorry, I do not have any error output 
>>> file, because of breaking up the simulation by myself (The simulation 
>>> remains in the process of receiver detection).  But I tried the same 
>>> simulation with just one receiver and it works fine.
>>> Please let me know if you can fix the problem.
>>>
>>> Thanks
>>> Mo
>>>
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------------------------------- 
>>>
>>> hi Moritz,
>>>
>>> is there also an error output?
>>>
>>> I’m not aware that there should be such an issue with SPECFEM3D on 
>>> this newest GPU hardware. running simulations on Pascal GPUs with 
>>> multiple GB memory usage works just fine. so I would expect that the 
>>> run exits because of another issue, not because of the GPU memory part.
>>>
>>> your output_solver.txt stops at the receiver detection. based on your 
>>> setup which uses about 13,670 stations, i expect it to be a receiver 
>>> detection issue. this routines works fine for a few hundred station, 
>>> but becomes very slow for more than a few thousand on a single 
>>> process. this issue has been addressed in the global version, let me 
>>> see if i can implement it in a similar way in the SPECFEM3D devel 
>>> version.
>>>
>>> many thanks for pointing out,
>>> daniel
>>>
>>>
>>>
>>> On Jan 24, 2018, at 6:35 PM, Fehr, Moritz <Moritz.Fehr at 
>>> dmt-group.com<mailto:Moritz.Fehr at dmt-group.com>> wrote:
>>>
>>> Hallo,
>>>
>>> I have a problem simulating a CUBIT meshed model (1.000.000 elements) 
>>> on a Tesla GPU V100-SXM2 (amazon cloud CPU / GPU cluster). I am using 
>>> SPECEM3D Cartesian (V3.0) and the newest CUDA 9 lib. I want to share 
>>> this issue: It seems that the GPU memory usage is limited to a value 
>>> of 420 MiB although the maximum GPU memory is 16000 Mib.
>>> Do you have any idea about the origin of this limitation?
>>>
>>> Thanks
>>> Mo
>>>
>>> <image001.png>
>>>
>>>
>>>
>>> <CMTSOLUTION><output_solver.txt><gpu_device_info.txt><Par_file><output_generate_databases.txt><values_from_mesher.h>_______________________________________________ 
>>>
>>> CIG-SEISMO mailing list
>>> CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org> 
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>>
>>>
>>> ________________________________
>>> This message and its contents including attachments are intended 
>>> solely for the original recipient. If you are not the intended 
>>> recipient or have received this message in error, please notify me 
>>> immediately and delete this message from your computer system. Any 
>>> unauthorized use or distribution is prohibited. Please consider the 
>>> environment before printing this email.
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: 
>>> <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180124/7c4f0513/attachment.html> 
>>>
>>>
>>> ___________________________________________________________________________________________________ 
>>>
>>> Sitz der Gesellschaft/Headquarters: DMT GmbH & Co. KG * Am 
>>> Technologiepark 1 * 45307 Essen * Deutschland/Germany 
>>> Registergericht/County Court: Amtsgericht Essen * HRA 9091 * USt-ID 
>>> DE 253275653 Komplementär/Fully Liable Partner: DMT 
>>> Verwaltungsgesellschaft mbH, Essen Registergericht/County Court: 
>>> Amtsgericht Essen * HRB 20420 Geschäftsführer/Board of Directors: 
>>> Prof. Dr. Eiko Räkers (Vorsitzender/CEO), Dr. Maik Tiedemann, Ulrich 
>>> Pröpper, Jens-Peter Lux Vorsitzender des Aufsichtsrates/Chairman of 
>>> the Supervisory Board: Jürgen Himmelsbach TÜV NORD GROUP 
>>> ___________________________________________________________________________________________________ 
>>>
>>> Diese Nachricht enthält vertrauliche Informationen und ist nur für 
>>> den Empfänger bestimmt. Wenn Sie nicht der Empfänger sind, sollten 
>>> Sie die E-Mail nicht verbreiten, verteilen oder diese E-Mail 
>>> kopieren. Benachrichtigen Sie bitte den Absender per E-Mail, wenn Sie 
>>> diese E-Mail irrtümlich erhalten haben und löschen dann diese E-Mail 
>>> von Ihrem System.
>>>
>>> This message contains confidential information and is intended only 
>>> for the recipient. If you are not the recipient you should not 
>>> disseminate, distribute or copy this e-mail. Please notify the sender 
>>> immediately by e-mail if you have received this e-mail by mistake and 
>>> delete this e-mail from your system.
>>>
>>>
>>> ------------------------------
>>>
>>> Subject: Digest Footer
>>>
>>> _______________________________________________
>>> CIG-SEISMO mailing list
>>> CIG-SEISMO at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>>
>>> ------------------------------
>>>
>>> End of CIG-SEISMO Digest, Vol 120, Issue 5
>>> ******************************************
>>>
>>> ___________________________________________________________________________________________________ 
>>>
>>> Sitz der Gesellschaft/Headquarters: DMT GmbH & Co. KG * Am 
>>> Technologiepark 1 * 45307 Essen * Deutschland/Germany
>>> Registergericht/County Court: Amtsgericht Essen * HRA 9091 * USt-ID 
>>> DE 253275653
>>> Komplementär/Fully Liable Partner: DMT Verwaltungsgesellschaft mbH, 
>>> Essen
>>> Registergericht/County Court: Amtsgericht Essen * HRB 20420
>>> Geschäftsführer/Board of Directors: Prof. Dr. Eiko Räkers 
>>> (Vorsitzender/CEO), Dr. Maik Tiedemann, Ulrich Pröpper, Jens-Peter Lux
>>> Vorsitzender des Aufsichtsrates/Chairman of the Supervisory Board: 
>>> Jürgen Himmelsbach
>>> TÜV NORD GROUP
>>> ___________________________________________________________________________________________________ 
>>>
>>> Diese Nachricht enthält vertrauliche Informationen und ist nur für 
>>> den Empfänger bestimmt. Wenn Sie nicht der Empfänger sind, sollten 
>>> Sie die E-Mail nicht verbreiten, verteilen oder diese E-Mail 
>>> kopieren. Benachrichtigen Sie bitte den Absender per E-Mail, wenn Sie 
>>> diese E-Mail irrtümlich erhalten haben und löschen dann diese E-Mail 
>>> von Ihrem System.
>>>
>>> This message contains confidential information and is intended only 
>>> for the recipient. If you are not the recipient you should not 
>>> disseminate, distribute or copy this e-mail. Please notify the sender 
>>> immediately by e-mail if you have received this e-mail by mistake and 
>>> delete this e-mail from your system.
>>>
>>> <error_message_000004.txt><error_message_000005.txt><error_message_000011.txt><error_message_000010.txt><error_message_000014.txt><error_message_000017.txt><error_message_000006.txt><error_message_000008.txt><error_message_000007.txt><error_message_000001.txt><error_message_000009.txt><output_solver.txt><gpu_device_info.txt><output_generate_databases.txt>_______________________________________________ 
>>>
>>> CIG-SEISMO mailing list
>>> CIG-SEISMO at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>
>> _______________________________________________
>> CIG-SEISMO mailing list
>> CIG-SEISMO at geodynamics.org
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
>>
> 

-- 
Dimitri Komatitsch, CNRS Research Director (DR CNRS)
Laboratory of Mechanics and Acoustics, Marseille, France
http://komatitsch.free.fr