[CIG-SEISMO] CIG-SEISMO Digest, Vol 120, Issue 5

Mon Feb 12 08:25:40 PST 2018

Hi Daniel,

thanks a lot for adjusting the receiver detection routine. First the routine seems to work fine, but after finishing the routine the calculation immediately breaks up with CUDA memory errors (see error files). I have used two Tesla K40m GPUs (2 x 12 gig) and the model consists of 1.500.000 elements. I think the GPU memory should be enough for that model size. The same problem has occurred with the V100 GPUs.
Do you have any idea?

Thanks
Mo

-----Ursprüngliche Nachricht-----
Von: CIG-SEISMO [mailto:cig-seismo-bounces at geodynamics.org] Im Auftrag von cig-seismo-request at geodynamics.org
Gesendet: Donnerstag, 25. Januar 2018 11:02
An: cig-seismo at geodynamics.org
Betreff: CIG-SEISMO Digest, Vol 120, Issue 5

Send CIG-SEISMO mailing list submissions to
cig-seismo at geodynamics.org

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo
or, via email, send a message with subject or body 'help' to
cig-seismo-request at geodynamics.org

You can reach the person managing the list at
cig-seismo-owner at geodynamics.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of CIG-SEISMO digest..."

Today's Topics:

   1. Re: SPECFEM3D: GPU memory usage limited (Daniel B. Peter)
   2.  SPECFEM3D: GPU memory usage limited (Fehr, Moritz)

----------------------------------------------------------------------

Message: 1
Date: Wed, 24 Jan 2018 21:02:53 +0000
From: "Daniel B. Peter" <daniel.peter at kaust.edu.sa>
To: "cig-seismo at geodynamics.org" <cig-seismo at geodynamics.org>
Subject: Re: [CIG-SEISMO] SPECFEM3D: GPU memory usage limited
Message-ID: <5EC00584-31D0-4CFE-8708-4142FEC8D106 at kaust.edu.sa>
Content-Type: text/plain; charset="utf-8"

hi Moritz,

is there also an error output?

I’m not aware that there should be such an issue with SPECFEM3D on this newest GPU hardware. running simulations on Pascal GPUs with multiple GB memory usage works just fine. so I would expect that the run exits because of another issue, not because of the GPU memory part.

your output_solver.txt stops at the receiver detection. based on your setup which uses about 13,670 stations, i expect it to be a receiver detection issue. this routines works fine for a few hundred station, but becomes very slow for more than a few thousand on a single process. this issue has been addressed in the global version, let me see if i can implement it in a similar way in the SPECFEM3D devel version.

many thanks for pointing out,
daniel

On Jan 24, 2018, at 6:35 PM, Fehr, Moritz <Moritz.Fehr at dmt-group.com<mailto:Moritz.Fehr at dmt-group.com>> wrote:

Hallo,

I have a problem simulating a CUBIT meshed model (1.000.000 elements) on a Tesla GPU V100-SXM2 (amazon cloud CPU / GPU cluster). I am using SPECEM3D Cartesian (V3.0) and the newest CUDA 9 lib. I want to share this issue: It seems that the GPU memory usage is limited to a value of 420 MiB although the maximum GPU memory is 16000 Mib.
Do you have any idea about the origin of this limitation?

Thanks
Mo

<image001.png>

___________________________________________________________________________________________________
Sitz der Gesellschaft/Headquarters: DMT GmbH & Co. KG * Am Technologiepark 1 * 45307 Essen * Deutschland/Germany Registergericht/County Court: Amtsgericht Essen * HRA 9091 * USt-ID DE 253275653 Komplementär/Fully Liable Partner: DMT Verwaltungsgesellschaft mbH, Essen Registergericht/County Court: Amtsgericht Essen * HRB 20420 Geschäftsführer/Board of Directors: Prof. Dr. Eiko Räkers (Vorsitzender/CEO), Dr. Maik Tiedemann, Ulrich Pröpper, Jens-Peter Lux Vorsitzender des Aufsichtsrates/Chairman of the Supervisory Board: Jürgen Himmelsbach TÜV NORD GROUP ___________________________________________________________________________________________________
Diese Nachricht enthält vertrauliche Informationen und ist nur für den Empfänger bestimmt. Wenn Sie nicht der Empfänger sind, sollten Sie die E-Mail nicht verbreiten, verteilen oder diese E-Mail kopieren. Benachrichtigen Sie bitte den Absender per E-Mail, wenn Sie diese E-Mail irrtümlich erhalten haben und löschen dann diese E-Mail von Ihrem System.

This message contains confidential information and is intended only for the recipient. If you are not the recipient you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

<CMTSOLUTION><output_solver.txt><gpu_device_info.txt><Par_file><output_generate_databases.txt><values_from_mesher.h>_______________________________________________
CIG-SEISMO mailing list
CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org>
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

________________________________
This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180124/7c4f0513/attachment-0001.html>

------------------------------

Message: 2
Date: Thu, 25 Jan 2018 10:07:18 +0000
From: "Fehr, Moritz" <Moritz.Fehr at dmt-group.com>
To: "cig-seismo at geodynamics.org" <cig-seismo at geodynamics.org>
Subject: [CIG-SEISMO]  SPECFEM3D: GPU memory usage limited
Message-ID:
<b0f6d12197f24424b880f2ae10910a39 at H04EXC13.netz.tuev-nord.de>
Content-Type: text/plain; charset="utf-8"

Hi Daniel,

thanks for your rapid response. Sorry, I do not have any error output file, because of breaking up the simulation by myself (The simulation remains in the process of receiver detection).  But I tried the same simulation with just one receiver and it works fine.
Please let me know if you can fix the problem.

Thanks
Mo

-----------------------------------------------------------------------------------------------------------------------------------
hi Moritz,

is there also an error output?

I’m not aware that there should be such an issue with SPECFEM3D on this newest GPU hardware. running simulations on Pascal GPUs with multiple GB memory usage works just fine. so I would expect that the run exits because of another issue, not because of the GPU memory part.

your output_solver.txt stops at the receiver detection. based on your setup which uses about 13,670 stations, i expect it to be a receiver detection issue. this routines works fine for a few hundred station, but becomes very slow for more than a few thousand on a single process. this issue has been addressed in the global version, let me see if i can implement it in a similar way in the SPECFEM3D devel version.

many thanks for pointing out,
daniel

On Jan 24, 2018, at 6:35 PM, Fehr, Moritz <Moritz.Fehr at dmt-group.com<mailto:Moritz.Fehr at dmt-group.com>> wrote:

Hallo,

I have a problem simulating a CUBIT meshed model (1.000.000 elements) on a Tesla GPU V100-SXM2 (amazon cloud CPU / GPU cluster). I am using SPECEM3D Cartesian (V3.0) and the newest CUDA 9 lib. I want to share this issue: It seems that the GPU memory usage is limited to a value of 420 MiB although the maximum GPU memory is 16000 Mib.
Do you have any idea about the origin of this limitation?

Thanks
Mo

<image001.png>

<CMTSOLUTION><output_solver.txt><gpu_device_info.txt><Par_file><output_generate_databases.txt><values_from_mesher.h>_______________________________________________
CIG-SEISMO mailing list
CIG-SEISMO at geodynamics.org<mailto:CIG-SEISMO at geodynamics.org> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

________________________________
This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180124/7c4f0513/attachment.html>

___________________________________________________________________________________________________
Sitz der Gesellschaft/Headquarters: DMT GmbH & Co. KG * Am Technologiepark 1 * 45307 Essen * Deutschland/Germany Registergericht/County Court: Amtsgericht Essen * HRA 9091 * USt-ID DE 253275653 Komplementär/Fully Liable Partner: DMT Verwaltungsgesellschaft mbH, Essen Registergericht/County Court: Amtsgericht Essen * HRB 20420 Geschäftsführer/Board of Directors: Prof. Dr. Eiko Räkers (Vorsitzender/CEO), Dr. Maik Tiedemann, Ulrich Pröpper, Jens-Peter Lux Vorsitzender des Aufsichtsrates/Chairman of the Supervisory Board: Jürgen Himmelsbach TÜV NORD GROUP ___________________________________________________________________________________________________
Diese Nachricht enthält vertrauliche Informationen und ist nur für den Empfänger bestimmt. Wenn Sie nicht der Empfänger sind, sollten Sie die E-Mail nicht verbreiten, verteilen oder diese E-Mail kopieren. Benachrichtigen Sie bitte den Absender per E-Mail, wenn Sie diese E-Mail irrtümlich erhalten haben und löschen dann diese E-Mail von Ihrem System.

This message contains confidential information and is intended only for the recipient. If you are not the recipient you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

------------------------------

Subject: Digest Footer

_______________________________________________
CIG-SEISMO mailing list
CIG-SEISMO at geodynamics.org
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-seismo

------------------------------

End of CIG-SEISMO Digest, Vol 120, Issue 5
******************************************

___________________________________________________________________________________________________
Sitz der Gesellschaft/Headquarters: DMT GmbH & Co. KG * Am Technologiepark 1 * 45307 Essen * Deutschland/Germany
Registergericht/County Court: Amtsgericht Essen * HRA 9091 * USt-ID DE 253275653
Komplementär/Fully Liable Partner: DMT Verwaltungsgesellschaft mbH, Essen
Registergericht/County Court: Amtsgericht Essen * HRB 20420
Geschäftsführer/Board of Directors: Prof. Dr. Eiko Räkers (Vorsitzender/CEO), Dr. Maik Tiedemann, Ulrich Pröpper, Jens-Peter Lux
Vorsitzender des Aufsichtsrates/Chairman of the Supervisory Board: Jürgen Himmelsbach
TÜV NORD GROUP
___________________________________________________________________________________________________
Diese Nachricht enthält vertrauliche Informationen und ist nur für den Empfänger bestimmt. Wenn Sie nicht der Empfänger sind, sollten Sie die E-Mail nicht verbreiten, verteilen oder diese E-Mail kopieren. Benachrichtigen Sie bitte den Absender per E-Mail, wenn Sie diese E-Mail irrtümlich erhalten haben und löschen dann diese E-Mail von Ihrem System.

This message contains confidential information and is intended only for the recipient. If you are not the recipient you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000004.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0014.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000005.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0015.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000011.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0016.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000010.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0017.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000014.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0018.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000017.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0019.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000006.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0020.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000008.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0021.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000007.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0022.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000001.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0023.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: error_message_000009.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0024.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: output_solver.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0025.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: gpu_device_info.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0026.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: output_generate_databases.txt
URL: <http://lists.geodynamics.org/pipermail/cig-seismo/attachments/20180212/6da08418/attachment-0027.txt>