[cig-commits] r19618 - seismo/3D/SPECFEM3D/trunk

[dkomati1 at geodynamics.org]

Author: dkomati1
Date: 2012-02-13 13:35:07 -0800 (Mon, 13 Feb 2012)
New Revision: 19618

Modified:
   seismo/3D/SPECFEM3D/trunk/todo_list_please_dont_remove.txt
Log:
added a sentence about PT-SCOTCH;
also removed old items that are now implemented.


Modified: seismo/3D/SPECFEM3D/trunk/todo_list_please_dont_remove.txt
===================================================================
--- seismo/3D/SPECFEM3D/trunk/todo_list_please_dont_remove.txt	2012-02-11 05:38:06 UTC (rev 19617)
+++ seismo/3D/SPECFEM3D/trunk/todo_list_please_dont_remove.txt	2012-02-13 21:35:07 UTC (rev 19618)
@@ -4,7 +4,9 @@
 
 - add a checkpointing/restarting system with CRC-32 as in SPECFEM3D_GLOBE/tags/v4.1.0_beta_merged_mesher_solver_non_blocking_MPI/src
 
-- Partitioning should be done using a parallel partitioner such as ParMETIS and/or SCOTCH (PT-SCOTCH). Reading the (distributed or not) mesh, building the graph, distributing, and partitioning is not a problem, but redistributing the graph/mesh according to the partitioning obtained might well be. Should add support for parMETIS or PT-Scotch (i.e. decompose the mesh in parallel rather than serial; this will be required for very large meshes, too large to fit on a serial machine). Serial domain decomposition is already a limitation in the case of regional seismology, because serial meshes can be very large. One option would be to switch to ParMetis or PT-Scotch, i.e., the parallel version of these domain decomposition packages. I know that PT-Scotch works great and is pretty fast. Another option is ZOLTAN from Sandia National Labs, but I do not know if it is serial or parallel. I think adding support for PT-Scotch should not be too difficult if we give it a very simple initial "analytical" partition, e.g. if we have NSPEC spectral elements and NPROC domains, put the first 1..NPROC elements in domain 0, then NPROC+1...2*NPROC in domain 1 etc. That's your initial domain decomposition; then give that to ParMETIS or PT-Scotch and ask it to optimize by moving elements between partitions.
+- Partitioning should be done using a parallel partitioner such as PT-SCOTCH.
+Developers and users in Switerland recently mentioned that using a serial domain decomposer such as SCOTCH was very slow for large meshes.
+Reading the (distributed or not) mesh, building the graph, distributing, and partitioning is not a problem, but redistributing the graph/mesh according to the partitioning obtained might well be. Should add support for PT-Scotch (i.e. decompose the mesh in parallel rather than serial; this will be required for very large meshes, too large to fit on a serial machine). Serial domain decomposition is already a limitation in the case of regional seismology, because serial meshes can be very large. One option would be to switch to ParMetis or PT-Scotch, i.e., the parallel version of these domain decomposition packages. I know that PT-Scotch works great and is pretty fast. Another option is ZOLTAN from Sandia National Labs, but I do not know if it is serial or parallel. I think adding support for PT-Scotch should not be too difficult if we give it a very simple initial "analytical" partition, e.g. if we have NSPEC spectral elements and NPROC domains, put the first 1..NPROC elements in domain 0, then NPROC+1...2*NPROC in domain 1 etc. That's your initial domain decomposition; then give that to PT-Scotch and ask it to optimize by moving elements between partitions.
 
 - For binary files (for instance Carl Tape's m16 model) we should use NETCDF to store them and read them back; that is the only clean way of ensuring portability between different systems; and NETCDF is free and open source therefore there is no reason not to do it. 
 
@@ -100,113 +102,4 @@
 Things to add later to the manual:
 ----------------------------------
 
-- Here is how to run the code:
 
-1/ create a CUBIT mesh 
-Possibly decimate it with "decimate_mesh"
-
-2/ check its quality with Dimitri's Fortran code "check_mesh_quality_CUBIT_Abaqus"
-
-3/ export it with Emanuele's scripts, in decompose_mesh_SCOTCH/OUTPUT_FILES
-
-2/ decompose it with "decompose_mesh_SCOTCH" and move the output in your "LOCAL_PATH" (see DATA/Par_file)
-
-4/ create databases and matrices with "generate_databases"
-
-5/ compile and run the solver "specfem3D"
-
-- remove old things from SPECFEM3D_BASIN in the manual and add new things
-related to CUBIT, METIS, SCOTCH etc. Add new figures showing the asteroid
-example etc
-
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
-
-Done:
------
-
- - Added attenuation
-
- - Added moment sources
-
- - Put kernel calculations back
-
-- Regarding domain decomposition, METIS and ParMETIS can become inefficient
-for very large meshes. Pieyre has done detailed tests (Pieyre,
-could you please send the Excel file to all of us with a few lines
-of description of the figures?).
-SCOTCH (serial) and PT-SCOTCH (parallel) is then better.
-(faster, uses less memory etc.)
-Because of that, we have removed support for METIS in SPECFEM3D.
-It can still be called indirectly from SCOTCH (there is an option
-in SCOTCH to call METIS instead). But this is a bad idea because
-the partition obtained is worse.
-
-- NOTE (by Daniel Peter): the 3D code here uses a potential of (density * displacement),
-                  identical to the 2D code (see also the note in compute_forces_acoustic.f90). 
-                  thus, the remark below is obsolete, having 
-                  acoustic-acoustic discontinuities in your 3D model is fine.
-
-Date: Fri, 23 Jul 2010 01:30:16 +0200
-From: Dimitri Komatitsch <dimitri.komatitsch at univ-pau.fr>
-To: Anne Sieminski <anne.sieminski at obs.ujf-grenoble.fr>
-CC: Jeroen Tromp <jtromp at princeton.edu>, Min Chen <mchen at gps.caltech.edu>, Vala Hjorleifsdottir <vala at ldeo.columbia.edu>, Brian Savage <savage at uri.edu>, Shiann-Jong Lee <sjlee at earth.sinica.edu.tw>, Dimitri Komatitsch <dimitri.komatitsch at univ-pau.fr>, Roland Martin <roland.martin at univ-pau.fr>, Bernhard Schuberth <mail at bernhard-schuberth.de>, Carl Tape <carltape at fas.harvard.edu>, Anne Sieminski <anne.sieminski at obs.ujf-grenoble.fr>, Paul Friberg <p.friberg at isti.com>, Kasper van Wijk <kasper at cgiss.boisestate.edu>, Dylan Mikesell <dmikesell at cgiss.boisestate.edu>, Federica Magnoni <federica.magnoni at ingv.it>, Ebru Bozdag <bozdag at princeton.edu>, Hejun Zhu <hejunzhu at princeton.edu>, Pieyre Le Loher <pieyre.le_loher at inria.fr>, Christina Morency <cmorency at Princeton.EDU>, Emanuele Casarotti <emanuele.casarotti at gmail.com>, Piero Basini <basini at bo.ingv.it>, Emmanuel Chaljub <Emmanuel.Chaljub at obs.ujf-grenoble.fr>, Tarje Nissen-Meyer <tarje at princeton.edu>, Qinya Liu <liuqy at physics.utoronto.ca>, Yang Luo <yangl at Princeton.EDU>, Daniel Peter <dpeter at Princeton.EDU>, Ying Zhou <yingz at vt.edu>
-Subject: Re: milieux acoustiques
-
-Dear Anne,
-
-I answer in English and cc all the other developers because other people 
-may be interested. Yes, SPECFEM3D version 2 (which is a beta
-version right now) can handle acoustic media in addition to 
-(visco)elastic layers. Daniel Peter has added support for that
-(based in part on our implementation in the 2D version SPECFEM2D-6.0.0
-I guess, or on what was already implemented in SPECFEM3D_GLOBE, i.e.,
-the fluid outer core of the Earth).
-
-However, there is a hidden problem if you want to use acoustic layers 
-for oil industry simulations: our acoustic formulation in the 3D code
-(at least in SPECFEM3D_GLOBE for sure) can only handle one acoustic 
-layer with a smooth velocity gradient (which is the case in the outer 
-core) but not a first-order acoustic-acoustic discontinuity, which from 
-a physical point of view does not make much sense. *BUT* this implies
-that if you take an elastic oil industry model and make it acoustic 
-(with layers) by only considering Vp, as people often do in oil 
-companies, the code will fail (more precisely, and unfortunately, it 
-will not crash but will produce seismograms that will not be correct).
-
-This is because for acoustic-acoustic interfaces one should use a 
-potential of (density * displacement) rather than a potential of 
-displacement only in order to enforce continuity of pressure 
-automatically in the weak formulation of the problem.
-
-In the 2D code that's what I do (which means you can safely use
-the 2D code for acoustic oil industry models). But I think that
-this is not done in the 3D code because I assume Daniel took the 
-acoustic code from SPECFEM3D_GLOBE, in which case a potential
-of displacement is used and acoustic-acoustic interfaces are not handled 
-properly.
-
-I guess one day we should switch to a potential of (density * 
-displacement) in the 3D code, as I did in the 2D code two years ago.
-Otherwise I think one day some users will use it for oil industry models
-and will get erroneous seismograms. That's (very) easy to do but 
-unfortunately I do not have time for now...
-
-Hope this helps,
-Cheers,
-
-Dimitri.
-
-On 06/23/2010 05:42 PM, Anne Sieminski wrote:
-> Bonjour Dimitri,
->
-> Je vois que SPECFEM2D-6.0.0 (disponible sur le site de CIG) permet de
-> faire des calculs dans des modeles acoustiques. La meme option est-elle
-> en projet pour SPECFEM3D (ancien SPECFEM3D_BASIN) ? J'ai un peu perdu le
-> fil de toutes les nouveautes... Desolee !
->
-> Merci,
-> Anne
-