[cig-commits] r12834 - seismo/3D/SPECFEM3D_GLOBE/trunk

[dkomati1 at geodynamics.org]

Author: dkomati1
Date: 2008-09-10 10:09:14 -0700 (Wed, 10 Sep 2008)
New Revision: 12834

Modified:
   seismo/3D/SPECFEM3D_GLOBE/trunk/todo_list_please_dont_remove.txt
Log:
updated the todo list


Modified: seismo/3D/SPECFEM3D_GLOBE/trunk/todo_list_please_dont_remove.txt
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/todo_list_please_dont_remove.txt	2008-09-10 01:19:17 UTC (rev 12833)
+++ seismo/3D/SPECFEM3D_GLOBE/trunk/todo_list_please_dont_remove.txt	2008-09-10 17:09:14 UTC (rev 12834)
@@ -1,9 +1,10 @@
 
-To-do list for SPECFEM3D_GLOBE, by Dimitri Komatitsch and David Michea
-----------------------------------------------------------------------
+To-do list for SPECFEM3D_GLOBE, by Dimitri Komatitsch:
+------------------------------------------------------
 
-- Add this to Cuthill-McKee sorting, to detect non adjacent graphs as done in the 2D code:
-"plutôt que de s'arrêter par un stop, on peut ressortir de la routine sans la finir et mettre un flag "cuthill_failed" et alors générer à la place une permutation identité ; ceci revient à dire que si l'on voit que Cuthill ne va pas marcher, plutôt que d'arrêter le code on peut tout simplement désactiver Cuthill, afficher à l'écran le fait qu'on l'a désactivé, et continuer le run sans Cuthill"
+- The superbrick cannot be cut in two in its middle: David Michea implemented
+it but the implementation has a bug and the code crashes (when NEX_XI is a
+multiple of 8 rather than 16). We should fix this bug in version 4.1 below at some point.
 
 Things that could be done in a future version 4.1:
 
@@ -54,81 +55,6 @@
 careful analysis we have found a way of doing better and getting rid of all
 the GATHERs and *all* the extra memory (this way we won't have to limit the number of sources to a maximum of 100000, as currently done in the solver).
 
-- merge the mesher in the solver, suppress storage of large mesh files on local disks, but make sure we keep the creation of AVS/OpenDX mesh files (which should be the case since we use specific routines called write_AVS_DX_*.f90) 30/06/2007 18:19 Hi Jeroen, Leif and David, I thought about merging the mesher and the solver, there might be a hidden difficulty because of memory allocation: the solver allocates almost all the available memory on a given CPU in static arrays, and the solver allocates a similar amount dynamically with allocate() statements. Therefore combining them would roughly double the required amount of memory and will not fit because the Par_file is tuned in order for the solver to use almost all the available memory on each CPU (in the case of large runs).  On systems (such as Linux) with a "first touch" memory allocation policy things should be ok because (if I understand correctly, but I am not sure) the system will not physically allocate the static memory for the solver until it is actually called, i.e. once the mesher has finished, even if both are combined in the same program.  But on other systems I guess we could have problems and the run would not start and exit with an "out of memory" error?  I don't know how to solve this problem. (*if* it needs to be solved; if *all* modern systems use a first touch policy, then we are all set; please let me know; I don't know much about operating systems). Leif, would you have any idea about how to handle this?  If the problem turns out to be too complex, I suggest *not* doing it in v4.0 and waiting until we start working on v4.1 after the October 9-11 workshop. Dimitri.
-Hi Jeroen,
-I would say 20%, which is probably not much and as you say it is
-probably the price to pay to run on a diskless machine.
-The question is: do we keep and maintain two versions (a merged version
-and also two separate codes)? I think what we should do it keep two
-codes and ask David to write a third (very small) code (an interface)
-which would call the mesher and the solver as subroutines and use memory
-instead of the disk to copy the output information of the mesher
-to the input of the solver. The mesher could still use dynamic
-allocation, the solver would remain static, and on any machine
-with a "first touch" policy in the operating system we would be all set.
-This way the code would still be portable, could run on any
-system (even systems without "first touch", using two codes
-and the local disks). The only type of system such a code
-could not run on would be a machine with no local disks
-*and* no first touch policy. But such a machine is very
-unlikely to exist (most modern operating systems
-allocate static memory when it starts to be used
-because there is no reason to do otherwise, because
-this would mean locking resources for no reason;
-therefore we can safely assume that "first touch"
-will become standard in the future; it is already almost standard.
-Dimitri.
-Jeroen Tromp wrote:
-> Hi Dimitri:
-> But perhaps the hit in extra memory should simply be regarded as the
-> extra cost of doing business on a petascale machine. What is the
-> percentage increase in memory that would result from the merge?
-> Jeroen
-> Dimitri Komatitsch wrote:
->> Hi Jeroen,
->> Yes, but there is a hidden problem: the current mesher needs four more
->> (big) arrays than the solver. I can reduce this to two maybe using
->> some clever "equivalence" statements, but not to zero; i.e.
->> if we merge the solver and the mesher and use static memory for both
->> we will create a merged version in which the mesher will control
->> memory size again, not the solver (as in v3.6).
->> I have solved this problem in version 4.0 by using dynamic memory
->> allocation cleverly in the mesher: I call create_region_mesh twice and
->> create half the (dynamic) arrays each time, therefore
->> I need only half the memory in each call, and this is always
->> below the memory size of all the static arrays in the solver,
->> i.e. we are back to the normal situation in which it is the solver
->> that controls memory size. But we would lose this if we merged
->> the codes because my trick would not work anymore...
->> (because I would not be able to deallocate the memory dynamically
->> after the first call)
->> Of course, we know that Linux does not allocate static arrays
->> until they are actually used, which means that on all Linux
->> machines we could merge the current v4.0 codes without
->> any modification, keeping the mesher dynamic, letting it finish,
->> and then Linux would create the static arrays of the solver
->> only when the solver actually starts and all the arrays used
->> by the mesher have been deallocated.
->> Therefore the question is: will we ever run the merged code
->> on a supercomputer that will not run Linux (or any other
->> system with a similar "first touch" policy). Difficult to know
->> in advance...
->> Dimitri.
->> Jeroen Tromp wrote:
->>> Hi Dimitri:
->>> Your ideas for reducing the memory requirements of the mesher made me
->>> think about the problems related to merging the mesher and solver.
->>> Can you see a way of have a small serial code, like
->>> create_header_file.f90, that one would run *before* you run either
->>> the mesher or the solver, that determines the correct array
->>> dimensions from the Par_file settings. This code is then used to
->>> produce the right header file for the combined static mesher-solver.
->>> The read_arrays_solver call in the solver is turned into a call to
->>> the mesher, which effectively becomes a subroutine.
->>> Jeroen
-
-- peut-etre supprimer write_AVS_mesh_quality, check_buffers*.f90, check_mesh_quality*.f90 etc du serveur SVN (svn remove) une fois que le mesher et le solver auront ete fusionnes; sauf si on decide de garder deux versions du code: une fusionnee et une non fusionnee qui utiliserait toujours le stockage sur des disques locaux. Les enlever du Makefile.in aussi. Do not store x y z in the files that contain the MPI buffers anymore, since they are used only by check_buffers*. (in the solver they are read back as xdummy, ydummy, zdummy); eventuellement supprimer aussi le fichier ecrit par le mailleur sur le disque et relu plus tard par ce meme mailleur, s'il n'est pas utilise par le solver (c'est a dire s'il est utilise seulement par les codes serial check*.f90, qui sont amenes a disparaitre)
-
 ---------------------------------------------------------------------------------
 ---------------------------------------------------------------------------------
 ---------------------------------------------------------------------------------
@@ -151,13 +77,3 @@
 
 - supprimer sections qui decrivent write_AVS_mesh_quality, check_buffers*.f90, check_mesh_quality*.f90 etc du manuel une fois que le mesher et le solver auront ete fusionnes
 
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
-
-Done:
------
-
-- Cuthill-McKee sorting to reduce cache misses: finished by David Michea in March 2008
-