[aspect-devel] Progress in writing the mantle convection code Aspect

[Wolfgang Bangerth]

> Good question.  I did a few tests on Stampede and it looks like it will
> scale reasonably well.  If we assume 1e9 particles with id+XYZ (4 doubles =
> 32 bytes) per particle, we need to broadcast ~30 GB from the root process
> in many small packages.

> [...]
>
> Each of these is the average of 100 broadcasts.  At the kind of transfer
> rates we see above, sending 30GB of particle data would take a minute or
> two, which would occur once during initialization.  This can be improved
> further by intelligently sending particles only to processes that have a
> chance of containing them (track bounding boxes at the root process).  And
> transmission likely won't be the slowest part, but rather determining if a
> particle is in a particular cell, which has to occur no matter what method
> we use.
>
> I'll look in the literature to see if there's a better way of doing this
> sort of thing but this seems reasonable from what I can tell.

How about this:

Every processor has a fraction 0<=V_i<=1 of the total probability, so that
   sum_i V_i = 1
and V_i is the integral over the local subdomain. Let's do an all-to-all on 
these. Then, each processor draws N numbers between zero and one from a random 
number generator that is initialized in exactly the same way (easy to do with 
the RNGs from BOOST). For one of these numbers x_k, if
   sum_{i=0}^{subdomain_id-1} V_i <= x_k <= sum_{i=0}^{subdomain} V_i
then we need to generate a particle locally. Otherwise, we simply draw the 
next number x_k. This ensures an accurate total number of particles, and it 
randomly distributes where they are located across the subdomains. If we do 
need to create a particle on the current processor, then we do the same 
process on the current processor to find out which cell and where inside the 
cell to create the particle.

I think this way you get around the communication and you get the same 
statistical properties.

Best
  W.


-- 
------------------------------------------------------------------------
Wolfgang Bangerth               email:            bangerth at math.tamu.edu
                                 www: http://www.math.tamu.edu/~bangerth/