[aspect-devel] Writing a plugin for a new data format/source

[Wolfgang Bangerth]

James,

> We working on a NSF-funded project (BALTO) to develop a ‘data broker’ and our PI uses Aspect in her work,, which serves as one of the use-cases for the project. We would like guidance on developing an Aspect plugin to read data not from a local ASCII file, but from a (remote) web service.
> 
> If a new plugin is not the best way forward, what would be the best way to extend Aspect so that it could read other data formats and/or data sources (i.e., from remote web services and not local files)?

Timo has already mentioned a number of issues, in particular the fact 
that on most clusters, individual nodes don't have access to the wider 
internet. This makes working with data sources located elsewhere awkward.

Off the top of my head, what would be interesting would be, for example, 
to deal with large data sets that may be stored elsewhere. Data sets 
such as CRUST 1.0 might be interesting, or the STS elevation data for 
the earth surface. In all of these cases, however, if the data is too 
large to store locally, then it is also too large to download in its 
entirety from a WebDAV (or OpenDAP) server. A more interesting method 
would then of course be if the data could be obtained by sending a query 
to some server that then processes or subsets the data stored there. I 
don't know whether that's part of you're working on.

An alternative -- maybe more interesting -- would be to not deal with 
input data but with output data. A common use case is if a simulation 
produces visualization data that is then visualized on some other 
system. Transferring this data after the run is often annoying and 
awkward. A nicer way would be if it is automatically transferred as part 
of the simulation run -- though one of course gets into the same issue 
with internet access from big clusters.


So my recommendation would be to first find a good use case and then 
think about how to implement it, rather than the other way around.

Best
  Wolfgang


-- 
------------------------------------------------------------------------
Wolfgang Bangerth          email:                 bangerth at colostate.edu
                            www: http://www.math.colostate.edu/~bangerth/