[aspect-devel] Writing a plugin for a new data format/source

[James Gallagher]

> On Sep 21, 2018, at 09:35, Timo Heister <heister at clemson.edu> wrote:
> 
> James,
> 
>> We working on a NSF-funded project (BALTO) to develop a ‘data broker’ and our PI uses Aspect in her work,, which serves as one of the use-cases for the project. We would like guidance on developing an Aspect plugin to read data not from a local ASCII file, but from a (remote) web service.
> 
> We are reading various kind of files and formats in different plugins
> (initial conditions, gravity, material model, to name a few). Search
> for "data file name" in https://urldefense.proofpoint.com/v2/url?u=https-3A__tjhei.github.io_aspect-2Dwww-2Dmanual_&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=M72AauGyerMMa0b-5d6UAdfFfqi5dhWbxYokW5N_y94&s=nv5Ev6xyH8qE7UV-Ga5U9axYfwBn2t1xTVUY0x-pY-A&e= for
> example. I am not sure what you are trying to do precisely.
> 
>> If a new plugin is not the best way forward, what would be the best way to extend Aspect so that it could read other data formats and/or data sources (i.e., from remote web services and not local files)?
> 
> A plugin would only allow for one of the many ways a user can read
> data in to work. For example, you could write a
> RemoteFileGravityAsciiData plugin.

> I would think a better option would
> be to enhance read_and_distribute_file_content() in aspect/utilities.h
> to accept URLs and downloading them as needed. This function is used
> by various current and future plugins. You need to worry about a few
> things though:
> - on a cluster you might not have internet access
> - in a parallel run you don't want every single of your 1000+
> processes to download the files
> - caching of large files?

This sounds like an option we should look into - it fits with the use case we have for the BALTO project. We’ll look into this and ask more questions as needed. You raise three good points, but we have experience with caching and sharing data among multiple processes.

Thanks,
James

> Best,
> Timo
> --
> Timo Heister
> http://www.math.clemson.edu/~heister/
> _______________________________________________
> Aspect-devel mailing list
> Aspect-devel at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel

--
James Gallagher
jgallagher at opendap.org




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/aspect-devel/attachments/20180924/efa254c1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.geodynamics.org/pipermail/aspect-devel/attachments/20180924/efa254c1/attachment.sig>