AG Tech Workshop Jan 06: Work Group Three A

The Group

• Markus Dolensky (ESO)
• Pekka Järveläinen (CSC)
• Johan Lindroos (CSC)
• Pedro Osuna (ESA)
• Francesco Pierfederici (ESO)
• Jesus Salgado (ESA)

How to use AG Infrastructure to implement VOTech Multi-waveband Image Tool and SED Builder?

Well, let's start with a ...

Typical scenario

Requirements on MySpace/ACR as far as moving files efficiently and co-locating code go are:

1. Users (astronomers) identify datasets of interest on the VO ( e.g. by querying DataScope).
2. Users move the datasets of interests onto their MySpace/VOSpace directories.
3. Users execute a scientific application (i.e. our tool) on (a subset of) the data in their MySpace/VOSpace directories.
4. Users choose to store the results of the computation (images and/or data files) onto MySpace/VOSpace.

Non trivial implications

1. The amount of data the tools operate on is large ( i.e. hundreds of megabytes or gigabytes).
2. The data is distributed over >1 FileStores (and the problem is not embarrassingly parallel).
3. The code is not colocated with the data.

The reason why the implications become non trivial is that large data transfers are impassively inefficient.

Implications of co-locating data and processing

1. It should be possible to modify both MySpace and CEA so that, if data and code are co-located, efficient file access is implemented.
2. It should be possible to specify a given FileStore instance when storing data on MySpace (implemented in the MySpace implementation, not yet in client applications).
3. It should be possible to move data from one FileStore to another using efficient data transfer protocols (e.g. GridFTP) instead of simple HTTP GET.

Limitations of current implementation Jan 06

1. MySpace supports only HTTP GET as file transfer mechanism.
2. no VOStore interface to MySpace which allows modular code to also access SRB or NGAS file servers
3. CEA does not currently access files stored on a local FileStore directly using the underlying file system.
4. ACR does not seem to support specifying a FileStore when uploading files.
5. No SSA support in ACR
6. Sharing input/output in a group not supported.

Misc

• For each application there is a design choice to make concerning packaging: Either fully integrate it with the workbench - or - selectively bundle the application with specific components only such as ACR API.

Reference

• Computer Networks for the AVO, Davenhall et al.

--Francesco & Markus