WP-A5.5: STP pilot - Time-series data

Latest revision to pilot at WPA5Focus20020527

(WPM: Perry - Staff: Perry, 0.50 FTE)

WWW Site: http://www.cluster.rl.ac.uk/spacegrid/wp_5_5.htm

This test federation will investigate the Grid-enabled federation of heterogeneous time-series data. This is of particular relevance to Solar Terrestrial Physics (STP) data sets due to the large number of in situ, multi-point and remote sensing measurements made across a wide range of scales in both time and space. STP data sets are relatively small compared to the other AstroGrid domains. The main issues for the Grid infrastructure to address come from the complexity of the analysis and in particular the need to locate, search, extract, manipulate and combine multiple data sets. It is also important to consider the international perspective since many of the key datasets that will be required by UK STP scientists originate from non-UK instruments and facilities.

Example science case:

A scientist wishes to study the propagation and effect of a coronal mass ejection. This requires use of: (i) the coronagraph on SOHO; (ii) upstream solar wind measurements from ACE; (iii) Cluster plasma and field measurements near the magnetopause; (iv) plasma composition measurements in the mid altitude cusp; (v) ring current enhancements, in situ, remote sampling and ground-based geomagnetic indices; (vi) position and timing information.

These data sets range from simple scalar time series data, to sequences of images and higher dimensionality arrays. They currently have different locations, query specifications and are returned in different formats. The data may need to be transformed into a consistent co-ordinate frame or combined to produce ancillary products. A uniform, and flexible, metadata specification is therefore crucial to ensure that manipulation of data from different archives can be done in a consistent and correct way.

Objectives:

• To identify and evaluate implementation options for the efficient query, manipulation and delivery of heterogeneous time series data.
• To implement a test bed system to assess the problems associated with the integration of legacy archives.
• To provide a simple web-based interface that can be used to demonstrate the end-to-end functionality of the test bed.

Inputs:

• Candidate archives - UKCDC and WDC.
• Existing middleware software/libraries and standards - STPDF, XDF, XSIL, CDFML etc.
• Inputs from technology work packages, particularly WP-A2.

Outputs:

• Time series federation test bed.
• Feedback on prototype system from test users.
• Inputs of STP requirements to WP-A1.
• Inputs of implementation issues to WP-A2, WP-A3, WP-A4 and WP-A9.
• Inputs to the development of the Phase B plan.

Tasks:

• 5.5.0 Design and Requirements
  • Define the requirements of the test bed federation. This will include some test Use Cases that will be used as part of the test bed evaluation (5.5.7). It will also include inputs from other Grid activities (e.g. SpaceGRID).
  • Develop a top-level architecture for the test bed.

• 5.5.1 Develop metadata translation layer
  • Assess various XML based schemas for their suitability in handling STP time series catalogues and data. Current specifications to be assessed are XDF, XSIL and CDFML.
  • Identify particular domain specific descriptors and supply to WP-A2 as part of the construction of AstroGrid metadata specification.
  • For each of the archives in the test bed develop a metadata translation layer to convert the archive specific metadata to the XML based standard.

• 5.5.2 Develop a data export layer
  • For each archive in the test bed develop a data export layer to convert between the local archive format and the internal format to be used within the test federation. This will build on Dave Giaretta's STPDF software library, which already includes conversion utilities for a number of commonly used STP data formats.

• 5.5.3 Implement a simple query layer
  • Assess query and join requirements for distributed time series data. Since STP does not generally produce science catalogues, it is important to include the ability to query both the catalogues and the data. This requires intelligent use of the metadata.
  • Identify sub-set to be implemented within the test bed.
  • Implement test bed query layer.

• 5.5.4 Implement an authorisation layer
  • Assess any particular authentication and authorisation issues arising from interaction with the test bed archive and provide input to WP-A2. (N.B. The original plan was to implement an authorisation layer to map between the test bed authentication and the local authorisation system to allow access within the test bed to proprietary data. This has been dropped and the existing archives will only supply publicly available data. Within each of the archives test bed access will be mapped to the public access account.)

• 5.5.5 Integrate system with Grid middleware
  • Interface the archive specific interface layers with the Grid infrastructure. Initially this will use the networking capabilities of the STPDF system. If time permits then an option based around Globus/GridFTP will also be tested to allow for some comparative assessment of the different systems.

• 5.5.6 Develop a simple web-based UI
  • Assess the minimum functionality that is required to adequately demonstrate the end-to-end operation of the test bed federations.
  • Implement a simple web-based interface that provides this functionality. This is likely to be based on one of the archives existing web-based interfaces since these already provide basic quicklook functionality and the main updates will be to handle the extra XML metadata and enhanced query specification.

• 5.5.7 Evaluate test-bed implementation
  • Test system, including evaluation against the test scenarios identified in 5.5.0.
  • Gather inputs from developers and users.
  • Produce a technical note on the lessons learnt.
  • Provide inputs to the Phase B plan.

-- BobMann - 26 Feb 2002