WP-A4: Database Technology

The original definition of this work package is:

The Database Technology work-package essentially covers any aspect of AstroGrid where there is a need for persistent storage of structured information. This means that the areas of interest have evolved considerably as the design of the overall architecture has developed.

Inputs

• Science cases and use-cases from WP-A1.
• Information from related projects especially AVO, US-NVO, and EGSO and various interoperability discussions and workshops.
• Visits to other sites including Jodrell Bank and MSSL to get information on requirements in radio, solar, and STP areas.
• Information from other grid projects, especially from liaison with the DBTF.
• Information on the technology used in a number of existing astronomical data archives.

Main tasks

• Interoperability: the federation of existing data archives depends on the development of standards for the interchange of all types of data. Fortunately all the VO projects see this as a high priority, and there has been general agreement that these can best be developed by adopting the middleware protocols forming part of the Web Services paradigm: XML, SOAP, WSDL, and associated standards. Members of the WP-A4 team have played an active part in the workshops and discussions which have led to the development of the VOTable proposal for the encapsulation of tabular results in XML. Further work is needed to extend this to cover homogeneous datasets such as images, and to develop standards for queries. We have also had useful contacts with EPCC where a group is working on methods of encapsulating binary data in XML.

• Architecture: we have been involved in extensive discussions of the overall architecture. Although many details remain to be resolved, it is now generally agreed that the solution involves portals to provide the user interface, a resource discovery mechanism, and a data warehouse to support I/O-intensive or cpu-intensive processing such as data mining.

• Resource register: it has gradually become clear that some form of central register of the data resources available to AstroGrid needs to be set up. Ideally this would be a joint resource for the world's VO projects. It is not yet known whether some existing system such as UDDI is suitable; if not than an XML-based database management system might provide the required facilities. If the resources were all interfaced via WSDL, this register could be updated automatically.

• Astronomical Data Warehouse: the facilities provided by many existing data archives are limited to answering queries about points or small patches of sky. The most valuable scientific results will arise from joint use of two or more large datasets, for example the cross-identification of sources at two or more wavelengths, and from statistical investigations and data mining operations on large datasets. This brings up a number of problems which are areas of current study:

  • Indexing the sky: joining tables on imprecise celestial coordinates, the so-called fuzzy join problem is difficult (or at least extremely slow) when using standard business-oriented DBMS. Possible solutions include two-dimensional indexes such as the R-tree provided with (or available as add-ons to) DBMS, and mapping functions to one-dimensional indexes such as HTM and HEALPIX. We plan to evaluate these.

  • Query languages: SQL is poorly matched to the many types of astronomical queries, and is best considered as an advanced API to a database. We may need to develop an astronomicall-oriented query language, which can be translated into SQL or XQUERY.

  • Metadata preservation is vital in all astronomical data processing. Standards such as FITS and UCD have been developed to this end, but persuading DBMS to preserve metadata is not easy.

  • Using parallel hardware: it should be possible to spread a many type sof database and data mining operation over the nodes of a cluster (Beowulf or similar) but we need to get experience of these and develop suitable algorithms. We expect to procure hardware shortly to begin work.

  • Managing workspace: users of the data warehouse will want to copy standard datasets from remote sites and generate their own new datasets. The MySpace concept has been devised to handle the latter. A database may be needed to manage these resources.

  • Evaluation of DBMS: we plan to evaluate a number of free and commercial DBMS using sample astronomical datasets. Plans for this are now at a fairly advanced stage.

  Plans for the astronomical data warehouse involve working closely with WP-A3 over data storage and computational resources, and with WP-A2 over grid software which will provide its authentication and authorisation facilities, and software such as gridFTP to copy large datasets efficently.

Outputs

• Status reports on DBMS activities generally.

• Report on sky indexing methods.

• List of test datasets, list of DBMS to be evaluated, and evaluation criteria for the DBMS evaluation exercise.

• Assessment of DTI Database Task Force plans and initiatives.

• Reports on each of the DBMS evaluations.

• Proposals for an astronomical database query language.

• Draft requirements for the resource registry, and an outline of possible technical solutions to be circulated around all VO projects.

-- Clive Page - 19 Apr 2002