PhaseAReport

(5) Report on Grid Technology

(5.1) Introduction

(5.1.1) What is "grid technology"?

Grid technology can usefully be defined as technology that helps a user to exploit distributed computing without needing to know a priori the disposition of resources about the network, or the exact nature of those resources. By extension, technology that allows computing jobs to run remotely without continual human intervention also counts as Grid technology. This definition widens the scope of the enquiry beyond the triad of products - Globus Toolkit, Condor and Storage Resource Broker - identified by the core e-Science programme at AstroGrid's inception.

(5.1.2) AstroGrid's approach to the Grid

AstroGrid is about astronomy, not computer science. Our remit is to support working scientists and to show how a virtual observatory may be built. We use Grid technology as and when it suits our purposes and not to demonstrate the grid to other disciplines.

Our preferred approach is to define a system architecture from the science requirements and then to choose available technology to suit that architecture. We wish to build our own grid technology only where no existing products suit our purposes.

However, technology choices themselves tend to dictate architecture. We have therefore followed an iterative approach of defining architecture first according to what is natural for available technology, then altering this abstract architecture to bring it back in line with AstroGrid requirements, and finally reassessing the technological products against the revised architecture. Since there are many griddish products to choose from, our early priorities have been to detect and avoid the unsuitable ones.

As a result, most of our investigations into Grid technology to date have been thought experiments rather than executable prototypes. This is a cheaper way, in terms of staff effort, to eliminate the unhelpful products. Now that the architecture is known, we are starting to run more practical experiments to verify our positive choices of technology.

(5.2) The natural architectures

We looked initially at the architectures that are natural to grid products, on the off-chance that one of these might entirely meet AstroGrid's needs. There are three principal approachs, outlined below, and a Grid can combine them as necessary. However, particular products tend to promote just one of the three architectures.

(5.2.1) Compute-grid

A natural compute-grid values processing power ahead of network bandwidth and richness of applications. The resources in the grid are "unfurnished" processors which are leased by users for the duration of a job and to which all data and executable software must be shipped in; results must be shipped out at the end of the job.

Compute grids are good when jobs are limited by CPU power; when software is self-containing and easy to run on arbitrary computers; and when data sets are small and simple enough to be readily portable between nodes of the grid. These grids work poorly when data sets and software are large or complex.

(5.2.2) Data-grid

A natural data grid values the software resources at any given location above processing power and network bandwidth. The resources in the grid are storage locations from which any programme can easily read and write data. The grid is effectively a distributed file-system from which data are copied to a central location for processing. Data-grids tend to work in terms of files rather than database tables.

Data grids are good when sites have large collections of complex software that needs to run locally, but need data from elsewhere. These grids do poorly when the data-sets for a job are too big or complex to move easily, or when the site with the software has too little CPU power.

(5.2.3) Service-grid

A natural service grid values data locality and the conservation of network bandwidth above CPU power and locality of application software. It makes operations on data available as network services at the sites where the relevant data are held. In the purest form of a service grid, no data ever passes from service to service and only a bare minimum between services and the user's client-software.

Service grids are excellent when data sets are too large to move about in toto, or when access to the data sets depends on local infrastructure such as a particular DBMS. Service grids also help in the curation of data, since they keep both the data and the specialist software for those data at the curators' sites. These grids do less well when a job requires data and software from many services to be combined; e.g. when the job implies a joining of database tables from several data services. In that situation, it may be possible to speed up the job by first migrating the entire data-sets to a central location, merging them there, and then runing the job on the merged data-set. I.e., the pure service-grid approach may run less efficiently than the pure data-grid approach for some complex jobs.

(5.3) Astrogrid's technology needs

AstroGrid's architecture is described in section 3 of this report. AstroGrid is now seen as basically a service grid, but with a data-grid incorporated to move data between services and to store intermediate results. At present, there are no plans to include a compute grid. Although some substantial processors (e.g. Beowulf clusters) are available to AstroGrid, these will initially be the sites for pre-installed application-services, not resources to which code is uploaded by end-users. The crucial points of the architecture are as follows.

All access to bulk data-sets in archives is through data-selection services that isolate data-extracts small enough to move about the data-grid. In the rare cases where a job needs to access all of a major data-set, then the processing services for that job are constrained to run at the data-centre where the data-set lives. Hence, large-scale data-mining operations are likely to be at major data-centres, not at arbitrary places on the grid where there is spare CPU power or storage.

Data-selection services are needed for data stored in flat files (mainly pixel data) and also for structured data in DBMS.

Data-processing services are needed that wrap up existing, general software in astronomy, such as the Interactive Data Language, the Astronomical Image Processing System and the Starlink Software collection. These services should be available at many points on the Grid. Where possible, they should be available at the places where data extracts are made, such that the extracts do not have to cross the network.

Most of AstroGrid's work will be the analysis of data in archives that have already been processed in standard data-reduction "pipelines". AstroGrid's mission is, in part, to promote the use of these standard reductions in preference to reprocessing of the raw data by individual researchers. However, there will be cases where pre-reduced data are not archived (notably in solar and radio astronomy) or where re-reduction with improved techniques is desirable. Specialist data-processing services are needed that wrap up the data-reduction pipelines for the major surveys and facilities. These will naturally be co-located with the data they process.

Many jobs will require the combining of services. There must be some means of defining this combination as a workflow that states the sequence of operation and the flow of data. Some workflows for common jobs may be coded by professional developers and provided by AstroGrid; in future, many workflows may be invented and coded by end-users using simple tools, possible a graphical programming system. Ideally, each running workflow is itself a service, which is started interactively but then runs unattended. The user should be able to detach from a workflow and later reattach to it to check intermediate results and to steer it.

If data are to be exchanged between services, then they must be intelligible to the receiving services. We need standards for data and particularly for metadata that make the services interoperable.

Some data may come into the system from the end-users' desktops. We need a way of acquiring this data, decorating it with sufficient metadata to make the data intelligible to the system, and making the decorated data accessible to services.

Services may migrate around the grid as facilities change. The details of services may evolve with time. Therefore, client and workflow programmes should not hard-code the location and nature of services, but should refer to a registry of services. We need technology to set up that registry.

Results from services must remain in the data-grid for later use. A user typically wishes to feed the results back into another calculation; or to share the results with collaborators; or to publish the results directly from the Grid. We need a general system for storing and cataloguing results. "MySpace" is its working title.

Some data and services must have restricted access. This applies at least to "personal" results in MySpace and to running workflows, even in the extreme case where all archives have read-only access to public data. In fact, some of the data centres also have access policies to their data that must be respected. We need an access-control mechanism built into the service infrastructure that limits access according to the user's identity and assigned roles. By extension, we must have a way of authenticating that identity that works all across the grid.

These then are the technological needs. The codes in paratheses refer to the technology-choice matrix below.

• A basic invocation system for services. (Inv.)
• A way or representing state in service between invocations. (State)
• The ability to pass metadata to and from services (Pass-metad.)
• The ability to describe details of services in machine-readable form.
• A uniform interface to data archives (Data-sel.)
• A file-transfer capability (FT)
• A DB-transfer capability (DBT)
• A way of caching and publishing data in the grid. (MySpace)
• A facility for registering and discovering services (Reg.)
• Interoperable metadata for data extracts (Interop.)
• Access control for data and resources, allowing delegation of acccess rights by users (IAA)
• A workflow engine (Workflow)
• The ability to detach and re-attach clients to workflows (Detach)

(5.4) Technology choices

(5.4.1) Globus Toolkit (GT) (1)

The Globus Resource Allocation Manager (GRAM) is the part of GT that enables submission of compute jobs on remote machines: it is the core of Globus. GRAM is intended for pure compute grids. It provides reasonable support for long-running jobs on major processors such as supercomputers, but we find it unwieldy for finer-grained work on smaller installations. GRAM has built-in support for grid authentication of users but inadequate support for authorization. We do not intended to use GRAM; we prefer web services for remote computation.

The Metacomputing Data Service (MDS) provides a registry of services for use in a GT-based grid. MDS stores for each processing resource metadata listing capabilities, current load, etc. In principle, MDS can store any programmer-defined schema for any resource, and could therefore be the basic for AstroGrid's registry of services. In practice, MDS has not been tested as a repository for very rich schema. Users of GT are now predicting that MDS will be completely replaced in subsequent releases of GT. This means that the current MDS is not useful to AstroGrid, and the replacement will not be available or stable soon enough for AstroGrid's timescales. Therefore, we do not intend to use MDS.

GridFTP is an extension of the commonly-used File Transfer Protocol to support grid security. It also provides some high-performance features that are standard but optional and commonly left out of normal FTP implementations. GridFTP is Globus' data-grid support for its compute-grids. We find GridFTP very suitable for our data-grid. It is efficient for large data-files; it has streaming semantics (c.f. HTTP); it allows reference to data on the Grid by URL. We intend to use GridFTP for moving data-extracts between services.

(5.4.2) Other compute-grid kits

Condor (2) is a well-known, mature system for running a compute grid inside one trust domain: typically, the trust domain is one university department. Condor-G is a recent version that interoperates with Globus. With Condor-G, a Globus grid can treat an entire Condor network as one computing resource (Condor handles the local allocation to queues and processors for jobs accepted from Globus). Alternately, a Condor network can "sub-contract" its excess load to a Globus compute-grid.

Grid Engine (3), from SUN Microsystems, is a proprietary product similar to Condor in scope. It may have better integration with SUN hardware (e.g. their Throughput Engine series of processor farms) and with Solaris. Grid Engine does not interoperate with Globus at the moment.

Neither Condor nor Grid Engine solve any AstroGrid problems. It might be useful for a service provider to run a private grid behind a service facade, but this is not really AstroGrid's responsibility. Thus, we have no plans to use Condor or Grid Engine for our own grids.

(5.4.3) SRB (4)

Storage Request Broker (SRB) from San Diego Supercomputing Centre implements a pure data-grid as a distributed file-system. Application software calls a low-level API in SRB to access files and a network of broker daemons makes the file contents available. SRB has many utilities for managing the data-grid by transparent replication and migration of files.

We initially investigated using SRB to build a data-grid that was exposed to applications. When AstroGrid was recast as a service grid, we dropped this idea. In any case, SRB as a data-grid cannot use data in a DBMS. We may now reconsider SRB for the data-grid hidden behind MySpace.

(5.4.4) Web services

Web services come in many forms. The kind that we find most apt are those that work by exchange of messages in virtual envelopes defined by the Simple Object Access Protocol (SOAP) and pass those messages using the Hypertext Transport Protocol (HTTP). This is the chosen method for invoking our services. The SOAP envelopes allow arbitrary XML fragments to be transmitted, so SOAP is good for moving our metadata between clients and services.

It is now becoming standard to describe web services using the Web Service Definition Language (WSDL). We intend to use this to describe some aspects of our services; it will be part of our service-discovery mechanism and our registry. However, WSDL will not describe all aspects of the services that AstroGrid software needs to understand. We have to provide extra metadata.

In e-Commerce, services may be registered and discovered using the Universal Description and Discovery (UDDI) system. We have considered using UDDI for our service registry, but we find it too tightly bound to North American business conventions. It is not at all clear that we can express our service metadata in UDDI; we may check this again when the ontological experiments have given us a clearer idea of what needs to be recorded. For now, we do not intend to use UDDI.

The Open Grid Services Architecture (OGSA) (5) extends normal web-services with facilities to express statefulness of services. In particular, OGSA covers the case where the state of a given service-instance is part of a private transaction between a user and the service provider (c.f. the case where all users see the same state of the service and the same changes of state). This is important for a small but significant part of our system. Most services in AstroGrid don't need to remember and express per-user state. However, the MySpace services and the workflow service do need this feature. OGSA defines facilities for producing private instances of services that remember a user's state, and for managing that state (e.g. resources allocated by a service instance can be automatically released and recycled if the user loses contact with the service instance.) OGSA also provides a much-needed framework for using Grid security with web services.

Web-services run in a hosting environment at the server. We have considered four such environments and tested two of them, as described below. We plan to use Axis (6), from the Apache Foundation, because early support for OGSA is based on Axis; the SOAP-Lite (7) module from the Comprehensive Perl Archive Network (CPAN) where a service needs to be coded in Perl; and probably WebSphere (8) (from IBM) when we have a free choice of hosting environment. Microsoft's .NET system is a web-service hosting-environment for Microsoft Windows; it is not currently portable to other platforms. Since few service sites run Windows, we have not investigated .NET in detail.

(5.4.5) Data-selection services

The OGSA-DAI project (9) (OGSA Data Access and Integration) has defined standard interfaces to XML and relational databases. Implementations of these interfaces wrap existing database-engines and databases in grid services following OGSA conventions. The OGSA-DAI standard defines a GridDataService port-type for operations on the database engine itself and a GridDataTransport port-type controlling access to and communication of the results of database queries. The current specification states an intention to include later a standard port-type for translating data fed to or read from the data service.

AstroGrid is an "early adopter" of this technology. We have had considerable input into the early stages of the OGSA-DAI project. The port-types in the current specification will be very useful to AstroGrid as a standard form for our own services. The utility of reference implementations of the specifications is less clear. Most existing databases in astronomy abstract the details of the database schema by transforming queries and results. An implementation that exposes the schema directly is not ideal and AstroGrid may need to extend the service to implement the transformations or to make the standard grid data-service subordinate to another service that does those transformations. The proposed extension of the OGSA-DAI specification to cover transformations would be very valuable to AstroGrid.

Spitfire (10), from EU DataGrid, is a grid-enabled database-access product. It is already in operation for GridPP, and a version with a SOAP/HTTP interface is due out soon. We may have a use for this product, and will investigate the SOAP version when it arrives.

(5.4.6) Access Control

The Grid Security Infrastructure (11) (GSI) is a system for authenticating identity of users with a Public Key Infrastructure (PKI); i.e. it uses public-key cryptography instead of passwords. GSI is the central and common part of "grid security"; it largely defines what makes a Grid. GSI differs from other PKIs (e.g. in e-Commerce) in that it allows "delegation by impersonation". A service can use this feature to call restricted, subsidiary services in the name of the user. The delegation feature is essential to grids, but is contentious: many commentators consider GSI to be too insecure for e-Commerce and the IETF rules that certificates of identity suitable for use with GSI are technically invalid (this affects all certificates issued by the e-Science certificate authority). For now we are happy to use GSI; it is standard in e-Science, it works well enough for our low-security system, and we need the delegation feature. We expect GSI to change radically or to be discarded as grid technology evolves to support e-Commerce. For that reason, we intend that no other parts of our grid should depend on the fine details of the authentication; we must be able to switch to a new standard when it emerges.

The Community Authorization Service (12) (CAS) is a database of authorization information. It is intended to replace and expand the inadequate authorization system in GRAM. CAS allows service providers to "outsource" the sociological part of the database - the record of groupings in the community of users - to central authority, leaving the service sites to manage only the allocation of privileges to users. By extension, if users are allowed a limited ability to change the authorization database, CAS can support the ability to share data in the grid by delegating access rights. We like the idea of CAS; it supports the grid metaphor nicely; we like the idea of reducing management workload by reducing duplication; we think that the CAS principle is the way to get the delegation ability that we need. However, the current (alpha) implementation of CAS is unserviceable, and the current design depends too much on GSI. We are looking at a reimplementation of the ideas: see the section on CoPS below.

The Virtual Organization Management System (13) (VOMS), from EU DataGrid, is similar to CoPS but is a finished, working product. We may be able to use VOMS in our production grid.

(5.4.7) Metadata

We express as much metadata as possible in XML. Some existing astronomical systems such as the Flexible Image Transport System (14) (FITS) will also be retained for compatibility with existing programmes.

Extracts of pixel data will be transported in FITS files. The file-header system built in to FITS will express some basic metadata such as the size and shape of the rasters and the world coordinate systems. It is not yet decided whether other metadata for the extracts will be in the FITS header or in accompanying XML fragments.

Data-extracts that are tabular will be expressed in VOTable (15) format. VOTable exploits Unified Column Descriptors (16) (UCDs) to make its tables machine readable, so UCDs are necessarily a part of AstroGrid. We may come to use UCDs in building our registry of services.

UCDs encode semantics of data, but we also need to record the relationships between data: e.g. between the columns of a table. This is the field of ontology; it is a research area, both for AstroGrid and for computer science generally. AstroGrid's experiments to date (17) have used the Darpa Agent Mark-up Language (DAML) and the Ontology Inference Layer (OIL).

One precursor of a service grid, "VizieR" (18), uses a system called Générateur de Liens Uniformes (19) (GLU) as an abstracting layer. VizieR is extremely successful and there is some motivation to use GLU, or its successor, in AstroGrid. A GLU dictionary could be part of our service registry. Further investigation is needed.

(5.5) Technology-choice matrix

Key: C = confirmed choice; P = probable choice, pending validation; ? = possible choice; R = technology considered and rejected.

(5.6) Experiments

(5.6.1) Starlink's data-grid

The Starlink project contributed to AstroGrid the results of their experiment "StarGrid". This system includes a pure data-grid, built to test the data-grid technology in the Globus Toolkit.

Starlink's non-networked application software has an abstraction layer for access to files of pixel data. In StarGrid, this layer was altered to call "Globus Access to Secondary Storage" which itself calls GridFTP to access remote files at known locations on the Grid.

StarGrid is a considerable technical finesse, and its data-grid aspects were cheap: they needed little change to existing code. The existence of StarGrid validates the use of GridFTP to build a data-grid.

However, StarGrid does not address the production of data-extracts at the data archives, so does not scale to large data sets. The experiment was not extended to indexing the remote data; the calling software has to know a priori the location of each file. Hence, StarGrid is not a complete solution for AstroGrid's data grid.

(5.6.2) Web portal (21)

In 2001, a small data-grid was assembled from WWW technologies. It had a portal on the web by which a user could browse metadata for data-products on the Grid, and could then make informed selection about which data to download. We provided a custom user-interface on client computer in order to get the best possible presentation of the metadata and an acceptable visualization of the data. This allowed us to test ideas about metadata and presentation of results, and to try out the basic concept of a service-grid. (At the time, most grids were seen as compute-grids or data-grids.)

The prototype demonstrated:

• The basic strengths of the service-grid concept.
• The need for uniform interfaces to services.
• The validity of separate channels for data and metadata.
• The need for additional contextual metadata over and above those normally provided in astronomy.
• The need for processing power at various points in the grid (i.e. the system would not have worked had it been a pure data-grid).

(5.6.3) CoPS (22)

The Community Privilege Service (CoPS) is an authorization service: data services can call CoPS to ask if an identified individual has the necessary roles and privileges to carry out requested work.

CoPS was written by AstroGrid. It was inspired by the Community Authorization Service (CAS) from the Globus project and uses many of the same ideas. We could have used the CAS code, but it proved easier for the experiment to build our own sub-system. When CoPS has proven the ideas, it may be best to feed the results back into the CAS product.

The current CoPS experiment is working with authorization structures to show how services can share knowledge about the human and economic structure of the astronomical community. CoPS will allow end-users to modify these structures in a controlled way to share their data with collaborators and the community. To date, the CoPS prototype can respond to requests from users working from a fixed database of authorities, but the code to update the database has not been finished.

(5.6.4) Web services (23)

We are learning to build web services. There is no doubt that this technology works, since it is already widely used, but it is a very flexible technology and the best way of using it is not yet clear. We are building prototypes to determine best practices for AstroGrid.

To date, we have investigated the techniques and tooling for web services in Java, using the Axis toolkit, and in Perl, using the SOAP::Lite module. Simple services have been run sucessfully at Cambridge and accessed sucessfully from other sites. We picked Axis not because it is good - it is unfinished and badly documented - but because initial support for OGSA is based on Axis.

Sample grid-services have also been produced using the preview issue of OGSA infrastructure from the Globus project. A simple "echo" service has been run at several AstroGrid sites. We have not yet tested the higher functions of OGSA. The OGSA specification is currently being revised and we are waiting for it to stabalise before investing much time in tests.

The third stage of this experiment is to publish protoype data-selection and data-transformation services. These will appear initially at Cambridge, Leicester and Edinburgh.

(5.6.5) Trial grid using the Globus Toolkit (GT2) (24)

We are building a trial grid using components of the Globus toolkit: GRAM and GridFTP. This grid will allow members of the project to obtain and try out their identity certificates, and let us begin building our data-grid using the GT2 version of GridFTP.

So far, the grid covers Cambridge, Leicester, MSSL, Jodrell bank and RAL. It will shortly expand to link all AstroGrid-project sites.

Findings so far:

• GRAM is not a comfortable or efficient technology; it compares badly with web-services for usability. Its virtue is its intrinsic use of grid security.
• GridFTP works well. However, there are problems in making the existing GridFTP server work through a firewall.
• Interactions with the UK e-Science certificate authority have been successful so far.

(5.7) References

1. Globus Toolkit: http://www.globus.org/toolkit/default.asp

2. Condor: http://www.cs.wisc.edu/condor/

3. Grid Engine: http://wwws.sun.com/software/gridware/

4. Storage Resource Broker: http://www.npaci.edu/DICE/SRB/

5. Open Grid Services Architecture: http://www.globus.org/ogsa/

6. Apache Axis: http://xml.apache.org/axis/index.html

7. SOAP::Lite: http://theoryx5.uwinnipeg.ca/CPAN/data/SOAP-Lite/SOAP/Lite.html

8. WebSphere: http://www-3.ibm.com/software/webservers/appserv/

9. Grid Database Service Specification: http://www.cs.man.ac.uk/grid-db/papers/DAIS:StatementSpec.pdf

10. Spitfire: http://hep-proj-spitfire.web.cern.ch/hep-proj-spitfire/doc/index.html

11. Grid Security Infrastructure: http://www.globus.org/security/

12. Community Authorization Service: http://www.globus.org/security/CAS/

13. Virtual Organization Management Service:

14. Flexible Image Transport Service: http://fits.gsfc.nasa.gov/

15. VOTable: http://cdsweb.u-strasbg.fr/doc/VOTable/

16. Unified Column Descriptors: http://cdsweb.u-strasbg.fr/doc/UCD.htx

17. AstroGrid ontology experiments: http://wiki.astrogrid.org/bin/view/Astrogrid/OntologyDemo

18. VizieR: http://archive.ast.cam.ac.uk/vizier/

19. Générateur de Liens Uniformes: http://simbad.u-strasbg.fr/glu/glu.htx

21. Web-portal experiment: http://www.ast.cam.ac.uk/astrogrid/publications/adass-gtr-20010930/

22. Community Privilege Service experiment: http://wiki.astrogrid.org/bin/view/Astrogrid/CASDemo

23. Web-service experiments: http://wiki.astrogrid.org/bin/view/Astrogrid/WebServiceExperiments

24. Demonstration grid: [http://wiki.astrogrid.org/bin/view/Astrogrid/GridDemo

-- Perpetrated by GuyRixon on 16 Sep 2002, using ideas from everybody in the project. Thanks to KonaAndrews for good advice and comments. Last updated by GuyRixon on 13 Dec 2002.