Grid Research in iteration 4

People

• KonaAndrews
• ElizabethAuden
• GuyRixon (group leader)

Mission

Our primary task is to build a data warehouse that exploits OGSA-DAI grid-services; please see GridDataWarehouse for details. In doing so, we establish techniques for using grid services running outside AstroGrid, e.g. on the UK e-Science grid, without "contaminating" the AstroGrid architecture with OGSA dependencies.

Technical information

Please see GridDataWarehouse for detailed information.

Milestones

Set by AstroGrid TSP

Set by the group

Weekly reports

2003-10-24

Achievements

2MASS has been copied to disc at Cambridge. It will not be made available in Postgres until some more work has been done on the best way to load the warehouse.

The first draft of the registry entry for the warehouse has been posted to the wiki for review. This draft follows v0.8.3 of IVOA's VOResource standard.

We have a web service that can talk to OGSA-DAI. The service currently does a synchronous query and receives the results on the SOAP control-channel. Later versions of the web service will support asynchronous queries and delivery from OGSA-DAI by GridFTP.

OGSA-DAI is running at Cambridge, attached to a MySql^? database containing the FIRST object-catalogues. It will be re-plumbed to use the Postgres database later in the iteration.

Issues

Elizabeth is unavailable for much of the next few weeks due to attendence at conferences etc.

Tomcat, for the warehouse web-service, is problematic and makes the development very slow.

Maven is working, but is slow. The cumulative delay in using it for all test builds is significant.

There are problems in connecting to OGSA-DAI, especially from the web service. These may be configuration errors in OGSA-DAI or they may indicate that we don't properly understand how to write an OGSA-DAI client.

OGSA-DAI itself is easy to set up, but the message-level security feature, which we need in order to use GridFTP, is less easy. The OGSA-DAI documents are good, but the details for MLS are hidden in an appendix, have a critical typographic error and don't tell quite enough of the story. We will need to provide our own summary of how to set up OGSA-DAI for AstroGrid use.

2003-11-09

Achievements

The group's plans were completed and presented at the consortium meeting.

The design has been completed sufficiently for serious coding. At the time of writing, the most-complete version of the design has not been uploaded to the web; that should be fixed today.

A prototype of the warehouse has run a query successfully using the delegate, web service, OGSA-DAI back-end and a prototype of the database in MySQL.

The group declared its first CVS baseline: Itn04-2003-11-07.

The problems with connecting to OGSA-DAI have been fixed. They were eventually found to be due to a version clash between axis.jar as distributed by Apache (needed by the warehouse web-service) and axis.jar as distributed by Globus (needed by GT3 and OGSA-DAI). The current solution is to re-host the web-service part of the warehouse onto GLUE, leaving Axis only for the connection to OGSA-DAI. This technique is used in the baseline Itn04-2003-11-07. Using GLUE also solves the problem with Tomcat, since GLUE can run outside Tomcat during development.

More data have arrived at Cambridge for pre-loading into the warehouse. USNOB1 and 2MASS are both present. USNOB1 is being loading; the loading should finish today. The loading of 2MASS will start next week, probably on Tuesday.

Issues

Elizabeth is unavailable for much of the next few weeks due to attendence at conferences etc.

At the consortium meeting, it was agreed that the warehouse package should re-use classes from the data-centre package in iteration 4, as opposed to refactoring the packages into one during iteration 5. Therefore, we have to learn about the data-centre package in detail. There may be problems in running the merged code with GLUE.

The warehouse package is currently not being build by Maven every night. It's not on the build list.

Maven is working, but is slow. The cumulative delay in using it for all test builds is significant.

We are currently unable to make OGSA-DAI drive GridFTP for message delivery. All tests end in authorization failures. These failures are errors of some kind, in GT3, in GridFTP or in our configuration.

2003-01-01

Achievements

1XMM, USNOB1, 2MASS and FIRST are now loaded at both Cambridge (on cass123) and at Leicester (on hydra). INT-WFS is being prepared as a data-centre installation at Cambridge; the public part of this survey should be copied to Leicester by the end of the iteration.

OGSA-DAI has been repackaged as a WAR (see, in AstroGrid CVS, the module astrogrid/ogsa/gdw-war-builder) and installed on hydra. It has been connected to the PostgreSQL database engine for the GDW and has successfully executed queries.

A refactoring has been agreed that works round the problems with Axis and Globus. This refactoring is underway and should be complete by the end of this week.

All GLUE-related code has been removed from the GDW following the project decision not to use GLUE in any component.

Since the last report, warehouse is now included in the nightly build.

The AngloAustralianGridDemo2003 was shown at Supercomputing 2003 where it ran successfully and generated much interest.

Issues

The warehouse package is highly dependent on the datacenter package but the latter is still not stable; much refactoring is in progress. Recent changes in datacenter stop warehouse from building until warehouse catches up with the changes. We are running out of time to build and test our product.