EditAttachPrintable
r12 - 29 Jun 2005 - 09:13:00 - KeithNoddleYou are here: TWiki >  Astrogrid Web  >  DaveMorris > EurovoDataCenterNotes

Install notes for EuroVo Data Center Infrastructure workshop

Instructions on how to install and configure an AstroGrid Data Set Access (DSA) service to connect to a local database.

Note, this is based on the current build of the AstroGrid DSA webapp as-is, and includes some known bugs. The notes are intended to show not only how to install a Data Set Access service, but what error messages you may encounter during the process.

-- DaveMorris - 20 Jun 2005

Installing the demo system

The EuroVo? workshop is based on a pre-packaged demo system, containing a set of AstroGrid services in a Tomcat server configured to use the loopback address http://localhost:8080/

Tomcat home

Due to the way that the AstroGrid EuroVo? demo system is configured, the CATALINA_HOME environment variable should not be set on your local machine.

The system was configured this way so that the AstroGrid EuroVo? demo system does not conflict with any existing Tomcat installed on your system.

However, the default Tomcat start and stop shell scripts will try to use the CATALINA_HOME environment variable if it is set.

If you have an existing installation of Tomcat on your machine, you will need to remove or unset the CATALINA_HOME environment variable when trying to run the workshop versions of Tomcat.

On linux 

unset CATALINA_HOME

On windows XP, right click on MyComputer?, select Properties, select Advanced, and click the Environment Variables at the bottom of the dialog. Look for CATALINA_HOME in the list of System variables, and either delete it or rename it.

EuroVo? home

The following workshop instructions will use the term EUROVO_HOME to refer to the home directory for the demo Tomcat server

Whenever EUROVO_HOME appears in the instructions, you will need to substitute the home directory for the demo Tomcat server.

For example, if you unpacked the zip file into C:\EuroVoDemo\ on your laptop, then the Tomcat home directory will be C:\EuroVoDemo\tomcat\.

If the instructions refer to EUROVO_HOME/webapps/eurovo-twomass/, you will need to look in C:\EuroVoDemo\tomcat\webapps\eurovo-twomass\ on your laptop.

Running the Tomcat server

In order to make the demo system portable, the components are configured to use relative file paths to refer to file locations.

This makes the demo system portable, so you can unpack the zip file anywhere you want on your local file system.

However, it does mean that you need to start Tomcat from within the EUROVO_HOME/bin directory. 

    EUROVO_HOME
        |
        +-- bin
        |    |
        |    +- ....
             |
             +- catalina.sh
             |
             +- catalina.bat
             |
             +- startup.sh
             |
             +- startup.bat
             |
             +- shutdown.sh
             |
             \- shutdown.bat

You must cd to the EUROVO_HOME/bin directory before you run the startup script.

If you invoke the startup script from somewhere else, then the current working directory will be different, and some of the components will not be able to read their configuration and data files.

Starting Tomcat from the EUROVO_HOME/bin directory works. 

    [user@host home]$ cd EUROVO_HOME/bin
    [user@host bin]$  ./catalina.sh run
    ....
    20-Jun-2005 19:44:05 org.apache.catalina.startup.Catalina start
    INFO: Server startup in 20057 ms

Starting Tomcat from another directory appears to work, but some of the components will not be configured correctly.

Installing the workshop DSA

The following instructions assume that you have installed the AstroGrid EuroVo? demo system.

Download the DSA war file

Download the latest version of the DSA war file, astrogrid-pal-skycatserver-1.0.1-b006pl.war from the AstroGrid maven repository http://www.astrogrid.org/maven/org.astrogrid/wars/

Install the war file

Copy the war file into your Tomcat webapps directory EUROVO_HOME/webapps/. 
    EUROVO_HOME
        |
        +-- bin
        |
        +- ....
        |
        \-- webapps
               |
               +- ....
               |
               \-- astrogrid-pal-skycatserver-1.0.1-b006pl.war

If Tomcat is configured to auto-detect new webapps (on by default), then it will automatically unpack the war file and install the webapp. If not, you may need to restart Tomcat to get it to recognise the new webapp. 

Tomcat log :
    20-Jun-2005 14:26:32 org.apache.catalina.core.StandardHostDeployer install
    INFO: Installing web application at context path /astrogrid-pal-skycatserver-1.0.1-b006pl
    from URL file:EUROVO_HOME/webapps/astrogrid-pal-skycatserver-1.0.1-b006pl

You can check this by looking in the Tomcat webapps directory. 

    EUROVO_HOME
        |
        +-- bin
        |
        +- ....
        |
        \-- webapps
               |
               +- ....
               |
               +-- astrogrid-pal-skycatserver-1.0.1-b006pl.war
               |
               \-- astrogrid-pal-skycatserver-1.0.1-b006pl
                       |
                       +- ....
                       |
                       +-- META-INF
                       |
                       \-- WEB-INF

This should have installed the basic DSA webapp as astrogrid-pal-skycatserver-1.0.1-b006pl.

You can check this works by pointing your browser at http://localhost:8080/astrogrid-pal-skycatserver-1.0.1-b006pl/.

Creating a new context

In many situations, a Data Center will need to publish several different DSA services, typically one for each dataset.

The LEDAS Data Center at Leicester is a typical examples of this. The complete LEDAS DataCenter? has over 400 data sets, each of which will need to be published as a separate DSA service.

It is possible to deploy a separate copy of the WAR file for each dataset. However, this would mean 400 copies of the webapp files, 107 items, totalling 10.7 MB for each webapp.

A better way to handle this is to use Tomcat context files to add multiple webapps based on the same document root.

Tomcat allows you to deploy multiple virtual webapps sharing the set of source files.

Create an XML file called eurovo-twomass-context.xml containing the following element : 

    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >
    </Context>

Add this new context file to the Tomcat context files in EUROVO_HOME/conf/Catalina/localhost. 

    EUROVO_HOME
        |
        +-- bin
        |
        +- ....
        |
        \-- conf
             |
             +- ....
             |
             \-- Catalina
                    |
                    \-- localhost
                           |
                           +-- admin.xml
                           |
                           +-- ....
                           |
                           \-- eurovo-twomass-context.xml

If Tomcat is configured to auto-detect changes to context files (on by default), then it should automatically detect the new context file and deploy a new webapp. If not, you may need to restart Tomcat to get it to recognise the new context file. 

Tomcat log :
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer remove
    INFO: Removing web application at context path /eurovo-twomass
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer install
    INFO: Processing Context configuration file URL file:EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml

Once Tomcat has processed the new context file, we should now have two instances of the DSA webapp deployed in tomcat.

The second webapp uses the same set of source files, but creates a completely separate webapp using the path defined in the context file.

If you look in the Tomcat work directory, you should see that it has created a separate working directory for the new webapp. 

    EUROVO_HOME
        |
        +-- bin
        |
        +- ....
        |
        \-- work
             |
             +- ....
             |
             \-- Catalina
                    |
                    \-- localhost
                           |
                           +-- ....
                           |
                           +-- astrogrid-pal-skycatserver-1.0.1-b006pl
                           |        |
                           |        \-- ....
                           |
                           \-- eurovo-twomass
                                    |
                                    \-- ....

Configuring new context

We can now use the new context file to configure the eurovo-twomass webapp by adding elements to the context file.

Before you edit the context file, have a look at the home page for the eurovo-twomass webapp. The first section of text in the welcome section should contain something like this:

  • Welcome to the DSA interface to (Unnamed) AstroGrid Datacenter.

Next, add the following to the eurovo-twomass-context.xml context file: 

    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

        ....

        <!--+
            | DataCenter name and description.
            +-->
        <Environment name="datacenter.name" override="false" type="java.lang.String"
           value="EuroVo workshop datacenter"/>
        <Environment name="datacenter.description" override="false" type="java.lang.String"
           value="EuroVo workshop datacenter, publishing a local copy of the TwoMass dataset"/>

        ....

    </Context>

If Tomcat is configured to auto-detect changes to context files (on by default), then it should automatically detect the change to the context file and re-deploy the eurovo-twomass webapp. If not, you may need to restart Tomcat to get it to recognise the changes. 

Tomcat log :
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer remove
    INFO: Removing web application at context path /eurovo-twomass
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer install
    INFO: Processing Context configuration file URL file:EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml

Once Tomcat has reloaded the webapp, have another look at the home page for the eurovo-twomass webapp. The first section of text in the welcome section should now contain something like this:

  • Welcome to the DSA interface to EuroVo workshop datacenter.

Configuring the admin account

The eurovo-twomass webapp uses a combination of settings in the context file and the default properties shipped in the WAR file to configure the WebService.

You can check where the webapp is getting the configuration settings from by checking the webapp fingerprint page. However, before you can access the fingerprint page, you will need to configure the DSA admin account.

The DSA webapp uses a Tomcat role and user account to control access to the DSA administration pages.

Edit the tomcat-users.xml file in the EUROVO_HOME/conf directory, and add the following role and user: 

    [file://EUROVO_HOME/conf/tomcat-users.xml]
    <?xml version='1.0' encoding='utf-8'?>
        <tomcat-users>
            ....
            <role rolename="paladmin"/>
            ....
            <user username="eurovo" password="eurovo" roles="paladmin"/>
            ....
        </tomcat-users>

Tomcat will not automatically load the new user and role information, so you will need to stop and start Tomcat to get it to recognise the new user account.

Once you have restarted Tomcat, you should be able to login and access the fingerprint page.

The fingerprint page shows which configuration properties are being read from the JNDI values configured in the context file, and which properties are being read from the default.properties file shipped in the WAR file.

At this stage, the fingerprint page should show that the datacenter.name and datacenter.description are being read from the JNDI properties configured in the context file. 

    [http://localhost:8080/eurovo-twomass/admin/fingerprint.jsp]
    JNDI Names:
        ....
        datacenter.name: java.lang.String = EuroVo workshop datacenter
        datacenter.description: java.lang.String = EuroVo workshop datacenter, publishing a local copy of the TwoMass dataset

Configuring JDBC connection

The AstroGrid DSA uses a standard JDBC connector to connect to the underlying database server.

The database server in the EuroVo? workshop is a PostgreSQL database containing a copy of the 2MASS dataset.

To enable the DSA service to connect to the database, we need to install and configure the JDBC driver classes for the target database server.

To start with, add the following to the eurovo-twomass-context.xml context file: 

    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

        ....

        <!--+
            | Use the JDBC database plugin.
            +-->
        <Environment name="datacenter.querier.plugin" override="false" type="java.lang.String"
            value="org.astrogrid.tableserver.jdbc.JdbcPlugin"/>
        <!--+
            | Select the PostgreSQL specific plugin
            +-->
        <Environment name="datacenter.querier.plugin.sql.translator" override="false" type="java.lang.String"
            value="org.astrogrid.tableserver.jdbc.postgres.PostgresSqlMaker"/>

        <!--+
            | PostgreSQL JDBC connection details.
            +-->
        <Environment name="datacenter.plugin.jdbc.drivers" override="false" type="java.lang.String"
            value="org.postgresql.Driver"/>
        <Environment name="datacenter.plugin.jdbc.url" override="false" type="java.lang.String"
            value="jdbc:postgresql://HOSTNAME:5432/wsdb"/>
        <Environment name="datacenter.plugin.jdbc.user" override="false" type="java.lang.String"
            value="USERNAME"/>
        <Environment name="datacenter.plugin.jdbc.password" override="false" type="java.lang.String"
            value="PASSWORD"/>

        ....

    </Context>

Need to configure the HOSTNAME, USERNAME and PASSWORD to match workshop server configuration

If Tomcat is configured to auto-detect changes to context files (on by default), then it should automatically detect the change to the context file and re-deploy the eurovo-twomass webapp. If not, you may need to restart Tomcat to get it to recognise the changes.

Once the webapp has been reloaded, check the fingerprint page to see if the new properties have been recognised. 

    [http://localhost:8080/eurovo-twomass/admin/fingerprint.jsp]
    JNDI Names:
        ....
        datacenter.querier.plugin.sql.translator: java.lang.String = org.astrogrid.tableserver.jdbc.postgres.PostgresSqlMaker
        datacenter.description: java.lang.String = EuroVo workshop datacenter, publishing a local copy of the TwoMass dataset
        datacenter.plugin.jdbc.url: java.lang.String = jdbc:postgresql://HOSTNAME:5432/wsdb
        datacenter.plugin.jdbc.user: java.lang.String = USERNAME
        datacenter.plugin.jdbc.password: java.lang.String = PASSWORD
        datacenter.plugin.jdbc.drivers: java.lang.String = org.postgresql.Driver
        ....

Once these properties are configured, we can test the database connection by trying to generate the table metadata from the database.

Try accessing the MetaDocGenerator? page at this location http://localhost:8080/eurovo-twomass/admin/GenerateMetaDoc

At this stage, the page displayed the following error :

  • JDBC Driver error: java.lang.ClassNotFoundException: org.postgresql.Driver

This means that the new configuration properties have been recognised, but Tomcat does not have access to the PostgreSQL JDBC driver classes.

Bug 1248 :

Bug 1249 :

  • If the JDBC drivers are not installed, the metadoc generator displays a java.lang.ClassNotFoundException stack trace
  • The metadoc generator servlet should catch the exception and display a more user friendly message.

Installing the JDBC driver

The PostgreSQL JDBC drivers are available from the PostgreSQL website.

The EuroVo? workshop database server is running PostgreSQL 7.4, and the Tomcat on your laptop should be running in Java JDK 1.4, so we need to use the JDBC 3 database drivers.

You can download the pg74.216.jdbc3.jar JAR file from the PostgreSQL website.

Download the JAR file, and save it in the EUROVO_HOME/common/lib directory. 

    EUROVO_HOME
        |
        +-- bin
        |
        +- ....
        |
        \-- common
              |
              +- classes
              |
              +- endorsed
              |
              \-- lib
                   |
                   +-- ....
                   |
                   \-- pg74.216.jdbc3.jar

Tomcat will not automatically detect new JAR files added to the EUROVO_HOME/common/lib directory, so you will need to stop and start Tomcat at this point.

Once you have restarted Tomcat, you should be able to login and use the metadata generator page.

Creating the table metadata

The metadata generator generates a template table metadata document, based on the metadata available via the JDBC connection to the database.

The generated metadata should include details of all the tables and columns available in the database. 

    <DatasetDescription targetNamespace='urn:astrogrid:schema:TableMetaDoc:v1'>
        <Catalog>
            <Table ID='twomass_psc_dec_idx' >
                ....
            </Table>
            <Table ID='twomass_psc_ra_idx' >
                ....
            </Table>
            <Table ID='twomass_psc' >
                ....
            </Table>
        </Catalog>
    </DatasetDescription>

Use your web browser to save the generated XML document to a local file called eurovo-twomass-tables.xml in your EUROVO_HOME/conf/ directory. 

    EUROVO_HOME
        |
        +-- bin
        |
        +- ....
        |
        \-- conf
             |
             +- ....
             |
             \- eurovo-twomass-tables.xml

Configuring the table metadata

Next, we need to update the DSA configuration to point to the location of our new table metadata file.

If you look at the metadata JSP page, you should see the metadata for the SampleStars and SampleGalaxies tables that is shipped as part of the default DSA webapp.

To configure the DSA service to use the table metadata for the workshpp database you need to add the following element to your eurovo-twomass-context.xml context file: 

    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

        ....

        <!--+
            | Location of the table metadata.
            +-->
        <Environment name="datacenter.metadoc.file" override="false" type="java.lang.String"
            value="EUROVO_HOME/conf/eurovo-twomass-tables.xml"/>

        ....

    </Context>

  • Note this needs to be set to the full path of the metadata file on your laptop.
  • You need to replace EUROVO_HOME with the location of the unzipped Tomcat on your laptop.
  • If you unpacked the zip file into C:\EuroVoDemo\, then the full path will be C:\EuroVoDemo\conf\twomass-two.tables.xml.

If Tomcat is configured to auto-detect changes to context files (on by default), then it should automatically detect the change to the context file and re-deploy the eurovo-twomass webapp. If not, you may need to restart Tomcat to get it to recognise the changes. 

Tomcat log :
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer remove
    INFO: Removing web application at context path /eurovo-twomass
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer install
    INFO: Processing Context configuration file URL file:EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml

Once Tomcat has reloaded the eurovo webapp, the metadata JSP page, should now display the metadata for the 2MASS workshop database.

  • Table 'twomass_psc_dec_idx'
  • Table 'twomass_psc_ra_idx'
  • Table 'twomass_psc'

At the moment, this includes elements for the main dataset table twomass_psc, and the table indexes, twomass_psc_dec_idx and twomass_psc_ra_idx.

The current metadata generator requests metadata for all of the available objects within the target database. As a result, the generated metadata may contain details of tables and indexes that we do not want to publish via the DSA service.

Edit the eurovo-twomass-tables.xml file, and remove the Table elements for twomass_psc_dec_idx and twomass_psc_ra_idx indexes. 

    [file://EUROVO_HOME/conf/eurovo-twomass-tables.xml]
    <DatasetDescription targetNamespace='urn:astrogrid:schema:TableMetaDoc:v1'>
        <Catalog>

            <!-- REMOVE THIS ELEMENT -->
            <Table ID='twomass_psc_dec_idx' >
                <Name>twomass_psc_dec_idx</Name>
                <Description> </Description>
                <Column ID='twomass_psc_dec_idx.decl' indexed='false' >
                    <Name>decl</Name>
                    <Datatype>float</Datatype>
                    <Description> </Description>
                    <Units> </Units>
                    <DimEq> </DimEq>
                    <Scale> </Scale>
                    <UCD> </UCD>
                    <UcdPlus> </UcdPlus>
                </Column>
            </Table>

            <!-- REMOVE THIS ELEMENT -->
            <Table ID='twomass_psc_ra_idx' >
                <Name>twomass_psc_ra_idx</Name>
                <Description> </Description>
                <Column ID='twomass_psc_ra_idx.ra' indexed='false' >
                    <Name>ra</Name>
                    <Datatype>float</Datatype>
                    <Description> </Description>
                    <Units> </Units>
                    <DimEq> </DimEq>
                    <Scale> </Scale>
                    <UCD> </UCD>
                    <UcdPlus> </UcdPlus>
                    <SkyPolarCoord>RA</SkyPolarCoord>
                </Column>
            </Table>

            ....

        </Catalog>
    </DatasetDescription>

Save the changes to the eurovo-twomass-tables.xml table metadata and refresh the metadata JSP page.

The metadata JSP page should now only show metadata for the main twomass_psc table.

Running a simple query

Once we have the initial table metadata in place, we can run a simple test query against the database.

Goto the ADQL(sql) page, and enter the following query : 

    SELECT twomass_psc.ra, twomass_psc.decl, twomass_psc.designation 
   FROM twomass_psc WHERE ra > 100 AND ra < 100.0005

Bug 1252

  • The ADQL(sql) page does not handle ADQL syntax errors properly.
  • The ADQL query SELECT ra, decl FROM twomass_psc WHERE ra > 100 AND ra < 100.0005 displays a java.lang.IllegalArgumentException and a stack trace.
  • The JSP page should catch the exception and display a more user friendly message.

The JSP page passes the ADQL query to the DSA service, which queries the database and returns the results as a VOTABLE XML document. 

    [http://localhost:8080/eurovo-twomass/adqlSqlForm.jsp]
    <VOTABLE .... >
        <RESOURCE>
            <TABLE>
                <FIELD name="ra" ID="twomass_psc.ra" datatype="float"/>
                <FIELD name="decl" ID="twomass_psc.decl" datatype="float"/>
                <FIELD name="designation" ID="twomass_psc.designation" datatype="char" arraysize="*"/>
                <DATA>
                    <TABLEDATA>
                        <TR>
                            <TD>100.000036</TD>
                            <TD>-76.525902</TD>
                            <TD>06400000-7631332 </TD>
                        </TR>
                        <TR>
                            <TD>100.000155</TD>
                            <TD>-76.649216</TD>
                            <TD>06400003-7638571 </TD>
                        </TR>

                        ....

                    </TABLEDATA>
                </DATA>
            </TABLE>
        </RESOURCE>
    </VOTABLE>

The VOTABLE generator uses the table metadata in the eurovo-twomass-tables.xml metadata file: 

    [file://EUROVO_HOME/conf/eurovo-twomass-tables.xml]
    <DatasetDescription targetNamespace='urn:astrogrid:schema:TableMetaDoc:v1'>
        <Catalog>
            <Table ID='twomass_psc'>
                <Column ID='twomass_psc.ra' indexed='false'>
                    <Name>ra</Name>
                    <Datatype>float</Datatype>
                    ....
                </Column>
                <Column ID='twomass_psc.decl' indexed='false'>
                    <Name>decl</Name>
                    <Datatype>float</Datatype>
                    ....
                </Column>
                <Column ID='twomass_psc.designation' indexed='false'>
                    <Name>designation</Name>
                    <Datatype>string</Datatype>
                    ....
                </Column>
                ....
            </Table>
        </Catalog>
    </DatasetDescription>

to fill in the FIELD elements in the VOTABLE results document. 

    [http://localhost:8080/eurovo-twomass/adqlSqlForm.jsp]
    <VOTABLE .... >
        <RESOURCE>
            <TABLE>
                <FIELD name="ra" ID="twomass_psc.ra" datatype="float"/>
                <FIELD name="decl" ID="twomass_psc.decl" datatype="float"/>
                <FIELD name="designation" ID="twomass_psc.designation" datatype="char" arraysize="*"/>
                ....
            </TABLE>
        </RESOURCE>
    </VOTABLE>

Configuring the ConeSearch?

The DSA webapp can be configured to provide a simple ConeSearch? service for the dataset.

In order to generate the ConeSearch? query, the DSA webapp needs to know which colums to use for the ra and dec values. Unfortunately, because DEC is a SQL keyword, most databases cannot contain a column called dec.

This means that the DSA webapp cannot guess which table or column(s) to use for a ConeSearch? query.

To configure the ConeSearch?, edit the eurovo-twomass-context.xml file and add the following elements: 

    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

       ....

        <!--+
            | Cone search configuration
            | These are the columns that will be used when a cone search is submitted to a RDBMS.
            +-->
        <Environment name="conesearch.table" override="false" type="java.lang.String"
            value="twomass_psc"/>
        <Environment name="conesearch.ra.column" override="false" type="java.lang.String"
            value="ra"/>
        <Environment name="conesearch.dec.column" override="false" type="java.lang.String"
            value="decl"/>
        <Environment name="conesearch.columns.units" override="false" type="java.lang.String"
            value="deg"/>

        <!--+
            | Set to false if the SQL Trig functions expect arguments in degrees.
            +-->
        <Environment name="db.trigfuncs.in.radians" override="false" type="java.lang.String"
           value="true"/>

       ....

    </Context>

If Tomcat is configured to auto-detect changes to context files (on by default), then it should automatically detect the change to the context file and re-deploy the eurovo-twomass webapp. If not, you may need to restart Tomcat to get it to recognise the changes. 

Tomcat log :
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer remove
    INFO: Removing web application at context path /eurovo-twomass
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer install
    INFO: Processing Context configuration file URL file:EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml

Once Tomcat has reloaded the webapp, goto the ConeSearch page, and enter the following values :

  • Right ascension : 100
  • Declination : -80
  • Search radius : 0.05

Running a ConeSearch? at this point will display an error message :

  • Unknown units '' for conesearch columns, only 'rad', 'deg' or 'marcsec' supported

In order to perform the ConeSearch? calculation, the DSA webapp needs to know if the ra and decl columns are in degrees or radians.

You can find out the correct units for the ra and decl columns in the 2MASS dataset from the metadata available on the 2MASS page of the Caltech website.

To configure the column units, edit the eurovo-twomass-tables.xml table metadata file, and set the units for the ra and decl columns to deg. 

    [file://EUROVO_HOME/conf/eurovo-twomass-tables.xml]
    <DatasetDescription targetNamespace='urn:astrogrid:schema:TableMetaDoc:v1'>
        <Catalog>
            <Table ID='twomass_psc'>

                <Column ID='twomass_psc.ra' indexed='false'>
                    ....
                    <Units>deg</Units>
                    ....
                </Column>
                <Column ID='twomass_psc.decl' indexed='false'>
                    ....
                    <Units>deg</Units>
                    ....
                </Column>

                ....
            </Table>
        </Catalog>
    </DatasetDescription>

Save the changes to the table metadata and run the ConeSearch? query again (use the browser back button and re-submit the form).

The ConeSearch? should now return a VOTABLE document containing the results of the ConeSearch? : 

    <VOTABLE .... >
        <RESOURCE>
            <TABLE>
                <FIELD name="ra" ID="twomass_psc.ra" datatype="float" unit="deg"/>
                <FIELD name="decl" ID="twomass_psc.decl" datatype="float" unit="deg"/>
                <FIELD name="err_maj" ID="twomass_psc.err_maj" datatype="float"/>
                <FIELD name="err_min" ID="twomass_psc.err_min" datatype="float"/>
                ....

                <DATA>
                    <TABLEDATA>
                        <TR>
                            <TD>100.106977</TD>
                            <TD>-80.041023</TD>
                            <TD>0.1</TD>
                            <TD>0.07</TD>
                            ....
                        </TR>
                        <TR>
                            <TD>99.961308</TD>
                            <TD>-80.036819</TD>
                            <TD>0.09</TD>
                            <TD>0.06</TD>
                            ....
                        </TR>
                        ....
                    </TABLEDATA>
                </DATA>
            </TABLE>
        </RESOURCE>
    </VOTABLE>

Additional JDBC properties

To complete the JDBC database configuration, add the following properties to the eurovo-twomass-context.xml context file: 
    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

        ....

        <!--+
            | Limit results - Sets the maximum number of rows/nodex that can be returned by a query.
            | This is a crude safety limit to avoid full datacenter readouts in VOTable...
            +-->
        <Environment name="datacenter.max.return" override="false" type="java.lang.String"
            value="2000"/>
        <!--+
            | Limits number of simultaneous queries.
            +-->
        <Environment name="datacenter.max.queries" override="false" type="java.lang.String"
            value="4"/>
        <!--+
            | Limits time for each query.  0 = no limit
            +-->
        <Environment name="datacenter.sql.timeout" override="false" type="java.lang.String"
            value="0"/>

       ....

    </Context>

These limits provide a crude safety net to prevent someone running your server into the ground by requesting all rows and all columns from a large dataset.

The values you use will depend on the size of dataset, the capabilities of your server, and the type of queries you expect people to run.

Generating registry metadata

The next stage of the configuration involves registering the service with the local AstroGrid registry.

The AstroGrid registry provides a resource discovery service for the AstroGrid services, dataset and tools. The AstroGrid EuroVo? demo system contains a registry service, available at http://localhost:8080/astrogrid-registry-SNAPSHOT/.

The DSA webapp contains a servlet for generating the service resource description from the configuration properties and table metadata. You can request a resource description from the servlet via the XML metadata link in the navigation links http://localhost:8080/eurovo-twomass/GetMetadata.

The generated XML contains three resource description elements.

The primary resource description describes the dataset tables and columns, based on the data in the table metadata file. 

    [http://localhost:8080/eurovo-twomass/GetMetadata]
    <resources>

        <Resource .... xsi:type='tdb:TabularDB'>>
            <title>EuroVo workshop datacenter</title>
            <identifier>ivo://(unknown)/(unknown)/TDB</identifier>
            <curation>
                <publisher></publisher>
                <contact>
                    <name></name>
                    <email></email>
                </contact>
            </curation>
            <content>
                <description></description>
                <referenceURL>http://localhost:8080/pal</referenceURL>
                <type>Catalog</type>
            </content>
            <tdb:db>
                <tdb:name></tdb:name>
                <tdb:description/>
                <tdb:table xmlns='http://www.ivoa.net/xml/VODataService/v0.5'>
                    <name>twomass_psc</name>
                    <description></description>
                    <column>
                        <name>ra</name>
                        <description></description>
                        <dataType>float</dataType>
                        <unit>deg</unit>
                    </column>
                    <column>
                        <name>decl</name>
                        <description></description>
                        <dataType>float</dataType>
                        <unit>deg</unit>
                    </column>
                ....
                </tdb:table>
            </tdb:db>
        </Resource>

        ....

    </resources>

The first item that needs to be configured is the resource identifier. This is the globally unique identifier that will be used to register the service in the registry. 

    [http://localhost:8080/eurovo-twomass/GetMetadata]
    <resources>

        <Resource .... xsi:type='tdb:TabularDB'>
            ....
            <identifier>ivo://(unknown)/(unknown)/TDB</identifier>
            ....
        </Resource>

        ....

    </resources>

The resource identifier is made up of two parts, the registry authority identifier and the local resource identifier.

The registry authority identifier should be set to match the authority identifier of the registry that the service will be registered in.

  • The registry service in the AstroGrid EuroVo? demo system is configured with the authority identifier org.eurovo8080.
  • The the local resource identifier identifies the DSA service within the local registry.

To configure the resource identifier, edit the eurovo-twomass-context.xml context file and add the following elements: 

    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

        ....

        <!--+
            | The registry authorityId and resourceKey used to register the service.
            +-->
        <Environment name="datacenter.authorityId" override="false" type="java.lang.String"
            value="org.eurovo8080"/>
        <Environment name="datacenter.resourceKey" override="false" type="java.lang.String"
            value="eurovo-twomass"/>

       ....

    </Context>

If Tomcat is configured to auto-detect changes to context files (on by default), then it should automatically detect the change to the context file and re-deploy the eurovo-twomass webapp. If not, you may need to restart Tomcat to get it to recognise the changes. 

Tomcat log :
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer remove
    INFO: Removing web application at context path /eurovo-twomass
    20-Jun-2005 22:01:51 org.apache.catalina.core.StandardHostDeployer install
    INFO: Processing Context configuration file URL file:EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml

Once Tomcat has reloaded the webapp, the three resource descriptions should now contain the correct resource identifier. 

    [http://localhost:8080/eurovo-twomass/GetMetadata]
    <resources>

        <Resource .... xsi:type='tdb:TabularDB'>
            ....
            <identifier>ivo://org.eurovo8080/eurovo-twomass/TDB</identifier>
            ....
        </Resource>

        <Resource .... xsi:type='cea:CeaServiceType'>
            ....
            <identifier>ivo://org.eurovo8080/eurovo-twomass/ceaService</identifier>
            ....
        </Resource>

        <Resource .... xsi:type='cea:CeaApplicationType'>
            ....
            <identifier>ivo://org.eurovo8080/eurovo-twomass/ceaApplication</identifier>
            ....
        </Resource>

    </resources>

The second item that needs to be configured is the curation metadata. 

    [http://localhost:8080/eurovo-twomass/GetMetadata]
    <resources>

        <Resource .... xsi:type='tdb:TabularDB'>
            ....
            <curation>
                <publisher></publisher>
                <contact>
                    <name></name>
                    <email></email>
                </contact>
            </curation>
            ....
        </Resource>

        ....

    </resources>

To configure the curation information, edit the eurovo-twomass-context.xml context file and add the following elements: 

    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

        ....

        <!--+
            | Optional but 'well behaved' datacenters should have these set.
            +-->
        <Environment name="datacenter.publisher" override="false" type="java.lang.String"
            value="Euro Vo workshop"/>
        <Environment name="datacenter.contact.name" override="false" type="java.lang.String"
            value="[your name here]"/>
        <Environment name="datacenter.contact.email" override="false" type="java.lang.String"
            value="[your email here]"/>

       ....

    </Context>

Once Tomcat has reloaded the webapp, the resource descriptions should now contain the correct curation information. 

    [http://localhost:8080/eurovo-twomass/GetMetadata]
    <resources>

        <Resource .... xsi:type='tdb:TabularDB'>
            ....
            <curation>
                <publisher>Euro Vo workshop</publisher>
                <contact>
                    <name>[your name here]</name>
                    <email>[your email here]</email>
                </contact>
            </curation>
            ....
        </Resource>

        ....

    </resources>

The third item that needs to be configured is the external access URL. This is used in a number of places in the metadata to generate the published URLs that external services can use to access the service.

To configure the access URL, edit the eurovo-twomass-context.xml context file and add the following element. 

    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

        ....

        <!--+
            | The base URL that external services should use to reach this service.
            | Note - this MUST end in a '/'
            +-->
        <Environment name="datacenter.url" override="false" type="java.lang.String"
            value="http://localhost:8080/eurovo-twomass/"/>

       ....

    </Context>

Once Tomcat has reloaded the webapp, the resource descriptions should now contain the correct access URLs. 

    [http://localhost:8080/eurovo-twomass/GetMetadata]
    <resources>

        <Resource .... xsi:type='tdb:TabularDB'>
            ....
            <content>
                ....
                <referenceURL>http://localhost:8080/eurovo-twomass/</referenceURL>
                ....
            </content>
            ....
        </Resource>

        <Resource .... xsi:type='cea:CeaServiceType'>
            ....
            <content>
                ....
                <referenceURL>http://localhost:8080/eurovo-twomass/</referenceURL>
                ....
            </content>
            <interface xsi:type='vs:WebService'>
               <accessURL use='full'>http://localhost:8080/eurovo-twomass/CommonExecutionConnectorService</accessURL>
            </interface>
            ....
        </Resource>

        <Resource .... xsi:type='cea:CeaApplicationType'>
            ....
            <content>
                ....
                <referenceURL>http://localhost:8080/eurovo-twomass/</referenceURL>
                ....
            </content>
            ....
        </Resource>

    </resources>

This property needs to be configured manually, because although Tomcat may be able detect the local host name the service is running on, if the service is installed behind a firewall or HTTP proxy the external URL may be different.

Configuring registry endpoints

In order to be able to use the local registry to register the service and to resolve identifiers for other services, the DSA webapp needs to know the endpoint address of the local registry service.

To configure the registry endpoint addresses, edit the eurovo-twomass-context.xml context file and add the following elements: 

    [file://EUROVO_HOME/conf/Catalina/localhost/eurovo-twomass-context.xml]
    <Context path="/eurovo-twomass"
        docBase="astrogrid-pal-skycatserver-1.0.1-b006pl"
        debug="9"
        reloadable="true"
        >

        ....

        <!--+
            | The registry (admin) endpoint to register the datacenter resources.
            +-->
        <Environment name="org.astrogrid.registry.admin.endpoint" override="false" type="java.lang.String"
            value="http://localhost:8080/astrogrid-registry-SNAPSHOT/services/RegistryUpdate"/>

       <!--+
           | The registry (query) endpoint(s) to use to resolve other services and resources.
           +-->
       <Environment name="org.astrogrid.registry.query.endpoint" override="false" type="java.lang.String"
           value="http://localhost:8080/astrogrid-registry-SNAPSHOT/services/RegistryQuery"/>
       <Environment name="org.astrogrid.registry.query.altendpoint" override="false" type="java.lang.String"
           value="http://localhost:8080/astrogrid-registry-SNAPSHOT/services/RegistryQuery"/>

       ....

    </Context>

The EuroVo? demo system only contains one registry service, so both of the query endpoints point to the same service. For a live DSA service, these should be set to point to two different registry services, enabling the system to function if one of the registry services fails.

Registering the service

Once all of the properties have been configured, we can register the DSA service with the local registry.

Before registering the DSA service, use the local registry browse page to list the services that are already registered in the local registry.

If you set the 'Find IVORNs including' text box to twomass, you should see an emplty list.

The DSA webapp contains a servlet that will register the service resource descriptions in a local registry. The registration servlet can be invoked from the register link in the Admin pages navigation.

To register the DSA in the local registry, leave the registry URL blank, and click the register button.

Bug 1254

  • If the registration JSP page succeeds, it display a blank page with no links.

Once the DSA has been registered, refresh the local registry browse page to verify that the three eurovo-twomass resources are in the registry.

Now if you set the 'Find IVORNs including' text box to twomass, you should see three new entries, one for the dataset itself, and two CEA entries for the service.

The local registry in the EuroVo? demo system is not configured to replicate its data with other registry services. On a live system, registering the DSA would publish it in the global registry, making it available for other grid components to use.

Astronomy metadata

At the moment, the DSA service is registered with minimal information about the database columns.

If you have a look at the metadata it only contains the name, dataType and units for a couple of the table columns.

For the new DSA service to be useful to other astronomers via the VO, we need to add more detail to the table metadata.

The the column description, units and UCD values will need to be supplied by the dataset owner.

For the 2MASS dataset most of the data is available from the 2MASS pages at the Caltech website.

The UCD identifiers are available from VizieR

toggleopenShow attachmentstogglecloseHide attachments
Topic attachments
I Attachment Action Size Date Who Comment
xmlxml eurovo-twomass-context.xml manage 4.4 K 28 Jun 2005 - 21:30 DaveMorris Example context file
xmlxml eurovo-twomass-tables.xml manage 20.7 K 28 Jun 2005 - 21:34 DaveMorris Example table metadata
Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r12 < r11 < r10 < r9 < r8 | More topic actions
 
AstroGrid Service Click here for the
AstroGrid Service Web		 This is the AstroGrid
Development Wiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback