Standard for managements of web applications

Most of AstroGrid's products are server-side components packaged as J2EE web-appplications. This has worked fairly well so far and is a good, uniform approach. However, the packaging, configuration and management procedures are not standard, and are a weakness. Currently, deploying AstroGrid software is very difficult. This soaks up project time and is off-putting for potential service providers. We need a more refined approach.

This will become the standard for how web-applications are packaged, deployed, configured and updated. Currently, I've left it as a descriptive document. After discussion, I will turn it into normative text.

Packaging

AstroGrid components shall continue to be created as J2EE web-applications and shall be packaged in web-archive (WAR) files. These applications shall run in a servlet/JSP container. Tomcat 5 is the supported container (see below for notes about workable versions), but we should avoid coding anything into the WARs that makes them unusable in other servlet engines.

It should not be necessary to unpack a WAR before deploying it on the server. Any necessary configuration should be possible after the WAR is deployed.

The configuration of web-applications may be changed in situ, and we need to support this. The code should not be updated in situ. If a code change is needed, the WAR should be replaced. Often, it will be necessary to upgrade to a new WAR while retaining the previous configuration. This leads us to the arrangement in the next section.

Mapping of WARs to web contexts

In a working servlet-engine, clients talk to a context which leads to the code and documents of a web application plus a configuration. By default, in Tomcat, each context has its own documents and code and is self-contained. We need to move to something a little more elaborate.

Following MartinHill's suggestion we need to put the configuration and code/documents for a web-application in separate contexts; the latter contexts hold the contents of the WARs. The client sees the contexts carrying the configuurations as separate web-applications and these contexts are configured to get documents and code from the other contexts. This gets us the benefits that:

1. We can update a code/document context, by loading a new WAR, without destroying the configuration. Currently, we lose the context's configuration when we update.
2. We can have many configuration context reading one code/document context. This allows us to update them all at once, and may cut the memory usage (it's conceivable that only one copy of all the code gets loaded, although I've yet to test this).
3. It becomes easy to clone a configuration context (same configuration, new context path) and then adjust the clone to use, e.g. a different version of the code.
4. We can have multiple versions of code and documents (for one web-application) in separate contexts and switch between them.
5. We can use the names of the configuration contexts to express the uses of the web-application (e.g. INT-WFS object catalogue, SWIRE IR images etc.) and the names of the document/code contexts to express the origin of the web-application (e.g. pal-v7.1, astrogrid-cea-http-v8.4).

The down-side is that Tomcat allows this (I think servlet 2.3+ requires that it be possible) but doesn't support it in any of its tools. Therefore, with basic Tomcat, one has to hack directly on the configuration files to set up the context.

The actual mapping is done by setting the context-configuration file to hold text like this:

<Context path="/pal-rave"
         docBase="/usr/bin/tomcat/webapps/pal-vx.y.z"
         reloadable="true">
</Context>

where /pal-vx.y.z is the context holding the documents and code.

The file to hold this will be, in this example, $CATALINA_HOME/conf/Catalina/localhost/pal-rave.xml, i.e. it takes the name of the context with .xml appended. It's the same file where context-specific configuration is stored (as child elements of the Context element; more on this below).

This arrangement should be made uniform across all AstroGrid applications.

Access to the server

We should make it so that web-applications can be deployed, configured and managed using only HTTP access to Tomcat and no direct access to the server file-system. Hacking directly on the server files is awkward and error prone. It also requires that the installer be able to log in to the host, which may be difficult to arrange.

This means that we use something like Tomcat's manager web-application to deploy the web applications and something like Tomcat's admin web-application to control the configuration. I say "something like" because the current versions of those web-applications (in Tomcat 5.0.28) are too limited for our purposes; more on this below.

We need our web configuration-interface to be password-protected. This should be done using security settings configured in the WAR (so that it works as soon as the WAR is loaded), as prototyped in astrogrid-registry and PAL. This method uses a Tomcat role (set in $CATALINA_HOME/conf/tomcat-users.xml) and the role must be set up by the owner of Tomcat. The astrogrid-registry example uses the manager role, on the basis that it is (almost) always there already. The PAL example uses paladmin, on the basis that the Tomcat owner may not want to allocate the manager role. I suggest that we standardize on one role, not manager, for all astrogrid web-applications. The "realm" name is set to AstroGrid and this is displayed in the log-in prompts for Tomcat. I suggest that we make the role AstroGrid to match.

Configuration of contexts

For the name-value pairs in a configuration we've tried several ways of setting them:

• Properties files inside the web applications.
• Properties files outside the web application, location of the files advertized via JNDI
• Individual keys in JNDI
• Values in web.xml, installed from the WAR.

The individual JNDI-keys have the advantage that they can be viewed and edited over the web using Tomcat's admin web-application, which meets the no-hacking-on-the-files criterion above. The values in web.xml appear in the admin web-application as default values. Conversely, values in properties files are invisible via the web interfaces unless the web-application includes custom JSPs to reveal these values.

In AstroGrid-2, all name-value pair configuration should be set in JNDI; properties files should not be used in the deployed web-applications, although they are OK for unit tests.

Any property that is mandatory -- i.e. any that must be sent for the web-application to work -- must be given a default value in web.xml such that the property shows up in the admin interface. The default value need not be suitable to make the web-application run, but we need to make the configuration such that the installer only needs to set the properties, not to create them from scratch (and thus has no chance to create the properties with incorrect names).

Special JSPs and servlets, built into our web-applications, for setting configuration values in JNDI would be very welcome. However, we should not force ourselves to build and maintain one of these applications for every AstroGrid component. Instead, we should use Tomcat's admin web-application for most configuration, of which more below.

Enhanced admin and manager web-applications

The admin and manager web-applications in Tomcat do not support what we need to do.

1. The admin application has silly limits on the length of JNDI keys and their values.
2. The manager web-application does not support separate configuration and code/document contexts at all.
3. Both web-applications are unstable and unreliable.

I suggest that we replace both applications, as follows.

We patch the admin web application to remove the name/value length-limits, and possibly to fix any gross bugs that matter to us. We aim to get these changes merged back into Tomcat very soon (in fact, a fix for one is already in Tomcat 5.0.30) such that we only distribute our own version for one iteration at most. After this, we should aim to use Tomcat's standard version.

We build our own version of the manager web-application, starting from the baseline in Tomcat 5.0.x. Our version will support split contexts and should have enhanced controls for managing contexts in running, production servers. It would be get our version adopted as part of Tomcat, but we should be prepared to maintain it separately indefinitely. In the medium term, we may end up replacing all the code inherited from Tomcat with better code. This web-application would be the first thing installed in a Tomcat that is starting to run AstroGrid software. We might build into it some checks on the environment; e.g. check for jars in common that that are needed by Axis.

Building our own manager will be a substantial peice of work, but I expect that we'll win back the effort in easier installations.

-- GuyRixon - 29 Nov 2004