Current Iteration Beta Testing

Changes to this page

20040823: NW: notes on workflow configuration files added

Portal address

The 'release' Iteration portal is available at:

• http://www.astrogrid.org/release

Username:

You need to get yourself a username. If you previously got your self a user name during Iteration 3 testing, that unfortunately will no longer be valid. But don't worry, Keith Noddle (ktn@star.le.ac.uk) has set up username/password combinations for the betatesters and will email those out to all concerned. Other interested people can email Keith directly for a username.

Iteration 5: What's to test?

The key point for science testing in Iteration 5 include:

• Access to more sophisticated data and tools allowing workflow processing

• Registry:
• Job Manager
• Data Query
• MySpace

These notes aim to show how:

• How to build up a workflow incorporating a number of application steps
• How to carry out database queries

Release Notes

General help on using the release is given at Itn05ReleaseNotes

Access AstroGrid via the Portal

Login to the system at http://www.astrogrid.org/release using your username/password combination.

Please note that a period of inactivity of more than five minutes will see you logged out of the system automatically. Unfortunately you are not warned of this - you will see that you can no longer view your files once logged out - and will have to log back in again. See Bug#413 to check on status of this problem.

ItnSixPortalUsabilityReport discusses some of the issues and likely enhancements to be made to the portal in the following iterations 6 and 7.

Browse registry:

Allows a return of resources currently register to the system. This is developmental just now, making use of latest IVOA registry standards.

Browse MySpace:

This gives you access to your online work space. Here you store your queries,and workflows. Submitted jobs return their results here. You can then view and manipulate these.

You might like to note that ItnSixMySpaceExplorerUsabilityReport discusses the changes being made to MySpace (in iterations 6 and 7) to allow for increased usability.

Data Query:

Create a database query

• The query for now needs to be generated in this rather opaque ADQL - which is based on evolving IVOA standards in this area (what we are moving to is a QL for astronomy, such that one query format can be fired off to many types of underlying databases).

Submitting a simple query

Itn05EndToEnd1 describes how to create a data query which can be sent to a remote database.

Example Brown Dwarf Search

PortalTestItn05 gives a step by step account of how to construct a query of the type required to search for brown dwarf candidates from optical data as held in a database. This contains more details including screenshots to illustrate each step.

Job Manager:

How to put the data query into the simplest workflow and run that to perform the actual data query is described in PortalTestItn05.

Photometric Redshift Workflow

Generation of more complex workflows is demonstrated by means of this example which enables the generation of photometric redshifts. In order to simplify the generation of the actual workflow a 'helper' app has been put together.

This process is one whereby photometric redshifts are generated from ACS four colour data released by the GOODS project team. The ACS data is from the GOODS team v1.0 release.

For each colour image, this is fed into the Sextractor application. Version 2.3.2 is used in this case. For each GOODS field, the outputs from the four image bands are cross federated, through a cross match process, to generate a single bviz input photometric file. This is then entered into the 'hyperz' application, and statistical redshifts returned.

1. Go to http://astrogrid.jb.man.ac.uk:8080/astrogrid-applications-Itn05_release/avodemo/. Select the area of the HDFN or CDFS that you wish to create redshifts for. Enter your username (as used in the portal, e.g. nicwalton) and hit the submit button.
2. Back in myspace:
   • (remember, when you first open the MySpace Explorer browser, the default end point is 'FILESYSTEM', you will have to select the 'Leicester' location from the drop-down menu box and hit the 'myspace-change' button before the directory listing appears for your files.)
   • you'll see a file has appeared in the 'workflow' directory - with a name similar to AVODEMO-sect11s (if you had selected area 11 south in this case).
   • there is also an empty directory - in this case - sect11s.
3. Go to 'workflow/job manager'
4. From the File menu select Open
   • a small microbrowser window will appear - careful, if you have opened this window at some earlier stage you must close (cancel) it first, as the updated one may not appear until the earlier one is closed!
5. From the microbrowser select the workflow to load - so in this case click on the folder icon ('nicwalton') which opens the tree, click on the workflow icon and click the file name 'AVODEMO-sect11s'
   • the IVORN reference to the file will appear at the top of the microbroswer
6. Click on the 'OK' button
   • the microbrowser window will close and the workflow ivorn will be transfered to the 'workflow' 'ivorn' box.
7. Due to a known bug, you know have to edit the 'ivorn' - replace the username#username bit with myspace#username
   • thus in this example ivo://org.astrogrid.itn05/nicwalton#nicwalton/workflow/AVODEMO-sect11s becomes ivo://org.astrogrid.itn05/myspace#nicwalton/workflow/AVODEMO-sect11s
8. Now hit the read-workflow button
   • The stored workflow will now be displayed in the workflow screen
9. In the 'File' menu - select 'submit workflow'
   • the workflow is now running!
10. Check the status of the workflow by selecting 'Workflow/Job Monitor' from the top line of menus
    • You will see a reference to your lastest job - check description, and time. Select the link to see status. This shows an example of a successful run.
11. After a while the job will complete and the output files will appear in myspace.
12. Go to myspace/explorer and select the directory where the files are deposited - e.g. nicwalton/sect11s in this example. The file 'hyperzout' is the output file for this run, with the other files being the intermediate results files
    • sexout_b, sexout_v, sexout_i, sexout_z are the outputs of the sextractor runs on the four input image files
    • merged is the cross matched photometric input file for hyperz

The Sextractor and HyperZ configuration files

The workflow generates a series of linked process, these are given below together with the default configuration files used in these:

• Sextractor
  • Config file for the b (F435W) images
    • Note that the detection filter refered to in this and the following three config files (FILTER_NAME detect.conv # name of the file containing the filter) is actually hard wired in this iteration. In Iteration six, this parameter will be user definable.
  • Config file for the v (F606W) images
  • Config file for the i (F775W) images
  • Config file for the z (F850LP) images
  • std.param: the list of output columns sent to the output file
    • Note - this file std.param replace the reference to 'PARAMETERS_NAME phot.param' in say h_goods_sz_r1.0z_phot_sex.txt because it is given effectively on the command line as an 'override parameter' through the workflow - see the PARAMETERS_NAME box in the workflow entry.
• Cross Federation
• HyperZ
  • zphot.param, the top level configuration file. This in turn contains references to a number of embedded configuration files:
    • A0V_KUR_BB.SED - the Vega SED used
    • FILTER.RES - the filter transmission file
    • cdfs-bviz.param - the filters file - noting which filters to use from the above file and the zero point magnitudes.
    • spectra.param - the models files for the templates (the evolved E, Sa, Sc and Im templates are used here - see the hyperz documentation for details).

For the current iteration (five) it is not possible to alter the parameters in a number of the lower level parameter files. Iteration Six will see full access to these files from a user point of view.

Getting the results:

1. Files in myspace can be viewed on screen by left-clicking the file disk icon. if you get strange results you might want to 'view source' in the browser, this is a problem with the way that different browsers handle xml files.
2. To save to a local disk, right-click on the icon. The saved file can now be viewed with a standard plotting package such as Starlink's TopCat or any other tool which reads VOTable.

Helper Applications

There are a number of applications which you might like to have available which will aid in visualising output results. All rely are able to read VOTable format files.

(In the next deployment release, you will be able to click on a file in myspace and it will automatically associate that file with the approriate 'viewer' - for now though you have to 'right-click' on the file, save to disk and view locally.)

list to be extended

Bugs

Please remeber to file any bugs that you come across using the astrogrid bugzilla at http://www.astrogrid.org/bugzilla. If you could enter bugs under the 'BetaTesting' category - and select version 'itn05-Deployment', that will help in tracking bugs found.

Known Bugs

1. When you first open the MySpace Explorer browser, the default end point is 'FILESYSTEM', you will have to select the 'Leicester' location from the drop-down menu box and hit the 'myspace-change' button before the directory listing appears for your files.

The Itn05 bugs listed in Bugzilla can be found here. This listing shows the bugs and the current status of those bugs.

Comments on the Deployment System

Please add comments on this release to the forum topic at http://forum.astrogrid.org/viewtopic.php?t=793

Technical information for the release:

The notes above concentrate on the use of this release from the end user standpoint.

The SoftwareRelease page provides information on all iteration releases to the technical user, and is of relevance to developers and system administrators.

Previous iteration Beta Testing Notes

Beta Testing in Iteration 3

Beta Testing in Iteration 5^?

slightly ammended by -- AnitaRichards - 06 Aug 2004