acd@roe.ac.uk)This note reports brief investigations of several packages which might contain facilities relevant to, or which should be incorporated in, the VO. The packages investigated are:
They are all produced by either ESO or Starlink.SkyCat and GAIA can access a reasonably extensive remote collection of standard astronomical catalogues and a few image surveys, principally the HST Digitised Sky Survey (DSS). The principal purpose of remote catalogue searches in SkyCat and GAIA is to find objects which overlay an image that has already been displayed by the application (though searches can be made which are not connected with any image). Consequently, the only type of remote search supported is the `cone search' to find objects within a specified angular separation (or `radius') of a specified central celestial coordinate. Optionally, the name of an astronomical object may be given instead of a central coordinate and the SIMBAD or NED name-resolver is used to replace the object name with the corresponding coordinates. For some catalogues additional selections are also supported on pre-defined columns. For example, it may be possible to select objects which lie in the specified region of sky and which also lie within a given magnitude range.
Regions of sky can be extracted from image surveys by specifying the central coordinates and size of the field required. Again, optionally, an object name can be substituted for the central coordinates.
SkyCat and GAIA have a convenient user-interface which is well-integrated with the rest of the display functions. Retrieved objects are automatically plotted on top of a displayed image if they overlay it. It is easy to highlight a given object in both a table of the selected objects and in an image overlay plot.
The list of remote catalogues and image surveys available to SkyCat and GAIA is held as a text file. This arrangement is good in that the list is not hard-wired into the code and can be customised, but is bad in that the file has to be edited manually, rather than maintained automatically by a `resource register' of the sort that we have been discussing.
Queries are submitted and results returned using HTTP protocols. The query format is somewhat restricted (and is similar to, but not identical with, the ASU query standard). Tables are returned in the Tab-Separated Table (TST) format, which is somewhat deficient in catalogue metadata, though it does contain enough information to define how objects are to be plotted on overlays (ellipses etc). Images are returned as FITS files.
A VO client or portal would need to provide at least all the remote access facilities of SkyCat and GAIA. Their principal disadvantage is that they can only search one catalogue at a time.
Version Tested
GAIA version 2.6, derived from SkyCat version 2.4.
Documentation
On-line documentation for SkyCat is available from its home page.
SC/17.1: The GAIA Cookbook, A.C. Davenhall and P.W. Draper, 31 December 2001 (Starlink).
SUN/214.9: GAIA - Graphical Astronomy and Image Analysis Tool, P.W. Draper, N. Gray and D.S. Berry, 13 September 2001 (Starlink).
SSN/75.1: Writing Catalogue and Image Servers for GAIA and CURSA, A.C. Davenhall, 26 July 2000 (Starlink).
The remote catalogue and image survey access facilities in JSkyCat are essentially identical to those in SkyCat: it provides the same functionality, uses the same mechanisms and formats for submitting queries and returning tables of results and has the same advantages and disadvantages.
Version Tested
JSkyCat version 1.2.
Documentation
On-line documentation is available from the JSky home page.
xcatview and
from the Unix command-line by using the application catremote.
The only type of remote search supported by either application is the `cone
search' to find objects within a specified angular separation of a specified
central celestial coordinate. Optionally, the name of an astronomical
object may be given instead of a central coordinate and the
SIMBAD or
NED name-resolver is used to
replace the object name with the corresponding coordinates. For some
catalogues catremote also allows additional selections on
pre-defined columns (for example, limiting the selected objects in the
specified region of the sky to also lie in a given magnitude range).
CURSA uses exactly the same mechanisms and formats as SkyCat and GAIA for
submitting queries to a remote catalogue and returning the table of results,
and has the same advantages and limitations. The most notable limitation
is that it is only possible to search one remote catalogue at a time.
xcatview's GUI for searching remote catalogues has a rather
different layout to SkyCat's. catremote is suitable for
embedding in scripts as well as for interactive use from the command-line.
Searching elements of the VO from within scripts which perform
specialised, bespoke tasks seems a likely requirement for the VO.
Version Tested
CURSA version 6.4.
Documentation
SUN/190.10: CURSA - Catalogue and Table Manipulation Applications, A.C. Davenhall, 4 November 2001 (Starlink).
SSN/76.1: CATREMOTE - a Tool for Querying Remote Catalogues, A.C. Davenhall, 24 May 2001 (Starlink).
The `external server search' option is more interesting and innovative. Here the user submits a query to a remote catalogue archive (such as LEDA or the NASA ADC catalogue collection), which is quite separate from the data centre holding the image surveys, in order to search one or more catalogues according to an arbitrary criterion the user has supplied. The remote catalogue archive returns a list of objects which satisfy the query. Querator takes this list and retrieves images for all the objects listed.
Access to Querator is solely through a Web interface, which is generally
easy to use. However, constructing the query for the remote archive in the
`external server search' option is complicated. The query is constructed
using the native syntax of the remote service and thus varies between
different services. Querator seems to still be under active development.
The query pages crash netscape running on a Compaq/Alpha but
are ok on Sun/Solaris (though this could be a bug in netscape
for all that I know). The surveys currently accessible mostly seem to
consist of pointed observations. However, presumably, there is no reason
why Querator could not access contiguous surveys, such as the DSS.
Querator has a number of features which seem likely to be required in the VO, including the ability to retrieve a stack of images from several surveys in one query and something analogous to the `external server search' option. However, to make the latter easy to use a unified (and simple) query syntax to specify queries on all the remote catalogues is required.
Version Tested
No version number was given. The tests were conducted on 13 February 2002.
Documentation
A brief introduction is available from the Querator home page.
`Querator: an advanced multi-archive data mining tool' F. Pierfederici, 2001, Proc. SPIE, 4477, pp246-253 (gzipped postscript format, 1.4Mb).
ftp: ftp.roe.ac.uk
directory: /pub/acd/various
file: stardoc.tar.Z
Decompress the file with uncompress (sic).
The High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA/GSFC has produced all the archive services described in this section; they are all accessible from their main page at http://heasarc.gsfc.nasa.gov/ or by using individual URLs.
Astrobrowse is a prototype VO in that it allows users to query hundreds of different catalogs and services around the world using a single form. These comments apply to version 1.7. There is a choice of a quick or full search, the latter form giving many more options and allowing a more selective search of archive sites. Queries are based on celestial positions, but with both types of interface you can search by object name (using either Simbad or NED as name resolver) or coordinates. You can also specify bandpass, data type, or other keyword to refine the selection. The user can also select which types of service to interrogate (e.g. optical or radio), or which individual servers, with considerable flexibility.
Astrobrowse uses AstroGLU software, a development of the GLU system written by the CDS at Strasbourg. This has a list of URLs of each data archive service, and information on how to generate a CGI query with suitable parameters. It then waits for the various replies to come back. There is obviously a potential problem if you select a large number of servers around the world: the results page has a side-bar reporting the status of each service, with an option to refresh this at intervals. In practice it takes only a few seconds to get results from the most servers, but it may be necessary to wait for a few minutes for all of them to respond (and at times some responses never appear). Astrobrowse has not attempted to solve the problem of integrating the results, which appear in a wide variety of formats. This is obviously a difficult problem, but one which the VO needs eventually to solve (e.g. with the aid of VOTable and UCDs).
A version of Astrobrowse is also installed at Harvard/SAO. The Astrobrowse software can be downloaded from HEASARC.
Browse is essentially a search engine for tabular data, originally designed for data from high-energy observatories, but now much broader in scope, including many radio and optical catalogues, as well as links to Vizier. The underlying DBMS is currently Sybase, but some attempt has been made to keep the software DBMS-independent. Version 6.3 was current when these comments were made. As with many astronomical archives, the primary search is by position or object name, resolved using Simbad or NED. Results can be produced in four formats: plain text, HTML tables, FITS tables, or Astrores XML.
Browse has two particular strengths:
HEASARC's web-based browse service was for a time known as W3Browse to distinguish it from the original BROWSE service accessed by telnet. The software was originally written in ESOC around 1980 as the interface to the EXOSAT Observatory's data archive. The service later moved to ESTEC, then to HEASARC, and versions were subsequently installed at other sites including LEDAS (Leicester) and MPE (Garching). The original telnet service is still available at LEDAS, although usage is dwindling. A few of the useful features of the original Browse have been lost in the web version, e.g. the ability to save the results of a filtering (select) operation and then make further selections on that. These features may be beyond the scope of a simple web service, but certainly ought to be provided by a data mining service.
SkyView describes itself in these terms: SkyView is a Virtual Observatory on the Net generating images of any part of the sky at wavelengths in all regimes from Radio to Gamma-Ray.
The Skyview server contains copies of images of the sky taken in a wide range of wavebands from radio to gamma ray, mostly (perhaps all) stored as FITS files. The SkyView software, written at GSFC, selects and overlays these images, giving results in one's chosen resolution, and it automatically handles rotation, precession, coordinate transformations, and pixel re-sampling. The results can be seen on the screen, or an FITS image can be downloaded from an FTP area in a number of formats including FITS, TIFF, GIF, and PostScript. There are actually five different interfaces: for the non-astronomer, basic, advanced, Java, and X-windows. The latter is regarded as obsolescent, now that Java can provide the required controls. More advanced image options, such as changes to color tables, overlays on extent images, image rescaling, zooming, etc. require a Java-enabled browser.
Advanced options include the ability to overlay data from two or three different data sources, perhaps mapping each to a different primary colour, producing a pseudo-colour result. This is even possible for those using 8-bit displays. SkyView can also implement boxcar averaging of an image, to obtain a smoothed result.
The comments here apply to Web Version 4.1, with version 3.2 of their Geometry Engine. A new interface is currently (2002-02-01) on beta-test and allows the interface to be customized; a few functions did not seem to be working correctly when I tested it.
There is also a batch option, with Perl scripts which can be downloaded and run on a Unix/Linux system. The software is freely available for download.
Overall the SkyView facilities are so comprehensive, and so well covered by the documentation, that it is hard to think of features still lacking. The FAQ covers a very wide range of questions, including How do I become an astronaut? The software, written by Tom McGlynn and his team, is all available for downloading and external use.
The ISAIA project was in intended to develop a number of virtual observatory concepts, such as the integration of results from queries such as those sent out by systems like Astrobrowse. As far as I can see the project is no longer active, and those involved are now part of the NVO team. The website contains several useful documents, but these are now being overtaken by more recent developments.
SkyMorph specialises in searches for variable, moving or transient objects. It provides convenient access to optical images and catalogs generated by the Near Earth Asteroid Tracking (NEAT) program. These include more than 67,000 CCD images covering a large fraction of the sky. The same region is typically observed several times each night, and is revisited on monthly and yearly timescales. As far as I can tell SkyMorph is based on SkyView, and does not seem to have any unique features of VO importance.
http://archive.stsci.edu/mast.html
MAST is the optical/UV/near-IR component of NASA's distributed Space Science Data Services (SSDS) and aims to provide integrated access to data from a range of missions/projects, namely: Hubble Space Telescope (HST) , Far Ultraviolet Spectroscopic Explorer (FUSE) , International Ultraviolet Explorer (IUE) , Extreme Ultraviolet Explorer (EUVE) , Hopkins Ultraviolet Telescope (HUT) , Ultraviolet Imaging Telescope (UIT) , Wisconsin Ultraviolet Photo Polarimetry Experiment (WUPPE) , Copernicus (OAO-3) , Orbiting and Retrieval Far and Extreme Ultraviolet Spectrograph (ORFEUS) , Berkeley Extreme and Far-UV Spectrometer (BEFS) , Interstellar Medium Absorption Profile Spectrograph(IMAPS) (first flight), Tübingen Echelle Spectrograph (TUES) , Digitized Sky Survey (DSS) & Guide Star Catalog II (GSCII) , Sloan Digital Sky Survey (SDSS) , FIRST (VLA radio data) , Roentgen Satellite (ROSAT) (N.B. some of these clearly go beyond the optical/UV/near-IR remit).
MAST offers a series of Cross-mission Search Tools, which vary from tools directed to specific science cases (e.g. find all data in MAST archives close to Abell clusters) to a Single Target quick search interface, where one can enter coordinates or source name (to be resolved by NED or SIMBAD), and get a list of data available within MAST: e.g. entering M31 yields the predictable long list of data - one nice feature is that preview versions of images (in GIF format) are provided. This interface also enables searches by data type - e.g. checking the "X-ray spectra" box returns a link to the top page of MAST's ROSAT site, as well as a helpful note that HEASARC provide access to a wider range of high-energy data. There is also the MAST Scrapbook, which offers users "representative images or spectra of an astronomical object" (specified by coordinates or resolvable name, as before) - basically another way of asking for preview data.
MAST also hosts a series of Prepared Science Products. Examples of these are the Hubble Deep Field (North and South) datasets, various UV spectral atlases and the SDSS Quasar Catalog , derived from the Early Data Release of the Sloan Digital Sky Survey. A page on Data Analysis Software lists "some of the data analysis software packages used for the MAST archived data", mostly comprising IRAF packages written for specific instruments. Data transfer varies between the different archives in the MAST system, but is usually either via anonymous ftp or direct downloading from a WWW browser.
Overall, MAST is a good example of a current-generation archive site: it provides access to data from a number of sources in a reasonably coherent and user-friendly manner, and with a reasonable amount of documentation (the documentation for the SDSS EDR site has improved markedly in recent months) and a Helpdesk which is staffed Mon-Fri 09.00-17:00 (EST), but it seems very interactive - there's no obvious batch mode facility, or any such means of submitting large numbers of queries in an automated fashion. There is basically nothing on the site to describe what technology underpins MAST.
Its blurb says that "StarView is an astronomical database browser and research analysis tool. Developed in Java, StarView provides an easy to use, highly capable user interface that runs on any Java enabled platform as a standalone application." Download and installation (under Windows NT) was remarkably simple, and the Java GUI is very nice. Starview can be used to search for data in MAST archives, examining the calibrations used for a particular dataset, and look at proposal information relating to past HST projects.
Downloading of data through Starview requires registration with the STScI archive, and is performed either by leaving the results file in an anonymous ftp site, or by ftping them to the user's machine: the latter requires supplying the user's password, and this is very unpopular with some system managers. One nice feature is that you can track the progress with your query on a WWW site.
Queries are defined by starting with a form (a number are provided, as templates for searching particular archives or making particular kinds of query) and then the user adds qualifications to narrow the search. Queries can be written out as SQL, which is nice, and there is also a Cross-Qualifier feature, which allows the results of one query to be used as input to constructing a second: this seems a very useful feature, but the instructions are not clear enough to enable a user to use this option at a cursory reading.
The results of a query are listed in another GUI, and datasets can be selected from that window for futher operations - e.g. previewing images or spectra, looking in ADS for references known to have resulted from that HST proposal, overlaying the instrument footprint on a DSS image - while the list itself can be exported as an ASCII file. One very nice feature is that the list of returned datasets can include proprietary ones, for which the date of public access is listed: slightly annoyingly, one has to remove those datasets from the list manually before asking to retrieve the data...surely a better default is not to include them, and then only the PI (who can access them) would have to do anything. A variety of data types can be retrieved - it is interesting that one can retrieve data quality information and/or observing logs, in addition to the data themselves.
All in all, this is a very interesting tool, displaying much of the functionality that one would require for the VO. As with the MAST WWW interface, this is still very interactive, but it does have the advantage that one can store and reload queries one has formulated interactively using the GUI. Again as for the MAST WWW interface, there is no description of the technology used, beyond Java.
http://archive.stsci.edu/starcast/about.html
Starcast is MAST's prototype implementation of Astrobrowse, which is a service allowing users to interrogate a large number of distributed archives with a single query, launched from a WWW form: at the heart of Astrobrowse is a database of "profiles", each of which gives the location of a data service and the means of querying it. The Starcast implementation currently uses a Perl interface to the profile database, not theCDS GLU system as used in the original Astrobrowse prototype at HEASARC, but is intended to migrate to using GLU at some point: this will mean that the Starcast administrator will not have to input the profiles manually, as is currently the case.
The Starcast query form allows the user to search for data around a sky position or an object with a name that can be resolved into a sky position by NED or SIMBAD. The user then specifies the Bandpass (running from radio to gamma ray) , Data Source (with choices Any, Derived, Observations, Pointed, Proposal, Survey, Survey Data), the Data Type (Any, Catalog, Image, Images, Other, Spectra, Spectrum, Time-series), and Location (Any, or a selection from a list of about 30 international data centres) and sets the query running. The browser moves to a new page, with two frames, one of which lists the specification of the query, and the second gives links to the services which might have data satisfying the query: next to each of these links is an icon showing whether the search on that resource is running, has completed successfully or has crashed, and these may be updated by pressing a Check status button at the top of the frame.
The implementation of this service seems incomplete, in some sense. For example, a test query asking for EUV data from Any Source of Any Type and at Any Location within 10 arcmin of 10 00 00 -10 00 00 returned a number of links, one of which was to the IMPReSS interface at the NASA ADC at Goddard. Clicking on that link took me to a WWW page generated within the IMPReSS system, which listed the sky position for my search and presented me with a list of archives (not just EUV, but also X-ray and optical) with data around that position, asking me which I wanted to choose. Clearly, since I'd already specified my query on the Starcast WWW form, I should have been taken one stage further within IMPReSS. This is something of minor quibble, for what is, after all, just a work-in-progress prototype implementation, but it does highlight the difficulty of fitting a top-level query interface on top of existing data centres, each of which provide access to their archives in different ways.
This report describes some existing facilities on the WWW that do some of the tasks expected of the VO.
CDS provide Simbad, Vizier, Aladin and a bibliographic archive. As with much of the CDS product, these services are interwoven and it's hard to see which part does what. Apparently:
NASA/IPAC provides NED. I quote from the introduction on the NED web-site: "It is built around a master list of extra-galactic objects for which cross-identifications of names have been established, accurate positions and redshifts entered to the extent possible, and some basic data collected. Bibliographic references relevant to individual objects have been compiled, and abstracts of extra-galactic interest are kept on line. Detailed and referenced photometry, position, and redshift data, have been taken from large compilations and from the literature. NED also includes images for over 700,000 extra-galactic objects from 2MASS, from the literature, and from the Digitized Sky Survey. NED's data and references are being continually updated, with revised versions being put on-line every 2-3 months."
In order to find out what these services can do, I tried to use them to do the Astrogrid use cases. I didn't have much success. The section names in the following are the Wiki-names of the use cases in the VO Wiki-web.
Vizier and Simbad can do the main flow of this use case, using Aladin as the display tool. Ironically, Aladin itself can't make the necessary selection. None of these tools can merge the tables of results.
NED can do some of this work. It can't select on radius from the search centre, but it can select on ranges of RA and dec., which is almost as good. It understands "QSOs" and "QSO clusters" as selection criteria. The plotter ("skyplot") from NED is poor (line graphics only) and is non-interactive: there is no way to select objects on the display and get to their details from the catalogue.
This feature is available in Simbad. Results of searches carry hyperlinks to entries in CDS' bibliographic service. However, only selected references are shown (to explain where the Simbad data came from, not to refer to the science). It is possible to query the bibliographic service directly, but the number of references returned is surprisingly small (e.g. 7 for a search on NGC1068).
NED allows one to query the database of abstracts directly by object name. This finds many (all?) references (e.g. 1249 for NGC1068).
I can't see a way to get any actual spectral pixel-data from any of these systems except one small part of IPAC. The SWAS mission, available through IPAC, serves spectra as either web graphics or in FITS files.
None of the systems seem to allow an instrument footprint as a search area in a query.
Aladin allows a use to draw one of a set of limited footprints as an overlay on an existing plot. If one then measures the footprint in Aladin one can get a search radius that encloses the footprint, and can search on that radius in Vizier. This allows the work to be done manually.
None of the systems do this use case. There are no links to observation-proposal systems.
As far as I can see, none of the systems cover this case. I haven't found any references to software for interconverting magnitudes and fluxes except in NED, which is trying to go in the opposite direction, from photometry to coarse spectra.
None of the systems even attempt this except for 2MASS (accessed through IPAC) which has a batch system for producing image extracts.
None of the systems can do these cases as written. There is no support for actually doing the astrometric fit, nor for plotting the residuals on the fit.
Vizier and Simbad can do most of SelectAstrometricStandards, but they cannot select the "best" catalogue out of the many available. Aladin doesn't help with this case, since the idea is to automate the process, not to do it interactively.
NED isn't very helpful, since stars are needed, not extra-galactic objects.
NED can do this very nicely, but only for one object at a time. The initial selection of data is not quite as general as in the use case. The plot is done as a web graphic displayed in a web browser.
Simbad, Vizier and Aladin can't do this work. It isn't even straight-forward to extract the photometry so that one can do it manually.
None of the systems allow this work to be done as stated.
NED allows references to be looked up, but not using bibcodes.
Vizier, Simbad and Aladin do not support bibcodes as a search term, but they do return bibcodes in the results of their results. The CDS bibliographic service does good searches by bibcodes.
None of the software helps with handling the list of data and bibcodes as suggested in the use case. Aladin could be used to display the objects in the user's catalogue and the user could then cross them off as the bibcodes were checked by drawing into the graphics overlay.
As far as I can see, none of the systems inspected here deal with these issues.
Some of the systems look very much like "the VO", especially Vizier and the fancier bits of NED. However, all the systems have the same basic philosophy: "display lots of data and metadata in a web page and give chains of hyperlinks to even more data." This is all one can do easily with the existing infrastructure, but it isn't "the VO".
Most of the use cases weren't supported because they involved the technique "do a search, then do something specific with the results of the search". The VO-like archives aren't set up to handle the "do something with the results" part, since they only represent the results as web-pages, not as semantically-useful data held for further processing. The exception is the making of synthetic spectra in NED, and this is a specific application - a vertical integration - that's clearly been coded in specially. It's not the kind of processing that a user can set up using a script and separate services at NED.
Some of the use-cases failed because the various archive don't have uniform criteria for selecting objects. In any given query, the selection criteria must either be on quantities that the interface designer coded into the UI, or there must be a free-form interface for specifying other criteria: a query language known to the user. The existing systems don't expose a query language, and their web interfaces only deal with a few quantities.
The use case GetReducedSpectra fails because the systems don't seem to provide reduced spectra. They only deal in images and tables.
The systems don't seem to deal in identified usage. Presumably, this means that they allow less access to data than a given user is entitled to.
In general, the systems reviewed let you look up more easily data that you could get by trawling through paper journals or by using interfaces to individual large archives. They require you either to know what you are looking for at the start (e.g. which catalogues to search) or to be prepared to spend a long time browsing. The output of the search is as for searches in paper collections: text you can read, but not machine-readable data products.
This begs the question: how much of what the VO ought to do is just a better version of this automated paper-searching?