Planning for Oxford workshops 4-8 Dec 2006

See workshop pages http://wiki.astrogrid.org/bin/view/Astrogrid/AgRadionetWorkshopDec06

Jabbercon 20061117

10 am GMT Friday 17 Nov, join chat room radiovo

See Jabber discussion 2006.11.17

For the interests of 'participants' please see http://wiki.astrogrid.org/bin/view/Astrogrid/AgRadionetWorkshopDec06

Telecon 20061025

Apols: Keith, Noel

Present: Kevin Kona Dave Nic Phil Guy Anita Jeff Eduardo

1. Need to identify and circulate details of which delegates have real data to publish now or who needs to use a test data set provided by AstroGrid.
   • Real data publishing - individual help
   • Rest work through exercised with help from one engineer and one scientist or as available.
2. Check re Keith's plan to send technical people in advance to help with publishing - probably not introduced till Liverpool mid-Dec.?
3. What community to register people with? Some e.g. Nancay may have already? (Dave/Guy)
4. Notes added 27 Oct: Set-up meeting to be arranged with Anita, Gary and Oxford Sys Admin Chris Hunter Tues 28 Nov. Dave available electronically for advice. Most of kit to be shipped from Cambridge, Gary to bring some from Leicester.
5. See attendance list on main page for overview of interests. Specific requests are listed below. In cases involving data publishing I will check what is available at home institutes and suggest other options.
   • Would it be possible to add a SLAP layer on top of the DSA?
     • Suggest Kona? See also VOSpec.
   • Pipelining visibility data for VO access (several)
     • This would mostly involve building a parselTongue or other pipeline with access to AIPS or other suitable data in a specialised environment, and interfacing that to the CEA. Anita to investigate whether Jodrell facailities can be used.
   • Grid computing for analysis of radioastronomy data
     • Check more details
   • Publishing tabular data (several)
     • It might be possible to provide a small DB on a Leicester VM but far better performance would be possible if data were on a home machine (or possibly FIRST at Cambridge?) and visible to Leicester DSA
     • What are the requirements for a home BD e.g. does it have to be on the same machine as the tomcat?
   • Publishing Nancay Radio Telescope data (single dish e.g. spectra)
     • Check what is involved - probably already some installations at Nancay.
   • Publishing images/spectra etc - as I understand it, that either has to be via a CEA application (e.g. python script) or via metadata stored in a DB, so populating the DB might be a prerequisite? Part of the exercise?
     • Question: Is MeXx (ESO/Rino) or the DALIngestor for spectral metadata (ESAC/Barbieri, Salgardo) useful to generate the metadata?

AstroGrid attendance

Please check/indicate days available and book accomodation

Science workshop 4-5 Dec

• Technical: Jeff, Gary
• Science: Anita, Nic, Eduardo +? (check with Oxford if any Solar people coming)
• TopCat + Mark T

Radio workshop

In addition to formal registrees (as of 27 Oct) there is interest from ASTRON (WSRT/LOFAR), NRAO (CASA etc.) and ESO (ALMA).

Overview

The first day and a half will be a 'normal' science workshop for Oxford people plus a few radio people who are interested in both.

The rest of the workshop will be more technical. The primary motivation from the radio side is:

1. To publish tabular data
2. To provide access to interferometry data using command-line software wrapped as a CEA application (e.g. the MERLIN Imager)

Most of the RadioNet people will be from anywhere in Europe (possibly further) and are likely to be archive developers, astronomical software engineers etc. and to have had some previous VO exposure but it can't be assumed. However it is hoped that if everyone meets the requirements given, they will at least have enough experience to e.g. run a .jar executable, follow instrutions to install Tomcat etc.

Preparation for delegates

Need instruction sheets - what should you ahve set up it you have a collection of visibility data, tabular data etc.

Need to prepare for:

1. Installing Java and Tomcat
   • Note - some Linux distros come with Java and Tomcat already installed. Unfortunately, these are often the wrong version and the AstroGrid services don't work properly in them. However, this isn't too much of a problem, as you can have multiple versions of both Java and Tomcat installed on the same machine. So they don't have to uninstall the default one, we can just add our version in a separate directory, and remove it when the workshop is finished.
2. Remote machine access - home institutes
   • Software
   • Firewalls/Ports
3. AstroGrid test machine - can have up to 4 VMs at Leicester but if one has to host a DB that may be less. Plain VMs for workshop services can be just vanilla Fedora with recent Java JDK 1.5 in them. Dave can provide scripts to Gary

Notes from Dave and Gary on requirements for Oxford

 
Internet access.

   Wired network connections for 20-30 laptops, with good bandwidth.

   All of the delegates and workshop tutors will need
   Internet access from their laptops during the workshop.

   Ideally, we would ask the institute to provide enough
   wired network connections for all of the delegates and
   workshop tutors.

   Most of the delegates will probably be running Linux;
   there will probably be a few Macs and Windows machines;
   if there is any OS limitation please let us know but Linux is vital.

   Based on past experience, we have found problems configuring
   some laptops to use wireless networks, and the volume of traffic
   during the workshop may saturate a standard wireless network.

   The ideal is if you can provide a standard (not Windows specific)
   IP address for each laptop via DHCP.
   We will be happy to ask all the participants to register the MAC
   addresses of the machines they intend to use.

People need to be able to 

       * Use ftp or sftp to download software and data onto their laptops
       * Use ssh to external sites and work remotely
       * Use the Internet including AstroGrid and RadioNet wikis

The AstroGrid software we are demonstrating during the workshop connects to
multiple external grid services using SOAP/HTTP messages.
Most of the services use port 80 or 8080, although there may be some that use
ports 8081-8088.


   All the delegates and tutors will need to be able to connect to external
   AstroGrid HTTP/SOAP services on ports 80 and 8080 to 8088.  Also 7512.
   The delegates and tutors will also need to be able to login to external
   machines at AstroGrid and at their home institute using SSH (port 22).

   In addition, the workshop tutors will need to be able to access
   our Jabber server on port 5222.

   The primary external (outbound) connections we would need would be
       * SSH on 22
       * FTP on 20 and 21
       * HTTP on 80
       * 7512
       * HTTP on 8080
       * Jabber on 5222

   Access to external SMTP and POP3 would be useful.

   Please contact our sys-admin, Gary Gilchrist ,
   to confirm details.

   If you have any problems, e.g. limited number of ethernet sockets,
   we may be able to provide additional hubs and cables etc.

   We will be teaching people how to publish data from their own institutes
   using remote log-ins to install AstroGrid services on their machines.
   If any of the Oxford delegates want to publish Oxford data then please
   see the requirements for an externally visible machine.

Jabber discussion 2006.11.17

Registry

• Sites installing their own services will have their own publishing registry - we show them how to set one up during the workshop.
• Users installing workshop services will register their services as org.radionet.{user} in the shared workshop registry.
• Workshop publishing registry installed by Kev at mssl.
  • Will everyone use the main query registry at Leicester ?
• Good practices regarding metdata should be instilled! e.g.
  • Proper descriptions of catalogues
  • There is no easy way at rpesent to get coverage info from raw radio astronomy data but we should think about it (maybe some delegates will ahve plans!)

CEA

• We show them how to setup an example CEA using STILTS (Mark/Guy to write instructions and add them to cvs).
  • These will be example services, so they will be registered in the workshop registry.
  • Probably most people will find this useful, but they can work in pairs/small groups.
  • Some people may need to use VM machines at Leicester
    • Leicester VM machines will be shutdown after the workshop.
  • We will teach people how to wrap Python scripts, which has many uses for astronomers - these could be published using Leics VM CEA if simple e.g. no external data involved?

• Most processing of radio astronomical data (e.g. turning visibilities into images) is done using specialised software eg AIPS but thees packages can be run from a python wrapper e.g. parselTongue, which can be published as a task in CEA.
  • Thus, AstroGrid only has to worry about wrapping an application and returning the output (probably a VOTable containing image or other data URLs) to the user.

• If they have their own data AND a processing toolkit (AIPS+parselTongue or equivalent and the right permissions to install tomcats etc.), we show them how to install CEA at their own site.
  • The services will be installed on their own machines.
  • They will need their own publishing registry, registered with the main AG registry.

• If they do not have suitable data and processing at their home insititute, then we give them an account at JBO, and show them how to install CEA.
  • Of people who want to publish data using AIPS, so far the ALMA people have definately indicated an interest and I suspect that the GMRT survey people do.
  • So regarding Jodrell that is 2-3 groups
  • CEA will need to be installed in a Tomcat on aife at JBO.
  • These will be example services, so they will be registered in the workshop registry.
  • Delegates will need ssh login accounts to aife at JBO, and access to a Tomcat.
  • The Tomcats will need to be configured on different port numbers, and use a httpd proxy to expose them on port 80 or 8080.
  • The services on aife will need to be able to access external services on ports 80 and 8080.
  • Someone from AG will need to have root access to tweak the httpd proxy settings.
  • When they get their own data processing setup they will use what they have learned to setup their own CEA.
  • How long will the accounts/tomcats at JBO be available ?
    • Only up to the end of the workshop

• If they have their own processing tool but no data .... ?
  • Unlikely! But plenty of test data will be at a public URL for the parselTongue tutorial

Telecon on set-up at JBO

Guy, Anita, Ant Holloway, Bob Dickson (JBO computer managers)

• Will set up 3 separate local machines (shuttles1-3, probably) to run AIPS, parselTongue and Tomcat (5.5.20 or .21; java 5)
  • One external user account per machine
    • Participants will use these to learn how to install CEA
    • Can do most of preparation of scripts on laptops
    • Will need to copy scripts to JBO machines and also ssh to edit/move files.
  • One machine to be available for testing before workshop
  • During workshop, external access will be restricted to IP addresses in the Oxford range if possible.
  • Guy to provide example python script for testing CEA installation
  • Anita and Mark K to provide test AIPS data
  • Anita to provide very simple AIPS data and parselTongue script for testing those. Also test publishing to CEA.

DSA

• We show them how to setup an example FIRST using DB on AG01.

• If they have their own database, we show them how to install DSA to publish it.
  • These will be production services, registered in their own registry.
  • They will need their own publishing registry, registered with the main AG registry.

• If they don't have their own DB, then they can install a DSA using FIRST on AG01.
  • These will be example services, registered in the workshop registry.
  • DSAs will use temp JDBC accounts to MySql on AG01.
  • JDBC accounts will be invalidated after the workshop (this means that their DSA will stop working once we disable the account).
  • AG01 firewall will only allow JDBC connections from specific hosts.

• If people have tabular data but not a DB we may be able to ingest the data using STILTS or similar into the Cambridge DB - advance notice and Kona's agreement that it is not too time consuming required.
  • A short tutorial on how to use STILTS to put data into a DB might be useful for people setting their own DBs up in future

• If they don't have their own machine, they can use a VM at Leicester to add DSA to FIRST on AG01.
  • Leicester VM machines will be shutdown after the workshop.