AG-SAG - Full Meeting #12

Date and Location: Imperial College, London: 12 June 2007, 10.00-16.00, 630 Blackett Laboratory

Supporting Paperwork:

Technical highlights in cycle 5: TechnicalUpdateAGSAGFM12.

Agenda : DRAFT as of 30 May 2007

09.45 Coffee available in Blackett Common Room (8th floor)  
10.00 Welcome NW
10.05 Review of Feb 07 meeting minutes (SAG11) and actions AJ/NW
10.15 SAG - new members & welcomes AJ
10.25 Input to next AstroGrid Oversight Committee Meeting, Jun 20 2007 AJ/JY
10.40 Future of AstroGrid - AG3 outcomes ( [.pdf] ) NW/AJ
11.00 - 11.30 Coffee  
11.30 AstroGrid May 2007 tools call: Responses and review procedure NW
12.15 Demo of current functionality updates in release 2007.1 (Open to ALL) (Time TBC) NW
12.30 - 13.00 Lunch Buffet
13:00-- 13.30 Demo of upcoming VoExplorer NW
13.30 - 13.50 Closed Session AJ
13.50 AstroGrid User Survey - Outcomes and Issues NW/ AJ
  Progress of Fitness Review (pdf) AL
14.40 AG Solar/STP services update SD
14.50 Technical update (inc IVOA issues) GTR
15.10 Euro-VO  
  Euro-VO Science Advisory Committee - Issues AJ
  Update on VOTECH stage 5 (Oct 06 - Mar 07) plans: new developments NW
  EuroVO Data Centre Alliance: Census of Data Centres and Workshops on VO Spectroscopy and Data Publishing to VO JAT
15.30 Increasing the use of the AstroGrid system JY
  - feedback from workshops - see AgWorkshops  
  - data and application priorities AJ
15.50 Future AstroGrid meetings and Workshops NW
15.55 AOB AJ
16.00 Close; Tea in Blackett Common Room  



Notes of Meeting

AstroGrid 3 Funding

  • Activities in Proposal
    • Meeting page FM11
    • Agreed revised programme
      • AGOC and STFC agreement
      • Cut in the R&D work package and engineering support
        • New technology development not explicitly supported
          • Issue of future participation in standards process
        • Less support for engineering activity
    • Euro-vo activities not affected - AstroGrid is still a project

FP7 Planning

  • May very busy lots of moving around wokring out how to respsond to the calls
  • VO type bids into 2 calls
    • AIDA - CDS led bid incl Edinburgh
      • standards development, help to data centres, publishing data - standards and deployment
      • digital repositories call
      • few mill euros
    • AGENA grid science infrastructure bid
      • led by INAF (going round the big groups)
      • interface computation grid computing groups - brings in german grid project AstroGrid-D - upload code run on big beowulf cluster
  • equivalent 2FTE of per effort per programme
  • these start approx. next spring - negotiable
  • fund about 5 people?

  • AstroGrid-D
    • German computational grid
    • upload code run on big beowulf cluster
  • Fire up workflow on system and run on big computer
    • JAY mentioned problems with running grid jobs on workstations is security e.g. firewalls
      • part of the reason for vospace
      • can do last bit through http
      • testing at UCL
        • non- AstroGrid core site
        • testing on a condor pool - use from within AstroGrid
        • ACTION DKW needs to talk to David Rage
  • Condor, local grid solution....fairly good compliment to CEA
  • Campus grids, just flocks of condor pools
  • Need our own condor pool
  • Lots of problems with 3rd party software...

Tools call

  • approx 6 staff months of effort AG side
    • configuring components
    • access in front of catalogues
  • invited PIs from programs to next consortium meeting
    • Janet Drew, Bob Mann and Julio.... on June 21st
    • Discussion with Janet about early data release for IPHAS.
  • ACTION Nic to send report on what happened for the 3 programmes to the SAG
  • solar work interesting
    • chianti
    • execution stuff of solar to be added
  • XCS data set available via DSA when catalogue itself released (imminent)
    • will show at consortium meeting next week
    • issues re security for private consortia such as this
    • being able to test has been quite interesting
    • certain elements have been added but not all
  • technical feedback has been handy

  • If a tool requires a lot of new effort then will require much more work
  • current tools used stuff that needed to be done anyway

  • are PIs from projects actually using tools?
    • students and postdocs starting to access the data through the system

2nd tools call

  • 5 proposals
    • slightly disappointing there weren't a few more but good quality (better than the 9 before)
  • eurovo call had 9 responses
    • only 1 which was reasonable (non from UK)

  1. Belfast led packaging up work to support pan stars data and sne discovery
  2. Integrating some models/cross matching - galaxy evolution UCLan
  3. Solar stuff
  4. Paresh Prema (Camb) - updated from last year
  5. Publish updated Hipparcos catalogue and ancillary stuff

  • All technically doable at some level
    • perhaps 2, one big one and one small one?
    • see which ones are worthwhile and then decide what can be supported
  • amount of effort to put in is about 6 to 9 staff months
  • invite (successful?) proposers to consortium meeting in December.

  • IPHAS is complete
    • primary catalogue is up
    • image retrieval
    • dsa access to catalogues
    • simple web page access (cone search)
    • documentation of how to access the data
    • official IPHAS EDR in July 2007
    • only way to access the data
    • CASU purchased server and database front end
    • provided for the IPHAS consortium
    • testing up to 4Gb queries returned from IPHAS
  • Use case for a lot of biggish projects



  • ACR improved, make myspace go better
    • updated query builder
    • workbench interface to current acr
    • voexplorer interface to updated ace

  • Data
    • dr2 now available
    • hinode
      • now accessible
    • iphas

  • applications wish
    • starting to add more of the applications on list

  • science
    • solar flare paper
      • lyndsay silvia
    • iphas
      • eduardo showing some usage of iphas through Agrid
    • Swift GRB
      • JAT + Swift team

  • robustness
    • much better - lots of re-engineering
    • major good news was dsa
    • storage improving
    • lots of stress testing on the registry
    • a lot of weak links have been found

  • Comments from Guy
    • all of system has to be brought up to a standard
    • DSA well on way - new IVOA standard protocol
    • registry doesn't support enough users but not urgent
    • query builder ok
    • myspace is difficult - won't improve until changed to vospace - hopefully be xmas
    • plastic ok but at mercy of those who plasticise applications
    • can do all but will be hardwork

  • workflow components not that brilliant - deprecated moving them to python examples
  • science docs improving
  • technical docs need to be made more accessible.

  • workshops - cardiff, belfast
  • some interest generated - belfast
  • caltech visit - met roy williams
    • useful discussions with those guys
    • managed to get test service - montage
    • 8 staff years of work
    • conserves flux (merges properly)
    • can be run from the workbench - returns the result to myspace
  • san diego write access to SRB
    • collaboration with san diego super computing

Programming Languages

  • Bill - idl, fortran, matlab, little bit of c
  • Lyndsay - PhD student uses java as well as the others IDL but some solar data in matlab
  • Alan - fortran, IDL (PPARC IDL license only applicable at more than 1 AU), matlab
  • Jeremy - fortran c c++ python perl matlab idl java - nothing dominates any more, students getting better with scripting languages
  • Andrew - mathematica, nobody uses matlab (license), increasingly python instead of IDL
  • Jonathan - IDL dominates among younger, active researchers in X-ray + multiwavelength science. Mixture of other languages used at lower levels.
  • Ed - grad students only learn IDL
  • ACTION to be asked to the rest of the SAG


  • Python
    • quite easy to use
    • lots of documentation
    • can work with myspace, create files etc.
    • can send tables from python to aladdin (PLASTIC)
    • ed has packaged some low level calls (helper class) to high level calls to simplify using python

  • Taverna
    • beta in october, operational in December/January


  • e-science stuff we are doing is ahead of the game
  • ACTION dkw and jay need to create a course for teaching postgrads astrogrid
  • probably best to do in January, will do when everything has settled down
  • 2 day course for UCL and IC
  • standardise training best role out method.....


Closed meeting feedback

  • we do share a concern to see the interface stabilised
  • positive stuff has been done
  • interface should be stabilised at expense at the technically best solution
  • workflows that are no longer usable
    • not much evidence of workflows being run
    • not enough development effort
  • user interface
    • think it will get better
    • as it moves into operation do get users
    • need to make sure changes are as backwards compatible as possible
    • want to avoid annoying the current or future user base where possible
      • but recognised reaching significantly more users is essential for AG3
    • ACTION worth contacting all people with usernames about the changes that have been made and why
      • news letter sent out as email

  • recalling astrogrid 10 science cases
    • might be worth visiting them or something like them, 2 or 3 science projects to be done with astrogrid in the next year
    • goal is to produce science-> papers with AstroGrid
    • SAG want to part of that and help as much as possible
    • can be with the very explicity participation of the scientitist on the AstroGrid side
    • concern that the demonstrations actually shown are very simple and need to show that real science can be done.
    • these experiments will be evolvable, exemplar....

  • Nic
    • galactic plane survey, swift (JAT), silvia and Lyndsay with solar stuff - galaxy clustering
      • Anita got reasonably far with Goald belt stuff with Jeremy
      • Aim to get these finished this year
      • AGLI thinking about the science

  • Very important for funding if can show real science accomplishments
    • Prove that there is science output that couldn't have been done without AstroGrid - needs to be seen to be competitive
    • Contact SAG
    • SAG wants to reaffirm that SAG is an important part of AG
      • ACTION SAG terms of reference to look over again
  • what were the things in AG 10 science cases that were difficult to do
  • AGO science case - quasars * required a lot of bespoke engineering
  • AstroGrid will hopefully prove to be the most efficient way to do things in the future, not necessarily the only way
  • revisit old case to see if have overcome the previous problems


  • Needs Metadata to be correctly filled in
  • VOExplorer Pages
    • Comments from SAG, send to mailing list
  • Overall structure already laid out but still some issues (e.g. layout)

User survey

  • now closed - link will be made available - andy lawrence wrote summary paper
  • 37 responses - not great but to be expected - 500 who have registered for AG accounts
  • most responded probably from workshops, worst comments from early workshops?
  • issues
    • nearly all focused on robustness
    • lack of applications or relevant data (in some cases data is there)
  • scripting
    • 50% said that they were using IDL
  • only a few said that they would never use it in the future - others said that they would be using it 6 months in the future
  • workshops went down well
    • content good
    • service was very slow during workshops
  • a number said they weren't using astrogrid but were using a number of things discovered through astrogrid (such as topcat)
  • bottom line
    • range of issues, many already known

Ergonomics review

  • resulted in voexplorer (improvements to) - andy lawrence refers to in paper

Fitness Review

  • user survey
  • useage monitoring - very important
  • cycle 6
    • finished vo explorer
    • working query builder
    • crossmatch utility/GUI
    • deployment across sites
    • UCL, wafu, LEDAS, cambridge etc.
    • ALMA centre in Cambridge will use DSA
    • to be wrapped up at the consortium meeting in a few weeks.

  • New solar dataset: Hinode EIS
    • EIS level 0 and 2 file are available via Helioscope (STAP on MSSL)
    • also ADQL queries - documentation

    • docs on web site
    • imported into MySql database
    • Taverna scripts?

  • Solar science case
    • flare productivity of newly emerged paired and isolated regions - suggested by Lyndsay
    • paper in A&A

  • Stuff from birmingham workshop and other workshops should perhaps be chased up

  • elizabeth auden esdo visualisation tool - streams data from a number of databases - gave paper at NAM

  • Guy
    • tabled report on wiki
    • backlog on 3 sections of possible useful work
    • new features - vo explorer
      • mosaicing for 2mass data - grid of computers behind
      • taverna - making taverna finished comes in two stages
        1. make low level thing to do vo stuff - finihsed by end of year
        2. afterwards refinement thing, package things up - becomes a lot nicer to use - ongoing task - totally open ended - keep them separate
        • could come from community
        • falls under AstroGrid 2
      • stilts - works more robust - will add redundancy service
      • france - cea wrapper for photon dominating regions
        • total number of users not high, but way it is wrapped is good
        • a lot of parameters have written custom gui to handle parameter constraints - can save and use in taverna workflow?
    • scaleability and reliability
      • size of returns
      • number of users (20 people doing reasonable size simultaneous queries)
      • stress tests
        • controlled environment
        • random failure rate about 1%
        • needs to be worked on
      • live production stuff
        • probable bug in the AR
        • overall reliability will go up if fixed
      • performance test show memory most important - do not run out, definitely not swapping
      • put queuing in - response time down but more reliable
      • characterise how much memory these services actually need.
      • to make remote processing actually work, need to get leverage off small computing clusters. - work with big grids in parallel
      • JAY - problems is getting access to them, people purchased computers through grant won't happen with current grant setup
      • NIC - perhaps only get kit if it will be made available to others as well.
      • spare computer nodes and archive....
      • JAY - what about the NGS
      • Guy - technically very hard, made very difficult to make it work
      • perhaps acccess control to limit local clusters to be used by local uses
      • will concentrate on getting usage out of local computer users
      • without programming will need condor to run on cluster
      • oxford/cardiff exemplar of good service, condor pool upgrade
      • would be useful to be able to access departmental condor pools with access pool
    • ivoa standards
      • vospace - want because better engineered than myspace
      • not free to set own standard
      • table access protocol
      • to be added to dsa catalogue
      • slightly dumbed down version of what is already being done.
      • allows adql queries against data model
      • ssap finally at v 1.0 - implemented via esac
      • new services ivoa
      • new way of doing registration - metadata
      • final piece of security - credentials - in by the autumn


  • science advisory committee andrew and nic going next week
  • EuroVO Data Centre Alliance (DCA) -
    • data publishing side - see links in agenda above
    • Leicester nominated UK partner (JAT+AMSR)
    • spectroscopy requirements gathering workshop at ESAC with 120+ participants
      • proceedings coming out
  • data publishing workshop, ESAC 25-29 June
    • bring together people from large data centres
    • major themes where publishing tools mature: catalogues, images, spectra
    • also time domain and transient data
    • AG leading catalogue DSA session + transient session
    • AG also leading gathering of requirements and feedback
  • JENAM (joint european national astronomy meeting) in armenia with VO session (JAT attending?)

Increasing the use of AG

  • good course
  • written up so that staff members can use it as well, read in their own time
  • demonstrating the usefulness of it
    • training
    • substantive demonstrations

  • access to eso data would be nice - ACTION statement by SAG?
  • ESA don't have a problems with the data....!
  • UKIDDS example resolves the security issue

Upcoming workshops/meetings

  • workshop in leicester likely postponed after July07
    • target specific community
  • sheffield and UCLan expressed interest
  • DCA workshop as above
  • ag consortium meeting
  • all hands meeting
  • adass sep 2007

  • another DAG at some point

  • ACTION Could everyone look at application wish, what should be on the list wiki word is application wish


  • next meeting Monday 29th october in Southampton


Claims for expenses can be made on the Leicester Expenses Claim Form (pdf). Please download, fill in, attach receipts (Leics will reject claims without receipts unless 'no receipt' is explicitly stated against an item), and send to:

Janet Bee
University of Leicester,
Dept of Physics & Astronomy,
University Road,

Please mark the Claim as 'AstroGrid: Science Advisory Group Meeting'

Travel and Accomodation

Blackett Laboratory is the Physics Department Building at Imperial, at the corner of Prince Consort Road and Queen's Gate. Nearest tube is Gloucester Road (Circle, District and Picadilly lines). More information is here.

If you require accommodation, please check

Wireless and Network Connections


-- AndrewJaffe - 30 May 2007

Topic revision: r22 - 2007-06-14 - 16:09:39 - DuganWitherick
AstroGrid Service Click here for the
AstroGrid Service Web
This is the AstroGrid
Development Wiki

This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback