ASTROGRID-2 : Lifetime Plan

June 2005 Draft-3 June 8

Part-F : Roadmap

Here is a summary of the key milestones on the Roadmap. The logic behind these is explained below.

2005 : Cycles 1-2 
------------------
2005 Feb   VOTECH Project Plan 
2005 Apr   working system demo at NAM
2005 Apr   Initial AstroGrid workbench
2005 May   V1.0 release of portal and software suite
2005 Jul   V1.1 s/w release ; full documentation
2005 Sep    VOTECH Science Framework Document
2005 Oct   Multiple third party applications in workbench
2005 Oct   Registry Exploration Browser
2005 Nov   Release AstroGrid Common Runtime
2005 Dec   Fully specified Registry with Xquery interface
2005 Dec   Universal Worker Service Working Draft V0.9
2005 Dec   International storage sharing : VOStore
2005 Dec   Initial server side applications

2006 : Cycles 3-4
------------------
2006 Feb   Personalised tools Call-1
2006 Apr   Query Builder and Workflow Builder for Workbench
2006 Apr   Deployment release : demo at NAM
2006 Apr   Query Builder and Workflow Builder for Workbench
2006 Apr   VOTECH Infrastructure Study Report
2006 May   Complex search interface
2006 Jun   Major package release
2006 Jun   Demonstration of large data volume problem
2006 Jun   Single Sign-on for VO services
2006 Jul   V1.5 s/w release
2006 Jul   VOTECH Revised Project Plan
2006 Aug   Prototype ontology use
2006 Sep   VOTECH Data Exploration Study Report
2006 Oct   Universal Worker Service Recommendation V1.0
2006 Oct   Client side scripting for workflow
2006 Dec   ADEF Prototype release
2006 Dec   Federated access : VOSpace
2006 Dec   Secure authorised access to VO services 
2006 Dec   New Look Portal 

2007 : Cycles 5-6
-----------------
2007 Feb   Personalised tools Call-2
2007 Apr   Deployment release : demo at NAM
2007 Apr   VOTECH Tools Study Report   
2007 Jun   Full access control
2007 Jun   Major package release
2007 Jun    First Resource Discovery Prototype
2007 Jul   V1.8 s/w release   
2007 Jul   Full client side integration
2007 Sep   VOTECH Resource Discovery Study Report
2007 Oct   Integrated ontology services
2007 Dec   V2.0 final s/w and deployment release
2007 Dec    Second Resource Discovery Prototype
2007 Dec    Working ADEF


2008 : post AG
--------------
2008 Dec    Euro-VO Reference Architecture

Release Plan

AstroGrid will be making releases both of its software suite and of its deployed working system. At first these are simultaneous but will gradually become distinct, especially as multiple deployments emerge, and as AstroGrid deployments will increasingly use modules from other projects. The deployments cannot be produced by the AstroGrid project alone, requiring a commitment of effort from UK data centres. However, AstroGrid leads this "Greater AstroGrid" effort. The best arrangements for long term provision of VO deployment in the UK is a separate matter that we won't address here.

The critical milestone is the V1 release of spring/summer 2005. This is the first point at which a working system for end-users is released, together with a simultaneous release of the software suite, including documentation, configuration scripts, and so on. Following this, we will release new versions of components separately as they emerge, so that releases will be in almost continuous drips . Because of the mix and match nature of AstroGrid software, and its intended use by external parties, new components will need to be compatible with multiple versions of other components. However this cannot be indefinitely sustained, and so we will also need to occasionally announce the deprecation of older versions of components.

There will still be a planning and PR value for annual packaged up announcements which can constitute public releases. Likewise, although science functionality in deployments will continuously improve, we will aim at annual announcements tied to the UK National Astronomy Meetings.

Milestones :

2005 Apr   working system demo at NAM
2005 May   V1.0 release of portal and software suite
2005 Jul   V1.1 s/w release ; full documentation
2006 Apr   deployment release : demo at NAM
2006 Jul   V1.5 s/w release
2007 Apr   deployment release : demo at NAM
2007 Jul   V1.8 s/w release   
2007 Dec   V2.0 final s/w and deployment release

External Science context

(Nic to write ?)

SWIFT

WFCAM

VISTA

ALMA ?

Infrastructure Components

AstroGrid has some kind of working component in all the key areas - Registry, User Interface, Dataset Access, Workflow, Virtual Storage, Community. We do not yet have the full original intended functionality, and expect this to emerge gradually.

Identity, Authentication, and Authorisation. This is the area where we are furthest behind our original intentions, partly because of the difficulty of international standardisation, and acceptance of new methods by data centres. We expect to achieve internationally consistent community based single sign-on by mid-2006, allowing authorised read access; and secure access by the end of 2006, allowing full control and write access to relevant services.

2006 Jun   Single Sign-on for VO services
2006 Dec   Secure authorised access to VO services 

Virtual Storage This is already an AstroGrid hit with Myspace - essentially, remote storage services are provided by the Community that a user logs on to. By the end of 2005 we aim to have international sharing of storage via the VOStore protocol. By the end of 2006 we aim to have federated access to data (merging the MySpaces) together with intelligent data routing (loosely referred to as VOSpace). The user should also then be able to interface to Storage Resource Broker as well as MySpace. Finally we should allow full access control by 2007.

2005 Dec   International storage sharing : VOStore
2006 Dec   Federated access : VOSpace
2007 Jun   Full access control

Applications Integration The key to our success so far has been the definition of a Common Execution Architecture (CEA), and Common Execution Connectors, which define a Web Service interface to applications of all kinds, and allow asynchronous activity. We aim to engineer a new WS-RF based version for standardisation, called the Universal Worker Service (UWS). Another central concept is that of the AstroGrid Common Runtime (ACR), which should complete in 2005 and subsequently allow integration of packages such as IRAF and MIDAS.

2005 Nov   Release AstroGrid Common Runtime
2005 Dec   Universal Worker Service Working Draft V0.9
2006 Oct   Universal Worker Service Recommendation V1.0

User Interface. AstroGrid-1 was entirely browser based, with web pages for Registry, MySpace, Query Builder, Workflow Builder, and Job monitoring. Complex functionality was rendered through HTML at the server via Cocoon. This proved to have a heavy development cost. A top priority during Cycle-1 was therefore to develop an additional "Workbench" interface launched through Java Webstart. We will add functionality to this during 2005, whilst maintaining and improving the Portal without significant new functionality. During 2006 we will produce versions of Query Builder and Workflow Builder for the Workbench, whilst developing a new-look portal, which will be released at the end of 2006.

2005 Apr   Initial AstroGrid workbench
2005 Oct   Multiple third party applications in workbench
2006 Apr   Query Builder and Workflow Builder for Workbench
2006 Dec   New Look Portal 

Workflow and client integration Workflow depends on the CEA and on the Job Execution Service (JES) which will be gradually enhanced. Building workflows is currently through a browser based tool which is ingenious but has a heavy development cost. We will therefore target a Webstart workflow builder for mid 2006. Apart from these interface issues, the key advance during 2006 and 2007 will be client side scripting for workflow, so that end users can write Python, Perl scripts, or whatever they wish, and put it into their own workflow. Similarly, a user should be able to write a Python script inside which they can call VO services in a standardised way. They should be able to upload algorithms to the server, to run next to the data, and see their own local directories in MySpace. In general your laptop should be part of the VO.

2006 Apr   Query Builder and Workflow Builder for Workbench
2006 Oct   Client side scripting for workflow
2007 Jul   Full client side integration

Registry The registry is a rich and potentially powerful structure, and we have system for querying it, but there is a feeling that we have yet solved the ergonomic problems of how to unlock its power. The first key goal is therefore to develop an improved query interface and registry browser . Meanwhile we are leading international standardisation in the Registry area, which we expect to complete in 2005, with a key change being use of Xquery rather than SQL. By mid-2006 we expect to expose the full power of the Registry to the user, with complex searches. More exciting and speculative improvements in resource discovery will come through the relevant VOTECH research area, with the use of ontologies.

2005 Oct   Registry Exploration Browser
2005 Dec   Fully specified Registry with Xquery interface
2006 May   Complex search interface
2006 Aug   Prototype ontology use
2007 Oct   Integrated ontology services

IVOA standards roadmap

International standardisation is the key to the VO. Agreement between projects is aggressively pursued through the International Virtual Observatory Alliance (IVOA), with mature standards then passed on to Commission 5 of the IAU for formal global endorsement. The IVOA has a clear Technical Roadmap, which AstroGrid staff have been very active in helping to define. This gives both a framework within which to evolve, and a target for new infrastructure - in places where we are moving ahead of international competition, we must plan standards that we would wish the IVOA to adopt.

IVOA milestones :

2005 May      REG      VO-identifiers1.0(R)
2005 May      UCD      UCD1+-V1.0 (R)
2005 Jun      UCD      create vocab and tech boards
2005 Jul      DM      STC-V1.0 (R)
2005 Jul       REG      RM-V1.1 (R)  
2005 Jul      REG      VOResource-V1.0 (WD)
2005 Jul      DAL      SSA-V0.9 (WD)
2005 Jul?      VOQ      ADQL-V1.0 (R)
2005 Jul?      VOQ      SkyNode-V1.0 (R)
2005 Jul      UCD      UCD-vocab (R)
2005 Jul      DM      SpecLine-V0.1 (WD)
2005 Aug      VOT      VOT-V1.2 (R)
2005 Aug      GWS      VO-SupportInterface-V1.0 (R)  
2005 Sep      VOQ      ADQL-V1.x (WD) (DAL integn)
2005 Oct      UCD      change name/nature of WG
2005 Oct          DAL      SSA-V1.0 (WD) 
2005 Oct      VOT      review fuure of WG
2005 Oct      TIG      Theory Use Case Doc
2005 Nov      REG      RegOfReg-V1.0 (R or Note)
2005 Nov       GWS      VO-WS-V1.0 (R)
2005 Dec      VOE      VOEvent-V1.0 (R?)
2005 Dec      GWS      VOStore-V1.0 (R)
2005 Dec      DAL      SIA-V1.1 (WD)
2005 Dec      DM      SED-V1.0 (R)
2005 Dec      DM      Catalog-V0.5 (WD)

2006 Jan      DM      Characterisation-V1.0 (R)
2006 Jan      GWS      SSO-V1.0 (R) 
2006 Jan      GWS      UWS-V1.0 (WD)
2006 Jan      GWS      VOSpace 1.0 (R)
2006 Feb      REG      VOResource-V1.0 (R)
2006 Feb      REG      RegInterface-V1.0 (R)
2006 Sep      VOQ      ADQL-1.x (R)
2006         DM      Obs-Provenance (R?)
2006         DM      Quantity-V1.0 ?
2006               new semantics WG ?
2006               Ontology 1.0 ?
2006               Preservation 1.0 ?

Tools and Science Services

Nic to write ?

VOTECH research and feedthrough

With our European partners in VOTECH, AstroGrid2^? will carry out a research programme aimed at development of future components. This is not blue-skies research, or academic computer science research. Rather, it takes new technologies being developed elsewhere, assesses them, builds prototypes, and then designs components to take forward into the mature infrastructure. As well as looking out for new components, this work also aims at identifying as early as possible the need for new standards and protocols, which we would then take forward to the IVOA agenda. The VOTECH work programme is divided into six separate "Design Study" areas.

DS1 Consortium management Work is co-ordinated on a six monthly cycle, but 180 degrees our of phase with the rest of AstroGrid-2. Board meetings take place annually, and meetings of the Technical Advisory Panel (TAP) every six months

2005 Apr   TAP meeting
2005 Sep   TAP meeting
2005 Nov   Board meeting
2006 Apr   TAP meeting
2006 Sep   TAP meeting
2006 Nov   Board meeting
2007 Apr   TAP meeting
2007 Sep   TAP meeting
2007 Nov   Board meeting

DS2 Technical Project Management This includes the responsibility to produce an overall project plan, a science analysis document, a sequence of software releases, and a final Euro-VO architecture.

2005 Feb   Project Plan 
2005 Sep    Science Framework Document
2006 Jan   Baseline software release-1
2006 Jul   Revised Project Plan
2007 Jan   Baseline software release 2
2008 Jan    Baseline software release 3
2008 Dec    Euro-VO Reference Architecture
2008 Dec   Baseline software release 4

DS3 New Infrastructure

AstroGrid will be leading the new infrastructure design study. The substantive improvements anticipated are therefore exactly as discussed above, with the VOTECH component of AstroGrid-2 peforming technology assessment and prototyping, leading on to mature component construction within the PPARC component of AstroGrid-2. The VOTECH project imposes one additional commitment - a study report in early 2006.

2006 Apr   VOTECH Infrastructure Study Report

DS4 New User Tools

This is understood to mean client side tools, rather than server-side applications which are pursued within DS6. The framework provided by CEA and ACR within AstroGrid makes it relatively easy to adapt tools for the VO, and so we expect a steady stream of these. However, we aim to adapt major packages, such as IRAF, in distinct releases; and to run a community call for new personalised tools, in combination with workshop events.

2006 Feb   Personalised tools Call-1
2006 Jun   Major package release
2007 Feb   Personalised tools Call-2
2007 Apr   VOTECH Tools Study Report   
2007 Jun   Major package release

DS5 Intelligent Resource Discovery

This is the most open-ended but potentially the most exciting part of VOTECH developments. What we have already is standards for resource and service metatadata, registry interfaces for accessing these, and a dictionary of standard terms, the UCDs, which make possible standardised queries. The next steps come in a more structured approach to UCDs building towards ontologies, defining relations between terms, and operations applicable to terms, with the aim of more powerful and flexible queries becoming possible. Once again, we will not be doing any original research here, but rather testing emerging technologies and prototyping. However external developments in ontologies and the semantic web are coming very fast, making this a significant challenge, and leaving the final outcome the most uncertain part of AstroGrid-2.

2007 Jun    First Prototype releases
2007 Sep   VOTECH Resource Discovery Study Report
2007 Dec    Second Prototype releases

DS6 Data Exploration

Provision of server side applications is a key part of delivery of the VO vision, as every user will have access to fast search engines next to the very large databases we expect, and supercomputer analysis engines to produce correlation functions, cluster analyses, and so on. As with client side tools, lots of relevant packages for visualisation and data mining already exist, with the key task being integration into the VO framework. For very heavy duty applications however it is not yet clear the necessary infrastructure is complete, especially in grid-like aspects of bulk data movement and replication, and optimally allocating and re-routing long-lived jobs. We are working towards a concept currently referred to as the "AstroGrid Data Exploration Framework (ADEF)".

2005 Dec   Initial server side applications
2006 Jun   Demonstration of large data volume problem
2006 Sep   VOTECH Data Exploration Study Report
2006 Dec   ADEF Prototype release
2007 Dec    Working ADEF

-- AndyLawrence - 08 Jun 2005 -- AndyLawrence - 02 May 2005