Iteration 2 Focus Meeting(Day 2)

Registry – Elizabeth Auden

Registry and Astro MQ Overview

Portal makes contact with 4 types of services:

• DataArchive^?
• Data Storage
• Data Transformation
• Distributed Network of Registries

Expection with Registry Web Services communicate with Registry.

Technology used: Web Services(soap), DB2, Xpath/Xquery(XML Parsing)

registry services perform query parsing ad query processing

*Registry use cases implemented through different SQL queries, not different service methods.

Portal talk to registry(db2) using web services registry response to portal using Xquery)

AstroMQ -- XML formated messages to be queued or stored Technology: XML Blaster Message Queue: queue job, user and community message Message Log (queue with database persistence): store job, user, and community messages

Questions:

• Tony: whether we still do need MQ??? We Do need message log.

• JMS via XMLBlaster

• Use of JDBC queries.

Use Cases: Registry

1. Look up CommunityMQ^?
2. Look up database...
3. Stor messages
4. Look up locations
5. look up dataset details
6. Receive and parse queries
7. build and send xml
8. registry database query

MQ interface where user can talk to the queue. MQ send messages asynchronizly.

XMLBlaster inter process communicate, logs is for the user

Tony: job control component, job schedule... -- in same machine, but job controller and job schedule ...and data center will be on different server that's why we need MQ to assure the delivery is successful.

Use MQ to write into message log. Logs are persistent.

Datacenter send data to job controller ... and send info to logs as well.

MQ is a Technology, look up CommunityMQ^? Message log build by registry team.

The registry contains the resource for every server each resource has meta data associated with it.

Registry never returns the actual data set it returns a point to the sets. In order to get the full sets, the portal will have to go and get things like UCD(Unified Content Descriptors)...

Discussion: Different between 2.Look up database and 5.Look up database details: using same query – from same place but for different perpose, Elizabeth to make it more clear.

Data Mover – talks to mySpace registry, but for iteration 2, we are just moving between servers. Implementation will be the same.

Use Cases AstroMQ:

1. Store Messages
2. Look up Messages
3. Look up user messages

Interfaces: Provide

1. Service Registries
   1. Portal
   2. Workflow
   3. Job Control
   4. DataMover^?
   5. Job Scheduler
2. AstroMQ

Interfaces:Consume

1. Service Registry
   1. Portal
   2. Workflow
   3. Job Control
   4. DataMover^?
   5. Job Scheduler – Elizabeth to remove
2. AstroMQ
   1. job control
   2. Database
   3. VOTable Catenation
   4. Job Monitor
   5. Portal

Look up messages: only portal. Job control look data from resource registry, because job control component will have to arrange resource.

Registry Class Diagram: Metod: xmlToSQL, sqlToXML, parseQuery, ProcessQurey^?

AstroMQ Class Diagram: Methods: storeMessage, lookupMessage -- Queue manager at both ends, massage manager gets the message,

-- Component writes to MQ and MQ stores in Database Can you directly write MQ to database?

Developing date: 23/04 – 06/06

Registry WSDL

Discussion: Shot delivery: use MQ systems, message will get into queue eventfully. XmlBlaster^? will guarantee messages being delivered. Database choosing: DB2? Oracle?

Nothing will take things off the logs only expend longer and longer. Use web service of Logs to read logs response as xml formate. Message log format and attributes: userid, taskid, date, time.....

Portal – David Giaretta

Use Cases http://wiki.astrogrid.org/bin/view/Astrogrid/PortalIteration2Design

astroticket: is certificate – not doing in this iteration

Technology: html – jsp --- dhtml? Avoiding applet.

Clarify target Browser to the user:

• Mozilar1.0 plus
• Netscape 6 plus?
• IE 6 ...

Questions: Guy: DataCentre^? or DataService^???

Tony: Workflow GUI

Dataset should be multiple selection, AQL is not specified. Compare with VizieR^? interface: http://vizier.u-strasbg.fr/cgi-bin/VizieR

based on UCDs: http://cdsweb.u-strasbg.fr/viz-bin/UCDs

Registry should support queries including whether these are the valid UCDs?

Does every UCDs support same cone search?

Decision: several boxs contains different datasets submit combine the multiple selections.

Boxs: Type, WareLength^?, Keywords,... and then clik find UCDs, select one UCD, in order to build one single query against one or more databases.

Registry will hold meta data for catalogues which grows over time. Interface will be IVOA.

To summurize: Portal will send identifier(after user made the multi selections) as Xpath to Registry and Registry returns required/mached UCDs to portal as XML. Portal then ask registry with selected meta data set and registry returns back to portal, portal then send them to job controller.

Registry only returns meta data.

Portal people will make mock ups of sequence diagram/UI to submit the above.

Data Centre -- Jeff Lusted

http://wiki.astrogrid.org/bin/view/Astrogrid/AgCd08DataCentre

One query to one data set in iteration 2.

• Submit job is the only external interface and it is a web service job.

• Job controller, job scheduler and job monitor runs in parallel.

• Communicate between them is not been decide.

• Logged message should be able to viewed by the user.

Questions:

• who to split the job? Workflow or job controller? This iteration workflow engine to start jobs.

• Job steps – where is coming from?

• Negotiate cache space – how big?-- get permission to write to a directory.

Job status is the communicate point. Job status is a message queue but not decided yet.

Consider IVOA standards for the queries.

Use Cases:

If fail to wake up scheduler pass back the xml file with some error code. Server manager should keep trying to start job scheduler.

Questions:

• Should Job identifier returns (end of submission) after step 3 (Create job centre)?

• Submit job – get job identifier – return

• If fails to wake up job scheduler where should it return?

• Job scheduler act as an demon – what is it? Servlet?

• DataCentre^?: one per community or job controller?

There will be few alternate flow after failing, for iteration 2 we pick one thing to do.

ToDo: Change term “wake up”, how many job scheduler will there be?

Decistion:

For submit job, take out entry 4(wake up scheduler) and 5(insert log messages).

Guy: Performance issues have to be tested and decided...

Cancel job: different cases may occur after running for a long time.

priority of the job: not in this iteration.

Should be able to select more then one job.

Interface is between submit query and run query.

Data Centre --Clive Page (Continued)

Query Language proposal for ad-hoc

cone-search

Send one query to several VOTables or send to job controller.

No joins, no sub-selects, no sorting or grouping.. at present.

ToDo: Will be on forum. Link.

Select and same time.

Possible datasets

DB2 will produce xml tables.

APM catalogues – Cambridge

6df galaxy survey - Edinburgh USNO-B - Leicester

STP datasets - RAL

USNO-B - Leicester

SuperCSMOS^? Edinburgh

Vizier Collection – Leicester

Decision: Datacentre must implement a Web Service catalogues. One of the most useful scientific cases. ToDo: open forum link..

DB2 and xml try not to duplicate out put.

Question:

How to set up an asynchronous service?

Code runs on database captures VOTable should be added into the web service.

Chris H. Perry's Slides

ToDo: Clive to put on wiki after granding permission from Chris, Clive to email powerpoint slides showed.

-- CatherineQin - 16 Apr 2003