Streaming Registry Client.

-- PaulHarrison - 21 Aug 2006

The distinction between service metadata in the capability and how to call the service in the interface isn't reflected in the examples for Cone and Siap. Coverage is placed outside the capability element. Metadata such as 'max search radius', 'max file size', 'whether verbosity parameter is accepted' are in the capability not the interface

It seems to me that placing managedApplications in the capability makes CEA more uniform with all the other schema types.

Your example of a SIAP service with a CEA interface is interesting, but seems quite contrived. Embedding a 'cea' interface within a 'siap' capability is just going to make things harder to find. Notice that capability has a standardID attribute - this could either be the standard for 'siap' or for 'cea'. Not both.

I thought it more likely that a registration for a cea server would have a capability that lists the managedApplications, and then 1 or more interface element, which list different versions of the CEA protocol - each of which provides access to the managed applications. -- NoelWinstanley - 22 Aug 2006

I think that the example of a SIAP service with a CEA interface is not contrived, but is actually the central use case for the Capability Interface model - Iook at the Registry Metadata document section 5 quoting, Capability metadata, which describe what the service does, its limitations, and other behavioral characteristics. and Interface metadata, which describe how to access the service—the inputs and the outputs - my contention is that CEA is really just an interface definition - you can tell me what a SIAP server does, but you can only tell me how a CEA server does stuff.

Incidentally - I think that the SIAP description in the test registry is old - compare with the latest example from the IVOA twiki http://www.ivoa.net/internal/IVOA/RegUpgradeSummer2006/sia.xml and see text in http://www.ivoa.net/Documents/WD/ReR/VOResource-20060530.html - but you are right coverage has been placed outside even capability, as there seems to have been created a new overall CatalogService Resource level type - I think that there was a 'historical reason' for doing this - but have forgotten exactly - e.g. see end of http://www.ivoa.net/forum/registry/0606/1666.htm

-- PaulHarrison - 22 Aug 2006

Registry Explorer

The Registry Explorer UI in the workbench now uses the new reg client. The speedup is noticable. In particular, it displays the first results while the later results are still being parsed - and doesn't crash from exhausted memory. From the 150806 snapshot at AstroRuntime, try entering 'galaxy' or 'abell' into the registry explorer (not app launcher - which only searches a subset of the whole registry). You'll see there's an initial wait while the query document is constructed on the server, but after that the results are displayed incrementally in the registry browser.

Streaming Registry Service

So, the most noticable delay in querying at the moment is on the server-side. The current registry implementation builds a DOM of the results before returning them, over SOAP, to the client. This buffering of the results slows down the process, and also makes it likely that the registry server will fail with an out-of-memory error on large queries.

I think it'd be sensible to use XFire to re-implement the registry query service in a streaming manner - then there's be no buffering of results in a DOM - instead the resultset from exist could be iterated through and written straight out. This should improve the speed and reliability of our registry implementation. I think Matthew Graham is looking at using XFire within Carnivore - as he posts to the XFire users mailing list occasionally.

Current Registry gets this ResourceSet^? which is an collection of your results, (but you don't actually get the XML into memorry till you call getResource(int)) in your collection. But since I did not have streaming currently it tends to take this resourceset call this getMembersAsResource method which essentially makes one big XML DOM from the whole collection and return it as you note causing things to be quite slow. The xfire implementation I am working on will take the ResourceSet^? and go through the collection piping it to the Stream connected to a XMLStreamReader^? and what should produce a significant response time. By the way any idea how to actually see/confirm it is streaming back? -- KevinBenson

Caching

Within the registry client, the cache keys are the XQuery string, and the results are the array of Resource object. I use a small memory cache backed by a larger disk cache, and the cache entries are set to expire after 24 hours. Once AG becomes more stable, I may make the expiration period longer (a week even?). I've added an operation to the 'Edit' menu to flush the cache.

This cache gives a significant speedup when the same query is done repeatedly (try querying for 'galaxy' again) - as serialized results are read from local disk, rather than running a new query on the registry service. Another place where this shows is in AstroScope & Helioscope. The lengthy query that lists all Cone services, for example, is only passed to the Registry service on the first run. All other runs that day use the cached copy. This is faster for the user, and reduces the load on the registry service.

I think it'd be sensible to use EhCache^? to cache results on the Registry server too. Reading a serialized result from the cache should be faster than running the query in the Exist xml database. Just a small cache - would speed things up by ensuring the answers to the most common queries (e.g. resolving endpoints for communities, list all cone services, etc) were at hand.

Querying.

The workbench now uses XQuery throughout to query the registry. It's quite hard to develop these queries at the moment (mostly because I'm still learning). It'd help to know some more about how xquery works in Exist.

What version of exist are we running? - where is this information available from.

For development, is there some way of running a query directly on the Galahad Exist? - e.g. some kind of XQuery dialog / console?

Are xqueries passed straight through to the exist database, or does the registry client manipulate / sanitize / transform them in any way?

Our root is actually /Astrogrid/Resource or for 0.10 case it would be /Astrogrid/vor:Resource. The RootResource^? syntax is probably only on the test registry not galahad in which you pass in RootResource^? and it translates it to the correct Resource on the backend.

Straight XQueries via the Service for the most part is straight through except it does one thing wrap an element around the results. Which is this is because it does all the soap itself.

If you were at galahad then there are some ways of querying directly at eXist and if you could login to galahad there is a commandline console. There is also a xmlrpc client but it would be hidden if your not at leicester. -- KevinBenson

Indexing

For example, doing a string match search using the standard XQuery 'contains' proved to be very slow. Using the '&=' operator is much faster. However, '&=' is an exist extension - not part of the XQuery language. I think it's important to keep the workbench free of implementation-specific details - so that someone else could connect it to a different registry implmenetation, etc. Would indexes, or different ways of arranging the resources, speed this up?

Collection Root

It's not documented how to refer to the root of the registry document collection. I've tried /vor:Resource which doesn't work. The new Registry Standard says that the collection root is called RootResource. I'm unable to make that work either. At the moment, all the xqueries start with //vor:Resource (which means 'match a Resource element, anywhere, at any depth in the hierarchy). This is very inefficient - as it needs to search throughout every registry record for a match. What is the collection root?

Summary

1. Documentation on how to write efficient queries
2. Indexing
3. Caching
4. Streaming

Other Questions

Has an endpoint been published yet for the Registry Of Registries? Or is there a temporary version somewhere I can take a look at. In particular, I'd like to know if the query interface for a registry is listed in the RoR^?, or is it just the harvesting interface?

Schema evolution

I'm concerned with the evolution of the schemata in respect of CEA.

First, I agree with Paul'sNoel's point above concerning CEAService v1.0. We should have a CEA capability instead of mutating the interface element.

Second, assuming that we want to end up with CEAService 1.0, with new structure to exploit capability, we should be careful about how we get there. If we make it one big change then lots of AG software has to change to suit. If we have an intermediate step, CEAService 0.3 say, which is the existing record type refactored onto VOResource 1.0, then we have an easier transition. It should be easier (trivial?) to deconstruct this in the AR, because it's the same structure in a new namespace. If it happens that all consumers of VOService just look for the ManagedApplications^? element and ignore the container then this intermediate form does help as much.

Third, as ever, I urge everybody not to issue sucessive versions of a schema without changing the namespace URI.

Fourth, if I understand correctly, CEAService is still internal to AG, even if its namespace URI starts with http://ivoa.net/. Therefore, we can have as many intermediate versions as we need to get it right before beatification in the IVOA. It's not subject to Roy's "one more schema change ever" fiat. What we have isn't very 1.0 yet.

-- GuyRixon - 22 Aug 2006

The evolution of the CEA schema has been made a IVOA level issue - it is on the Registry WG Roadmap http://www.ivoa.net/twiki/bin/view/IVOA/InterOpMay2006ResReg#Session_6_Roadmap - however, I agree that the only sensible approach for AG to take is to develop a CEAService 0.3 that has a smaller number of changes to bridge the gap.

- PaulHarrison - 22 Aug 2006