VoSpace

The original VoSpace was based on the concept of a "virtual file system", which implied a distributed storage system with some form of hierarchical directory structure.

VoStore

Discussion on the specification for VoSpace was stalled at an abstract level, due to different expectations and perceptions of what a VoSpace system actually needed to do.

In an effort to reach a concensus, we split the specifications into two parts, similar to the two MySpace services currently implemented in AstroGrid, FileManager and FileStore .

• VoSpace , which would represent the higher level abstraction of the virtual file stsrem

• VoStore , which would implement the lower level storage nodes

VoSpace services would provide the metadata management functions required to maintain the virtual file system view of the distributed storage system. VoStore services would provide the lower level storage of objects.

The VoSpace service(s) would provide the high level hierachical view of the users data, and VoStore services would provide a flat, non-hierarchical, view of the objects in the each individual VoStore .

Each user would be allocated a 'home directory' in a VoSpace service, which represented the top of the tree of their 'virtual space'. Giving them a central point to access all of their files in the virtual space.

Users (or client software) would interact with the VoSpace service, asking it to import and export data from nodes in the VoSpace tree. The VoSpace service would act as a broker service, talking to the individual VoStore services on the users behalf.

Users would access data in the virtual space using paths and file names that represented the node location of the files in the VoSpace tree, e.g. /mydata/results.vot.

The VoSpace service would keep track of the actual VoStore location of a file in the node metadata and contact the relevant VoStore service to perform the required action.

However, during the discussions leading up to the Ivoa InterOp conference at Kyoto, the functional separation between the VoSpace and VoStore services gradually reduced.

In order to avoid problems with unique naming of files, the original VoStore specification only dealt with server allocated unique identifiers e.g. 23657Bd-6657A.

The VoSPace service would handle use allocated file paths and names e.g. /mydata/results.vot and the VoSpace service would maintain the mapping between the user allocated name and the actual VoStore identifiers.

/mydata/results.vot => object 23657Bd-6657A on ivo://filestore-xyz

This relied on the user (or client software) only interacting with the VoSpace service, and did not interact directly with the underlying VoStore services. The user could refer to a file by the file name and path, e.g. /mydata/results.vot, and the VoSpace service would handle the mapping between the VoSpace file path and the physical VoStore identifier.

During the discussions, it became clear that we wanted the client software to be able to interact directly with the VoStore services, without using a VoSpace service to handle the filename mapping. This means that the VoStore services will need to handle user allocated names and paths e.g. /mydata/results.vot.

As a result, the current the VoStore specification requres the VoStore to be able to handle user allocated file names, which may include '/' file path characters e.g. /mydata/results.vot. However, the VoStore does not need to implement any additional functionality to support the impled directory nodes e.g. list all files in the /mydata/ directory.

This represents something of a half way house between the original intent of the VoStore and VoSpace specifications.

Allowing user defined file names in the VoStore service means that the service must keep a separation between files with the same names stored for different users.

• User 'fred' can have a file called /mydata/results.vot.
• User 'john' can have a file called /mydata/results.vot.

The current plan is that the VoStore service will know the identity of the user account that is trying to access a file, using the IVOA SingleSignOn tools currently being developed.

Although we can use the identity from the SSO component to assign the file owner, we will need some form of URI syntax which enables us to identify individual files in a VoStore . The eventual aim of VoSpace /VoStore is to allow users to share files within VoSpace . In which case, we will need to provide a mechanism for john to be able to refer to, and access, the data in fred's /mydata/results.vot file.

Within AstroGrid, this is handled by the FileManager (VoSpace ) component in collaboration with the Community which supplies the user account identifiers.

If we want to support user defined filenames at the VoStore level, without using a VoSpace service, then we will need to provide a uniform way of referring to individual files across user account boundaries.

The VoStore specification prior to the Kyoto conference had two MIME attachment methods, get and put, plus four data transfer methods.

get - Get data in message.

• Client sends the name or identifier of the file it wants.
• Service replies with file contents as a MIME attachment in the SOAP message.

put - Send data in message.

• Client sends (optional) filename it wants to use.
• Client sends file contents as a MIME attachment in the SOAP message.
• If the client does not supply a filename, the service will allocate a new identifier for the file.
• Service replies with the filename and status result of the transfer.

importInit - Request to send data from the client.

• Client sends (optional) filename it wants to use.
• If the client does not supply a filename, the service will allocate a new identifier for the file.
• Service replies with the filename and a URL that the client should send the data to.
• If client supplied a callback address, the service sends an update message when the transfer completes.

importData (synchronous) - Import data from a remote location.

• Client sends the filename it wants to use, and the URL of where to get the data from.
• If the client does not supply a filename, the service will allocate a new identifier for the file.
• The service fetches the data from the remote location.
• Service replies with the filename, and set of properties for the file and the transfer.

importData (asynchronous) - Import data from a remote location.

• Client sends the filename it wants to use, and the URL of where to get the data from.
• Client sends (optional) callback address.
• If the client does not supply a filename, the service will allocate a new identifier for the file.
• The service schedules the transfer to occur after the method completes.
• Service replies with the filename, and set of properties for the file and the transfer.
• If client supplied a callback address, the service sends an update message when the transfer completes.

exportInit - Request to get some data from the store.

• Client sends the name or identifier of the file it wants.
• Server replies with a URL indicating where the client can download the data from.

exportData (synchronous) - Send some data to a remote location.

• Client sends the filename it wants to transfer, and the URL of where to send the data to.
• The service sends the data to the remote location.
• Service responds with results of the transfer.

exportData (asynchronous) - Send some data to a remote location.

• Client sends the filename it wants to transfer, and the URL of where to send the data to.
• Client sends (optional) callback address.
• Service schedules the transfer to occur after the method completes.
• Service replies with set of properties for the file and the transfer.
• If client supplied a callback address, the service sends an update message when the transfer completes.

list - List the files for a user account.

• Client sends user identifier or (null or '*') for all users.
• Service responds with a list of the matching files.

delete - Delete a file from the store.

• Client sends file name or identifier of the target file.
• Service responds status.

At the Kyoto conference, there was a suggestion that all of the transfer methods could be implemented in just two SOAP methods, get and put. Using xsi:type attributes to handle the different message structures for the direct and indirect transfers.

At the time, this seemed like a GoodIdea(tm). However, subsequent investigations suggest that this might not be the best way to handle things.

Firstly, the main objection to the four transfer methods seemed to be that the method names did not clearly represent the distinction between the direct and indirect transfers. Using xsi:type attributes in the SOAP messages just moves the naming problem from the SOAP methods to the the XML element schema. We would still need to have descriptive names to for the different transfer methods.

Secondly, combining everything into just get and put means that the SOAP methods would have to handle information for both the direct and indirect transfers, including the option to transfer using SOAP MIME attatchments and the synchronous and asynchronous transfer options.

I think this will complicates the message syntax more that is necessary, and it may be better to leave them as separate methods, at least for now. Based on this, I would suggest we treat each of the methods separately, and implement each of them one step at a time. If we really want to wrap them all up into two complex methods we can do this later, once we have the core service functionality in place.

-- DaveMorris - 26 Jul 2005

Detailed functional definition

TBD

SOAP message schema

TBD