Protocols

A Protocol describes the parameters required to perform a data transfer using a particular protocol.

A Protocol structure has the following members:

• • Protocol. uri : the URI of the transfer protocol, eg: Protocol identifier.
    • ivo://net.ivoa.vospace/protocols/http-get
    • ivo://net.ivoa.vospace/protocols/dime-get
  • Protocol. endpoint : the endpoint for the transfer protocolURL to use for the data transfer.
  • Protocol. param : A list ofThese name-value pairs that specify any additional arguments required to define the transfer operationto transfer the data.

Protocol identifiers

Every new type of Protocol requires a unique URI to identify the Protocol and how to use it.

The rules for the Protocol identifiers are similar to the rules for namespace URIs in XML schema. The only restriction is that it must be a vaild (unique) URI.

• An XML schema namespace identifier can be just a simple URN, e.g. urn:my-namespace
• Within the IVOA, the convention for namespace identifiers is to use a HTTP URL pointing to the namespace schema, or a resource describing it

The current VOSpace schema defines Protocol identifiers as anyURI. The only restriction is that it must be a vaild (unique) URI.

• A Protocol URI can be a simple URN, e.g. urn:my-protocol

This may be sufficient for testing and development on a private system, but it is not scalable for use on a public service.
For a production system, any new Protocols should have unique URIs that can be resolved into to a description of the Protocol.

Ideally, these should be IVO registry URIs that point to a description registered in the IVO registry.

• e.g. ivo://my-registry/vospace/protcols/my-protocol

Using an IVO registry URI to identify Protocols has two main advantages

• IVO registry URIs are by their nature unique, which makes it easy to ensure that different teams do not accidentally use the same URI.
• If the IVO registry URI points to a description registered in the IVO registry, this provides a mechanism to discover how to use the Protocol.

Protocol descriptions

If the URI for a particular Protocol is resolvable, i.e. an IVO registry identifier or a HTTP URL, then it should point to an XML resource that describes the Protocol.

A ProtocolDescription should describe the underlying transport protocol, and how it should be used in this context.

A ProtocolDescription should have the following members:

• uri : the formal URI of the Protocol
• Description : Text block describing describe the underlying transport protocol, and how it should be used in this context
• DisplayName : Display name of the Protocol

However, at the time of writing, the schema for registering ProtocolDescriptions in the IVO registry has not been finalised.

UI display name

If a client is unable to resolve a Protocol identifier into a description, then it may just display the identifier as a text string.

• Download using urn:my-protocol

If the client can resolve the Protocol identifier into a description, then the client may use the information in the description to display a human readable name and description of the Protocol.

• Download using standard HTTP GET

Standard protocols

The VOSpace team intend to register Protocol URIs and ProtocolDescriptions for the core set of standard transport protocols.

e.g.

• Standard HTTP get and put
• Standard FTP get and put
• etc

However, this is not intended to be a closed list, different implementations are free to define and use their own VOSpace transfer Protocols.

Custom protocols

There are several use cases where a specific VOSpace implementation may want to define and use a custom VOSpace transfer Protocol, either extending an existing Protocol, or defining a new one.

SRB use case

One example would be a VOSpace service that was integrated with a SRB system. In order to enable the service to use the native SRB transport protocol to transfer data, the service developers would need to register a new ProtocolDescription to represent the SRB transport protocol.

The ProtocolDescription would refer to the technical specification for the SRB transport protocol, and define how it should be used to transfer data to and from the VOSpace service.

Clients that do not understand the SRB transport protocol would not recongnise the URI for the SRB Protocol, and would ignore Transfer protocol options offerd using this Protocol.

Clients that were able to understand the SRB transport protocol would reconginse the URI for the SRB Protocol, and could use the srb:// endpoint address in a Transfer protocol option to transfer data using this Protocol.

Enabling different gorups to define, register and use their own custom Protocols in this way means that support for new transport protocols can be added to VOSpace systems without requiring changes to the core VOSpace specification.

In this particular example, it is expected that one group within the IVOA will work with the SRB team at SDSC to define and register the Protocol URI and ProtocolDescription for using the SRB protocol to transfer data between VOSpace systems.

Other implementations that plan to use the SRB transport protocol in the same way could use the same Protocol URI and ProtocolDescription to describe data transfers using the SRB transport protocol.

The two implementations could then use the common Protocol URI to negotiate data transfers using the SRB transport protocol.

Local NFS use case

Another example of a custom Protocol use case would to transfer data using the local NFS filesystem within an institute.

If an institute has one or more VOSpace services co-located with a number of data processing applications, all located within the same local network, then it would be inefficient to use HTTP get and put to transfer the data betweeen the services if they could all access the same local filesystem.

In this case, the local system administrators could register a custom ProtocolDescription which described how to transfer data using their local NFS filesystem.

• ivo:///my.institute/vospace/protocols/internal-nfs

Data transfers using this Protocol would be done using file:// URLs pointing to locations within the local NFS filesystem. e.g.

• file:///net/host/path/file

These URLs would only have meaning to services and applicaitions located within the local network, and would not be useable from outside the network.

Services and applications located within the local network would be configured to recognise the custom Protocol URI, and to use local filesystem operations to move files within the NFS filesystem.

Services and applications located outside local network would not recognise the custom Protocol URI and so would not attempt to use the internal file URLs to transfer data.

Note that in this example the custom Protocol URI and the associated ProtocolDescription refer to data transfers using file URLs within a specific local NFS filesystem.

If a different institute wanted to use a similar system to transfer data within their own local network, then they would have to register their own custom Protocol URI and associated ProtocolDescription.

The two different Protocol URIs and ProtocolDescriptions describe how to use the same underlying transport protocol (NFS) in different contexts.

Enabling different gorups to define, register and use their own custom Protocols in this way means that systems can be configured to use the best available transport protocols for transferring data without conflicting with other systems who may be using similar a transport protocol in a different context.

Transfers

A Transfer describes This defines the details of a data transfer to or from a space.

A Transfer The transfer structure has the following members:

• Transfer. view : a View structure specifying the transfer view and data format.
  • The list of available formats for the service can be obtained from the getViews operation.
  • The list of supported formats for a StructuredDataNode can be obtained from the properties of the node – the default is highlighted and will be used when no format is specified in the URI.</s[an>
  • For the transfer to be valid, the specified view must match one of the View elements listed in the accepts or provides list of the Node.

• Transfer. protocol : one or more a Protocol structureelements specifying the available transfer protocols to use.
  • A Transfer may contain more than one Protocol element with different Protocol URIs.
  • A Transfer may contain more than one Protocol element with the same Protocol URI with different endpoints.

Synchronous transfers

Two of the VOSpace transfer methods are synchronous - the service performs the data transfer during the scope of the SOAP call.

In these methods, the client constructs a Transfer request containing details of the Node and View and one or more Protocol elements with valid endpoint addresses.

The service may ignore Protocol elements with Protocol URIs that it does not recognise.

If the server is unable to handle any of the requested Protocols in a Transfer request, then it responds with an exception message.

The order of the Protocol elements indicates the order of preference that the client would like the server to use However, this is only a suggestion, and the server is free to use its own order of preferences to select which Protocol it uses.

The service selects the first Protocol it wants to use from the list of requested Protocols attempts to transfer the data using the Protocol endpoint.

If the first attempt fails, the server may choose another Protocol from the list and re-try the transfer using that Protocol endpoint.

The server may attempt to transfer the data using any or all of the Protocol elements in the Transfer until either, the data transfer succeds, or there are no more Protocol options in the list.

The server is only allowed to use each Protocol option once.

Once one of the Protocol options succeeds the transfer is considered to be completed, and the server is not allowed to use any of the remaining Protocol options. This allows a data source to issue one time URLs for data access, and to cancel any unused URLs once the transfer has been completed.

Some Protocols may require the service to call a callback address when a data transfer completes. This behaviour is specific to the Protocol, and should be defined in the ProtocolDescription.

If none of the Protocol options succeed, then the transfer is considered to have failed, and the service returns an exception containing details of the Protocol options it tried.

Asynchronous transfers

Two of the VOSpace transfer methods are asynchronous - an external actor performs the data transfer outside the scope of the SOAP call.

In these methods, the client sends a template Transfer request to the server.

The Transfer request should contain details of the Node and View and one or more incomplete Protocol elements without endpoint addresses .

In effect, the client is sending a list of Protocols that it (the client) wants to use for the transfer.

The service may ignore Protocol elements with Protocol URIs that it does not recognise.

The service selects the Protocol elements from the request that it is capable of handling, and builds a Transfer response containing the selected Protocol elements filling in valid endpoint addresses for each of them.

In effect, the server is responding with the subset of the requested Protocols that it (the server) is prepared to offer.

If the server is unable to accept any of the requested Protocols, then it responds with an exception message.

On receipt of the response, the client can use the list of Protocols itself, or pass them on to another agent to perform the data transfer on its behalf.

The agent may ignore Protocol elements with Protocol URIs that it does not recognise.

The agent selects the first Protocol it wants to use from the list of Protocols and attempts to transfer the data using the Protocol endpoint.

If the first attempt fails, the agent may choose another Protocol from the list and re-try the transfer using that Protocol endpoint.

The agent may attempt to transfer the data using any or all of the Protocol elements in the Transfer until either, the data transfer succeds, or there are no more Protocol options in the list.

The agent is only allowed to use each Protocol option once.

Once one of the Protocol options succeeds the transfer is considered to be completed, and the agent is not allowed to use any of the remaining unused Protocol options. This allows a data source to issue one time URLs for data access, and to cancel any unused URLs once the transfer has been completed.

Some Protocols may require the agent to call a callback address when a data transfer completes. This behaviour is specific to the Protocol, and should be defined in the ProtocolDescription.

If none of the Protocol options succeed, then the transfer is considered to have failed.

Access control

The access control policy for a VOSpace is defined by the implementor of that space according to the use cases for which the implementation is intended. A space may have any access control policy for its nodes but the policy shall be uniform across all nodes in the space and shall be declared in human-readable form in the registry metadata for the space. A human-readable description of the implemented access policy must be declared in the registry metadata for the space.

These are the most probable access policies.

• No access control is asserted. Any client can create, read, write and delete nodes anonymously.
• No authorization is required, as in policy #1, but clients must authenticate an identity (for logging purposes) in each request to the space.
• Clients may not create or change nodes (i.e. the contents of the space are fixed by the implementation or set by some interface other than VOSpace), but any client can read any node without authentication.
• Nodes are considered to be owned by the user who created them. Only the owner can operate on a node.

No operations to varymodify the access policy (e.g. to set permissions on an individual node) are included in this standard. Later versions may add such operations.

Where the access policy requires authentication of callers, the VOSpace service shall support one of the authentication methods defined in the IVOA Single Sign On profile.

-- DaveMorris - 28 Feb 2007