r2 - 16 Nov 2005 - 18:25:04 - DaveMorrisYou are here: TWiki > 
Astrogrid Web > DaveMorris > VoSpaceDIME
Astrogrid Web > DaveMorris > VoSpaceDIMEIndirect DIME transfer
Author- DaveMorris - 15 Nov 2005
- GuyRixon
- Matthew Graham (Caltech)
- PaulHarrison (ESO)
Current implementation
DIME put
In the current VoStore? specification, a client can send data directly to a VoStore? by calling the DIME put method on the store.
Client VoStore VoSpace
|
| DIME put
\------------------------------\
[Auth : x509 cn=user.. ] |
[Data : .............. ] |
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=user.. ]*
*[Data : ................ ]*
*[Size : xxxx ]*
This makes it easy for a client to send data to the VoStore?.
However, because the data is included with the control message, it is difficult to redirect
the messages via a VoSpace? layer service.
If we want the new data to appear in the users VoSpace? tree, then we need to tell
the VoSpace? service about the data.
The simplest implementation would be to send the put message to the user's
VoSpace? service.
The VoSpace? service creates the corresponding metadata node, and then forwards the
message on to the relevant VoStore? service.
Client VoStore VoSpace
|
| DIME put
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Data : ............... ] |
|
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
|
DIME put |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| [Data : ............... ]
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
*[Data : ................ ]*
*[Size : xxxx ]*
|
\---------------------------------\
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
*[File : ivo://vostore/.. ]*
However, this means that all of the data is sent through the VoSpace? service.
If one VoSpace? service is acting as a manager for a large number of VoStore? service,
then this will place a heavy load on the VoSpace? service.
Indirect (HTTP put)
In order to support indirection via a VoSpace? layer, the current VoStore? specification includes methods for performing an indirect transfer using HTTP put. This splits the transfer into two stages. First, the client contacts the VoStore? and asks to initiate an import.
Client VoStore VoSpace
|
| importInit
\------------------------------\
[Auth : x509 cn=user.. ] |
[Protocol : http.put ] |
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=user ]*
|
/------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
The VoStore? service replies with a URL that the client can send the data to.
The client can then transfer the data using the URL supplied by the VoStore? service.
Client VoStore VoSpace
|
| importInit
\------------------------------\
[Auth : x509 cn=user.. ] |
[Protocol : http.put ] |
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=user ]*
|
/------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
|
| HTTP put
\------------------------------\
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=user ]
*[Data : ................ ]*
*[Size : xxxx ]*
Using a two stage process to transfer the data means that the control information and data
are transferred in separate messages.
This makes it much easier to integrate the VoStore? service with a VoSpace? metadata layer.
In this system, the client would send the initial request to the VoSpace? service rather than
direct to the VoStore? service.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Protocol : http.put ] |
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
The VoSpace? service creates the metadata node for the new item, and then calls the VoStore?
service to initiate the transfer.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Protocol : http.put ] |
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
|
importInit |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| [Protocol : http.put ]
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
The VoStore? service creates a new container for the data and replies with a the file identifier
and a URL to send the data to.
The VoSpace? service makes a note of the file identifier, and then passes the information back the client.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
importInit |
/---------------------------------/
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
|
\---------------------------------\
[File : ivo://vostore/.. ] |
[URL : http://vostore/.. ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
*[File : ivo://vostore/.. ]*
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
The client can then use the URL to transfer the data directly into the VoStore? service.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
|
| HTTP put
\------------------------------\
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=user ]
*[Data : ................ ]*
*[Size : xxxx ]*
At this point, the VoSpace? service knows where the data should be, but it does not know if the data has arrived yet.
In order to complete the sequence, the VoStore? service needs to call the VoSpace? service to notify it that
the data has arrived, and update the metadata with things like the data size etc.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
|
/----------------------------------------------------------------/
|
| HTTP put
\------------------------------\
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=user ]
*[Data : ................ ]*
*[Size : xxxx ]*
|
| ImportDone
\---------------------------------\
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
*[Size : xxxx ]*
*[MD5 : ################ ]*
The full sequence looks like this.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Protocol : http.put ] |
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
|
importInit |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| [Protocol : http.put ]
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
|
\---------------------------------\
[File : ivo://vostore/.. ] |
[URL : http://vostore/.. ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
*[File : ivo://vostore/.. ]*
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
|
| HTTP put
\------------------------------\
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=user ]
*[Data : ................ ]*
*[Size : xxxx ]*
|
| ImportDone
\---------------------------------\
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
*[Size : xxxx ]*
*[MD5 : ################ ]*
This is how the current AstroGrid FileManager? (VoSpace?) and FileStore? (VoStore?) implementations
work.
However, there is an inherent security problem in using HTTP put to transfer the data into the VoStore?.
As yet, we have not found a simple, interoperable, way of adding secure authentication to a standard
HTTP put transfer.
Which means that at the moment, anyone can send data to the URL supplied by the VoStore? service and
overwrite the contents of the container.
Indirect (DIME put)
As part of the work on the current VoStore?, each of the groups involved have already implemented a secure write mechanism, using signed SOAP messages with DIME attachments. In which case, it makes sense to use parts of the existing implementations to see if we can implement a secure indirect transfer using DIME attachments. The initial stages of the sequence would be identical to the HTTP put outlined above. However, when the client sends theimportInit() request, it specifies SOAP.DIME
as the transport protocol.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
*[Protocol : soap.dime ]* |
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
|
importInit |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| *[Protocol : soap.dime ]*
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
In response, the VoStore? service would reply with the URL of a SOAP webservice that can receive
a DIME put message, which the VoSpace? service passes back to the client.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
importInit |
/---------------------------------/
|
\---------------------------------\
[File : ivo://vostore/.. ] |
*[URL : http://vostore/.. ]* |
*[Protocol : soap.dime ]* |
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
| [Protocol : soap.dime ]
The client can then use the SOAP endpoint to send the data as a DIME attachment.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
| [Protocol : soap.dime ]
|
| DIME put
\------------------------------\
[Auth : x509 cn=user ] |
[File : ivo://vostore/.. ] |
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
*[Data : ................ ]*
*[Size : xxxx ]*
Once the data has been transferred, the VoStore? service needs to call the VoSpace? service
to notify it of the completed transfer and update the VoSpace? metadata.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
/----------------------------------------------------------------/
|
| DIME put
\------------------------------\
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
*[Data : ................ ]*
*[Size : xxxx ]*
|
| ImportDone
\---------------------------------\
[Auth : x509 cn=vostore ] |
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
*[Size : xxxx ]*
*[MD5 : ################ ]*
Problems to be solved
At this point we still have a number of problems with the authentication of the DIMEput
and importDone() messages.
The file in the VoStore? service was created by the VoSpace? service, which sets the file ownership to
be the identity of the VoSPace? service. Which is what we want.
However, the DIME put call to send the data is comming direct from the client, and so will be authenticated
with the user's certificate and identity.
In order to allow the user to send the data to the file in the VoStore?, we need to add an additional
field to the importInit() message to enable the VoSpace? service to grant write access
to the user's x509 certificate.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Protocol : soap.dime ] |
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
|
importInit |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| *[Write : x509 cn=user ]*
| [Protocol : soap.dime ]
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
*[Write : x509 cn=user ]*
If the VoStore? service allows write access to the specified identity, then the client can
send the data as a DIME attachement, using the user's x509 identity to authenticate the
transfer.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
/----------------------------------------------------------------/
|
| DIME put
\------------------------------\
*[Auth : x509 cn=user ]* |
[File : ivo://vostore/.. ] |
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
*[Write : x509 cn=user ]*
[Data : ................ ]
[Size : xxxx ]
This means adding a slightly more complex permission handling to the VoStore? than was currently intended.
We also have a similar problem with ownership of the VoSpace? metadata node.
The node was originally created by a call from the client, using the user's x509 identity,
so logically, VoSpace? should restrict write access of the node to the user's identity.
However, until we are able to handle delegated proxy certificates, the update call from the VoStore? service
to the VoSpace? service can only be authenticated using the VoStore? service's identity.
Client VoStore VoSpace
|
| DIME put
\------------------------------\
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
[Data : ................ ]
[Size : xxxx ]
|
| ImportDone
\---------------------------------\
*[Auth : x509 cn=vostore ]* |
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
[Size : xxxx ]
[MD5 : ################ ]
In order to enable the VoStore? to update the node status in the VoSpace? service, and still limit
write access to the node to general public, we may need to add an additional field to the node
metadata, listing the other identities that are allowed to update the node e.g. the VoStore? service.
In order to avoid having to make the VoSpace? use a lookup mechanism to find the x509 identity of
the VoStore? service, this information can be passed back in the response to the original
importInit() call from VoSpace? to VoStore?.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
importInit |
/---------------------------------/
|
\---------------------------------\
[File : ivo://vostore/.. ] |
[URL : http://vostore/.. ] |
[Protocol : soap.dime ] |
*[Store : x509 cn=vostore ]* |
|
Node
*[Owner : x509 cn=user ]*
*[Write : x509 cn=vostore ]*
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
|
/----------------------------------------------------------------/
|
This would then enable the VoStore? service to authenticate using its own identity when it makes the
importDone() call back to the VoSpace?.
Client VoStore VoSpace
|
| DIME put
\------------------------------\
|
| ImportDone
\---------------------------------\
*[Auth : x509 cn=vostore ]* |
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Owner : x509 cn=user ]
*[Write : x509 cn=vostore ]*
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
[Size : xxxx ]
[MD5 : ################ ]
In order to prevent the user from modifying or deleting the file, we may need to change this to a 'write once' permission, preventing the user from modifying the file once it has been imported. However, if we have already implemented the
importDone() callback from the VoStore? to the VoSpace?
which is called when the data import has completed, then we could possibly use a similar mechanism to update
the VoSpace? service whenever the user modifies the file in the VoStore?.
There are another two options which I haven't worked through yet: Setting the initial file ownership to the user's identity, and then using the adopt mechnism described earlier to transfer ownership to the VoSpace? service once the data transfer has completed. Probably very messy and overly complicated. Using proxy certificates and authenticating all of the actions using the user's own identity. This would mean that both the file and node were owned by the user's identity and not by the VoSpace? service. This does allow the user the ability to modify the file, but as I mention above, with some form of
fileModified() callback in place, this may not be a problem. |
Click here for the AstroGrid Service Web |
This is the AstroGrid Development Wiki |
|
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors. Ideas, requests, problems regarding TWiki? Send feedback