Indirect DIME transfer
Author- DaveMorris - 15 Nov 2005
- GuyRixon
- Matthew Graham (Caltech)
- PaulHarrison (ESO)
Current implementation
DIME put
In the current VoStore specification, a client can send data directly to a VoStore by calling the DIME put method on the store.
Client VoStore VoSpace
|
| DIME put
\------------------------------\
[Auth : x509 cn=user.. ] |
[Data : .............. ] |
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=user.. ]*
*[Data : ................ ]*
*[Size : xxxx ]*
This makes it easy for a client to send data to the VoStore .
However, because the data is included with the control message, it is difficult to redirect
the messages via a VoSpace layer service.
If we want the new data to appear in the users VoSpace tree, then we need to tell
the VoSpace service about the data.
The simplest implementation would be to send the put message to the user's
VoSpace service.
The VoSpace service creates the corresponding metadata node, and then forwards the
message on to the relevant VoStore service.
Client VoStore VoSpace
|
| DIME put
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Data : ............... ] |
|
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
|
DIME put |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| [Data : ............... ]
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
*[Data : ................ ]*
*[Size : xxxx ]*
|
\---------------------------------\
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
*[File : ivo://vostore/.. ]*
However, this means that all of the data is sent through the VoSpace service.
If one VoSpace service is acting as a manager for a large number of VoStore service,
then this will place a heavy load on the VoSpace service.
Indirect (HTTP put)
In order to support indirection via a VoSpace layer, the current VoStore specification includes methods for performing an indirect transfer using HTTP put. This splits the transfer into two stages. First, the client contacts the VoStore and asks to initiate an import.
Client VoStore VoSpace
|
| importInit
\------------------------------\
[Auth : x509 cn=user.. ] |
[Protocol : http.put ] |
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=user ]*
|
/------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
The VoStore service replies with a URL that the client can send the data to.
The client can then transfer the data using the URL supplied by the VoStore service.
Client VoStore VoSpace
|
| importInit
\------------------------------\
[Auth : x509 cn=user.. ] |
[Protocol : http.put ] |
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=user ]*
|
/------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
|
| HTTP put
\------------------------------\
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=user ]
*[Data : ................ ]*
*[Size : xxxx ]*
Using a two stage process to transfer the data means that the control information and data
are transferred in separate messages.
This makes it much easier to integrate the VoStore service with a VoSpace metadata layer.
In this system, the client would send the initial request to the VoSpace service rather than
direct to the VoStore service.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Protocol : http.put ] |
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
The VoSpace service creates the metadata node for the new item, and then calls the VoStore
service to initiate the transfer.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Protocol : http.put ] |
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
|
importInit |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| [Protocol : http.put ]
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
The VoStore service creates a new container for the data and replies with a the file identifier
and a URL to send the data to.
The VoSpace service makes a note of the file identifier, and then passes the information back the client.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
importInit |
/---------------------------------/
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
|
\---------------------------------\
[File : ivo://vostore/.. ] |
[URL : http://vostore/.. ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
*[File : ivo://vostore/.. ]*
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
The client can then use the URL to transfer the data directly into the VoStore service.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
|
| HTTP put
\------------------------------\
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=user ]
*[Data : ................ ]*
*[Size : xxxx ]*
At this point, the VoSpace service knows where the data should be, but it does not know if the data has arrived yet.
In order to complete the sequence, the VoStore service needs to call the VoSpace service to notify it that
the data has arrived, and update the metadata with things like the data size etc.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
|
/----------------------------------------------------------------/
|
| HTTP put
\------------------------------\
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=user ]
*[Data : ................ ]*
*[Size : xxxx ]*
|
| ImportDone
\---------------------------------\
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
*[Size : xxxx ]*
*[MD5 : ################ ]*
The full sequence looks like this.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Protocol : http.put ] |
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
|
importInit |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| [Protocol : http.put ]
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
|
\---------------------------------\
[File : ivo://vostore/.. ] |
[URL : http://vostore/.. ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
*[File : ivo://vostore/.. ]*
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
|
| HTTP put
\------------------------------\
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=user ]
*[Data : ................ ]*
*[Size : xxxx ]*
|
| ImportDone
\---------------------------------\
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
*[Size : xxxx ]*
*[MD5 : ################ ]*
This is how the current AstroGrid FileManager (VoSpace ) and FileStore (VoStore ) implementations
work.
However, there is an inherent security problem in using HTTP put to transfer the data into the VoStore .
As yet, we have not found a simple, interoperable, way of adding secure authentication to a standard
HTTP put transfer.
Which means that at the moment, anyone can send data to the URL supplied by the VoStore service and
overwrite the contents of the container.
Indirect (DIME put)
As part of the work on the current VoStore , each of the groups involved have already implemented a secure write mechanism, using signed SOAP messages with DIME attachments. In which case, it makes sense to use parts of the existing implementations to see if we can implement a secure indirect transfer using DIME attachments. The initial stages of the sequence would be identical to the HTTP put outlined above. However, when the client sends theimportInit() request, it specifies SOAP.DIME
as the transport protocol.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
*[Protocol : soap.dime ]* |
Node
*[Node : ivo://vospace/.. ]*
*[Path : /home/path/file ]*
|
importInit |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| *[Protocol : soap.dime ]*
|
File
*[File : ivo://vostore/.. ]*
*[Owner : x509 cn=vospace ]*
In response, the VoStore service would reply with the URL of a SOAP webservice that can receive
a DIME put message, which the VoSpace service passes back to the client.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
importInit |
/---------------------------------/
|
\---------------------------------\
[File : ivo://vostore/.. ] |
*[URL : http://vostore/.. ]* |
*[Protocol : soap.dime ]* |
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
| [Protocol : soap.dime ]
The client can then use the SOAP endpoint to send the data as a DIME attachment.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
/----------------------------------------------------------------/
| [File : ivo://vostore/.. ]
| [URL : http://vostore/.. ]
| [Protocol : soap.dime ]
|
| DIME put
\------------------------------\
[Auth : x509 cn=user ] |
[File : ivo://vostore/.. ] |
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
*[Data : ................ ]*
*[Size : xxxx ]*
Once the data has been transferred, the VoStore service needs to call the VoSpace service
to notify it of the completed transfer and update the VoSpace metadata.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
/----------------------------------------------------------------/
|
| DIME put
\------------------------------\
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
*[Data : ................ ]*
*[Size : xxxx ]*
|
| ImportDone
\---------------------------------\
[Auth : x509 cn=vostore ] |
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
*[Size : xxxx ]*
*[MD5 : ################ ]*
Problems to be solved
At this point we still have a number of problems with the authentication of the DIMEput
and importDone() messages.
The file in the VoStore service was created by the VoSpace service, which sets the file ownership to
be the identity of the VoSPace service. Which is what we want.
However, the DIME put call to send the data is comming direct from the client, and so will be authenticated
with the user's certificate and identity.
In order to allow the user to send the data to the file in the VoStore , we need to add an additional
field to the importInit() message to enable the VoSpace service to grant write access
to the user's x509 certificate.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
[Auth : x509 cn=user ] |
[Path : /home/path/file ] |
[Protocol : soap.dime ] |
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
|
importInit |
/---------------------------------/
| [Auth : x509 cn=vospace ]
| *[Write : x509 cn=user ]*
| [Protocol : soap.dime ]
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
*[Write : x509 cn=user ]*
If the VoStore service allows write access to the specified identity, then the client can
send the data as a DIME attachement, using the user's x509 identity to authenticate the
transfer.
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
/----------------------------------------------------------------/
|
| DIME put
\------------------------------\
*[Auth : x509 cn=user ]* |
[File : ivo://vostore/.. ] |
[Data : ............... ] |
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
*[Write : x509 cn=user ]*
[Data : ................ ]
[Size : xxxx ]
This means adding a slightly more complex permission handling to the VoStore than was currently intended.
We also have a similar problem with ownership of the VoSpace metadata node.
The node was originally created by a call from the client, using the user's x509 identity,
so logically, VoSpace should restrict write access of the node to the user's identity.
However, until we are able to handle delegated proxy certificates, the update call from the VoStore service
to the VoSpace service can only be authenticated using the VoStore service's identity.
Client VoStore VoSpace
|
| DIME put
\------------------------------\
|
File
[File : ivo://vostore/.. ]
[Owner : x509 cn=vospace ]
[Data : ................ ]
[Size : xxxx ]
|
| ImportDone
\---------------------------------\
*[Auth : x509 cn=vostore ]* |
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
[Size : xxxx ]
[MD5 : ################ ]
In order to enable the VoStore to update the node status in the VoSpace service, and still limit
write access to the node to general public, we may need to add an additional field to the node
metadata, listing the other identities that are allowed to update the node e.g. the VoStore service.
In order to avoid having to make the VoSpace use a lookup mechanism to find the x509 identity of
the VoStore service, this information can be passed back in the response to the original
importInit() call from VoSpace to VoStore .
Client VoStore VoSpace
|
| importInit
\----------------------------------------------------------------\
|
importInit |
/---------------------------------/
|
\---------------------------------\
[File : ivo://vostore/.. ] |
[URL : http://vostore/.. ] |
[Protocol : soap.dime ] |
*[Store : x509 cn=vostore ]* |
|
Node
*[Owner : x509 cn=user ]*
*[Write : x509 cn=vostore ]*
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
|
/----------------------------------------------------------------/
|
This would then enable the VoStore service to authenticate using its own identity when it makes the
importDone() call back to the VoSpace .
Client VoStore VoSpace
|
| DIME put
\------------------------------\
|
| ImportDone
\---------------------------------\
*[Auth : x509 cn=vostore ]* |
[File : ivo://vostore/.. ] |
[Size : xxxx ] |
[MD5 : ################ ] |
|
Node
[Owner : x509 cn=user ]
*[Write : x509 cn=vostore ]*
[Node : ivo://vospace/.. ]
[Path : /home/path/file ]
[File : ivo://vostore/.. ]
[Size : xxxx ]
[MD5 : ################ ]
In order to prevent the user from modifying or deleting the file, we may need to change this to a 'write once' permission, preventing the user from modifying the file once it has been imported. However, if we have already implemented the
importDone() callback from the VoStore to the VoSpace
which is called when the data import has completed, then we could possibly use a similar mechanism to update
the VoSpace service whenever the user modifies the file in the VoStore .
There are another two options which I haven't worked through yet: Setting the initial file ownership to the user's identity, and then using the adopt mechnism described earlier to transfer ownership to the VoSpace service once the data transfer has completed. Probably very messy and overly complicated. Using proxy certificates and authenticating all of the actions using the user's own identity. This would mean that both the file and node were owned by the user's identity and not by the VoSpace service. This does allow the user the ability to modify the file, but as I mention above, with some form of
fileModified() callback in place, this may not be a problem. Edit | Attach | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2005-11-16 - 18:25:04 - DaveMorris
 |
Click here for the AstroGrid Service Web |
This is the AstroGrid Development Wiki |
|
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors. Ideas, requests, problems regarding TWiki? Send feedback