Downloading a large file from MySpace using VOExplorer

Motivation

It appears that downloading files from MySpace takes an unreasonably long time

Method

A large (1,200,066,560 byte) file was downloaded from MySpace using the "download" facility.

For comparison the same file was downloaded from myspace by using the AR to directly discovering the URL for the file with

ms.getReadContentURL('ivo://uk.ac.le.star/paulharrison#result/twomass2.vot')

and then the file was transferred using the standard unix command line command "wget".

Results

The file failed to transfer via the "download" facility as after about 2 hours the java process running VOExplorer failed with an "Out of Memory" exception (with approximately 700M transferred). The transfer process was monitored whilst it was occuring by using the wireshark packet sniffer, and the data transfer rate for a portion of that time is shown below.

• data transfer rate from myspace:
  

The average rate of data transfer is approximately 1Mbyte in 100s in the middle of this plot

In comparison, direct transfer from the filestore url with wget achieved 9.81 MB/s

Conclusion

Indeed it does take a long time to transfer a file from Myspace when using the "download" facility from the Fileexplorer - about 2 orders of magnitude slower in this case....

This requires further investigation, but early analysis on the available evidence would tend to suggest that a the transfer is buffered in memory in some way, and that the reason for the bursty nature of the transfer is possibly that "garbage collection", or some other non optimal use of memory/disk io is happening in the "quiet" phases.

-- PaulHarrison - 11 Jan 2008

As far as I can see this problem has gone away - I see around 5MB/s for VOSpace->local disk downloads, reliable and sustained. See Bugzilla 2536.

-- MarkTaylor - 28 Mar 2008