Download Manager

["Markus M. May" <mm...@gmx.net>]

Hello, here are some ideas for the Download-Manager. What do you think?
- Repository gets new attribute "trusted", then no check of MD5
- Artifact gets the same attribute
- if attribute is set to true, then no check of MD5 and the file is
   directly copied to the local repository
- get MD5-file
- copy file to a tmp-Directory with tmp-Name
- check tmp file to MD5
- if correct, copy tmp file to local repository with correct name
- also copy MD5 file to the local repository, so that later checks could
   be also performed
Question: do we use cmdLine md5hash or do we use java? - I like the java 
idea, but this could be pretty bad, if the file is large or the md5 
implementation of java does not support the bsd/linux md5

- how to unit test this, without a download? - not possible, so the 
internet connection should be up and running for the testcase

Re: Download Manager

[Nick Chalko <ni...@chalko.com>]

Markus M. May wrote:

> Hello, here are some ideas for the Download-Manager. What do you think?
>
>
> - how to unit test this, without a download? - not possible, so the 
> internet connection should be up and running for the testcase

For test check the files into cvs and use a repo url of 
file:///${basedir}/src/test-repo


R,
Nick

Re: Download Manager

["Adam R. B. Jack" <aj...@trysybase.com>]

> Ah, well, yeah, why not? Currently I am browsing through the API and add 
> some JavaDocs. Would be great, if you would check this, so that these 
> are correct.

Sure, will do.

regards

Adam

Re: Download Manager

["Adam R. B. Jack" <aj...@trysybase.com>]

> Ah, well, yeah, why not?

Well, it might be  a tad painful, 'cos I suspect there are some barriers.
;-)

For some reason I think I created a copy from one repo to another, but no
'getStream' or anything like that. Maybe due to trying to keep 'hands off'
the protocols.

The Avalon Repository work has some interesting code in this regard worth
looking at. Not sure I followed it, but I think the had an URLConnection, or
something.

> Currently I am browsing through the API and add
> some JavaDocs. Would be great, if you would check this, so that these
> are correct.

Looks good so far, I'll edit if/when I find things. Thanks for doign this,
it is a good job to keep up on.

regards

Adam

Re: Download Manager

["Markus M. May" <mm...@gmx.net>]

Ah, well, yeah, why not? Currently I am browsing through the API and add 
some JavaDocs. Would be great, if you would check this, so that these 
are correct.

Markus

Adam R. B. Jack wrote:
> Markus:
> 
> 
>>Would you be willing to take the DownloadManager development on? I think
> 
> the
> 
>>'sharing' we'd get discussing it, and how to implement it, would be
>>education for this team as a whole. I think we'd work out more details
> 
> with
> 
>>doing it this way.
> 
> 
> ?
> 
> regards
> 
> Adam
> 
> 

Re: Download Manager

["Adam R. B. Jack" <aj...@apache.org>]

Markus:

> Would you be willing to take the DownloadManager development on? I think
the
> 'sharing' we'd get discussing it, and how to implement it, would be
> education for this team as a whole. I think we'd work out more details
with
> doing it this way.

?

regards

Adam

Re: Download Manager

["Mark R. Diggory" <md...@latte.harvard.edu>]

Adam R. B. Jack wrote:

>>One thing I can chime in on here is that the mirror folks sure do like
>>the idea of uploading to a staging area then copying to the desired
>>location. It benefits their server side scripts to do this since an
>>upload could take much longer time than a server side copy operation.
>>I'm not sure how rsync handles partially complete uploaded files, but I
>>know the signature scripts Henning is running via cron will generate an
>>md5 for a partial file. Then when you upload your md5 they clearly don't
>>match. Though this is a shortfall of his script, uploading to a staging
>>area just seems more robust in the long run.
>>
>>    
>>
>
>Good to know, thanks. Right now we are focusing on downloading jars from a
>pre-populated repository (kindly managed by folks such as yourself. :-)
>
>regards
>
>Adam
>
>  
>
Oh, thats RIGHT, your working on the DOWNLOAD Manager :-) [sorry]

Re: Download Manager

["Adam R. B. Jack" <aj...@trysybase.com>]

> >>
> One thing I can chime in on here is that the mirror folks sure do like
> the idea of uploading to a staging area then copying to the desired
> location. It benefits their server side scripts to do this since an
> upload could take much longer time than a server side copy operation.
> I'm not sure how rsync handles partially complete uploaded files, but I
> know the signature scripts Henning is running via cron will generate an
> md5 for a partial file. Then when you upload your md5 they clearly don't
> match. Though this is a shortfall of his script, uploading to a staging
> area just seems more robust in the long run.
>

Good to know, thanks. Right now we are focusing on downloading jars from a
pre-populated repository (kindly managed by folks such as yourself. :-)

regards

Adam

Re: Download Manager

["Mark R. Diggory" <md...@latte.harvard.edu>]

Adam R. B. Jack wrote:

>Ought we simple download it using Download-Manager w/ trusted? :)
>Ought we not simple copy it down to the local repository location?
>
>  
>
>>- copy file to a tmp-Directory with tmp-Name
>>- check tmp file to MD5
>>- if correct, copy tmp file to local repository with correct name
>>    
>>
One thing I can chime in on here is that the mirror folks sure do like 
the idea of uploading to a staging area then copying to the desired 
location. It benefits their server side scripts to do this since an 
upload could take much longer time than a server side copy operation. 
I'm not sure how rsync handles partially complete uploaded files, but I 
know the signature scripts Henning is running via cron will generate an 
md5 for a partial file. Then when you upload your md5 they clearly don't 
match. Though this is a shortfall of his script, uploading to a staging 
area just seems more robust in the long run.

2cents,
Mark

Re: Download Manager

["Adam R. B. Jack" <aj...@trysybase.com>]

This is good stuff.

> Hello, here are some ideas for the Download-Manager. What do you think?
> - Repository gets new attribute "trusted", then no check of MD5
> - Artifact gets the same attribute
> - if attribute is set to true, then no check of MD5 and the file is
>    directly copied to the local repository

All sounds good.

Further, right now we have a 'runtime attribute' on the ArtifactInstance
that tell it where the MD5 or ASC file is to be found. A virtual location.
If it isn't there, we could see if the repository was 'untrusted' and not
download?

> - get MD5-file

Ought we simple download it using Download-Manager w/ trusted? :)
Ought we not simple copy it down to the local repository location?

> - copy file to a tmp-Directory with tmp-Name
> - check tmp file to MD5
> - if correct, copy tmp file to local repository with correct name

Are there MD5-based streams we can use where the file isn't completed on
disk until verified? If not, your approach seems good.

BTW: The 'protocols abstraction' (that works over VFS or HttpClient or XXX)
might need some work to do this. Right now it dumbly copies from one repo to
another (i.e. local). I don't think it even has the concept of 'get'. I can
help work on that.

> - also copy MD5 file to the local repository, so that later checks could
>    be also performed
> Question: do we use cmdLine md5hash or do we use java? - I like the java
> idea, but this could be pretty bad, if the file is large or the md5
> implementation of java does not support the bsd/linux md5

I don't care about speed, we only download when we need to.

> - how to unit test this, without a download? - not possible, so the
> internet connection should be up and running for the testcase

Hmm, that will do for starters, but I'd like to figure this out eventually
using MockRepository, etc.

Would you be willing to take the DownloadManager development on? I think the
'sharing' we'd get discussing it, and how to implement it, would be
education for this team as a whole. I think we'd work out more details with
doing it this way.

regards

Adam