MRM-684 - Solving Archiva Blocking

[James William Dumay <ja...@atlassian.com>]

Hey guys,
I've been investigating the problem described in MRM-684 for the last
week or so and so far I have come up with the following.

* Data from a remote repository should be streamed back to the client as
it is being received.
* Both Wagon and Plexus webdav handle resources only as valid file
objects.
* Fixing Plexus Webdav is a relatively simple change. 
* Wagon makes a lot of sense in Maven in its current state - its a
library built for the needs of Maven and Maven only.
* Changing Wagon to support streams instead of files is difficult
without taking away what makes Wagon great for Maven (A simple resource
transport layer)

What I'm want to propose is the removal of Wagon and replacing it with a
more flexible library (perhaps we could cook something up with
commons-vfs?)

So what do you think? Suggestions?

Thanks
James Dumay

Re: MRM-684 - Solving Archiva Blocking

[Brett Porter <br...@apache.org>]

On 08/02/2008, at 2:21 PM, James William Dumay wrote:

> Looks like we only use Http/Https, ssh and file Wagon's in archiva -
> commons-vfs so far looks like it hits the spot.

Sounds like it's worth an evaluation regardless.

> I have a copy of Wagon checked out at the moment that works purely  
> with
> streams but I ran into a few of issues:
> * Wagon's transfer even model relies on being able to pass the File
> object that is being transfered to the listener - if its a streaming  
> the
> transfer events don't make any sense seeing that the consumer of Wagon
> would be doing all the reading/writing
> * There are about three different ssh/scp Wagon implementations that  
> can
> only deal with files. For example, there is a Wagon provider that will
> use the systems scp executable to upload/download files
>
> I really don't see the value in having to ditch Wagon implementations.

So if we do stick to this, sounds like it's not a convenience File  
method, but two implementations, and the stream one is only supported  
by some implementations and just throws an error on the others? (Those  
don't make sense for Archiva anyway). As far as the transfer events go  
- not sure I understand - isn't the wagon consuming from an  
InputStream and pushing to an OutputStream and can trace the byte  
count along the way?

Cheers,
Brett

--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/

Re: MRM-684 - Solving Archiva Blocking

[James William Dumay <ja...@atlassian.com>]

Hey Brett,

On Thu, 2008-02-07 at 15:20 +1100, Brett Porter wrote:
> I don't have any objection to using commons-vfs, since I think it  
> covers much of what Wagon does anyway - as long as it does have all  
> the features needed.

Looks like we only use Http/Https, ssh and file Wagon's in archiva -
commons-vfs so far looks like it hits the spot.

> That said - I don't really see any reason this is difficult in Wagon -  
> and do believe Wagon should be usable outside Maven. I think streaming  
> is a perfectly sensible thing to add...

I have a copy of Wagon checked out at the moment that works purely with
streams but I ran into a few of issues:
* Wagon's transfer even model relies on being able to pass the File
object that is being transfered to the listener - if its a streaming the
transfer events don't make any sense seeing that the consumer of Wagon
would be doing all the reading/writing
* There are about three different ssh/scp Wagon implementations that can
only deal with files. For example, there is a Wagon provider that will
use the systems scp executable to upload/download files

I really don't see the value in having to ditch Wagon implementations.

> Regardless, I certainly don't want to see *another* transport layer,  
> or lose something like remote file:// repositories :)

I understand your concern - I don't want to add yet another framework to
Archiva. All we need is a simple module to allow us to get files from
remote locations (http and ssh) and from the local file system.

> Does that make sense?

Perfectly :)

James

Re: MRM-684 - Solving Archiva Blocking

[Brett Porter <br...@apache.org>]

I don't have any objection to using commons-vfs, since I think it  
covers much of what Wagon does anyway - as long as it does have all  
the features needed.

That said - I don't really see any reason this is difficult in Wagon -  
and do believe Wagon should be usable outside Maven. I think streaming  
is a perfectly sensible thing to add...

Regardless, I certainly don't want to see *another* transport layer,  
or lose something like remote file:// repositories :)

Does that make sense?

On 07/02/2008, at 2:18 PM, James William Dumay wrote:

> Hey guys,
> I've been investigating the problem described in MRM-684 for the last
> week or so and so far I have come up with the following.
>
> * Data from a remote repository should be streamed back to the  
> client as
> it is being received.
> * Both Wagon and Plexus webdav handle resources only as valid file
> objects.
> * Fixing Plexus Webdav is a relatively simple change.
> * Wagon makes a lot of sense in Maven in its current state - its a
> library built for the needs of Maven and Maven only.
> * Changing Wagon to support streams instead of files is difficult
> without taking away what makes Wagon great for Maven (A simple  
> resource
> transport layer)
>
> What I'm want to propose is the removal of Wagon and replacing it  
> with a
> more flexible library (perhaps we could cook something up with
> commons-vfs?)
>
> So what do you think? Suggestions?
>
> Thanks
> James Dumay
>

--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/