wagon changes

[Brett Porter <br...@apache.org>]

Hi,

I'm starting to port all the fixes I made to 
maven-artifact-plugin/httputils to wagon so that I can look at dropping 
it into Maven 1.x. I have noticed a few issues, would anyone like to 
comment on whether I am approaching this the right way?

- need to be able to communicate file size to the observer
- no last-modified handling - this should be in abstract wagon
- change input/output stream to input/output source wrapper class that 
contains
timestamp & length (& checksum?)

^ I think this requires some refactoring to use more than just streams 
as the basis for most wagons. Does anyone mind?

- is there a reason there is no sftp wagon?
- is there any problem adding an ext wagon?

^ I'm not convinced we need an ext wagon, but it can definitely be used 
and it's easy to do. I feel it's easier to give than to try and handhold 
people through sshagent changes etc.

- ssh tests currently have to be online

^ any thoughts on how to do this some other way?

Cheers,
Brett



---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org

Re: wagon changes

[Brett Porter <br...@apache.org>]

> >> If I remember we have decided that we will have pointer files  
> >> containing the timestamp of the last deployed
> >> artifact. 
> >
> >
> > in the local repository?
> 
> 
> for accessing the file in the local repository you don't need wagon :)

that's not what I meant, but I see what you mean now. So we effectively write
last-modified to a file rather than using a header.

For compatibility with m1, is there any reason not to use last-modified if that
file doesn't exist (as nothing, including wagon, is attempting to create it now).

> >> Also we will need to handle timezones properly and we will need to 
> >> take into
> >> account that server's time zone might be different then user's 
> >> timezone. And stuff like that.
> >
> >
> > This is all handled correctly already.
> 
> 
> Is this a case of every type of the server?

Every type of server that currently uses last-modified, yes. We only support
http and file in m1.

> We planned to change bit in which snapshot artifact will be resolved but 
> it could be that we can still use it in m1...
> That is something which we really have to disuss to have a clear  vison  
> how it will work  in  m2 and compare
> it with the way in which works  in m1.

ok, so what do we need to sort out? should we start a separate thread?

> We have (or better say I have)  planned to put any extra metadata to 
> repository as set of extra files
> (like we do with .md5 files) and in wagon have a support for really dumb 
> get/put operations
> so we will be able to support any repository in uniform way.

ok, let's start getting this happening then.

> Such functionality can be already provided by Observers.
> They were really created from ultra fast processing so number of i/o 
> operation is limited to absolute maximum
> (e..g md5 cheksum is computed on the fly when artifact is transfered)
> but they were also supposed to be used for cases like progress bars in 
> guis like eclipse/idea.

ok, I'll take another look - I understand how the observers work, but IIRC the
oberver is created in get/putTransfer. What I'm proposing is that we modify that
(and the SSH one that doesn't use it) to take a richer definition of the
source/dest so it can construct the observers rather than continually appending
parameters.

I think it just needs an input/output def like get/put do themselves, with the
additional info populated by get/put.

This morning I removed one of the parameters to get/putTransfer (the local
FileInputStream/LazyFileOutputStream) because it was always created the same and
some of the handling was duplicated and unnecessarily different.

Actually, thinking about it, I'm not sure why SSH/HTTP wagon shouldn't just be a
stream wagon with a customised stream. That way more shared handling could be in
the core.

Cheers,
Brett


---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org

Re: wagon changes

[Michal Maczka <mm...@interia.pl>]

Brett Porter wrote:

>
>>
>> We will need to add the functionality which can access file size for 
>> every protocol.
>> I am not 100% sure if this is doable for all of them  but we can try 
>> to do this.
>
>
> I'll look at this then. If it is not possible, a known value such as 
> -1 can indicate it is not known and be handled appropriately.
>
+1

>>
>>> - no last-modified handling - this should be in abstract wagon
>>
>>
>>
>> It is needed only for snapshot artifacts, right?
>
>
> yep
>
>> If I remember we have decided that we will have pointer files  
>> containing the timestamp of the last deployed
>> artifact. 
>
>
> in the local repository?


for accessing the file in the local repository you don't need wagon :)
but I imagine that meta data in the local repsoitory should be identical 
to that in the remote repository
so any local repository can be easly turned into remote repository.


>
>> Not every protocol supports last-modified flag ( e.g I am not sure 
>> about ftp, sftp) - and we need to use "the lowest common denominator". 
>
>
> sure, although in some cases the LCD can be raised by special handling 
> the functionality in some cases.
>
>> Also we will need to handle timezones properly and we will need to 
>> take into
>> account that server's time zone might be different then user's 
>> timezone. And stuff like that.
>
>
> This is all handled correctly already.


Is this a case of every type of the server?

>
> So now I'm remembering a brief comment by Jason that snapshot handling 
> wasn't going to be in Wagon. Can we sort this out so we know the way 
> forward? It may be that it is not practical to use wagon in Maven 1.1 
> for dependency downloading after all.


We planned to change bit in which snapshot artifact will be resolved but 
it could be that we can still use it in m1...
That is something which we really have to disuss to have a clear  vison  
how it will work  in  m2 and compare
it with the way in which works  in m1.

First client project of wagon is m2. But we still can change things in 
wagon to support m1 as long
as we are not going to rewrite it from the scratch.

We have (or better say I have)  planned to put any extra metadata to 
repository as set of extra files
(like we do with .md5 files) and in wagon have a support for really dumb 
get/put operations
so we will be able to support any repository in uniform way.

>
>>
>>> - change input/output stream to input/output source wrapper class 
>>> that contains
>>> timestamp & length (& checksum?)
>>
>>
>>
>> Sorry but I don't understand this point :(
>
>
> I think having abstract wagon deal with input and output streams isn't 
> rich enough. I'd like to be able to attach properties to the source 
> and destination for comparison. Timestamps are for the above. Length 
> can be for reporting how long a download will take exactly. And then 
> you could add some optional extra-paranoid checking: check remote 
> dependencies for the same size as local, compare checksums or even 
> signatures again. This could be run occasionally to verify the 
> integrity of the local/remote repo without just redownloading everything.
>

Such functionality can be already provided by Observers.
They were really created from ultra fast processing so number of i/o 
operation is limited to absolute maximum
(e..g md5 cheksum is computed on the fly when artifact is transfered)
but they were also supposed to be used for cases like progress bars in 
guis like eclipse/idea.

When trasnfer is started an event, which contains some bits of 
artifact's metadata will be sent to all observers.
We can certainly extend that metadata and for example add "file length" 
fileld and whatever else which may be useful for them.

[...]

Michal


----------------------------------------------------------------------
Ponad 400 tysiecy kandydatow na meza? Poszukaj wlasnego!
>>> http://link.interia.pl/f183c


---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org

Re: wagon changes

[Brett Porter <br...@apache.org>]

>
> We will need to add the functionality which can access file size for 
> every protocol.
> I am not 100% sure if this is doable for all of them  but we can try 
> to do this.

I'll look at this then. If it is not possible, a known value such as -1 
can indicate it is not known and be handled appropriately.

>
>> - no last-modified handling - this should be in abstract wagon
>
>
> It is needed only for snapshot artifacts, right?

yep

> If I remember we have decided that we will have pointer files  
> containing the timestamp of the last deployed
> artifact. 

in the local repository?

> Not every protocol supports last-modified flag ( e.g I am not sure 
> about ftp, sftp) - and we need to use "the lowest common denominator". 

sure, although in some cases the LCD can be raised by special handling 
the functionality in some cases.

> Also we will need to handle timezones properly and we will need to 
> take into
> account that server's time zone might be different then user's 
> timezone. And stuff like that.

This is all handled correctly already.

So now I'm remembering a brief comment by Jason that snapshot handling 
wasn't going to be in Wagon. Can we sort this out so we know the way 
forward? It may be that it is not practical to use wagon in Maven 1.1 
for dependency downloading after all.

>
>> - change input/output stream to input/output source wrapper class 
>> that contains
>> timestamp & length (& checksum?)
>
>
> Sorry but I don't understand this point :(

I think having abstract wagon deal with input and output streams isn't 
rich enough. I'd like to be able to attach properties to the source and 
destination for comparison. Timestamps are for the above. Length can be 
for reporting how long a download will take exactly. And then you could 
add some optional extra-paranoid checking: check remote dependencies for 
the same size as local, compare checksums or even signatures again. This 
could be run occasionally to verify the integrity of the local/remote 
repo without just redownloading everything.

>
> we can easly add one. I think that scp is used much more frequently 
> and my private benchmarks have shown that
> transfers are much  faster with scp (at least wihout any compression) 
> . For anybody using ssh2 there  is no differnce if he uses scp or sftp 
> as both protocols
> must be supported by ssh2 server. As scp seems to be faster (my 
> bechmarks are not unltimate refernce here :) I guess that scp will be 
> preferable one.
>
ok. We might need to consult the users to see who is using it and why.

>
>> - is there any problem adding an ext wagon?
>>
>> ^ I'm not convinced we need an ext wagon, but it can definitely be 
>> used and it's easy to do. I feel it's easier to give than to try and 
>> handhold people through sshagent changes etc.
>>
> I think it is doable to add "ext" wagon. One of the reason (and in 
> fact the only reason which I can see) to  use such  wagon
> is becouse it can give more security.
> I figured out myself that passwords can be provided in a secure way to 
> long living applications like continuum (wen interface)
> or injected via call back method by observers which are displaying 
> dialogboxes
> I had an implementation of "password storage" which can be used in 
> continum/eclipse/idea etc.

Sorry, not sure why an ext wagon helps here, except maybe for doing ftp 
without a password - but you'd think anyone that security conscious 
would be using SSH with a private key anyway.

I was thinking it was for using SSH agents.

>
> I am testing ssh wagon inside the continuum. It will be quite hard to 
> find improment here and I am not sure if it is worthed it.

These seem like integration tests best run in continuum but not day to 
day. Is it best we pull them all out? ie wagon-provider-test could 
contain all the tests and depend on each provider. Continuum would run 
those tests when any provider changed, right?

Cheers,
Brett



---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org

Re: wagon changes

[Michal Maczka <mm...@interia.pl>]

Brett Porter wrote:

> Hi,
>
> I'm starting to port all the fixes I made to 
> maven-artifact-plugin/httputils to wagon so that I can look at 
> dropping it into Maven 1.x. I have noticed a few issues, would anyone 
> like to comment on whether I am approaching this the right way?
>
> - need to be able to communicate file size to the observer

We will need to add the functionality which can access file size for 
every protocol.
I am not 100% sure if this is doable for all of them  but we can try to 
do this.

> - no last-modified handling - this should be in abstract wagon

It is needed only for snapshot artifacts, right?
If I remember we have decided that we will have pointer files  
containing the timestamp of the last deployed
artifact. Not every protocol supports last-modified flag ( e.g I am not 
sure about ftp, sftp) - and we need to use "the lowest common 
denominator". Also we will need to handle timezones properly and we will 
need to take into
account that server's time zone might be different then user's timezone. 
And stuff like that.


> - change input/output stream to input/output source wrapper class that 
> contains
> timestamp & length (& checksum?)

Sorry but I don't understand this point :(

>
> ^ I think this requires some refactoring to use more than just streams 
> as the basis for most wagons. Does anyone mind?
>
> - is there a reason there is no sftp wagon?

we can easly add one. I think that scp is used much more frequently and 
my private benchmarks have shown that
transfers are much  faster with scp (at least wihout any compression) . 
For anybody using ssh2 there  is no differnce if he uses scp or sftp as 
both protocols
must be supported by ssh2 server. As scp seems to be faster (my 
bechmarks are not unltimate refernce here :) I guess that scp will be 
preferable one.


> - is there any problem adding an ext wagon?
>
> ^ I'm not convinced we need an ext wagon, but it can definitely be 
> used and it's easy to do. I feel it's easier to give than to try and 
> handhold people through sshagent changes etc.
>
I think it is doable to add "ext" wagon. One of the reason (and in fact 
the only reason which I can see) to  use such  wagon
is becouse it can give more security.
I figured out myself that passwords can be provided in a secure way to 
long living applications like continuum (wen interface)
or injected via call back method by observers which are displaying 
dialogboxes
I had an implementation of "password storage" which can be used in 
continum/eclipse/idea etc.


> - ssh tests currently have to be online


I am testing ssh wagon inside the continuum. It will be quite hard to 
find improment here and I am not sure if it is worthed it.

regards

Michal

----------------------------------------------------------------------
Startuj z INTERIA.PL!!! >>> http://link.interia.pl/f1837


---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org