You are viewing a plain text version of this content. The canonical link for it is here.
Posted to wagon-dev@maven.apache.org by Michal Maczka <mm...@interia.pl> on 2004/04/01 08:05:05 UTC
Re: New interfaces in Wagon
Jason van Zyl wrote:
[...]
>>Other example:
>>When we are searching for the latest version of the snapshot artifact,
>>we need to fetch from all visible repositories xx.artifact-version file
>>and find that repository whic contains the latest version of the given snapshot and compare it with the local version.
>>This file is quite small but still we don't want to save dozen of such files
>>to disk (finding unique name for each) and then read from them and delete them right after we are done with each of them.
>>
>>
>
>I don't see what the big deal is, so we save a bunch of files to disk
>first. But the API can be expanded out again and the first client Wagon
>will be fully integrated into will be Maven. Let me wire Wagon in
>completely to Maven and see what happens.
>
>
>
I am right now using quite a lot of large artifacts (mainly wars) - each
of them has - 20 MB+.
In the solution I have proposed it was possible to do the multiplexing
of streams and for example in single pass transfer a file to
the repository and compute in memory it's md5 and sha1 checksum and then
transfer thow two strings as artifiates directly from memory
If you have to read 20 MB three time from disk performance starts to matter.
For example in place of 7 seconds you might have 18 (I don't know real
numbers!)
>>My implementation of Wagon introduced 2 key interfaces: Wagon and Repository.
>>
>>I imagined theat simplification of usage of the API will be implemented one level higher than that.
>>
>>I was actually thinking about something like:
>>
>>class ArtifactRepository
>>{
>>
>> WagonFactory wagonFactory;
>>
>> Repository repository
>>
>> File getToFile( Artifact artifat ) throws ....;
>>
>> String getToString( Artifact artifat );
>>
>> /**
>> * Controls if md5 checksum are computed and verified
>> * during "get" operation and md5 checkusum are automatically
>> * computed and transfered
>> * to the repository during "put" operation (this means 2 calls to
>> * wagon.put are made - one for artifact second for md5sum
>> */
>> void useMd5s( boolean value);
>>
>> void useSha1s( boolean value );
>>
>> ...
>>
>>}
>>
>>
>
>I don't think mixing verification into the actual artifact is a good
>idea and there is no notion of searching in Wagon but we could add that.
>
>
>
It is not mixing of concern. Any "artifact verifier" can be plugged in
dynamically into the process. Even such verifiers which validate if
given file is valid XML file
or if given POM is valid. I just don't think that if you want to use
100 artifact validators to validate given artificat
you should read the same file 100 times ( I am exagerating here) . This
just don't scale well. You are validating the content.
So you can plug content validatorsas early as that content arrives.
I know that in reality we will have 2 maybe 3 validators...
>>etc. So users will even don't have to know that wagon exists.
>>
>>
>>I think that last changes in the API over simplified it and excluded some nice possibilties while the simplicity level for the end user is still not optimal.
>>
>>
>
>I'm going to go from end-to-end today with integrating Wagon into Maven
>so I'll be the first to tell you. But again, there were barely any tests
>for anything so I went to town.
>
I don't agree that everything was that badly tested! Wagon API module
was tested in 98% (clover) and that's where all those things were
implemented.
You can hardly do any better!!!
In there I defined interfaces like WagonSource WagonResult and
provided fully tested implementation of them like for example
MemoryResult which I wanted to use for transfering "to memory". Only
thing which was abolutly not tested were Wagon Providers.
But I completly agree that some intention which I had weren't that
obvious and clear and still that many thing could be simplified. E.g
instead of WagonResult probably just OutputStream might be used. That's
why I am trying to explain some things now and I am glad that finally I
have to defend my vison.
For example the exmplanaion why such "artificial" things like event in
Wagon exists is simple.
I wanted to use events for informing transfer observers like those which
can compute message digests of input stream. Other use case I wanted to
support is to display progress bars once Maven with Wagon will be
integrated with GUI/IDE. I hoped that this would make it even more
attractive.
All other liblaries I know of simply do not support such feature.
> I also believe the Wagon interface is a
>balance between simplicity and functionality. It's easy to make simple
>stream based wagons and the interface in the WagonManager/Conducter will
>eventually be. Let me do the round trip with integrating Wagon into
>Maven first.
>
>
>
OK. I still belive that too much of useful functinality was removed. I
also belive that we could build on top of that what it was
the layer which could be even simpler to use then what we have now. I
think that I know well what you want to achive as i feel that you are
repeting
my early ideas. I spent couple weeks implementing and during those few weeks
wagon was oscilting between something very easy and something horribly
complicated. I change my mind couple of times reagrding the API but I
tried to implement something simple and extendible.
Just to make thing even more clear: My idea was that we should have
three layers:
[ maven artifact ] (3)
[ wagon core API] (2)
[ wagon providers] (1)
and only the third layer should be normally used in application. You
should never use Wagon API directly in application unless you are doing
something extremly bizzare. Other importand design goal was to make
wagon providers very simple to write
that's why you don't see many methods in wagons - you won't find there
things like:
interface Wagon
{
void get( Artifact artifact, File destination )
throws TransferFailedException, ResourceDoesNotExistException,
AuthorizationException;
void put( File source, Artifact artifact )
throws TransferFailedException, ResourceDoesNotExistException,
AuthorizationException;
// not yet exists
void get( Artifact artifact, OutputStream destination )
throws TransferFailedException, ResourceDoesNotExistException,
AuthorizationException;
void put( InputStream inputStream, Artifact artifact )
throws TransferFailedException, ResourceDoesNotExistException,
AuthorizationException;
void getAsString( Artifact );
}
each Wagon (if I rember correctly ) had only
interfce Wagon
{
transfer( PutRequest )
transfer( GetRequest )
}
and this first layer were free from the from the entities like Artifact.
In exchange in this layer just "paths" relative to repositor root were
used.
So with any wagon provider you can put/get any file, even such which is
completly not obeing the layout which we are using in maven world.
So in my solution layers (1) and (3) were supposed to be super simple.
Layer 2 was rather finished and tested in ~100%.
Layer 1 was implemented (some providers exists) but not tetsted at all.
And layer 3 was not yet started.
Michal
---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org
Re: New interfaces in Wagon
Posted by Michal Maczka <mm...@interia.pl>.
Jason van Zyl wrote:
>On Thu, 2004-04-01 at 01:05, Michal Maczka wrote:
>
>
>>Jason van Zyl wrote:
>>
>>
>
>
>
>>I don't agree that everything was that badly tested! Wagon API module
>>was tested in 98% (clover) and that's where all those things were
>>implemented.
>>You can hardly do any better!!!
>>In there I defined interfaces like WagonSource WagonResult and
>>provided fully tested implementation of them like for example
>>MemoryResult which I wanted to use for transfering "to memory". Only
>>thing which was abolutly not tested were Wagon Providers.
>>But I completly agree that some intention which I had weren't that
>>obvious and clear and still that many thing could be simplified. E.g
>>instead of WagonResult probably just OutputStream might be used. That's
>>why I am trying to explain some things now and I am glad that finally I
>>have to defend my vison.
>>For example the exmplanaion why such "artificial" things like event in
>>Wagon exists is simple.
>>I wanted to use events for informing transfer observers like those which
>>can compute message digests of input stream. Other use case I wanted to
>>support is to display progress bars once Maven with Wagon will be
>>integrated with GUI/IDE. I hoped that this would make it even more
>>attractive.
>>All other liblaries I know of simply do not support such feature.
>>
>>
>>
>>>I also believe the Wagon interface is a
>>>balance between simplicity and functionality. It's easy to make simple
>>>stream based wagons and the interface in the WagonManager/Conducter will
>>>eventually be. Let me do the round trip with integrating Wagon into
>>>Maven first.
>>>
>>>
>>>
>>>
>>>
>>OK. I still belive that too much of useful functinality was removed. I
>>also belive that we could build on top of that what it was
>>the layer which could be even simpler to use then what we have now. I
>>think that I know well what you want to achive as i feel that you are
>>repeting
>>my early ideas. I spent couple weeks implementing and during those few weeks
>>wagon was oscilting between something very easy and something horribly
>>complicated. I change my mind couple of times reagrding the API but I
>>tried to implement something simple and extendible.
>>
>>
>
>It was still too complicated, and there were too many places where
>simple mistakes could be made in creating wagons that yielded a
>non-functional wagon. It happened to me twice and if it happened to me
>it will happen to others.
>
>
I know that it had some complicity. I just thought and still think that
this complexity could be completly removed
by the introduction of one more layer on top of Wagon.
interface ArtifactDownloader
>If you have knowledge of the client library all you have to do is know
>how to get an InputStream and an OutputStream. That's it. It doesn't get
>much simpler than that. And if you need to go down to the bare metal you
>can as you have to in the scp wagon.
>
>
>
I completly agree.
>Adding streaming capabilities might possibly be desirable but the bottom
>line is you haven't had time to touch the code in months and there were
>no visible tests for the wagons
>
The lack of time was not at all the reason why devlopment was stopped.
It was stopped as it was nothing with what wagon could be integrated
and some feature in plexus were missing. But first of all I waited for
that moment which come just now when somebody finally will give me some
searious
feedback!
>and the wagons themselves just too
>complicated as examples for people to follow.
>
>
>
I never designed Wagon to be the liblary used by the crowd of people. I
always thought about Maven as the only client of Wagon and people
using Wagon only with conjuction with the API which will be decided by
needs of Maven and exposed in Maven Components.
>This week I will finish a more exhaustive test suite, finish off some
>docs, alter the verifiers to work with streams and release 0.9 and
>integrate it into Maven.
>
>It's all stream-based underneath so it wouldn't be hard at all to expose
>those again but I'm willing to bet the vast majority of usage will be
>moving files around.
>
>
>
I might agree with that. But I did't want to predict what are the
possible use cases we might have and to leave some more possibilities open.
Coming back to my example when you want to transfer a war file which has
20 MB, MD5 and SHA1 files to the repository
What I wanted to support with Wagon is that you can have just one disk
I/O operation and the rest will happen on the fly.
So MD5 and SHA1 will be constructed in the memory and they will be
never written to disk.
Other scenrio just looks like:
1. Read war (20 MB) and send it to the repository
2. Read war (20 MB) and compute SHA1 message digiest
3. Save it to the file
4. Read that file and send it to the repository
5. Read war (20 MB) and compute MD5 message digest
6. Save it to the file
7. Read that file and send it to the repository
So totaly you have 7 I/O operations. And that's exactly what I am doing
now in maven-artifact plugin.
I just think that we can do way better then that!
Michal
---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org
Re: New interfaces in Wagon
Posted by Jason van Zyl <jv...@maven.org>.
On Thu, 2004-04-01 at 01:05, Michal Maczka wrote:
> Jason van Zyl wrote:
> I don't agree that everything was that badly tested! Wagon API module
> was tested in 98% (clover) and that's where all those things were
> implemented.
> You can hardly do any better!!!
> In there I defined interfaces like WagonSource WagonResult and
> provided fully tested implementation of them like for example
> MemoryResult which I wanted to use for transfering "to memory". Only
> thing which was abolutly not tested were Wagon Providers.
> But I completly agree that some intention which I had weren't that
> obvious and clear and still that many thing could be simplified. E.g
> instead of WagonResult probably just OutputStream might be used. That's
> why I am trying to explain some things now and I am glad that finally I
> have to defend my vison.
> For example the exmplanaion why such "artificial" things like event in
> Wagon exists is simple.
> I wanted to use events for informing transfer observers like those which
> can compute message digests of input stream. Other use case I wanted to
> support is to display progress bars once Maven with Wagon will be
> integrated with GUI/IDE. I hoped that this would make it even more
> attractive.
> All other liblaries I know of simply do not support such feature.
>
> > I also believe the Wagon interface is a
> >balance between simplicity and functionality. It's easy to make simple
> >stream based wagons and the interface in the WagonManager/Conducter will
> >eventually be. Let me do the round trip with integrating Wagon into
> >Maven first.
> >
> >
> >
>
> OK. I still belive that too much of useful functinality was removed. I
> also belive that we could build on top of that what it was
> the layer which could be even simpler to use then what we have now. I
> think that I know well what you want to achive as i feel that you are
> repeting
> my early ideas. I spent couple weeks implementing and during those few weeks
> wagon was oscilting between something very easy and something horribly
> complicated. I change my mind couple of times reagrding the API but I
> tried to implement something simple and extendible.
It was still too complicated, and there were too many places where
simple mistakes could be made in creating wagons that yielded a
non-functional wagon. It happened to me twice and if it happened to me
it will happen to others.
If you have knowledge of the client library all you have to do is know
how to get an InputStream and an OutputStream. That's it. It doesn't get
much simpler than that. And if you need to go down to the bare metal you
can as you have to in the scp wagon.
Adding streaming capabilities might possibly be desirable but the bottom
line is you haven't had time to touch the code in months and there were
no visible tests for the wagons and the wagons themselves just too
complicated as examples for people to follow.
This week I will finish a more exhaustive test suite, finish off some
docs, alter the verifiers to work with streams and release 0.9 and
integrate it into Maven.
It's all stream-based underneath so it wouldn't be hard at all to expose
those again but I'm willing to bet the vast majority of usage will be
moving files around.
--
jvz.
Jason van Zyl
jason@maven.org
http://maven.apache.org
happiness is like a butterfly: the more you chase it, the more it will
elude you, but if you turn your attention to other things, it will come
and sit softly on your shoulder ...
-- Thoreau
---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org