You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ant.apache.org by Steve Loughran <st...@apache.org> on 2007/05/04 09:56:48 UTC
auto download of antlibs
One thing I've been thinking of this week is how could we work with Ivy
for automatic antlib download.
No code right now, just some thoughts
1. add a -offline argument to say "we are offline". this will set a
property, (and.build.offline) and the <offline> test will work. It is
meant to tell things like Ivy that we are offline. At some point we
could add some way for Ant to guess whether the net is there or not, if
java integrates with the OS properly (there is an API call for this in
J2ME, just not Java SE)
2. when we encounter an element (or even an attr) in an unknown antlib
xmlns, and we want to map that to a projectcomponent, we hand off
resolution to an antlib resolver. We would have one built in (the
failing resolver), would default to the ivy one if it was present, and
provide some way to let people switch to a different one.
3. an antlib resolver would do the mapping from antlib package to
artifacts (problem one), then download the metadata for that artifact,
pull it down and all its artifacts
4. we would then <typedef> the lib with the classpath that is set up by
the resolver
we'd need a metadata tree mapping antlibs to well known packages, but
that is not too hard. JSON, perhaps.
-steve
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Xavier Hanin <xa...@gmail.com>.
On 5/10/07, Steve Loughran <st...@apache.org> wrote:
>
> Xavier Hanin wrote:
> > On 5/9/07, Steve Loughran <st...@apache.org> wrote:
> teresting.
> >>
> >> Maybe every repository in ivyconf.xml would be marked as offline,
> >> meaning they are available when there is no network.When you run ant
> (or
> >> ivy) with -offline, only offline repositories would be used.
> >
> >
> > What do you mean? If all repositories are marked as offline, there is
> no
> > added value. We are currently reviewing our cache management, but for
> the
> > moment Ivy can already use the cache when you are offline. The problem
> is
> > that if you ask for a latest version of something, Ivy will try to
> connect
> > to the repository. For the moment this fails if the repository is not
> > available, but we are planning to make it possible to use cache only in
> > this
> > case. But even with this improvement, trying to connect to a non
> available
> > repository may take time, so the idea of an offline mode would be to
> say:
> > "do not use this repository when offline, use only cache" for
> repositories
> > requiring a network connection, and for those which do not require this
> > connection (like a local repo), continue to use the repo and not only
> the
> > cache.
> >
> > - xavier
>
> I think I mean you could mark a repository (such as an a filesystem,
> ssh or http repo) as available when a system is offline. When there's an
> offline build, the stuff in cache is always there, but you'd only hit
> those repositories marked as available offline.
OK, I think I had a similar idea, but thinking about it the other way
around: repositories would have a online property, when set to true the
repository requires network connection, when false it is available offline.
By default an http repository would be considered online, and a filesystem
not, but you could change that. But now I tihnk using an offline property
would be better understood by users, especially because it matches the idea
of the offline mode.
the other trick is to hit every HTTP repo with a GET request on startup,
> but that can be misleading. the ibiblio root is dog slow, and proxy
> servers can lie, returning an old copy, even when the repository is
> missing
Yes, this is not easy to implement and can be very misleading. But maybe we
could see the problem of disabling a repository (as discussed for the
offline mode) in a more generic point of view, and be able to enable/disable
repositories based on a property value. The property could be
ant.build.offline, but it could be anything else. Then the user could pretty
easily define his own property when he knows that a particular repository is
not available, to disable only one repo.
Xavier
-steve
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
Re: auto download of antlibs
Posted by Xavier Hanin <xa...@gmail.com>.
On 5/10/07, Steve Loughran <st...@apache.org> wrote:
>
> Stephen McConnell wrote:
>
> >
> > It seems that there may be two distinct subjects in this thread:
> >
> > a) introduction of policy that restricts dynamic resolution of
> > resource to those available via a local file protocol (refer
> > Xavier's comments "By offline I mean with no network access")
> >
> > b) introduction of a policy that restricts dynamic resolution
> > of resources to a selection of 'safe'(?) repositories
> >
> > The first scenario correctly reflects the offline notion while the
> second
> > scenario does not have any relationship to the term. However, the
> second
> > scenario does start to recognize that the physical topology of a machine
> is
> > not equivalent to the definition of a policy.
>
>
> "offline" is maybe not the correct term. "partitioned" is more accurate.
> when the internet goes from our site, ibiblio is missing, a local
> repository is reachable. When the network goes from my laptop, only the
> localhost and all VMWare hosted machines are available. My laptop may
> still use ssh and mounted filesystem protocols to see the system, but
> nothing else.
>
> switching on file IO vs. network IO doesnt cut it, because NFS and
> networked mounted DAV filesystems may be on the wrong side of the
> partition.
>
> Like you say, it depends on network topologies, but I dont want to
> introduce the concept of partitioned network, as it scares people.
> Unless we hide it under network "configurations", where different
> configurations can have different proxy and repository options. That is
> a more realistic world view of how my laptop acts.
Yes, and if we introduce conditional enabling of dependency resolvers in Ivy
(disabled would still use cache), this is something that easily be done by
users (at least if they manually switched from one network configuration to
another). Note that you can already do that in Ivy by switching your
settings, but it would be easier with conditional resolver enablement.
Xavier
Then I can use DNS
> and WLAN ID analysis to determine the active configuration; this is
> something best done in C++ than java.
>
> -steve
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
Re: auto download of antlibs
Posted by Steve Loughran <st...@apache.org>.
Stephen McConnell wrote:
>
> It seems that there may be two distinct subjects in this thread:
>
> a) introduction of policy that restricts dynamic resolution of
> resource to those available via a local file protocol (refer
> Xavier's comments "By offline I mean with no network access")
>
> b) introduction of a policy that restricts dynamic resolution
> of resources to a selection of 'safe'(?) repositories
>
> The first scenario correctly reflects the offline notion while the second
> scenario does not have any relationship to the term. However, the second
> scenario does start to recognize that the physical topology of a machine is
> not equivalent to the definition of a policy.
"offline" is maybe not the correct term. "partitioned" is more accurate.
when the internet goes from our site, ibiblio is missing, a local
repository is reachable. When the network goes from my laptop, only the
localhost and all VMWare hosted machines are available. My laptop may
still use ssh and mounted filesystem protocols to see the system, but
nothing else.
switching on file IO vs. network IO doesnt cut it, because NFS and
networked mounted DAV filesystems may be on the wrong side of the
partition.
Like you say, it depends on network topologies, but I dont want to
introduce the concept of partitioned network, as it scares people.
Unless we hide it under network "configurations", where different
configurations can have different proxy and repository options. That is
a more realistic world view of how my laptop acts. Then I can use DNS
and WLAN ID analysis to determine the active configuration; this is
something best done in C++ than java.
-steve
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
RE: auto download of antlibs
Posted by Stephen McConnell <mc...@dpml.net>.
> -----Original Message-----
> From: Steve Loughran [mailto:stevel@apache.org]
> Sent: Thursday, 10 May 2007 8:20 PM
> To: Ant Developers List
> Subject: Re: auto download of antlibs
>
> Xavier Hanin wrote:
> > On 5/9/07, Steve Loughran <st...@apache.org> wrote:
> teresting.
> >>
> >> Maybe every repository in ivyconf.xml would be marked as offline,
> >> meaning they are available when there is no network.When
> you run ant
> >> (or
> >> ivy) with -offline, only offline repositories would be used.
> >
> >
> > What do you mean? If all repositories are marked as
> offline, there is
> > no added value. We are currently reviewing our cache
> management, but
> > for the moment Ivy can already use the cache when you are
> offline. The
> > problem is that if you ask for a latest version of
> something, Ivy will
> > try to connect to the repository. For the moment this fails if the
> > repository is not available, but we are planning to make it
> possible
> > to use cache only in this case. But even with this
> improvement, trying
> > to connect to a non available repository may take time, so
> the idea of
> > an offline mode would be to say:
> > "do not use this repository when offline, use only cache" for
> > repositories requiring a network connection, and for those which do
> > not require this connection (like a local repo), continue
> to use the
> > repo and not only the cache.
> >
> > - xavier
>
> I think I mean you could mark a repository (such as an a
> filesystem, ssh or http repo) as available when a system is
> offline. When there's an offline build, the stuff in cache is
> always there, but you'd only hit those repositories marked as
> available offline.
It seems that there may be two distinct subjects in this thread:
a) introduction of policy that restricts dynamic resolution of
resource to those available via a local file protocol (refer
Xavier's comments "By offline I mean with no network access")
b) introduction of a policy that restricts dynamic resolution
of resources to a selection of 'safe'(?) repositories
The first scenario correctly reflects the offline notion while the second
scenario does not have any relationship to the term. However, the second
scenario does start to recognize that the physical topology of a machine is
not equivalent to the definition of a policy.
Cheers, Steve.
--------------------------
Stephen McConnell
mailto:mcconnell@dpml.net
http://www.dpml.net
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Steve Loughran <st...@apache.org>.
Xavier Hanin wrote:
> On 5/9/07, Steve Loughran <st...@apache.org> wrote:
teresting.
>>
>> Maybe every repository in ivyconf.xml would be marked as offline,
>> meaning they are available when there is no network.When you run ant (or
>> ivy) with -offline, only offline repositories would be used.
>
>
> What do you mean? If all repositories are marked as offline, there is no
> added value. We are currently reviewing our cache management, but for the
> moment Ivy can already use the cache when you are offline. The problem is
> that if you ask for a latest version of something, Ivy will try to connect
> to the repository. For the moment this fails if the repository is not
> available, but we are planning to make it possible to use cache only in
> this
> case. But even with this improvement, trying to connect to a non available
> repository may take time, so the idea of an offline mode would be to say:
> "do not use this repository when offline, use only cache" for repositories
> requiring a network connection, and for those which do not require this
> connection (like a local repo), continue to use the repo and not only the
> cache.
>
> - xavier
I think I mean you could mark a repository (such as an a filesystem,
ssh or http repo) as available when a system is offline. When there's an
offline build, the stuff in cache is always there, but you'd only hit
those repositories marked as available offline.
the other trick is to hit every HTTP repo with a GET request on startup,
but that can be misleading. the ibiblio root is dog slow, and proxy
servers can lie, returning an old copy, even when the repository is missing
-steve
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Xavier Hanin <xa...@gmail.com>.
On 5/9/07, Steve Loughran <st...@apache.org> wrote:
>
> Xavier Hanin wrote:
> > On 5/7/07, Stephen McConnell <mc...@dpml.net> wrote:
> >>
> >>
> >> > -----Original Message-----
> >> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> >> > Sent: Friday, 4 May 2007 5:56 PM
> >> > To: Ant Developers List
> >> > Subject: Re: auto download of antlibs
> >> >
> >> > On 5/4/07, Steve Loughran <st...@apache.org> wrote:
> >> > >
> >> > > One thing I've been thinking of this week is how could we work with
> >> > > Ivy for automatic antlib download.
> >> > >
> >> > > No code right now, just some thoughts
> >> > >
> >> > >
> >> > > 1. add a -offline argument to say "we are offline". this will set
> >> > > a property, (and.build.offline) and the <offline> test will
> >> > > work. It is meant to tell things like Ivy that we are offline. At
> >> > > some point we could add some way for Ant to guess whether the net
> >> > > is there or not, if java integrates with the OS properly (there is
> >> > > an API call for this in J2ME, just not Java SE)
> >> >
> >> > This makes me think that we could improve how Ivy deal with
> >> > online/offline mode. Indeed for the moment Ivy doesn't really
> >> > know which repository requires a network access and which
> >> > doesn't. It would be nice if Ivy would know that, so that
> >> > even if offline mode you could still use alocal repository
> >> > (for antlib testing for instance).
> >>
> >> Are you describing a policy at the level of:
> >>
> >> a) a multi-project build decision?
> >> b) a specific target project build decision?
> >> c) a repository access decision?
> >> d) some or any of the above?
> > I'm describing how Ivy could be improved to better deal with a
> > situation where the user do not have network access. You can already
> > deal with that pretty efficiently in Ivy, but you have to define
> > several settings, to avoid using remote access when it's not
> > available. Having something easier for the user in this common
> > situation would be interesting.
>
> Maybe every repository in ivyconf.xml would be marked as offline,
> meaning they are available when there is no network.When you run ant (or
> ivy) with -offline, only offline repositories would be used.
What do you mean? If all repositories are marked as offline, there is no
added value. We are currently reviewing our cache management, but for the
moment Ivy can already use the cache when you are offline. The problem is
that if you ask for a latest version of something, Ivy will try to connect
to the repository. For the moment this fails if the repository is not
available, but we are planning to make it possible to use cache only in this
case. But even with this improvement, trying to connect to a non available
repository may take time, so the idea of an offline mode would be to say:
"do not use this repository when offline, use only cache" for repositories
requiring a network connection, and for those which do not require this
connection (like a local repo), continue to use the repo and not only the
cache.
- xavier
---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
Re: auto download of antlibs
Posted by Steve Loughran <st...@apache.org>.
Xavier Hanin wrote:
> On 5/7/07, Stephen McConnell <mc...@dpml.net> wrote:
>>
>>
>> > -----Original Message-----
>> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
>> > Sent: Friday, 4 May 2007 5:56 PM
>> > To: Ant Developers List
>> > Subject: Re: auto download of antlibs
>> >
>> > On 5/4/07, Steve Loughran <st...@apache.org> wrote:
>> > >
>> > > One thing I've been thinking of this week is how could we work with
>> > > Ivy for automatic antlib download.
>> > >
>> > > No code right now, just some thoughts
>> > >
>> > >
>> > > 1. add a -offline argument to say "we are offline". this will set
>> > > a property, (and.build.offline) and the <offline> test will
>> > > work. It is meant to tell things like Ivy that we are offline. At
>> > > some point we could add some way for Ant to guess whether the net
>> > > is there or not, if java integrates with the OS properly (there is
>> > > an API call for this in J2ME, just not Java SE)
>> >
>> > This makes me think that we could improve how Ivy deal with
>> > online/offline mode. Indeed for the moment Ivy doesn't really
>> > know which repository requires a network access and which
>> > doesn't. It would be nice if Ivy would know that, so that
>> > even if offline mode you could still use alocal repository
>> > (for antlib testing for instance).
>>
>> Are you describing a policy at the level of:
>>
>> a) a multi-project build decision?
>> b) a specific target project build decision?
>> c) a repository access decision?
>> d) some or any of the above?
> I'm describing how Ivy could be improved to better deal with a
> situation where the user do not have network access. You can already
> deal with that pretty efficiently in Ivy, but you have to define
> several settings, to avoid using remote access when it's not
> available. Having something easier for the user in this common
> situation would be interesting.
Maybe every repository in ivyconf.xml would be marked as offline,
meaning they are available when there is no network.When you run ant (or
ivy) with -offline, only offline repositories would be used.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Xavier Hanin <xa...@gmail.com>.
On 5/7/07, Stephen McConnell <mc...@dpml.net> wrote:
>
>
> > -----Original Message-----
> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > Sent: Friday, 4 May 2007 5:56 PM
> > To: Ant Developers List
> > Subject: Re: auto download of antlibs
> >
> > On 5/4/07, Steve Loughran <st...@apache.org> wrote:
> > >
> > > One thing I've been thinking of this week is how could we work with
> > > Ivy for automatic antlib download.
> > >
> > > No code right now, just some thoughts
> > >
> > >
> > > 1. add a -offline argument to say "we are offline". this will set
> > > a property, (and.build.offline) and the <offline> test will
> > > work. It is meant to tell things like Ivy that we are offline. At
> > > some point we could add some way for Ant to guess whether the net
> > > is there or not, if java integrates with the OS properly (there is
> > > an API call for this in J2ME, just not Java SE)
> >
> > This makes me think that we could improve how Ivy deal with
> > online/offline mode. Indeed for the moment Ivy doesn't really
> > know which repository requires a network access and which
> > doesn't. It would be nice if Ivy would know that, so that
> > even if offline mode you could still use alocal repository
> > (for antlib testing for instance).
>
> Are you describing a policy at the level of:
>
> a) a multi-project build decision?
> b) a specific target project build decision?
> c) a repository access decision?
> d) some or any of the above?
I'm describing how Ivy could be improved to better deal with a
situation where the user do not have network access. You can already
deal with that pretty efficiently in Ivy, but you have to define
several settings, to avoid using remote access when it's not
available. Having something easier for the user in this common
situation would be interesting.
>
> >
> > >
> > > 2. when we encounter an element (or even an attr) in an
> > > unknown antlib xmlns, and we want to map that to a
> > > projectcomponent, we hand off resolution to an antlib
> > > resolver. We would have one built in (the failing resolver),
> > > would default to the ivy one if it was present, and
> > > provide some way to let people switch to a different one.
> >
> > This sounds like a good idea.
> >
> > >
> > > 3. an antlib resolver would do the mapping from antlib package to
> > > artifacts (problem one),
> > Yes, and note that we have to consider the version too.
>
> If you assume you are keying of a url, then no ..
>
> In such a scenario you can bring thing back to the url protocol handler and
> delegate the problem to the handler. For example I may want to assert any
> of the following:
>
> a) a specific version artifact
> b) the latest version of an artifact
> c) an artifact with a versioned constraint range
>
> Using Metro/Depot/Transit the following may be equivalent:
>
> artifact:jar:org/apache/ant/ant#1.7
> link:jar:org/apache/ant/ant
>
> The first url references an absolute version, the second is like a symlink
> (typically referencing the latest version).
Fine, so you consider the version.
>
> >
> > > then download the metadata for that artifact, pull it down
> > > and all its artifacts
> > >
> > > 4. we would then <typedef> the lib with the classpath that
> > > is set up by the resolver
> >
> > One question here: is it the responsibility of the resolver
> > to keep artifacts in a cache, or put artifacts in an Ant
> > managed cache.
>
> Isn't that an implementation decision?
I don't think so. If Ant provides a pluggable mechanism for automatic
download of antlibs, it will have to define if once the files are
downloaded they are under Ant control, or stay under control of the
plugin.
>
> > This is important to specify how things are
> > going in offline mode. Ivy already has pretty good support
> > for offline mode, and I think we could improve it (see
> > above). But this is important to consider when specifying the
> > role of the antlib resolver.
>
> What do you mean by offline?
By offline I mean with no network access.
>
> Typically this subject is about policy on resource resolution which is not
> simply a question of establishing a remote connection. Are we making
> assumptions about cache content? Is the cache a trusted repository?
>
> > >
> > > we'd need a metadata tree mapping antlibs to well known
> > packages, but
> > > that is not too hard. JSON, perhaps.
> > Not too hard, except maybe for version information. I'm not
> > sure that it would be really nice to get the latest version
> > by default, making the build system automatically updated is
> > not necessarily a good idea (at least users have to keep very
> > good control over that).
>
> Yep - basically your describing the policy you want to apply with respect to
> artifact resolution. If its absolute versioning are you assuming Dewey
> versioning? If its latest do you mean latest build or latest stable build
> or latest released build?
This is fully configurable in Ivy, so the user can use whatever he wants.
- Xavier
>
> Cheers, Steve.
>
> --------------------------
> Stephen McConnell
> mailto:mcconnell@dpml.net
> http://www.dpml.net
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
RE: auto download of antlibs
Posted by Stephen McConnell <mc...@dpml.net>.
> -----Original Message-----
> From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> Sent: Friday, 4 May 2007 5:56 PM
> To: Ant Developers List
> Subject: Re: auto download of antlibs
>
> On 5/4/07, Steve Loughran <st...@apache.org> wrote:
> >
> > One thing I've been thinking of this week is how could we work with
> > Ivy for automatic antlib download.
> >
> > No code right now, just some thoughts
> >
> >
> > 1. add a -offline argument to say "we are offline". this will set
> > a property, (and.build.offline) and the <offline> test will
> > work. It is meant to tell things like Ivy that we are offline. At
> > some point we could add some way for Ant to guess whether the net
> > is there or not, if java integrates with the OS properly (there is
> > an API call for this in J2ME, just not Java SE)
>
> This makes me think that we could improve how Ivy deal with
> online/offline mode. Indeed for the moment Ivy doesn't really
> know which repository requires a network access and which
> doesn't. It would be nice if Ivy would know that, so that
> even if offline mode you could still use alocal repository
> (for antlib testing for instance).
Are you describing a policy at the level of:
a) a multi-project build decision?
b) a specific target project build decision?
c) a repository access decision?
d) some or any of the above?
>
> >
> > 2. when we encounter an element (or even an attr) in an
> > unknown antlib xmlns, and we want to map that to a
> > projectcomponent, we hand off resolution to an antlib
> > resolver. We would have one built in (the failing resolver),
> > would default to the ivy one if it was present, and
> > provide some way to let people switch to a different one.
>
> This sounds like a good idea.
>
> >
> > 3. an antlib resolver would do the mapping from antlib package to
> > artifacts (problem one),
> Yes, and note that we have to consider the version too.
If you assume you are keying of a url, then no ..
In such a scenario you can bring thing back to the url protocol handler and
delegate the problem to the handler. For example I may want to assert any
of the following:
a) a specific version artifact
b) the latest version of an artifact
c) an artifact with a versioned constraint range
Using Metro/Depot/Transit the following may be equivalent:
artifact:jar:org/apache/ant/ant#1.7
link:jar:org/apache/ant/ant
The first url references an absolute version, the second is like a symlink
(typically referencing the latest version).
>
> > then download the metadata for that artifact, pull it down
> > and all its artifacts
> >
> > 4. we would then <typedef> the lib with the classpath that
> > is set up by the resolver
>
> One question here: is it the responsibility of the resolver
> to keep artifacts in a cache, or put artifacts in an Ant
> managed cache.
Isn't that an implementation decision?
> This is important to specify how things are
> going in offline mode. Ivy already has pretty good support
> for offline mode, and I think we could improve it (see
> above). But this is important to consider when specifying the
> role of the antlib resolver.
What do you mean by offline?
Typically this subject is about policy on resource resolution which is not
simply a question of establishing a remote connection. Are we making
assumptions about cache content? Is the cache a trusted repository?
> >
> > we'd need a metadata tree mapping antlibs to well known
> packages, but
> > that is not too hard. JSON, perhaps.
> Not too hard, except maybe for version information. I'm not
> sure that it would be really nice to get the latest version
> by default, making the build system automatically updated is
> not necessarily a good idea (at least users have to keep very
> good control over that).
Yep - basically your describing the policy you want to apply with respect to
artifact resolution. If its absolute versioning are you assuming Dewey
versioning? If its latest do you mean latest build or latest stable build
or latest released build?
Cheers, Steve.
--------------------------
Stephen McConnell
mailto:mcconnell@dpml.net
http://www.dpml.net
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Xavier Hanin <xa...@gmail.com>.
On 5/4/07, Steve Loughran <st...@apache.org> wrote:
>
> One thing I've been thinking of this week is how could we work with Ivy
> for automatic antlib download.
>
> No code right now, just some thoughts
>
>
> 1. add a -offline argument to say "we are offline". this will set a
> property, (and.build.offline) and the <offline> test will work. It is
> meant to tell things like Ivy that we are offline. At some point we
> could add some way for Ant to guess whether the net is there or not, if
> java integrates with the OS properly (there is an API call for this in
> J2ME, just not Java SE)
This makes me think that we could improve how Ivy deal with
online/offline mode. Indeed for the moment Ivy doesn't really know
which repository requires a network access and which doesn't. It would
be nice if Ivy would know that, so that even if offline mode you could
still use alocal repository (for antlib testing for instance).
>
> 2. when we encounter an element (or even an attr) in an unknown antlib
> xmlns, and we want to map that to a projectcomponent, we hand off
> resolution to an antlib resolver. We would have one built in (the
> failing resolver), would default to the ivy one if it was present, and
> provide some way to let people switch to a different one.
This sounds like a good idea.
>
> 3. an antlib resolver would do the mapping from antlib package to
> artifacts (problem one),
Yes, and note that we have to consider the version too.
> then download the metadata for that artifact,
> pull it down and all its artifacts
>
> 4. we would then <typedef> the lib with the classpath that is set up by
> the resolver
One question here: is it the responsibility of the resolver to keep
artifacts in a cache, or put artifacts in an Ant managed cache. This
is important to specify how things are going in offline mode. Ivy
already has pretty good support for offline mode, and I think we could
improve it (see above). But this is important to consider when
specifying the role of the antlib resolver.
>
> we'd need a metadata tree mapping antlibs to well known packages, but
> that is not too hard. JSON, perhaps.
Not too hard, except maybe for version information. I'm not sure that
it would be really nice to get the latest version by default, making
the build system automatically updated is not necessarily a good idea
(at least users have to keep very good control over that).
Xavier
>
> -steve
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
--
Learn Ivy at ApacheCon: http://www.eu.apachecon.com/
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Stefan Bodewig <bo...@apache.org>.
On Fri, 4 May 2007, Peter Reilly <pe...@gmail.com> wrote:
> On 5/4/07, Stefan Bodewig <bo...@apache.org> wrote:
>> On Fri, 04 May 2007, Steve Loughran <st...@apache.org> wrote:
>> > we'd need a metadata tree mapping antlibs to well known packages,
>> > but that is not too hard. JSON, perhaps.
>>
>> Not sure. Who'd maintain it?
>
> It should be some xml format. I think that it should be on the ant
> site and ant committers would be the updaters of it. - this would
> be similar to the related projects page -
I know the projects page, and I know that sometimes we are a bit
sloppy in updating it, even if people send in patches. At one point
we started to recommend that people use the Wiki instead.
Using the Wiki is fine as long as the content is processed by humans,
but it is unusable for automated processing.
> of course this raises the issues of version. One may not want the
> lastest version of a particular antlib.
Could be a fragment identifier or an appended XPath or something like
that
antlib:org.apache.ant.antunit#1.1
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Peter Reilly <pe...@gmail.com>.
On 5/4/07, Stefan Bodewig <bo...@apache.org> wrote:
> On Fri, 04 May 2007, Steve Loughran <st...@apache.org> wrote:
>
> > 1. add a -offline argument to say "we are offline". this will set a
> > property, (and.build.offline) and the <offline> test will work.
>
> Do I sense oata.utils.NetworkUtils? Might contain some Proxy
> configuration (and if possible detection) code as well.
>
> > 2. when we encounter an element (or even an attr) in an unknown
> > antlib xmlns, and we want to map that to a projectcomponent, we hand
> > off resolution to an antlib resolver. We would have one built in
> > (the failing resolver), would default to the ivy one if it was
> > present, and provide some way to let people switch to a different
> > one.
>
> OK.
>
> > 3. an antlib resolver would do the mapping from antlib package to
> > artifacts (problem one),
>
> actually a pretty big problem.
>
> > then download the metadata for that artifact, pull it down and all
> > its artifacts
> >
> > 4. we would then <typedef> the lib with the classpath that is set up
> > by the resolver
>
> sounds right.
>
> > we'd need a metadata tree mapping antlibs to well known packages,
> > but that is not too hard. JSON, perhaps.
>
> Not sure. Who'd maintain it?
It should be some xml format.
I think that it should be on the ant site
and ant committers would be the updaters of it.
- this would be similar to the
related projects page - http://ant.apache.org/projects.html
but have a separate url for each antlib.
? somthing like: http://ant.apache.org/antlibdefintions/<package>.xml
for example: http://ant.apache.org/antlibdefintions/net.sf.antcontrib.xml
of course this raises the issues of version. One may not want the lastest
version of a particular antlib.
Peter
> Maintaining it for our own Antlibs is
> easy, but we wouldn't want the mechanism to only apply for them. And
> I'd be scared of the security implications of a Wiki driven list or
> something even close to that.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Stefan Bodewig <bo...@apache.org>.
On Fri, 4 May 2007, Xavier Hanin <xa...@gmail.com> wrote:
> On 5/4/07, Stefan Bodewig <bo...@apache.org> wrote:
>> On Fri, 04 May 2007, Steve Loughran <st...@apache.org> wrote:
>> > we'd need a metadata tree mapping antlibs to well known packages,
>> > but that is not too hard. JSON, perhaps.
>>
>> Not sure. Who'd maintain it? Maintaining it for our own Antlibs
>> is easy, but we wouldn't want the mechanism to only apply for them.
>> And I'd be scared of the security implications of a Wiki driven
>> list or something even close to that.
>
> You make a good point. So maybe this would require all information
> (module identifier and version) to be in the antlib URL, thus
> requiring another antlib url format (maybe with a distinct
> protocol), which is not really going in the same direction as you
> suggested, steve.
At least that would allow us to live without a central URI -> antlib
artifact mapping, I'd prefer that.
> Another option from the top of my head: build a module identifier
> from the package name, even if it's not very accurate, the only
> purpose is to get something unique. It could something like: org =
> package name; module = last part of the package name eg:
> org.apache.ivy.ant => org = org.apache.ivy.ant; module = ant This
> module would not be the antlib module, but only a module with its
> only artifact being metadata about the module containing the actual
> antlib. This metadata could be in a simple format, JSON, XML or
> properties file. Then we can use this metadata to actually download
> the antlib. The remaining problem is version information.
That would require an extra level of indirection.
I think I'm more leaning towards a different URI scheme that encoded
all information that we need (including version).
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Xavier Hanin <xa...@gmail.com>.
On 5/4/07, Stefan Bodewig <bo...@apache.org> wrote:
> On Fri, 04 May 2007, Steve Loughran <st...@apache.org> wrote:
>
> > 1. add a -offline argument to say "we are offline". this will set a
> > property, (and.build.offline) and the <offline> test will work.
>
> Do I sense oata.utils.NetworkUtils? Might contain some Proxy
> configuration (and if possible detection) code as well.
>
> > 2. when we encounter an element (or even an attr) in an unknown
> > antlib xmlns, and we want to map that to a projectcomponent, we hand
> > off resolution to an antlib resolver. We would have one built in
> > (the failing resolver), would default to the ivy one if it was
> > present, and provide some way to let people switch to a different
> > one.
>
> OK.
>
> > 3. an antlib resolver would do the mapping from antlib package to
> > artifacts (problem one),
>
> actually a pretty big problem.
>
> > then download the metadata for that artifact, pull it down and all
> > its artifacts
> >
> > 4. we would then <typedef> the lib with the classpath that is set up
> > by the resolver
>
> sounds right.
>
> > we'd need a metadata tree mapping antlibs to well known packages,
> > but that is not too hard. JSON, perhaps.
>
> Not sure. Who'd maintain it? Maintaining it for our own Antlibs is
> easy, but we wouldn't want the mechanism to only apply for them. And
> I'd be scared of the security implications of a Wiki driven list or
> something even close to that.
You make a good point. So maybe this would require all information
(module identifier and version) to be in the antlib URL, thus
requiring another antlib url format (maybe with a distinct protocol),
which is not really going in the same direction as you suggested,
steve.
Another option from the top of my head: build a module identifier from
the package name, even if it's not very accurate, the only purpose is
to get something unique. It could something like: org = package name;
module = last part of the package name
eg: org.apache.ivy.ant => org = org.apache.ivy.ant; module = ant
This module would not be the antlib module, but only a module with its
only artifact being metadata about the module containing the actual
antlib. This metadata could be in a simple format, JSON, XML or
properties file. Then we can use this metadata to actually download
the antlib. The remaining problem is version information.
Xavier
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
--
Learn Ivy at ApacheCon: http://www.eu.apachecon.com/
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Steve Loughran <st...@apache.org>.
Stephen McConnell wrote:
>> -----Original Message-----
>> From: Stefan Bodewig [mailto:bodewig@apache.org]
>
>>> 3. an antlib resolver would do the mapping from antlib package to
>>> artifacts (problem one),
>> actually a pretty big problem.
>
> Which may explain why two JSRs are dealing with the subject:
>
> http://jcp.org/en/jsr/detail?id=294
> http://jcp.org/en/jsr/detail?id=277
>
gosh, two! aren't we lucky!
-steve
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
RE: auto download of antlibs
Posted by Stephen McConnell <mc...@dpml.net>.
> -----Original Message-----
> From: Stefan Bodewig [mailto:bodewig@apache.org]
> > 3. an antlib resolver would do the mapping from antlib package to
> > artifacts (problem one),
>
> actually a pretty big problem.
Which may explain why two JSRs are dealing with the subject:
http://jcp.org/en/jsr/detail?id=294
http://jcp.org/en/jsr/detail?id=277
Cheers, Steve.
--------------------------
Stephen McConnell
mailto:mcconnell@dpml.net
http://www.dpml.net
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: auto download of antlibs
Posted by Stefan Bodewig <bo...@apache.org>.
On Fri, 04 May 2007, Steve Loughran <st...@apache.org> wrote:
> 1. add a -offline argument to say "we are offline". this will set a
> property, (and.build.offline) and the <offline> test will work.
Do I sense oata.utils.NetworkUtils? Might contain some Proxy
configuration (and if possible detection) code as well.
> 2. when we encounter an element (or even an attr) in an unknown
> antlib xmlns, and we want to map that to a projectcomponent, we hand
> off resolution to an antlib resolver. We would have one built in
> (the failing resolver), would default to the ivy one if it was
> present, and provide some way to let people switch to a different
> one.
OK.
> 3. an antlib resolver would do the mapping from antlib package to
> artifacts (problem one),
actually a pretty big problem.
> then download the metadata for that artifact, pull it down and all
> its artifacts
>
> 4. we would then <typedef> the lib with the classpath that is set up
> by the resolver
sounds right.
> we'd need a metadata tree mapping antlibs to well known packages,
> but that is not too hard. JSON, perhaps.
Not sure. Who'd maintain it? Maintaining it for our own Antlibs is
easy, but we wouldn't want the mechanism to only apply for them. And
I'd be scared of the security implications of a Wiki driven list or
something even close to that.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
RE: auto download of antlibs
Posted by Stephen McConnell <mc...@dpml.net>.
> -----Original Message-----
> From: Steve Loughran [mailto:stevel@apache.org]
> Sent: Friday, 4 May 2007 5:27 PM
> To: Ant Developers List
> Subject: auto download of antlibs
>
>
> One thing I've been thinking of this week is how could we
> work with Ivy for automatic antlib download.
>
> No code right now, just some thoughts
>
>
> 1. add a -offline argument to say "we are offline". this will
> set a property, (and.build.offline) and the <offline> test
> will work. It is meant to tell things like Ivy that we are
> offline. At some point we could add some way for Ant to guess
> whether the net is there or not, if java integrates with the
> OS properly (there is an API call for this in J2ME, just not Java SE)
Sounds like we are mixing a characteristic of a network connection with a
policy decision. When we talk about being offline - we are normally
describing a situation under which a TCP/IP connection is unavailable. When
developers discuss '-offline' as a policy what this often translates to is
that they want to assert a rule preventing the automatic downloading of
artifacts. Recognizing this difference enables recognition of a bunch of
other possibilities:
a) I have a artifacts that I depend upon I want to modify the logic
used in the resolution of said artifacts
- do I want to resolve artifacts over a remote network
connection?
- are internet resolvable connections ok?
- am I really talking about shades of gray relative to
a collection of repositories - in effect, am I designing
a artifact retrieval policy
- am I talking about trust?
- am I talking about artifact integrity?
b) What is the state of my cache?
- was the cached artifact established with the same policy
as the policy I'm currently asserting
- does my cache management system associate existence
policy with the artifact
- it the cached object verifiable
- does my build policy imply anything on my caching policy
- is my cache sharable and if it is, what am I asserting
in terms of policy
c) What is the relationship between build process, cache, and
shared repositories?
- am I trusted?
- how can clients validate me, my cache, my policy, my
artifacts
- does my build process trust my cache (given that interim
dependent builds may be using policies that are not under
my control - e.g. Eric uses Antlib X which has dependencies
on jar X, Y, and Z
> 2. when we encounter an element (or even an attr) in an
> unknown antlib xmlns, and we want to map that to a
> projectcomponent, we hand off resolution to an antlib
> resolver. We would have one built in (the failing resolver),
> would default to the ivy one if it was present, and provide
> some way to let people switch to a different one.
You can do this without mentioning Ivy so long as you have the mechanisms to
include URL protocol handlers.
Example of a working build.xml file:
<project default="install" xmlns:x="antlib:dpml.tools">
<x:import uri="local:template:dpml/tools/standard"/>
</project>
The above build works if I put around about 5 specialized jar files into my
./ant/lib directory and invoke:
$ ant
Or more typically, the mechanism I use on more than one hundred Ant based
projects (without anything in .ant/lib):
$ build
In both cases what I am doing is making Ant URL aware - as such
"local:template:dpml/tools/standard" is recognized as a protocol handler and
the handler recognizes content types and maps into place the appropriate
content handler which in this case simply drags in template build file. The
template file contains the following statements:
<project name="standard" default="install"
xmlns:x="antlib:dpml.tools" >
<target name="init">
<x:init/>
</target>
...
</project>
The <x:init/> task establishes an project helper that deals with uris for
things like antlib plugins (and a bunch of other protocol handlers that let
me deal with cached resources, resources on remote hosts, local preferences,
services based on independent virtual machines, deployment scenarios for
local or remote applications, basically most of the things you need in a
fully functional build environment. In effect - it basically does the setup
of the machinery needed to override Ant behaviour when resolving tasks and
data types using the URL machinery bundled in the JVM.
> 3. an antlib resolver would do the mapping from antlib
> package to artifacts (problem one), then download the
> metadata for that artifact, pull it down and all its artifacts
Sounds like a protocol handler that captures sufficient information to
represent a classloader chain together with som information about the
deployment target. One example that approaches this is the DPML part
definition which encapsulates (a) generic info, (b) a deployment strategy,
and (c) a classloader chain definition.
http://www.dpml.net/metro/parts/index.html
In the DPML model the deployment strategy is dynamic and in out environment
we have several strategies we use on a regular basis. One of these is a
antlib strategy which simply identifies the path to the antlib resource and
the namespace uri that the resource should be bound to.
> 4. we would then <typedef> the lib with the classpath that is
> set up by the resolver
Depending on a classpath is generally not sufficient although recognition of
classpath definitions is a part of the problem. In effect - you need to deal
with the definition of a classloader chain. Basically a class path defines
a classloader and a classloader is entry in a chain of classloaders
connected by parent relationships. The chain (versus path) abstraction is
needed in order to isolate implementation classes from service classes. In
the extreme scenario you will see examples involving a public API, and
management SPI, and implementation classloader. In some scenario you will
also see plugins (e.g. antlibs) being loaded as children of an SPI or impl
classloader (which result in chain extension).
>
> we'd need a metadata tree mapping antlibs to well known
> packages, but that is not too hard. JSON, perhaps.
It would be worth looking at the JSR 277 draft spec - in particular the
topics dealing with repositories, runtime module construction, and service
loading.
http://jcp.org/en/jsr/detail?id=277
Cheers, Steve.
--------------------------
Stephen McConnell
mailto:mcconnell@dpml.net
http://www.dpml.net
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org