Dependency management of the easyant plugins

[Nicolas Lalevée <ni...@hibnet.org>]

I would like to discuss the way the plugins are declaring the dependencies with each other.

In easyant 0.8, there was some easyant include. Since it is an include and not an import, each plugin is somehow embedding its dependencies at runtime without directly exposing them to the plugins which are dependencies of it. So if two plugins wants to share some targets, they shouldn't declare these dependencies, it would be the responsibility to the one which includes them both to also include the plugin in common.

In the refactoring of EASYANT-27 [1], I changed the includes with imports so there is no transitive issue about using extension point. Using extension point is kind of different of phases since the dependencies between targets are now explicit. Phases were the workflow every plugins was relying on. Now each plugin expose its points of extension, so each plugins is actually relying on each other. So include was not possible.
I hope this is OK for everyone ?

Then I would like to go further. Because each plugin manage its own dependency, each import is an Ivy resolve behind the hood, there is not much dependency management of the plugins. Ivy is here barely used more than just downloading build files. There is no transitive dependency or conflict management.
For instance:
* build-std-java imports compile-java and ivy-provisioning
* compile-java imports ivy-provisioning
In this example, ivy-provisioning is resolved two times, and it is possible that the two resolved versions are not the same.

I then suggest to move to a real dependency management of the plugins, by really using Ivy. Each plugin rather than importing himself its dependencies, it would just declares them. As a proof of concept, I implemented an Ant task that just does that. See the "experiment" folder I have just committed to test it.
To run the sample, put the jars of both easyant and ivy into sample/ea-jars.
Two special ant properties controls the task behaviour:
* easyant.refresh : set it to true to force a resolve. Otherwise the last resolve report will be used
* easyant.localrepo.basedir : set folder in which the scripts are retrieved, useful for embedding the build. If not set, the scripts are imported directly from the cache.

As you could see the plugin dependencies are nicely resolved. So nicely that a script can have dependencies reusable in the script itself. See the script of org.apache.easyant#groovyfrontend. The classpath to find the project helper is computed by Ivy from the ivy.xml of the plugin.

WDYT of the idea ?

Nicolas

Re: Dependency management of the easyant plugins

[Nicolas Lalevée <ni...@hibnet.org>]

Le 24 avr. 2011 à 01:21, Jean-Louis Boudart a écrit :

> Le 22 avril 2011 13:43, Nicolas Lalevée <ni...@hibnet.org> a écrit
> :
> 
>> I would like to discuss the way the plugins are declaring the dependencies
>> with each other.
>> 
>> In easyant 0.8, there was some easyant include. Since it is an include and
>> not an import, each plugin is somehow embedding its dependencies at runtime
>> without directly exposing them to the plugins which are dependencies of it.
>> So if two plugins wants to share some targets, they shouldn't declare these
>> dependencies, it would be the responsibility to the one which includes them
>> both to also include the plugin in common.
>> 
>> In the refactoring of EASYANT-27 [1], I changed the includes with imports
>> so there is no transitive issue about using extension point. Using extension
>> point is kind of different of phases since the dependencies between targets
>> are now explicit. Phases were the workflow every plugins was relying on. Now
>> each plugin expose its points of extension, so each plugins is actually
>> relying on each other. So include was not possible.
>> I hope this is OK for everyone ?
>> 
> Change on import vs include looks good for me even if we loose the ability
> to "alias" a module.
> Prefixing all targets of a plugins by hand will be error prone but maybe we
> can find a workaround for this.
> 
> If i understand correctly, buildtype are supposed to load a set of plugins
> and plug them through their extensionPoint, am i wrong ?
> At first glance it seems to offer much more flexibility than what we had
> before.
> 
> Let's now discuss about "build lifecycle".
> In easyant early stages we used a set of phases defining dependency between
> each others.
> Some plugins was "plugged by default" to standard phases. And buildtypes
> were loading standard phases and sets of plugins.
> There is at least two drawback of this solution :
> 
>   - As a enduser it was a bit confusing as you seen  all "PHASES" even if
>   they don't contains any target. I guess enduser will all the time call the
>   same phases (compile, test, package, verify, release) but will probably
>   never call most of existing phases like process-sources etc...
> 
> 
>   - Another drawback of this model was the coupling between plugins and
>   phases. How someone can use it's own "lifecycle" ? Should he rebind all
>   targets to his own phases ?
> 
> 
> First can be addressed by exposing "high level target" for end users such as
> compile, test, package, verify, etc.. (i'm not sure we need more). To limit
> "copy/paste" between all our buildtype we should put thoses "high level
> target" in a common plugin.
> 
> Second point is defacto addressed by your refactoring.
> 
> My last comment on the refactoring would be about relations between plugins.
> I don't think we need to link plugins except if we had a good reason for it.
> And if we need to link two plugins it would probably better to link on an
> extensionPoint provided by an abstract module. In other cases linking
> plugins should be done by buildtypes or by the enduser itself.
> 
> For example compile-java <import> ivy-provisioning. This sounds like a bad
> choice for me as provisioning (filling a classpath for compilation) can be
> done by others plugins such as "deps-lib".
> In opposite i really like how cobertura / emma plugins interacts with tests
> plugins.
> Both cobertura and emma plugins relies on test extension points defined in
> an abtract-test plugin.

I agree with the idea. Ultimately we never want to declare dependencies on implementations but always on APIs.

I may have a lot to talk about on this topic, as this make me think of the OSGi dependency model, which explicitly makes the differences between a dependency on an API and a dependency on an implementation [1]. But this is quite fuzzy in my mind at the moment, so let's keep it simple.

I choose here direct dependency rather than creating "abstract plugin":
* first by laziness :)
* creating an abstraction, an API, is always tedious, as it often means maintaing backward compatibility. So you need to be right before exposing it. Since Easyant is in the early stage, I preferred to not open doors.
* having plugins depends on abstraction rather than on implementation has the consequence that transitive dependencies are less useful for the end user. For instance for compile-java, if it depends on an "abstract-provisionning", then an end user wanting to depend on compile-java will have to also declare a dependency on an implementation of an "abstract-provisionning", like "ivy-provisionning". Since we only have one implementation today, it seems painful.

But I agree, there is no reason to not have abstraction between every modules. We should definitively tend towards that. But the reasons above make me go there slowly.
I have no objection if you change some of the current dependency and introduce some abstract plugins. Feel free to make some. I would just suggest that these abstract plugin should not be created automatically on each plugin implementation.


>> Then I would like to go further. Because each plugin manage its own
>> dependency, each import is an Ivy resolve behind the hood, there is not much
>> dependency management of the plugins. Ivy is here barely used more than just
>> downloading build files. There is no transitive dependency or conflict
>> management.
>> For instance:
>> * build-std-java imports compile-java and ivy-provisioning
>> * compile-java imports ivy-provisioning
>> In this example, ivy-provisioning is resolved two times, and it is possible
>> that the two resolved versions are not the same.
>> 
>> I then suggest to move to a real dependency management of the plugins, by
>> really using Ivy. Each plugin rather than importing himself its
>> dependencies, it would just declares them. As a proof of concept, I
>> implemented an Ant task that just does that. See the "experiment" folder I
>> have just committed to test it.
>> To run the sample, put the jars of both easyant and ivy into
>> sample/ea-jars.
>> Two special ant properties controls the task behaviour:
>> * easyant.refresh : set it to true to force a resolve. Otherwise the last
>> resolve report will be used
>> * easyant.localrepo.basedir : set folder in which the scripts are
>> retrieved, useful for embedding the build. If not set, the scripts are
>> imported directly from the cache.
>> 
>> As you could see the plugin dependencies are nicely resolved. So nicely
>> that a script can have dependencies reusable in the script itself. See the
>> script of org.apache.easyant#groovyfrontend. The classpath to find the
>> project helper is computed by Ivy from the ivy.xml of the plugin.
>> 
>> WDYT of the idea ?
>> 
> 
> BIG +1 :)

About that part I forgot to mention one of the consequence. If we manage dependencies between easyant plugins with ivy.xml files, do we want to continue to support the ea namespace in the module.ivy ? Because this will begin to look like writing an ivy.xml within an ivy.xml file.

Nicolas, having some spare agin :)

[1] http://www.osgi.org/blog/2011/05/unbearable-lightness-of-jigsaw.html

Re: Dependency management of the easyant plugins

[Nicolas Lalevée <ni...@hibnet.org>]

Just a little mail to say that I didn't forgot this thread, I just don't have much free time nowadays, I'm about to switch jobs.

see you
Nicolas

Le 24 avr. 2011 à 01:21, Jean-Louis Boudart a écrit :

> Le 22 avril 2011 13:43, Nicolas Lalevée <ni...@hibnet.org> a écrit
> :
> 
>> I would like to discuss the way the plugins are declaring the dependencies
>> with each other.
>> 
>> In easyant 0.8, there was some easyant include. Since it is an include and
>> not an import, each plugin is somehow embedding its dependencies at runtime
>> without directly exposing them to the plugins which are dependencies of it.
>> So if two plugins wants to share some targets, they shouldn't declare these
>> dependencies, it would be the responsibility to the one which includes them
>> both to also include the plugin in common.
>> 
>> In the refactoring of EASYANT-27 [1], I changed the includes with imports
>> so there is no transitive issue about using extension point. Using extension
>> point is kind of different of phases since the dependencies between targets
>> are now explicit. Phases were the workflow every plugins was relying on. Now
>> each plugin expose its points of extension, so each plugins is actually
>> relying on each other. So include was not possible.
>> I hope this is OK for everyone ?
>> 
> Change on import vs include looks good for me even if we loose the ability
> to "alias" a module.
> Prefixing all targets of a plugins by hand will be error prone but maybe we
> can find a workaround for this.
> 
> If i understand correctly, buildtype are supposed to load a set of plugins
> and plug them through their extensionPoint, am i wrong ?
> At first glance it seems to offer much more flexibility than what we had
> before.
> 
> Let's now discuss about "build lifecycle".
> In easyant early stages we used a set of phases defining dependency between
> each others.
> Some plugins was "plugged by default" to standard phases. And buildtypes
> were loading standard phases and sets of plugins.
> There is at least two drawback of this solution :
> 
>   - As a enduser it was a bit confusing as you seen  all "PHASES" even if
>   they don't contains any target. I guess enduser will all the time call the
>   same phases (compile, test, package, verify, release) but will probably
>   never call most of existing phases like process-sources etc...
> 
> 
>   - Another drawback of this model was the coupling between plugins and
>   phases. How someone can use it's own "lifecycle" ? Should he rebind all
>   targets to his own phases ?
> 
> 
> First can be addressed by exposing "high level target" for end users such as
> compile, test, package, verify, etc.. (i'm not sure we need more). To limit
> "copy/paste" between all our buildtype we should put thoses "high level
> target" in a common plugin.
> 
> Second point is defacto addressed by your refactoring.
> 
> My last comment on the refactoring would be about relations between plugins.
> I don't think we need to link plugins except if we had a good reason for it.
> And if we need to link two plugins it would probably better to link on an
> extensionPoint provided by an abstract module. In other cases linking
> plugins should be done by buildtypes or by the enduser itself.
> 
> For example compile-java <import> ivy-provisioning. This sounds like a bad
> choice for me as provisioning (filling a classpath for compilation) can be
> done by others plugins such as "deps-lib".
> In opposite i really like how cobertura / emma plugins interacts with tests
> plugins.
> Both cobertura and emma plugins relies on test extension points defined in
> an abtract-test plugin.
> 
> 
> 
> 
>> 
>> Then I would like to go further. Because each plugin manage its own
>> dependency, each import is an Ivy resolve behind the hood, there is not much
>> dependency management of the plugins. Ivy is here barely used more than just
>> downloading build files. There is no transitive dependency or conflict
>> management.
>> For instance:
>> * build-std-java imports compile-java and ivy-provisioning
>> * compile-java imports ivy-provisioning
>> In this example, ivy-provisioning is resolved two times, and it is possible
>> that the two resolved versions are not the same.
>> 
>> I then suggest to move to a real dependency management of the plugins, by
>> really using Ivy. Each plugin rather than importing himself its
>> dependencies, it would just declares them. As a proof of concept, I
>> implemented an Ant task that just does that. See the "experiment" folder I
>> have just committed to test it.
>> To run the sample, put the jars of both easyant and ivy into
>> sample/ea-jars.
>> Two special ant properties controls the task behaviour:
>> * easyant.refresh : set it to true to force a resolve. Otherwise the last
>> resolve report will be used
>> * easyant.localrepo.basedir : set folder in which the scripts are
>> retrieved, useful for embedding the build. If not set, the scripts are
>> imported directly from the cache.
>> 
>> As you could see the plugin dependencies are nicely resolved. So nicely
>> that a script can have dependencies reusable in the script itself. See the
>> script of org.apache.easyant#groovyfrontend. The classpath to find the
>> project helper is computed by Ivy from the ivy.xml of the plugin.
>> 
>> WDYT of the idea ?
>> 
> 
> BIG +1 :)
> 
> -- 
> Jean Louis Boudart
> Independent consultant
> Project Lead http://www.easyant.org

Re: Dependency management of the easyant plugins

[Jean-Louis Boudart <je...@gmail.com>]

Le 22 avril 2011 13:43, Nicolas Lalevée <ni...@hibnet.org> a écrit
:

> I would like to discuss the way the plugins are declaring the dependencies
> with each other.
>
> In easyant 0.8, there was some easyant include. Since it is an include and
> not an import, each plugin is somehow embedding its dependencies at runtime
> without directly exposing them to the plugins which are dependencies of it.
> So if two plugins wants to share some targets, they shouldn't declare these
> dependencies, it would be the responsibility to the one which includes them
> both to also include the plugin in common.
>
> In the refactoring of EASYANT-27 [1], I changed the includes with imports
> so there is no transitive issue about using extension point. Using extension
> point is kind of different of phases since the dependencies between targets
> are now explicit. Phases were the workflow every plugins was relying on. Now
> each plugin expose its points of extension, so each plugins is actually
> relying on each other. So include was not possible.
> I hope this is OK for everyone ?
>
Change on import vs include looks good for me even if we loose the ability
to "alias" a module.
Prefixing all targets of a plugins by hand will be error prone but maybe we
can find a workaround for this.

If i understand correctly, buildtype are supposed to load a set of plugins
and plug them through their extensionPoint, am i wrong ?
At first glance it seems to offer much more flexibility than what we had
before.

Let's now discuss about "build lifecycle".
In easyant early stages we used a set of phases defining dependency between
each others.
Some plugins was "plugged by default" to standard phases. And buildtypes
were loading standard phases and sets of plugins.
There is at least two drawback of this solution :

   - As a enduser it was a bit confusing as you seen  all "PHASES" even if
   they don't contains any target. I guess enduser will all the time call the
   same phases (compile, test, package, verify, release) but will probably
   never call most of existing phases like process-sources etc...


   - Another drawback of this model was the coupling between plugins and
   phases. How someone can use it's own "lifecycle" ? Should he rebind all
   targets to his own phases ?


First can be addressed by exposing "high level target" for end users such as
compile, test, package, verify, etc.. (i'm not sure we need more). To limit
"copy/paste" between all our buildtype we should put thoses "high level
target" in a common plugin.

Second point is defacto addressed by your refactoring.

My last comment on the refactoring would be about relations between plugins.
I don't think we need to link plugins except if we had a good reason for it.
And if we need to link two plugins it would probably better to link on an
extensionPoint provided by an abstract module. In other cases linking
plugins should be done by buildtypes or by the enduser itself.

For example compile-java <import> ivy-provisioning. This sounds like a bad
choice for me as provisioning (filling a classpath for compilation) can be
done by others plugins such as "deps-lib".
In opposite i really like how cobertura / emma plugins interacts with tests
plugins.
Both cobertura and emma plugins relies on test extension points defined in
an abtract-test plugin.




>
> Then I would like to go further. Because each plugin manage its own
> dependency, each import is an Ivy resolve behind the hood, there is not much
> dependency management of the plugins. Ivy is here barely used more than just
> downloading build files. There is no transitive dependency or conflict
> management.
> For instance:
> * build-std-java imports compile-java and ivy-provisioning
> * compile-java imports ivy-provisioning
> In this example, ivy-provisioning is resolved two times, and it is possible
> that the two resolved versions are not the same.
>
> I then suggest to move to a real dependency management of the plugins, by
> really using Ivy. Each plugin rather than importing himself its
> dependencies, it would just declares them. As a proof of concept, I
> implemented an Ant task that just does that. See the "experiment" folder I
> have just committed to test it.
> To run the sample, put the jars of both easyant and ivy into
> sample/ea-jars.
> Two special ant properties controls the task behaviour:
> * easyant.refresh : set it to true to force a resolve. Otherwise the last
> resolve report will be used
> * easyant.localrepo.basedir : set folder in which the scripts are
> retrieved, useful for embedding the build. If not set, the scripts are
> imported directly from the cache.
>
> As you could see the plugin dependencies are nicely resolved. So nicely
> that a script can have dependencies reusable in the script itself. See the
> script of org.apache.easyant#groovyfrontend. The classpath to find the
> project helper is computed by Ivy from the ivy.xml of the plugin.
>
> WDYT of the idea ?
>

BIG +1 :)

-- 
Jean Louis Boudart
Independent consultant
Project Lead http://www.easyant.org