Dependency handling for bundle lists

[Carsten Ziegeler <cz...@apache.org>]

Hi,

thanks to Justin we now have a nice way of defining the bundle list
while at the same time being fully integrated in the Maven lifecycle
(see SLING-2273).
Therefore I reopened SLING-2265 and SLING-2194.

I really think we should just provide one way for defining a bundle
list - as the current way with the xml file has no downsides anymore,
I don't think we should support the other two.

There has always been the argument that having the configuration
inside the plugin configuration in the pom allows to use tools like
m2e etc. but that's only true to some degree. If we want support for
our plugin, we need to write a m2e plugin (if that's the right term).
If we really want to go down this way, I would opt for using exactly
the same xml format we use for the external file but just inside the
configuration of the plugin. So people have just to learn one syntax.
But I really would only go this way if someone is providing the ide
integration plugin as well - without that, I don't see any point.
Especially as the configuration blows up the pom.

Carsten
-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Dependency handling for bundle lists

[Carsten Ziegeler <cz...@apache.org>]

It seems we have consensus here, let's keep the bundle list xml and
remove all others ways for this plugin.

Regards
Carsten
-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Dependency handling for bundle lists

[Bertrand Delacretaz <bd...@apache.org>]

On Thu, Nov 17, 2011 at 3:27 PM, Carsten Ziegeler <cz...@apache.org> wrote:
> 2011/11/17 Jukka Zitting <ju...@gmail.com>:
>> ...Will the launchpad plugin be able to pick up and
>> use such an upstream dependency (a zip file with all the relevant
>> bundles and configuration inside it in correct locations) when
>> building a launcher jar?
>>
> No, you need to extract this into the target folder

Which is easy to do with maven-dependency-plugin.

-Bertrand

Re: Dependency handling for bundle lists

[Carsten Ziegeler <cz...@apache.org>]

2011/11/17 Jukka Zitting <ju...@gmail.com>:
> Hi,
>
> On Thu, Nov 17, 2011 at 2:40 PM, Carsten Ziegeler <cz...@apache.org> wrote:
>> In addition, if people really want to use all the magic with
>> transitive dependencies etc. (which I really discourage from using in
>> this case), then they can simply use the dependency plugin. In the
>> end, the launchpad plugin creates a well documented folder structure
>> nd then zips everything. So it is possible to create the same
>> structure with the dependency plugin (or an ant task or whatever you
>> prefer) and then you're done.
>
> I can live with that. Will the launchpad plugin be able to pick up and
> use such an upstream dependency (a zip file with all the relevant
> bundles and configuration inside it in correct locations) when
> building a launcher jar?
>
No, you need to extract this into the target folder

Carsten

> BR,
>
> Jukka Zitting
>



-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Dependency handling for bundle lists

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Thu, Nov 17, 2011 at 2:40 PM, Carsten Ziegeler <cz...@apache.org> wrote:
> In addition, if people really want to use all the magic with
> transitive dependencies etc. (which I really discourage from using in
> this case), then they can simply use the dependency plugin. In the
> end, the launchpad plugin creates a well documented folder structure
> nd then zips everything. So it is possible to create the same
> structure with the dependency plugin (or an ant task or whatever you
> prefer) and then you're done.

I can live with that. Will the launchpad plugin be able to pick up and
use such an upstream dependency (a zip file with all the relevant
bundles and configuration inside it in correct locations) when
building a launcher jar?

BR,

Jukka Zitting

Re: Dependency handling for bundle lists

[Carsten Ziegeler <cz...@apache.org>]

I think, we should go forward with the extern xml configuration file
and remove the other ways. I think we shouldn't support several
different ways of doing this as this will cause more confusion than a
single well documented way (maybe we have to work on the docs, but
that's solvable).

In addition, if people really want to use all the magic with
transitive dependencies etc. (which I really discourage from using in
this case), then they can simply use the dependency plugin. In the
end, the launchpad plugin creates a well documented folder structure
nd then zips everything. So it is possible to create the same
structure with the dependency plugin (or an ant task or whatever you
prefer) and then you're done. So I think to add this to the launchpad
plugin as well.

Regards
Carsten
-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Dependency handling for bundle lists

[Justin Edelson <ju...@justinedelson.com>]

On Wed, Nov 16, 2011 at 7:38 AM, Carsten Ziegeler <cz...@apache.org> wrote:
> 2011/11/16 Jukka Zitting <ju...@gmail.com>:
>> Hi,
>>
>> On Wed, Nov 16, 2011 at 3:51 PM, Carsten Ziegeler <cz...@apache.org> wrote:
>>> For use cases you might have in mind, we have the partial bundle list
>>> which is a well defined list of modules.
>>
>> My point is that it would be very convenient if I *could* use
>> transitive dependencies (and other standard Maven dependency features)
>> to produce the partial bundle list. That way I wouldn't have to
>> maintain the dependency information in two different places.
>>
>> Of course it's good if there's way to explicitly override certain
>> dependencies, but for the most part my existing Maven dependency tree
>> already contains a perfectly good starting point for the bundle list.
>> Having to duplicate that information just opens up the risk of
>> accidental inconsistencies.
>>
> Yes, and I guess this is the core where we disagree - I really want to
> avoid transitive dependencies and other magic for defining a (partial)
> bundle list. In my opinion it should be a complete list

I agree with Carsten with respect to transitive dependencies, largely
because of the need to place different bundles in different start
levels. When you assemble a classpath for compilation or place a bunch
of JAR files into a WAR, then I see the usefulness of transitive
dependencies (although I've ended up with plenty of cruft in the
latter case), but in the case of a launchpad application, it seems to
add significant complexity (what start level do transitive
dependencies go in? how do you deal with dependency version conflicts
across start levels?).

Furthermore, at least in the case of Sling, we make extensive use of
the provided scope to indicate dependencies which are expected to be
provided by the container, so transitive dependencies would not be
useful for us. I'm always open to adding features to the launchpad
plugin which are not immediately useful to the Sling project, but
since this is the testbed, it does give me pause.

Justin

>
> Let's see what others think :)
>
> Regards
> Carsten
>
>> BR,
>>
>> Jukka Zitting
>>
>
>
>
> --
> Carsten Ziegeler
> cziegeler@apache.org
>

Re: Dependency handling for bundle lists

[Carsten Ziegeler <cz...@apache.org>]

2011/11/16 Jukka Zitting <ju...@gmail.com>:
> Hi,
>
> On Wed, Nov 16, 2011 at 3:51 PM, Carsten Ziegeler <cz...@apache.org> wrote:
>> For use cases you might have in mind, we have the partial bundle list
>> which is a well defined list of modules.
>
> My point is that it would be very convenient if I *could* use
> transitive dependencies (and other standard Maven dependency features)
> to produce the partial bundle list. That way I wouldn't have to
> maintain the dependency information in two different places.
>
> Of course it's good if there's way to explicitly override certain
> dependencies, but for the most part my existing Maven dependency tree
> already contains a perfectly good starting point for the bundle list.
> Having to duplicate that information just opens up the risk of
> accidental inconsistencies.
>
Yes, and I guess this is the core where we disagree - I really want to
avoid transitive dependencies and other magic for defining a (partial)
bundle list. In my opinion it should be a complete list

Let's see what others think :)

Regards
Carsten

> BR,
>
> Jukka Zitting
>



-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Dependency handling for bundle lists

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Nov 16, 2011 at 3:51 PM, Carsten Ziegeler <cz...@apache.org> wrote:
> For use cases you might have in mind, we have the partial bundle list
> which is a well defined list of modules.

My point is that it would be very convenient if I *could* use
transitive dependencies (and other standard Maven dependency features)
to produce the partial bundle list. That way I wouldn't have to
maintain the dependency information in two different places.

Of course it's good if there's way to explicitly override certain
dependencies, but for the most part my existing Maven dependency tree
already contains a perfectly good starting point for the bundle list.
Having to duplicate that information just opens up the risk of
accidental inconsistencies.

BR,

Jukka Zitting

Re: Dependency handling for bundle lists

[Carsten Ziegeler <cz...@apache.org>]

Hi,

the bundle list is about modules (bundles) and it is up to the
application which module it will assemble into its application. A
module only declares on what api it depends, but it should never tell
which exact other bundles should be packaged. This rules out
transitivie dependencies and to some degree dependency management.
Especially as it is up to the application to decide which version of a
module it uses.

For use cases you might have in mind, we have the partial bundle list
which is a well defined list of modules. The final application can
reference such partial bundle lists and the app can therefore be
assembled out of larger building blocks. But then, this building block
is well defined by the assembler of that block and not by any bundle
it contains.

Carsten

2011/11/16 Jukka Zitting <ju...@gmail.com>:
> Hi,
>
> On Wed, Nov 16, 2011 at 12:00 PM, Carsten Ziegeler <cz...@apache.org> wrote:
>> Hmm, first time users have to read the docs anyway
>
> We're talking about Maven users here, who already understand the POM
> format and how to manage normal dependencies, so you only need to
> learn extra stuff if you're dealing with a separate bundle list file.
> I'm talking about people who only need to do simple stuff like update
> the version of a bundle or add a new bundle to the list. Such stuff
> should be as simple and straightforward as possible.
>
>> The answer is easy: these things don't work and are by
>> intention this way.
>
> So if I do need features like these, I shouldn't be using the launchpad plugin?
>
>> The number one rule is: no surprises and therefore you have to
>> configure the full dependency list including versions. We explicitly
>> came up with the partial bundle list. If you want to assemble an
>> application you really want to specify what you want to have in there,
>> so transitive deps etc. should not be used at all here.
>
> A proper Maven dependency tree already contains all the information
> you need to assemble working set of bundles. Why should I need to
> throw that information away just to duplicate it in the bundle list
> file?
>
> If you're worried about bloating the POM file, I'd say not supporting
> transitive dependencies is a way bigger cause of bloat in the bundle
> list file.
>
> For example, I'm trying to manage the dependency list of a large
> component  (CRX server bundle) that requires about a dozen upstream
> bundles as dependencies. Instead of having just a single dependency in
> the partial bundle list project for this and all required transitive
> dependencies, I have to duplicate the entire list of bundles and
> manually make sure that their versions are correctly updated when the
> dependencies change.
>
>> And just by having dependencies inside the plugin configuration. this
>> doesn't work either.
>
> Correct. IMHO we should be using normal project level dependencies for
> the bundles included in the bundle list.
>
>> Ah, ok, we asked the maven gurus last week and they told us the
>> opposite. I haven't validated it yet. If that's the case then I'm fine
>> if we use the exact same format as we use for the external file inside
>> the plugin configuration.
>
> Not sure if we're talking about the same thing here, but here's a
> screenshot of the introspection at work with a vanilla Eclipse Indigo
> installation: http://people.apache.org/~jukka/2011/m2e-introspection.png
>
>> I really like the external XML file - it's imho the best way to
>> specify an application assembly.
>
> I'm not opposed to the idea of an external XML file as such, for
> example it works pretty well for the assembly plugin. But just like is
> done in the assembly plugin, I'd very much like to see the actual
> dependencies managed as normal Maven dependencies and only referenced
> in whatever configuration (plugin configuration in the POM or an
> external configuration file) is used to construct the resulting
> assembly.
>
> If we can keep the dependencies as normal <dependency> entries in the
> POM (with standard handling of transitive dependencies,
> dependencyManagement, etc.), I'm basically fine with whatever
> mechanism is used to configure the rest of the build.
>
> BR,
>
> Jukka Zitting
>



-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Dependency handling for bundle lists

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Nov 16, 2011 at 12:00 PM, Carsten Ziegeler <cz...@apache.org> wrote:
> Hmm, first time users have to read the docs anyway

We're talking about Maven users here, who already understand the POM
format and how to manage normal dependencies, so you only need to
learn extra stuff if you're dealing with a separate bundle list file.
I'm talking about people who only need to do simple stuff like update
the version of a bundle or add a new bundle to the list. Such stuff
should be as simple and straightforward as possible.

> The answer is easy: these things don't work and are by
> intention this way.

So if I do need features like these, I shouldn't be using the launchpad plugin?

> The number one rule is: no surprises and therefore you have to
> configure the full dependency list including versions. We explicitly
> came up with the partial bundle list. If you want to assemble an
> application you really want to specify what you want to have in there,
> so transitive deps etc. should not be used at all here.

A proper Maven dependency tree already contains all the information
you need to assemble working set of bundles. Why should I need to
throw that information away just to duplicate it in the bundle list
file?

If you're worried about bloating the POM file, I'd say not supporting
transitive dependencies is a way bigger cause of bloat in the bundle
list file.

For example, I'm trying to manage the dependency list of a large
component  (CRX server bundle) that requires about a dozen upstream
bundles as dependencies. Instead of having just a single dependency in
the partial bundle list project for this and all required transitive
dependencies, I have to duplicate the entire list of bundles and
manually make sure that their versions are correctly updated when the
dependencies change.

> And just by having dependencies inside the plugin configuration. this
> doesn't work either.

Correct. IMHO we should be using normal project level dependencies for
the bundles included in the bundle list.

> Ah, ok, we asked the maven gurus last week and they told us the
> opposite. I haven't validated it yet. If that's the case then I'm fine
> if we use the exact same format as we use for the external file inside
> the plugin configuration.

Not sure if we're talking about the same thing here, but here's a
screenshot of the introspection at work with a vanilla Eclipse Indigo
installation: http://people.apache.org/~jukka/2011/m2e-introspection.png

> I really like the external XML file - it's imho the best way to
> specify an application assembly.

I'm not opposed to the idea of an external XML file as such, for
example it works pretty well for the assembly plugin. But just like is
done in the assembly plugin, I'd very much like to see the actual
dependencies managed as normal Maven dependencies and only referenced
in whatever configuration (plugin configuration in the POM or an
external configuration file) is used to construct the resulting
assembly.

If we can keep the dependencies as normal <dependency> entries in the
POM (with standard handling of transitive dependencies,
dependencyManagement, etc.), I'm basically fine with whatever
mechanism is used to configure the rest of the build.

BR,

Jukka Zitting

Re: Dependency handling for bundle lists

[Carsten Ziegeler <cz...@apache.org>]

2011/11/16 Jukka Zitting <ju...@gmail.com>:
> Hi,
>
> On Wed, Nov 16, 2011 at 9:28 AM, Carsten Ziegeler <cz...@apache.org> wrote:
>> I really think we should just provide one way for defining a bundle
>> list - as the current way with the xml file has no downsides anymore,
>
> From my perspective as a user of the plugin there are still big
> problems with the separate XML file:
>
> 1) You have to learn a new dependency and configuration mechanism.
> I've witnessed a handful people interact with the partial bundle list
> mechanism for the first time, and they all come out confused trying to
> figure out where they should put a new dependency.

Hmm, first time users have to read the docs anyway -  so they need to
know whether and how they have to configure it in the configuration of
the plugin. The same goes for the external file - so there is no
difference.

>
> 2) You lose a lot of built-in Maven mechanisms. It's only recently
> when support for basic stuff like ${properties} was added, and I have
> no idea if or how things like transitive dependencies, inheritance,
> dependencyManagement, profiles, etc. work with the bundle list
> mechanism. They even might, but where do I find the documentation of
> how that stuff is supposed to work with the bundle list?

The answer is easy: these things don't work and are by intention this
way. The number one rule is: no surprises and therefore you have to
configure the full dependency list including versions. We explicitly
came up with the partial bundle list. If you want to assemble an
application you really want to specify what you want to have in there,
so transitive deps etc. should not be used at all here.

>
> 3) Having a custom dependency mechanism confuses tools like Jenkins
> that could otherwise automatically trigger CI builds based on changes
> in upstream dependencies. Or Eclipse that when opening a project gives
> the option for automatically opening also all upstream dependencies.

We're using Maven api to get this done, so I expect that other tools
leveraging maven work as well. If not, well....
And just by having dependencies inside the plugin configuration. this
doesn't work either. Try specifying a dependency inside the dependency
plugin without in addition mentioning it in the dependencies section
of your pom! And I don't want to declare deps twice.


>
>> There has always been the argument that having the configuration
>> inside the plugin configuration in the pom allows to use tools like
>> m2e etc. but that's only true to some degree. If we want support for
>> our plugin, we need to write a m2e plugin (if that's the right term).
>
> A plugin is not needed since m2e automatically introspects Maven
> plugins to find which configuration options are available. My Eclipse
> instance happily provides inline validation, code completion and
> context-sensitive help for all Maven plugins, including the launchpad
> one.

Ah, ok, we asked the maven gurus last week and they told us the
opposite. I haven't validated it yet. If that's the case then I'm fine
if we use the exact same format as we use for the external file inside
the plugin configuration.

I really like the external XML file - it's imho the best way to
specify an application assembly. The average developer rarely touches
these things and if, it's a well documented place with an easily
understandable format (well maybe we're missing docs atm, but we'll
fix that anyway).

Carsten

>
> Of course we'd get similar level of IDE support also for the bundle
> list XML file by adding an explicit XML schema or DTD reference, but
> the effort spent implementing something like that is just reinventing
> stuff that we'd already get for free by following standard Maven
> conventions.
>
> BR,
>
> Jukka Zitting
>



-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Dependency handling for bundle lists

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Nov 16, 2011 at 9:28 AM, Carsten Ziegeler <cz...@apache.org> wrote:
> I really think we should just provide one way for defining a bundle
> list - as the current way with the xml file has no downsides anymore,

>From my perspective as a user of the plugin there are still big
problems with the separate XML file:

1) You have to learn a new dependency and configuration mechanism.
I've witnessed a handful people interact with the partial bundle list
mechanism for the first time, and they all come out confused trying to
figure out where they should put a new dependency.

2) You lose a lot of built-in Maven mechanisms. It's only recently
when support for basic stuff like ${properties} was added, and I have
no idea if or how things like transitive dependencies, inheritance,
dependencyManagement, profiles, etc. work with the bundle list
mechanism. They even might, but where do I find the documentation of
how that stuff is supposed to work with the bundle list?

3) Having a custom dependency mechanism confuses tools like Jenkins
that could otherwise automatically trigger CI builds based on changes
in upstream dependencies. Or Eclipse that when opening a project gives
the option for automatically opening also all upstream dependencies.

> There has always been the argument that having the configuration
> inside the plugin configuration in the pom allows to use tools like
> m2e etc. but that's only true to some degree. If we want support for
> our plugin, we need to write a m2e plugin (if that's the right term).

A plugin is not needed since m2e automatically introspects Maven
plugins to find which configuration options are available. My Eclipse
instance happily provides inline validation, code completion and
context-sensitive help for all Maven plugins, including the launchpad
one.

Of course we'd get similar level of IDE support also for the bundle
list XML file by adding an explicit XML schema or DTD reference, but
the effort spent implementing something like that is just reinventing
stuff that we'd already get for free by following standard Maven
conventions.

BR,

Jukka Zitting