You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildr.apache.org by Assaf Arkin <ar...@intalio.com> on 2008/05/02 23:41:58 UTC
Transitive dependencies, version matching and OSGi support
Now that 1.3 is out, it's time to start talking about future releases.
We've got the 1.3.x line of releases, for dealing with outstanding issues,
new ones we'll find along the way, and building on new features that showed
up in 1.3.
Separately, there's 1.4, the next major release and a place to make more
significant changes. We just have to decide what those would be.
I propose that we focus 1.4 on improving the way we handle dependencies.
Specifically what I have in mind are transitive dependencies, version
matching, and OSGi support. So I'll start by talking about some things I've
been looking into recently.
For starters, adding a dependency method to artifact (which also includes
packages) that returns an array of all the known dependencies for that
artifacts. Ruby arrays are pretty powerful, you can do things like:
# Does it have a particular dependency?
foo.dependencies.any? { |dep| ... }
# Add new dependencies
bar.dependencies << ....
# Merge foo's dependencies into bar
bar.dependencies |= foo.dependencies
One problem that comes up frequently are broken dependency lists, they may
be missing a dependency, or reference the wrong version, or include one you
don't want to use. Mutable arrays would make it easy to fix these broken
dependency lists, for example:
# Change the artifact in memory
artifact( spec ).exclude( dont-want-this ).include( but want this )
# And use it
compile.with spec
It will be even better to introduce a Dependencies class that extends array
and adds all sorts of convenience methods and lazy expansion, similar to
Rake's FileList. Then you could do things like:
# Find dependency by its specification
foo.dependencies.has?( spec )
# Exclude all dependencies that match a pattern
foo.dependencies.exclude!( pattern )
# Expand all transitive dependencies and return flat list
foo.dependencies.expand
# Replace one with another
foo.dependencies.replace!( dont-want, want )
The API for methods like compile.with and compile.dependencies remain the
same, just changing the underlying implementation.
Packages would acquire dependencies from the project, for example,
package(:jar) would add all the dependencies from compile.dependencies,
which of course you can fix to your heart's content.
Version matching makes it possible to point at the most recent version that
satisfies a particular version requirement, so >=1.2 to get 1.2.0 or later,
or ~>1.2.3 to get any version between 1.2.3 and 1.3 (excluding).
OSGi is a set of big specs that boil a lot of oceans, the part I'm
particularly interested in, is managing dependencies. When we started
working on Buildr, OSGi was used almost exclusively by Eclipse. That
changed. We're seeing more and more servers and frameworks adopting OSGi.
Imagine that we had OSGi support. You will specify all the dependencies
once in the buildfile. Those will be used to build the project, and also
incorporated into the package manifest. Other projects could use these
packages, reading the dependency list from the package manifest. More
important, when you get to deploy those packages, the runtime can use that
information to bring in the right dependencies.
That's one of those things that, once you use it, you won't want to go back.
The problem is, OSGi and Maven are incompatible with each other. I'll try
to summarize the differences:
1. The OSGi package identifier is typically the same as the file name. The
Maven package identifier combines group and artifact identifiers, only the
later is used as the file name.
2. Both use four-part version numbers. The first three are numerical (e.g.
1.2.3) and managed the same way, the fourth one is the qualifier. OSGi
qualifiers are alphanumeric and compared lexically, so R1 comes before R2;
using number is not advised, 10 comes before 2. Maven qualifiers are
numeric for releases, and alphanumeric for pre-releases; the qualifier 10
comes after 2, and 2 comes after R2.
3. The rules for version matching are therefore not the same. The same
rule could match different versions, depending on whethr you use OSGi or
Maven.
There's also the issue that OSGi keeps the meta-data inside the package,
while Maven requires the additional POM file, and if both are present we
have to decide which one to rely on.
It's possible to some extent to support both at the same time, but there
will always be that impedence mismatch. If we do a good enough job, it will
probably only affect edge cases, but we need to be aware.
It will definitely be easier to pick one of the two, either OSGi or Maven,
and add transitive support and version matching based on that model, then do
translation when necessary (e.g. when reading POM file, generating OSGi
manifest).
If I had to start all over again, I would go for OSGi with Maven
compatibility layer for three simple reasons:
1. Runtime deployment
2. Simpler to implement, and if your build doesn't quite work, troubleshoot
and fix.
3. Works the same way as most other packaging schemes (e.g. Gems, Debian,
RPM).
Comments, ideas?
Assaf
Re: Transitive dependencies, version matching and OSGi support
Posted by Tommy Knowlton <to...@ke4kug.net>.
Would Buildr do better to accept the same version range syntax that
maven2 already accepts? In fact, if parsing POM's, it would seem that
such syntax must already be supported, although I confess I haven't
tried this.
Cheers,
Tommy
On May 2, 2008, at 3:41 PM, Assaf Arkin wrote:
> Version matching makes it possible to point at the most recent
> version that
> satisfies a particular version requirement, so >=1.2 to get 1.2.0 or
> later,
> or ~>1.2.3 to get any version between 1.2.3 and 1.3 (excluding).
Re: Transitive dependencies, version matching and OSGi support
Posted by Shane Witbeck <sh...@digitalsanctum.com>.
This looks like I can move to using environments instead of my pre-1.3
custom extension. Thanks for the pointer.
On Thu, May 8, 2008 at 12:03 AM, Assaf Arkin <ar...@intalio.com> wrote:
> On Wed, May 7, 2008 at 7:44 PM, Shane Witbeck <sh...@digitalsanctum.com>
> wrote:
>
>> Expanding on the notion of deploying, I extended Buildr to allow
>> deployments to various server environments (local, development,
>> production, etc.). The command looks something like this:
>>
>> buildr projectname:deploy-dev
>>
>> The above command automagically packages and deploys the artifact(s)
>> to a specific server which makes it nice for turning around changes
>> quickly. To accomplish this I used a combination of extending
>> Buildr::Project and the Net::SFTP project.
>
>
> Have you looked into environments in 1.3? You could do something like:
>
> buildr -e dev projectname:deploy
>
> or set the environment variable BUILDR_ENV=dev.
>
> http://incubator.apache.org/buildr/settings_profiles.html#environments
>
> Assaf
>
>
>>
>>
>>
>>
>>
>> On Wed, May 7, 2008 at 9:44 PM, Assaf Arkin <ar...@intalio.com> wrote:
>> > On Wed, May 7, 2008 at 6:19 PM, Shane Witbeck <sh...@digitalsanctum.com>
>> > wrote:
>> >
>> >
>> > > I second the idea of going the way of OSGI. It's clearly the next step
>> > > in Java's evolution.
>> > >
>> > > In terms of pre-1.3 dependency support, we resorted to doing something
>> > > like the following to support a subset of compile dependencies for
>> > > runtime packages:
>> > >
>> > > project.runtime.with(specs)
>> >
>> >
>> > +1 for adding runtime dependencies.
>> >
>> >
>> >
>> > >
>> > >
>> > > Because some of our environments had shared runtime classpaths, it
>> > > wasn't always necessary to include certain libraries. I did this by
>> > > extending Buildr::Project to support the different kinds of internal
>> > > projects we have such as WebProject, StandaloneProject, BatchProject,
>> > > etc.
>> > >
>> > > Something else that was needed in our projects was a way of putting
>> > > together different packages for different environments. Again, by
>> > > extending Buildr::Project, I introduced a way to package and deploy
>> > > (via SSH) the various projects.
>> > >
>> > > I mention the ways that I've extended Buildr to possibly touch on new
>> > > features to incorporate such as:
>> > >
>> > > 1. Being able to define different sets of dependencies for projects
>> > > (in my case it was compile vs. runtime). I like the direction Assaf
>> > > mentions with being able to merge dependency arrays and better
>> > > transitive support. This is huge in terms of being able to make the
>> > > buildfile less verbose.
>> > > 2. A way to easily deploy (via SSH, etc.) packages
>> >
>> >
>> > Can you expand on that?
>> >
>> >
>> >
>> > >
>> > > 3. Better integration with IDE's. It's a struggle to keep the
>> > > dependencies in sync without having to regenerate the IDE files and in
>> > > the process losing other project-specific IDE settings. Perhaps this
>> > > is better solved via IDE-specific plugins for Buildr?
>> >
>> >
>> > Good point. It might be possible to retain those project-specific IDE
>> > settings, though I don't know what it would look like.
>> >
>> >
>> >
>> > >
>> > > 4. In terms of additional language support, my vote is for Flex. With
>> > > the recent open sourced SDK, I believe there will be a surge in usage
>> > > among not only Java developers but developers of other languages as
>> > > well.
>> >
>> >
>> > We'll need someone who's using Flex to help develop it.
>> >
>> > Also, this is something we don't have to tie to the release schedule.
>> We
>> > finally have an API for adding new languages and delivering extensions
>> using
>> > Ruby Gems, so you can get people to use it without waiting for the next
>> > release. Separately, once we add it to the code base it will show up in
>> the
>> > next release, but I think independent Gem would be the way to start.
>> >
>> > Assaf
>> >
>> >
>> >
>> >
>> > >
>> > >
>> > > I'd be glad to elaborate further on the points I've brought up and
>> > > look forward to contributing more.
>> > >
>> > > Thanks,
>> > > Shane
>> > >
>> > >
>> > > On Fri, May 2, 2008 at 5:41 PM, Assaf Arkin <ar...@intalio.com>
>> wrote:
>> > > > Now that 1.3 is out, it's time to start talking about future
>> releases.
>> > > > We've got the 1.3.x line of releases, for dealing with outstanding
>> > > issues,
>> > > > new ones we'll find along the way, and building on new features
>> that
>> > > showed
>> > > > up in 1.3.
>> > > >
>> > > > Separately, there's 1.4, the next major release and a place to make
>> more
>> > > > significant changes. We just have to decide what those would be.
>> > > >
>> > > > I propose that we focus 1.4 on improving the way we handle
>> dependencies.
>> > > >
>> > > > Specifically what I have in mind are transitive dependencies,
>> version
>> > > > matching, and OSGi support. So I'll start by talking about some
>> things
>> > > I've
>> > > > been looking into recently.
>> > > >
>> > > >
>> > > > For starters, adding a dependency method to artifact (which also
>> > > includes
>> > > > packages) that returns an array of all the known dependencies for
>> that
>> > > > artifacts. Ruby arrays are pretty powerful, you can do things
>> like:
>> > > >
>> > > > # Does it have a particular dependency?
>> > > > foo.dependencies.any? { |dep| ... }
>> > > > # Add new dependencies
>> > > > bar.dependencies << ....
>> > > > # Merge foo's dependencies into bar
>> > > > bar.dependencies |= foo.dependencies
>> > > >
>> > > > One problem that comes up frequently are broken dependency lists,
>> they
>> > > may
>> > > > be missing a dependency, or reference the wrong version, or include
>> one
>> > > you
>> > > > don't want to use. Mutable arrays would make it easy to fix these
>> > > broken
>> > > > dependency lists, for example:
>> > > >
>> > > > # Change the artifact in memory
>> > > > artifact( spec ).exclude( dont-want-this ).include( but want this )
>> > > > # And use it
>> > > > compile.with spec
>> > > >
>> > > > It will be even better to introduce a Dependencies class that
>> extends
>> > > array
>> > > > and adds all sorts of convenience methods and lazy expansion,
>> similar to
>> > > > Rake's FileList. Then you could do things like:
>> > > >
>> > > > # Find dependency by its specification
>> > > > foo.dependencies.has?( spec )
>> > > > # Exclude all dependencies that match a pattern
>> > > > foo.dependencies.exclude!( pattern )
>> > > > # Expand all transitive dependencies and return flat list
>> > > > foo.dependencies.expand
>> > > > # Replace one with another
>> > > > foo.dependencies.replace!( dont-want, want )
>> > > >
>> > > > The API for methods like compile.with and compile.dependencies
>> remain
>> > > the
>> > > > same, just changing the underlying implementation.
>> > > >
>> > > > Packages would acquire dependencies from the project, for example,
>> > > > package(:jar) would add all the dependencies from
>> compile.dependencies,
>> > > > which of course you can fix to your heart's content.
>> > > >
>> > > >
>> > > > Version matching makes it possible to point at the most recent
>> version
>> > > that
>> > > > satisfies a particular version requirement, so >=1.2 to get 1.2.0
>> or
>> > > later,
>> > > > or ~>1.2.3 to get any version between 1.2.3 and 1.3 (excluding).
>> > > >
>> > > >
>> > > > OSGi is a set of big specs that boil a lot of oceans, the part I'm
>> > > > particularly interested in, is managing dependencies. When we
>> started
>> > > > working on Buildr, OSGi was used almost exclusively by Eclipse.
>> That
>> > > > changed. We're seeing more and more servers and frameworks
>> adopting
>> > > OSGi.
>> > > >
>> > > > Imagine that we had OSGi support. You will specify all the
>> dependencies
>> > > > once in the buildfile. Those will be used to build the project,
>> and
>> > > also
>> > > > incorporated into the package manifest. Other projects could use
>> these
>> > > > packages, reading the dependency list from the package manifest.
>> More
>> > > > important, when you get to deploy those packages, the runtime can
>> use
>> > > that
>> > > > information to bring in the right dependencies.
>> > > >
>> > > > That's one of those things that, once you use it, you won't want to
>> go
>> > > back.
>> > > >
>> > > >
>> > > > The problem is, OSGi and Maven are incompatible with each other.
>> I'll
>> > > try
>> > > > to summarize the differences:
>> > > >
>> > > > 1. The OSGi package identifier is typically the same as the file
>> name.
>> > > The
>> > > > Maven package identifier combines group and artifact identifiers,
>> only
>> > > the
>> > > > later is used as the file name.
>> > > >
>> > > > 2. Both use four-part version numbers. The first three are
>> numerical
>> > > (e.g.
>> > > > 1.2.3) and managed the same way, the fourth one is the qualifier.
>> OSGi
>> > > > qualifiers are alphanumeric and compared lexically, so R1 comes
>> before
>> > > R2;
>> > > > using number is not advised, 10 comes before 2. Maven qualifiers
>> are
>> > > > numeric for releases, and alphanumeric for pre-releases; the
>> qualifier
>> > > 10
>> > > > comes after 2, and 2 comes after R2.
>> > > >
>> > > > 3. The rules for version matching are therefore not the same. The
>> same
>> > > > rule could match different versions, depending on whethr you use
>> OSGi or
>> > > > Maven.
>> > > >
>> > > > There's also the issue that OSGi keeps the meta-data inside the
>> package,
>> > > > while Maven requires the additional POM file, and if both are
>> present we
>> > > > have to decide which one to rely on.
>> > > >
>> > > > It's possible to some extent to support both at the same time, but
>> there
>> > > > will always be that impedence mismatch. If we do a good enough
>> job, it
>> > > will
>> > > > probably only affect edge cases, but we need to be aware.
>> > > >
>> > > > It will definitely be easier to pick one of the two, either OSGi or
>> > > Maven,
>> > > > and add transitive support and version matching based on that
>> model,
>> > > then do
>> > > > translation when necessary (e.g. when reading POM file, generating
>> OSGi
>> > > > manifest).
>> > > >
>> > > > If I had to start all over again, I would go for OSGi with Maven
>> > > > compatibility layer for three simple reasons:
>> > > >
>> > > > 1. Runtime deployment
>> > > > 2. Simpler to implement, and if your build doesn't quite work,
>> > > troubleshoot
>> > > > and fix.
>> > > > 3. Works the same way as most other packaging schemes (e.g. Gems,
>> > > Debian,
>> > > > RPM).
>> > > >
>> > > >
>> > > > Comments, ideas?
>> > > >
>> > > > Assaf
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -Shane
>> > >
>> >
>>
>>
>>
>> --
>> -Shane
>>
>
--
-Shane
Re: Transitive dependencies, version matching and OSGi support
Posted by Assaf Arkin <ar...@intalio.com>.
On Wed, May 7, 2008 at 7:44 PM, Shane Witbeck <sh...@digitalsanctum.com>
wrote:
> Expanding on the notion of deploying, I extended Buildr to allow
> deployments to various server environments (local, development,
> production, etc.). The command looks something like this:
>
> buildr projectname:deploy-dev
>
> The above command automagically packages and deploys the artifact(s)
> to a specific server which makes it nice for turning around changes
> quickly. To accomplish this I used a combination of extending
> Buildr::Project and the Net::SFTP project.
Have you looked into environments in 1.3? You could do something like:
buildr -e dev projectname:deploy
or set the environment variable BUILDR_ENV=dev.
http://incubator.apache.org/buildr/settings_profiles.html#environments
Assaf
>
>
>
>
>
> On Wed, May 7, 2008 at 9:44 PM, Assaf Arkin <ar...@intalio.com> wrote:
> > On Wed, May 7, 2008 at 6:19 PM, Shane Witbeck <sh...@digitalsanctum.com>
> > wrote:
> >
> >
> > > I second the idea of going the way of OSGI. It's clearly the next step
> > > in Java's evolution.
> > >
> > > In terms of pre-1.3 dependency support, we resorted to doing something
> > > like the following to support a subset of compile dependencies for
> > > runtime packages:
> > >
> > > project.runtime.with(specs)
> >
> >
> > +1 for adding runtime dependencies.
> >
> >
> >
> > >
> > >
> > > Because some of our environments had shared runtime classpaths, it
> > > wasn't always necessary to include certain libraries. I did this by
> > > extending Buildr::Project to support the different kinds of internal
> > > projects we have such as WebProject, StandaloneProject, BatchProject,
> > > etc.
> > >
> > > Something else that was needed in our projects was a way of putting
> > > together different packages for different environments. Again, by
> > > extending Buildr::Project, I introduced a way to package and deploy
> > > (via SSH) the various projects.
> > >
> > > I mention the ways that I've extended Buildr to possibly touch on new
> > > features to incorporate such as:
> > >
> > > 1. Being able to define different sets of dependencies for projects
> > > (in my case it was compile vs. runtime). I like the direction Assaf
> > > mentions with being able to merge dependency arrays and better
> > > transitive support. This is huge in terms of being able to make the
> > > buildfile less verbose.
> > > 2. A way to easily deploy (via SSH, etc.) packages
> >
> >
> > Can you expand on that?
> >
> >
> >
> > >
> > > 3. Better integration with IDE's. It's a struggle to keep the
> > > dependencies in sync without having to regenerate the IDE files and in
> > > the process losing other project-specific IDE settings. Perhaps this
> > > is better solved via IDE-specific plugins for Buildr?
> >
> >
> > Good point. It might be possible to retain those project-specific IDE
> > settings, though I don't know what it would look like.
> >
> >
> >
> > >
> > > 4. In terms of additional language support, my vote is for Flex. With
> > > the recent open sourced SDK, I believe there will be a surge in usage
> > > among not only Java developers but developers of other languages as
> > > well.
> >
> >
> > We'll need someone who's using Flex to help develop it.
> >
> > Also, this is something we don't have to tie to the release schedule.
> We
> > finally have an API for adding new languages and delivering extensions
> using
> > Ruby Gems, so you can get people to use it without waiting for the next
> > release. Separately, once we add it to the code base it will show up in
> the
> > next release, but I think independent Gem would be the way to start.
> >
> > Assaf
> >
> >
> >
> >
> > >
> > >
> > > I'd be glad to elaborate further on the points I've brought up and
> > > look forward to contributing more.
> > >
> > > Thanks,
> > > Shane
> > >
> > >
> > > On Fri, May 2, 2008 at 5:41 PM, Assaf Arkin <ar...@intalio.com>
> wrote:
> > > > Now that 1.3 is out, it's time to start talking about future
> releases.
> > > > We've got the 1.3.x line of releases, for dealing with outstanding
> > > issues,
> > > > new ones we'll find along the way, and building on new features
> that
> > > showed
> > > > up in 1.3.
> > > >
> > > > Separately, there's 1.4, the next major release and a place to make
> more
> > > > significant changes. We just have to decide what those would be.
> > > >
> > > > I propose that we focus 1.4 on improving the way we handle
> dependencies.
> > > >
> > > > Specifically what I have in mind are transitive dependencies,
> version
> > > > matching, and OSGi support. So I'll start by talking about some
> things
> > > I've
> > > > been looking into recently.
> > > >
> > > >
> > > > For starters, adding a dependency method to artifact (which also
> > > includes
> > > > packages) that returns an array of all the known dependencies for
> that
> > > > artifacts. Ruby arrays are pretty powerful, you can do things
> like:
> > > >
> > > > # Does it have a particular dependency?
> > > > foo.dependencies.any? { |dep| ... }
> > > > # Add new dependencies
> > > > bar.dependencies << ....
> > > > # Merge foo's dependencies into bar
> > > > bar.dependencies |= foo.dependencies
> > > >
> > > > One problem that comes up frequently are broken dependency lists,
> they
> > > may
> > > > be missing a dependency, or reference the wrong version, or include
> one
> > > you
> > > > don't want to use. Mutable arrays would make it easy to fix these
> > > broken
> > > > dependency lists, for example:
> > > >
> > > > # Change the artifact in memory
> > > > artifact( spec ).exclude( dont-want-this ).include( but want this )
> > > > # And use it
> > > > compile.with spec
> > > >
> > > > It will be even better to introduce a Dependencies class that
> extends
> > > array
> > > > and adds all sorts of convenience methods and lazy expansion,
> similar to
> > > > Rake's FileList. Then you could do things like:
> > > >
> > > > # Find dependency by its specification
> > > > foo.dependencies.has?( spec )
> > > > # Exclude all dependencies that match a pattern
> > > > foo.dependencies.exclude!( pattern )
> > > > # Expand all transitive dependencies and return flat list
> > > > foo.dependencies.expand
> > > > # Replace one with another
> > > > foo.dependencies.replace!( dont-want, want )
> > > >
> > > > The API for methods like compile.with and compile.dependencies
> remain
> > > the
> > > > same, just changing the underlying implementation.
> > > >
> > > > Packages would acquire dependencies from the project, for example,
> > > > package(:jar) would add all the dependencies from
> compile.dependencies,
> > > > which of course you can fix to your heart's content.
> > > >
> > > >
> > > > Version matching makes it possible to point at the most recent
> version
> > > that
> > > > satisfies a particular version requirement, so >=1.2 to get 1.2.0
> or
> > > later,
> > > > or ~>1.2.3 to get any version between 1.2.3 and 1.3 (excluding).
> > > >
> > > >
> > > > OSGi is a set of big specs that boil a lot of oceans, the part I'm
> > > > particularly interested in, is managing dependencies. When we
> started
> > > > working on Buildr, OSGi was used almost exclusively by Eclipse.
> That
> > > > changed. We're seeing more and more servers and frameworks
> adopting
> > > OSGi.
> > > >
> > > > Imagine that we had OSGi support. You will specify all the
> dependencies
> > > > once in the buildfile. Those will be used to build the project,
> and
> > > also
> > > > incorporated into the package manifest. Other projects could use
> these
> > > > packages, reading the dependency list from the package manifest.
> More
> > > > important, when you get to deploy those packages, the runtime can
> use
> > > that
> > > > information to bring in the right dependencies.
> > > >
> > > > That's one of those things that, once you use it, you won't want to
> go
> > > back.
> > > >
> > > >
> > > > The problem is, OSGi and Maven are incompatible with each other.
> I'll
> > > try
> > > > to summarize the differences:
> > > >
> > > > 1. The OSGi package identifier is typically the same as the file
> name.
> > > The
> > > > Maven package identifier combines group and artifact identifiers,
> only
> > > the
> > > > later is used as the file name.
> > > >
> > > > 2. Both use four-part version numbers. The first three are
> numerical
> > > (e.g.
> > > > 1.2.3) and managed the same way, the fourth one is the qualifier.
> OSGi
> > > > qualifiers are alphanumeric and compared lexically, so R1 comes
> before
> > > R2;
> > > > using number is not advised, 10 comes before 2. Maven qualifiers
> are
> > > > numeric for releases, and alphanumeric for pre-releases; the
> qualifier
> > > 10
> > > > comes after 2, and 2 comes after R2.
> > > >
> > > > 3. The rules for version matching are therefore not the same. The
> same
> > > > rule could match different versions, depending on whethr you use
> OSGi or
> > > > Maven.
> > > >
> > > > There's also the issue that OSGi keeps the meta-data inside the
> package,
> > > > while Maven requires the additional POM file, and if both are
> present we
> > > > have to decide which one to rely on.
> > > >
> > > > It's possible to some extent to support both at the same time, but
> there
> > > > will always be that impedence mismatch. If we do a good enough
> job, it
> > > will
> > > > probably only affect edge cases, but we need to be aware.
> > > >
> > > > It will definitely be easier to pick one of the two, either OSGi or
> > > Maven,
> > > > and add transitive support and version matching based on that
> model,
> > > then do
> > > > translation when necessary (e.g. when reading POM file, generating
> OSGi
> > > > manifest).
> > > >
> > > > If I had to start all over again, I would go for OSGi with Maven
> > > > compatibility layer for three simple reasons:
> > > >
> > > > 1. Runtime deployment
> > > > 2. Simpler to implement, and if your build doesn't quite work,
> > > troubleshoot
> > > > and fix.
> > > > 3. Works the same way as most other packaging schemes (e.g. Gems,
> > > Debian,
> > > > RPM).
> > > >
> > > >
> > > > Comments, ideas?
> > > >
> > > > Assaf
> > > >
> > >
> > >
> > >
> > > --
> > > -Shane
> > >
> >
>
>
>
> --
> -Shane
>
Re: Transitive dependencies, version matching and OSGi support
Posted by Shane Witbeck <sh...@digitalsanctum.com>.
Expanding on the notion of deploying, I extended Buildr to allow
deployments to various server environments (local, development,
production, etc.). The command looks something like this:
buildr projectname:deploy-dev
The above command automagically packages and deploys the artifact(s)
to a specific server which makes it nice for turning around changes
quickly. To accomplish this I used a combination of extending
Buildr::Project and the Net::SFTP project.
On Wed, May 7, 2008 at 9:44 PM, Assaf Arkin <ar...@intalio.com> wrote:
> On Wed, May 7, 2008 at 6:19 PM, Shane Witbeck <sh...@digitalsanctum.com>
> wrote:
>
>
> > I second the idea of going the way of OSGI. It's clearly the next step
> > in Java's evolution.
> >
> > In terms of pre-1.3 dependency support, we resorted to doing something
> > like the following to support a subset of compile dependencies for
> > runtime packages:
> >
> > project.runtime.with(specs)
>
>
> +1 for adding runtime dependencies.
>
>
>
> >
> >
> > Because some of our environments had shared runtime classpaths, it
> > wasn't always necessary to include certain libraries. I did this by
> > extending Buildr::Project to support the different kinds of internal
> > projects we have such as WebProject, StandaloneProject, BatchProject,
> > etc.
> >
> > Something else that was needed in our projects was a way of putting
> > together different packages for different environments. Again, by
> > extending Buildr::Project, I introduced a way to package and deploy
> > (via SSH) the various projects.
> >
> > I mention the ways that I've extended Buildr to possibly touch on new
> > features to incorporate such as:
> >
> > 1. Being able to define different sets of dependencies for projects
> > (in my case it was compile vs. runtime). I like the direction Assaf
> > mentions with being able to merge dependency arrays and better
> > transitive support. This is huge in terms of being able to make the
> > buildfile less verbose.
> > 2. A way to easily deploy (via SSH, etc.) packages
>
>
> Can you expand on that?
>
>
>
> >
> > 3. Better integration with IDE's. It's a struggle to keep the
> > dependencies in sync without having to regenerate the IDE files and in
> > the process losing other project-specific IDE settings. Perhaps this
> > is better solved via IDE-specific plugins for Buildr?
>
>
> Good point. It might be possible to retain those project-specific IDE
> settings, though I don't know what it would look like.
>
>
>
> >
> > 4. In terms of additional language support, my vote is for Flex. With
> > the recent open sourced SDK, I believe there will be a surge in usage
> > among not only Java developers but developers of other languages as
> > well.
>
>
> We'll need someone who's using Flex to help develop it.
>
> Also, this is something we don't have to tie to the release schedule. We
> finally have an API for adding new languages and delivering extensions using
> Ruby Gems, so you can get people to use it without waiting for the next
> release. Separately, once we add it to the code base it will show up in the
> next release, but I think independent Gem would be the way to start.
>
> Assaf
>
>
>
>
> >
> >
> > I'd be glad to elaborate further on the points I've brought up and
> > look forward to contributing more.
> >
> > Thanks,
> > Shane
> >
> >
> > On Fri, May 2, 2008 at 5:41 PM, Assaf Arkin <ar...@intalio.com> wrote:
> > > Now that 1.3 is out, it's time to start talking about future releases.
> > > We've got the 1.3.x line of releases, for dealing with outstanding
> > issues,
> > > new ones we'll find along the way, and building on new features that
> > showed
> > > up in 1.3.
> > >
> > > Separately, there's 1.4, the next major release and a place to make more
> > > significant changes. We just have to decide what those would be.
> > >
> > > I propose that we focus 1.4 on improving the way we handle dependencies.
> > >
> > > Specifically what I have in mind are transitive dependencies, version
> > > matching, and OSGi support. So I'll start by talking about some things
> > I've
> > > been looking into recently.
> > >
> > >
> > > For starters, adding a dependency method to artifact (which also
> > includes
> > > packages) that returns an array of all the known dependencies for that
> > > artifacts. Ruby arrays are pretty powerful, you can do things like:
> > >
> > > # Does it have a particular dependency?
> > > foo.dependencies.any? { |dep| ... }
> > > # Add new dependencies
> > > bar.dependencies << ....
> > > # Merge foo's dependencies into bar
> > > bar.dependencies |= foo.dependencies
> > >
> > > One problem that comes up frequently are broken dependency lists, they
> > may
> > > be missing a dependency, or reference the wrong version, or include one
> > you
> > > don't want to use. Mutable arrays would make it easy to fix these
> > broken
> > > dependency lists, for example:
> > >
> > > # Change the artifact in memory
> > > artifact( spec ).exclude( dont-want-this ).include( but want this )
> > > # And use it
> > > compile.with spec
> > >
> > > It will be even better to introduce a Dependencies class that extends
> > array
> > > and adds all sorts of convenience methods and lazy expansion, similar to
> > > Rake's FileList. Then you could do things like:
> > >
> > > # Find dependency by its specification
> > > foo.dependencies.has?( spec )
> > > # Exclude all dependencies that match a pattern
> > > foo.dependencies.exclude!( pattern )
> > > # Expand all transitive dependencies and return flat list
> > > foo.dependencies.expand
> > > # Replace one with another
> > > foo.dependencies.replace!( dont-want, want )
> > >
> > > The API for methods like compile.with and compile.dependencies remain
> > the
> > > same, just changing the underlying implementation.
> > >
> > > Packages would acquire dependencies from the project, for example,
> > > package(:jar) would add all the dependencies from compile.dependencies,
> > > which of course you can fix to your heart's content.
> > >
> > >
> > > Version matching makes it possible to point at the most recent version
> > that
> > > satisfies a particular version requirement, so >=1.2 to get 1.2.0 or
> > later,
> > > or ~>1.2.3 to get any version between 1.2.3 and 1.3 (excluding).
> > >
> > >
> > > OSGi is a set of big specs that boil a lot of oceans, the part I'm
> > > particularly interested in, is managing dependencies. When we started
> > > working on Buildr, OSGi was used almost exclusively by Eclipse. That
> > > changed. We're seeing more and more servers and frameworks adopting
> > OSGi.
> > >
> > > Imagine that we had OSGi support. You will specify all the dependencies
> > > once in the buildfile. Those will be used to build the project, and
> > also
> > > incorporated into the package manifest. Other projects could use these
> > > packages, reading the dependency list from the package manifest. More
> > > important, when you get to deploy those packages, the runtime can use
> > that
> > > information to bring in the right dependencies.
> > >
> > > That's one of those things that, once you use it, you won't want to go
> > back.
> > >
> > >
> > > The problem is, OSGi and Maven are incompatible with each other. I'll
> > try
> > > to summarize the differences:
> > >
> > > 1. The OSGi package identifier is typically the same as the file name.
> > The
> > > Maven package identifier combines group and artifact identifiers, only
> > the
> > > later is used as the file name.
> > >
> > > 2. Both use four-part version numbers. The first three are numerical
> > (e.g.
> > > 1.2.3) and managed the same way, the fourth one is the qualifier. OSGi
> > > qualifiers are alphanumeric and compared lexically, so R1 comes before
> > R2;
> > > using number is not advised, 10 comes before 2. Maven qualifiers are
> > > numeric for releases, and alphanumeric for pre-releases; the qualifier
> > 10
> > > comes after 2, and 2 comes after R2.
> > >
> > > 3. The rules for version matching are therefore not the same. The same
> > > rule could match different versions, depending on whethr you use OSGi or
> > > Maven.
> > >
> > > There's also the issue that OSGi keeps the meta-data inside the package,
> > > while Maven requires the additional POM file, and if both are present we
> > > have to decide which one to rely on.
> > >
> > > It's possible to some extent to support both at the same time, but there
> > > will always be that impedence mismatch. If we do a good enough job, it
> > will
> > > probably only affect edge cases, but we need to be aware.
> > >
> > > It will definitely be easier to pick one of the two, either OSGi or
> > Maven,
> > > and add transitive support and version matching based on that model,
> > then do
> > > translation when necessary (e.g. when reading POM file, generating OSGi
> > > manifest).
> > >
> > > If I had to start all over again, I would go for OSGi with Maven
> > > compatibility layer for three simple reasons:
> > >
> > > 1. Runtime deployment
> > > 2. Simpler to implement, and if your build doesn't quite work,
> > troubleshoot
> > > and fix.
> > > 3. Works the same way as most other packaging schemes (e.g. Gems,
> > Debian,
> > > RPM).
> > >
> > >
> > > Comments, ideas?
> > >
> > > Assaf
> > >
> >
> >
> >
> > --
> > -Shane
> >
>
--
-Shane
Re: Transitive dependencies, version matching and OSGi support
Posted by Assaf Arkin <ar...@intalio.com>.
On Wed, May 7, 2008 at 6:19 PM, Shane Witbeck <sh...@digitalsanctum.com>
wrote:
> I second the idea of going the way of OSGI. It's clearly the next step
> in Java's evolution.
>
> In terms of pre-1.3 dependency support, we resorted to doing something
> like the following to support a subset of compile dependencies for
> runtime packages:
>
> project.runtime.with(specs)
+1 for adding runtime dependencies.
>
>
> Because some of our environments had shared runtime classpaths, it
> wasn't always necessary to include certain libraries. I did this by
> extending Buildr::Project to support the different kinds of internal
> projects we have such as WebProject, StandaloneProject, BatchProject,
> etc.
>
> Something else that was needed in our projects was a way of putting
> together different packages for different environments. Again, by
> extending Buildr::Project, I introduced a way to package and deploy
> (via SSH) the various projects.
>
> I mention the ways that I've extended Buildr to possibly touch on new
> features to incorporate such as:
>
> 1. Being able to define different sets of dependencies for projects
> (in my case it was compile vs. runtime). I like the direction Assaf
> mentions with being able to merge dependency arrays and better
> transitive support. This is huge in terms of being able to make the
> buildfile less verbose.
> 2. A way to easily deploy (via SSH, etc.) packages
Can you expand on that?
>
> 3. Better integration with IDE's. It's a struggle to keep the
> dependencies in sync without having to regenerate the IDE files and in
> the process losing other project-specific IDE settings. Perhaps this
> is better solved via IDE-specific plugins for Buildr?
Good point. It might be possible to retain those project-specific IDE
settings, though I don't know what it would look like.
>
> 4. In terms of additional language support, my vote is for Flex. With
> the recent open sourced SDK, I believe there will be a surge in usage
> among not only Java developers but developers of other languages as
> well.
We'll need someone who's using Flex to help develop it.
Also, this is something we don't have to tie to the release schedule. We
finally have an API for adding new languages and delivering extensions using
Ruby Gems, so you can get people to use it without waiting for the next
release. Separately, once we add it to the code base it will show up in the
next release, but I think independent Gem would be the way to start.
Assaf
>
>
> I'd be glad to elaborate further on the points I've brought up and
> look forward to contributing more.
>
> Thanks,
> Shane
>
>
> On Fri, May 2, 2008 at 5:41 PM, Assaf Arkin <ar...@intalio.com> wrote:
> > Now that 1.3 is out, it's time to start talking about future releases.
> > We've got the 1.3.x line of releases, for dealing with outstanding
> issues,
> > new ones we'll find along the way, and building on new features that
> showed
> > up in 1.3.
> >
> > Separately, there's 1.4, the next major release and a place to make more
> > significant changes. We just have to decide what those would be.
> >
> > I propose that we focus 1.4 on improving the way we handle dependencies.
> >
> > Specifically what I have in mind are transitive dependencies, version
> > matching, and OSGi support. So I'll start by talking about some things
> I've
> > been looking into recently.
> >
> >
> > For starters, adding a dependency method to artifact (which also
> includes
> > packages) that returns an array of all the known dependencies for that
> > artifacts. Ruby arrays are pretty powerful, you can do things like:
> >
> > # Does it have a particular dependency?
> > foo.dependencies.any? { |dep| ... }
> > # Add new dependencies
> > bar.dependencies << ....
> > # Merge foo's dependencies into bar
> > bar.dependencies |= foo.dependencies
> >
> > One problem that comes up frequently are broken dependency lists, they
> may
> > be missing a dependency, or reference the wrong version, or include one
> you
> > don't want to use. Mutable arrays would make it easy to fix these
> broken
> > dependency lists, for example:
> >
> > # Change the artifact in memory
> > artifact( spec ).exclude( dont-want-this ).include( but want this )
> > # And use it
> > compile.with spec
> >
> > It will be even better to introduce a Dependencies class that extends
> array
> > and adds all sorts of convenience methods and lazy expansion, similar to
> > Rake's FileList. Then you could do things like:
> >
> > # Find dependency by its specification
> > foo.dependencies.has?( spec )
> > # Exclude all dependencies that match a pattern
> > foo.dependencies.exclude!( pattern )
> > # Expand all transitive dependencies and return flat list
> > foo.dependencies.expand
> > # Replace one with another
> > foo.dependencies.replace!( dont-want, want )
> >
> > The API for methods like compile.with and compile.dependencies remain
> the
> > same, just changing the underlying implementation.
> >
> > Packages would acquire dependencies from the project, for example,
> > package(:jar) would add all the dependencies from compile.dependencies,
> > which of course you can fix to your heart's content.
> >
> >
> > Version matching makes it possible to point at the most recent version
> that
> > satisfies a particular version requirement, so >=1.2 to get 1.2.0 or
> later,
> > or ~>1.2.3 to get any version between 1.2.3 and 1.3 (excluding).
> >
> >
> > OSGi is a set of big specs that boil a lot of oceans, the part I'm
> > particularly interested in, is managing dependencies. When we started
> > working on Buildr, OSGi was used almost exclusively by Eclipse. That
> > changed. We're seeing more and more servers and frameworks adopting
> OSGi.
> >
> > Imagine that we had OSGi support. You will specify all the dependencies
> > once in the buildfile. Those will be used to build the project, and
> also
> > incorporated into the package manifest. Other projects could use these
> > packages, reading the dependency list from the package manifest. More
> > important, when you get to deploy those packages, the runtime can use
> that
> > information to bring in the right dependencies.
> >
> > That's one of those things that, once you use it, you won't want to go
> back.
> >
> >
> > The problem is, OSGi and Maven are incompatible with each other. I'll
> try
> > to summarize the differences:
> >
> > 1. The OSGi package identifier is typically the same as the file name.
> The
> > Maven package identifier combines group and artifact identifiers, only
> the
> > later is used as the file name.
> >
> > 2. Both use four-part version numbers. The first three are numerical
> (e.g.
> > 1.2.3) and managed the same way, the fourth one is the qualifier. OSGi
> > qualifiers are alphanumeric and compared lexically, so R1 comes before
> R2;
> > using number is not advised, 10 comes before 2. Maven qualifiers are
> > numeric for releases, and alphanumeric for pre-releases; the qualifier
> 10
> > comes after 2, and 2 comes after R2.
> >
> > 3. The rules for version matching are therefore not the same. The same
> > rule could match different versions, depending on whethr you use OSGi or
> > Maven.
> >
> > There's also the issue that OSGi keeps the meta-data inside the package,
> > while Maven requires the additional POM file, and if both are present we
> > have to decide which one to rely on.
> >
> > It's possible to some extent to support both at the same time, but there
> > will always be that impedence mismatch. If we do a good enough job, it
> will
> > probably only affect edge cases, but we need to be aware.
> >
> > It will definitely be easier to pick one of the two, either OSGi or
> Maven,
> > and add transitive support and version matching based on that model,
> then do
> > translation when necessary (e.g. when reading POM file, generating OSGi
> > manifest).
> >
> > If I had to start all over again, I would go for OSGi with Maven
> > compatibility layer for three simple reasons:
> >
> > 1. Runtime deployment
> > 2. Simpler to implement, and if your build doesn't quite work,
> troubleshoot
> > and fix.
> > 3. Works the same way as most other packaging schemes (e.g. Gems,
> Debian,
> > RPM).
> >
> >
> > Comments, ideas?
> >
> > Assaf
> >
>
>
>
> --
> -Shane
>
Re: Transitive dependencies, version matching and OSGi support
Posted by Shane Witbeck <sh...@digitalsanctum.com>.
I second the idea of going the way of OSGI. It's clearly the next step
in Java's evolution.
In terms of pre-1.3 dependency support, we resorted to doing something
like the following to support a subset of compile dependencies for
runtime packages:
project.runtime.with(specs)
Because some of our environments had shared runtime classpaths, it
wasn't always necessary to include certain libraries. I did this by
extending Buildr::Project to support the different kinds of internal
projects we have such as WebProject, StandaloneProject, BatchProject,
etc.
Something else that was needed in our projects was a way of putting
together different packages for different environments. Again, by
extending Buildr::Project, I introduced a way to package and deploy
(via SSH) the various projects.
I mention the ways that I've extended Buildr to possibly touch on new
features to incorporate such as:
1. Being able to define different sets of dependencies for projects
(in my case it was compile vs. runtime). I like the direction Assaf
mentions with being able to merge dependency arrays and better
transitive support. This is huge in terms of being able to make the
buildfile less verbose.
2. A way to easily deploy (via SSH, etc.) packages
3. Better integration with IDE's. It's a struggle to keep the
dependencies in sync without having to regenerate the IDE files and in
the process losing other project-specific IDE settings. Perhaps this
is better solved via IDE-specific plugins for Buildr?
4. In terms of additional language support, my vote is for Flex. With
the recent open sourced SDK, I believe there will be a surge in usage
among not only Java developers but developers of other languages as
well.
I'd be glad to elaborate further on the points I've brought up and
look forward to contributing more.
Thanks,
Shane
On Fri, May 2, 2008 at 5:41 PM, Assaf Arkin <ar...@intalio.com> wrote:
> Now that 1.3 is out, it's time to start talking about future releases.
> We've got the 1.3.x line of releases, for dealing with outstanding issues,
> new ones we'll find along the way, and building on new features that showed
> up in 1.3.
>
> Separately, there's 1.4, the next major release and a place to make more
> significant changes. We just have to decide what those would be.
>
> I propose that we focus 1.4 on improving the way we handle dependencies.
>
> Specifically what I have in mind are transitive dependencies, version
> matching, and OSGi support. So I'll start by talking about some things I've
> been looking into recently.
>
>
> For starters, adding a dependency method to artifact (which also includes
> packages) that returns an array of all the known dependencies for that
> artifacts. Ruby arrays are pretty powerful, you can do things like:
>
> # Does it have a particular dependency?
> foo.dependencies.any? { |dep| ... }
> # Add new dependencies
> bar.dependencies << ....
> # Merge foo's dependencies into bar
> bar.dependencies |= foo.dependencies
>
> One problem that comes up frequently are broken dependency lists, they may
> be missing a dependency, or reference the wrong version, or include one you
> don't want to use. Mutable arrays would make it easy to fix these broken
> dependency lists, for example:
>
> # Change the artifact in memory
> artifact( spec ).exclude( dont-want-this ).include( but want this )
> # And use it
> compile.with spec
>
> It will be even better to introduce a Dependencies class that extends array
> and adds all sorts of convenience methods and lazy expansion, similar to
> Rake's FileList. Then you could do things like:
>
> # Find dependency by its specification
> foo.dependencies.has?( spec )
> # Exclude all dependencies that match a pattern
> foo.dependencies.exclude!( pattern )
> # Expand all transitive dependencies and return flat list
> foo.dependencies.expand
> # Replace one with another
> foo.dependencies.replace!( dont-want, want )
>
> The API for methods like compile.with and compile.dependencies remain the
> same, just changing the underlying implementation.
>
> Packages would acquire dependencies from the project, for example,
> package(:jar) would add all the dependencies from compile.dependencies,
> which of course you can fix to your heart's content.
>
>
> Version matching makes it possible to point at the most recent version that
> satisfies a particular version requirement, so >=1.2 to get 1.2.0 or later,
> or ~>1.2.3 to get any version between 1.2.3 and 1.3 (excluding).
>
>
> OSGi is a set of big specs that boil a lot of oceans, the part I'm
> particularly interested in, is managing dependencies. When we started
> working on Buildr, OSGi was used almost exclusively by Eclipse. That
> changed. We're seeing more and more servers and frameworks adopting OSGi.
>
> Imagine that we had OSGi support. You will specify all the dependencies
> once in the buildfile. Those will be used to build the project, and also
> incorporated into the package manifest. Other projects could use these
> packages, reading the dependency list from the package manifest. More
> important, when you get to deploy those packages, the runtime can use that
> information to bring in the right dependencies.
>
> That's one of those things that, once you use it, you won't want to go back.
>
>
> The problem is, OSGi and Maven are incompatible with each other. I'll try
> to summarize the differences:
>
> 1. The OSGi package identifier is typically the same as the file name. The
> Maven package identifier combines group and artifact identifiers, only the
> later is used as the file name.
>
> 2. Both use four-part version numbers. The first three are numerical (e.g.
> 1.2.3) and managed the same way, the fourth one is the qualifier. OSGi
> qualifiers are alphanumeric and compared lexically, so R1 comes before R2;
> using number is not advised, 10 comes before 2. Maven qualifiers are
> numeric for releases, and alphanumeric for pre-releases; the qualifier 10
> comes after 2, and 2 comes after R2.
>
> 3. The rules for version matching are therefore not the same. The same
> rule could match different versions, depending on whethr you use OSGi or
> Maven.
>
> There's also the issue that OSGi keeps the meta-data inside the package,
> while Maven requires the additional POM file, and if both are present we
> have to decide which one to rely on.
>
> It's possible to some extent to support both at the same time, but there
> will always be that impedence mismatch. If we do a good enough job, it will
> probably only affect edge cases, but we need to be aware.
>
> It will definitely be easier to pick one of the two, either OSGi or Maven,
> and add transitive support and version matching based on that model, then do
> translation when necessary (e.g. when reading POM file, generating OSGi
> manifest).
>
> If I had to start all over again, I would go for OSGi with Maven
> compatibility layer for three simple reasons:
>
> 1. Runtime deployment
> 2. Simpler to implement, and if your build doesn't quite work, troubleshoot
> and fix.
> 3. Works the same way as most other packaging schemes (e.g. Gems, Debian,
> RPM).
>
>
> Comments, ideas?
>
> Assaf
>
--
-Shane
Re: Transitive dependencies, version matching and OSGi support
Posted by Tommy Knowlton <to...@ke4kug.net>.
Wow, I should look into this, and soon.
Thanks for providing all of the details you did with regard to OSGi.
Otherwise I don't have any comments (yet), just wanted to acknowledge
that your effort has spurred me to do more research.
Cheers,
Tommy
On May 2, 2008, at 3:41 PM, Assaf Arkin wrote:
> If I had to start all over again, I would go for OSGi with Maven
> compatibility layer for three simple reasons:
>
> 1. Runtime deployment
> 2. Simpler to implement, and if your build doesn't quite work,
> troubleshoot
> and fix.
> 3. Works the same way as most other packaging schemes (e.g. Gems,
> Debian,
> RPM).