You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Kevin Burton <bu...@spinn3r.com> on 2015/01/05 22:03:24 UTC
Resolving classpath hell issues by allowing libraries to only use the
version they request?
I spent a ton of time tonight in classpath hell.
Basically, Apache Spark, Cassandra, and Cassandra Unit, and Guava, Jackson
JSON, and Jetty have an INSANE dependency graph. They're all trampling on
each other with broken
dependencies. This results in a lot of <exclusion> work to get them to use
sane bindings.
And it’s not necessarily anyone’s fault. Library A will use library B
which depend on library C version 1.1 but then library D will use library C
which depends on 1.2…
I *mostly* end up with NoClassDefFoundException.
But I’m worried about other smaller bugs. For example, if my library
wasn’t tested with a later version of a library it isn’t necessarily as
reliable out of the box since it wasn’t tested with that config.
Has anyone done any work with only allowing a library to use the version of
a class it requested?
If you release your library fully tested on version 1.1.1 of some library,
and
someone accidentally starts using 1.1.2 becuase of dependency hell, then
we're
going to break the build and introduce bugs.
So what would happen is every dependency would only be sourced from the
version you require.
This could be done via a hierarchical class loader. Of course this would
probably end up burning up some memory but probably worth it.
The main issue, is it WORTH it.. would this introduce new bugs?
For example, it’s possible that libraries could step on each other by
creating duplicate threads with the same names, or messing up system
properties.
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Resolving classpath hell issues by allowing libraries to only use
the version they request?
Posted by Mark Derricutt <ma...@talios.com>.
On 6 Jan 2015, at 12:29, Kevin Burton wrote:
> I agree. in this case the issue is testing. I need to embed cassandra so I
> can test it but it’s conflicting with spark.
As a total side issue - have you considered using Docker?
http://blog.xebia.com/2014/10/13/fast-and-easy-integration-testing-with-docker-and-overcast/
I'm sorely tempted to switch a lot of our integration tests over to using overcast, assuming I can create a local registry of docker images ( I believe Artifactory can do it, but not nexus (yet?) ).
Mark
--
Mark Derricutt
http://www.theoryinpractice.net
http://plus.google.com/+MarkDerricutt
http://twitter.com/talios
http://facebook.com/mderricutt
Re: Resolving classpath hell issues by allowing libraries to only use
the version they request?
Posted by Barrie Treloar <ba...@gmail.com>.
On 6 January 2015 at 14:50, Ron Wheeler <rw...@artifact-software.com>
wrote:
> On 05/01/2015 6:29 PM, Kevin Burton wrote:
>
>> Breaking your application into smaller independent services may give you a
>>
>>> way to reduce the number of conflicts related to these different
>>> transitive
>>> dependencies but that may not be as easy as testing your app with the
>>> updated versions or doing some research with the teams that are building
>>> with the old libraries to see if they will upgrade and run their
>>> acceptance
>>> tests within a time frame that meets your project's timeline
>>> requirements.
>>>
>>
>> I agree. in this case the issue is testing. I need to embed cassandra so
>> I
>> can test it but it’s conflicting with spark.
>>
> These are both Apache projects that I don't know.
Same.
Or alternatively from my brief reading of these two it appears that they
can run independently.
If you can avoid having them running in the same process you can separate
out your dependency hell.
This may be as simple as creating a new module to do your integration
testing and to control starting the external dependencies needed to do your
integration testing.
Re: Resolving classpath hell issues by allowing libraries to only
use the version they request?
Posted by Ron Wheeler <rw...@artifact-software.com>.
On 05/01/2015 6:29 PM, Kevin Burton wrote:
> Breaking your application into smaller independent services may give you a
>> way to reduce the number of conflicts related to these different transitive
>> dependencies but that may not be as easy as testing your app with the
>> updated versions or doing some research with the teams that are building
>> with the old libraries to see if they will upgrade and run their acceptance
>> tests within a time frame that meets your project's timeline requirements.
>
> I agree. in this case the issue is testing. I need to embed cassandra so I
> can test it but it’s conflicting with spark.
These are both Apache projects that I don't know.
Have you asked the one using the older dependencies if they can upgrade
their dependencies and run their acceptance tests?
Even if you have to download the sources and upgrade the dependencies
and run their acceptance tests, this might be easier than trying to test
your own application to the extent required to ensure that the software
has no problems with the updated versions.
This assumes that these projects have strong test suites.
I would state your problems in the forum of the offending project. There
might be a lot of enthusiasm for an upgrade.
Ron
> Ideally I could just start a container and I think that’s the way we’re
> going to evolve things. Essentially, start a container for the daemons I
> need to test against and then make sure all my components work. Then just
> kill the containers.
>
> —
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Resolving classpath hell issues by allowing libraries to only
use the version they request?
Posted by Ron Wheeler <rw...@artifact-software.com>.
It seems that there is a lot of expertise devoted to mission-critical
projects that involve both Cassandra and Spark.
I would be surprised of you could not get the "right" solution" in the
Spark user group.
They would at least have worked out a set of dependencies that are
working well.
Or would be able to warn you about broken dependencies.
Ron
On 06/01/2015 3:27 AM, Stephen Connolly wrote:
> On Monday, January 5, 2015, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> Breaking your application into smaller independent services may give you a
>>> way to reduce the number of conflicts related to these different
>> transitive
>>> dependencies but that may not be as easy as testing your app with the
>>> updated versions or doing some research with the teams that are building
>>> with the old libraries to see if they will upgrade and run their
>> acceptance
>>> tests within a time frame that meets your project's timeline
>> requirements.
>>
>>
>> I agree. in this case the issue is testing. I need to embed cassandra so I
>> can test it but it’s conflicting with spark.
>
> Cassandra is not really designed for embedding. Better to start it in a
> separate JVM which would allow controlling the classpath
>
> Take a look at Cassandra-maven-plugin either as a solution or for some
> example code for forking a JVM with Cassandra
>
>
>> Ideally I could just start a container and I think that’s the way we’re
>> going to evolve things. Essentially, start a container for the daemons I
>> need to test against and then make sure all my components work. Then just
>> kill the containers.
>>
>> —
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Resolving classpath hell issues by allowing libraries to only use
the version they request?
Posted by Stephen Connolly <st...@gmail.com>.
On Monday, January 5, 2015, Kevin Burton <bu...@spinn3r.com> wrote:
> Breaking your application into smaller independent services may give you a
> > way to reduce the number of conflicts related to these different
> transitive
> > dependencies but that may not be as easy as testing your app with the
> > updated versions or doing some research with the teams that are building
> > with the old libraries to see if they will upgrade and run their
> acceptance
> > tests within a time frame that meets your project's timeline
> requirements.
>
>
> I agree. in this case the issue is testing. I need to embed cassandra so I
> can test it but it’s conflicting with spark.
Cassandra is not really designed for embedding. Better to start it in a
separate JVM which would allow controlling the classpath
Take a look at Cassandra-maven-plugin either as a solution or for some
example code for forking a JVM with Cassandra
>
> Ideally I could just start a container and I think that’s the way we’re
> going to evolve things. Essentially, start a container for the daemons I
> need to test against and then make sure all my components work. Then just
> kill the containers.
>
> —
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
--
Sent from my phone
Re: Resolving classpath hell issues by allowing libraries to only use
the version they request?
Posted by Kevin Burton <bu...@spinn3r.com>.
Breaking your application into smaller independent services may give you a
> way to reduce the number of conflicts related to these different transitive
> dependencies but that may not be as easy as testing your app with the
> updated versions or doing some research with the teams that are building
> with the old libraries to see if they will upgrade and run their acceptance
> tests within a time frame that meets your project's timeline requirements.
I agree. in this case the issue is testing. I need to embed cassandra so I
can test it but it’s conflicting with spark.
Ideally I could just start a container and I think that’s the way we’re
going to evolve things. Essentially, start a container for the daemons I
need to test against and then make sure all my components work. Then just
kill the containers.
—
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Resolving classpath hell issues by allowing libraries to only
use the version they request?
Posted by Ron Wheeler <rw...@artifact-software.com>.
You are right that jar hell and dll hell are not that easy to deal with.
OSGi is supposed to be a way to reduce this problem but IMHUO brings it
own problems.
Most reputable projects are pretty careful about compatibility since it
is really tough on developers if new versions break existing code.
Developers will just refuse to use the library if updates repeatedly
break their code.
Breaking your application into smaller independent services may give you
a way to reduce the number of conflicts related to these different
transitive dependencies but that may not be as easy as testing your app
with the updated versions or doing some research with the teams that are
building with the old libraries to see if they will upgrade and run
their acceptance tests within a time frame that meets your project's
timeline requirements.
If the project is open source, you may be able to run the acceptance
tests for the dependencies that use old libraries after you change their
build to use later versions.
That would give you a higher level of confidence and perhaps be a bit of
a pay-back to the team that is giving you a helpful library, if you give
them the feedback.
Ron
On 05/01/2015 4:03 PM, Kevin Burton wrote:
> I spent a ton of time tonight in classpath hell.
>
> Basically, Apache Spark, Cassandra, and Cassandra Unit, and Guava, Jackson
> JSON, and Jetty have an INSANE dependency graph. They're all trampling on
> each other with broken
> dependencies. This results in a lot of <exclusion> work to get them to use
> sane bindings.
>
> And it’s not necessarily anyone’s fault. Library A will use library B
> which depend on library C version 1.1 but then library D will use library C
> which depends on 1.2…
>
> I *mostly* end up with NoClassDefFoundException.
>
> But I’m worried about other smaller bugs. For example, if my library
> wasn’t tested with a later version of a library it isn’t necessarily as
> reliable out of the box since it wasn’t tested with that config.
>
> Has anyone done any work with only allowing a library to use the version of
> a class it requested?
>
> If you release your library fully tested on version 1.1.1 of some library,
> and
> someone accidentally starts using 1.1.2 becuase of dependency hell, then
> we're
> going to break the build and introduce bugs.
>
> So what would happen is every dependency would only be sourced from the
> version you require.
>
> This could be done via a hierarchical class loader. Of course this would
> probably end up burning up some memory but probably worth it.
>
> The main issue, is it WORTH it.. would this introduce new bugs?
>
> For example, it’s possible that libraries could step on each other by
> creating duplicate threads with the same names, or messing up system
> properties.
>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Resolving classpath hell issues by allowing libraries to only use
the version they request?
Posted by Curtis Rueden <ct...@wisc.edu>.
Hi Kevin,
I agree with Steven. One way of resolving this sort of problem without OSGi
is to use a consistent, meaningful versioning system such as Semantic
Versioning (http://semver.org/). Once you have the ability to reason about
forwards and backwards compatibility, it is easier to resolve version
conflicts in a more reliable (ideally automated) way.
Of course, all the tools you are using do not necessarily abide by such
versioning standards. So then conflict resolution becomes more complicated.
Steven's suggestion of "managing them upwards to the latest needed" is
often good enough...
Regards,
Curtis
On Mon, Jan 5, 2015 at 3:20 PM, Steven Schlansker <
stevenschlansker@gmail.com> wrote:
>
> On Jan 5, 2015, at 1:03 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> > I spent a ton of time tonight in classpath hell.
> >
> > Basically, Apache Spark, Cassandra, and Cassandra Unit, and Guava,
> Jackson
> > JSON, and Jetty have an INSANE dependency graph. They're all trampling
> on
> > each other with broken
> > dependencies. This results in a lot of <exclusion> work to get them to
> use
> > sane bindings.
> >
> > Has anyone done any work with only allowing a library to use the version
> of
> > a class it requested?
> >
> > So what would happen is every dependency would only be sourced from the
> > version you require.
> >
> > This could be done via a hierarchical class loader. Of course this would
> > probably end up burning up some memory but probably worth it.
> >
> > The main issue, is it WORTH it.. would this introduce new bugs?
>
> You are basically talking about OSGi or some other such module system.
> https://en.wikipedia.org/wiki/OSGi
>
> In my opinion, it is actually easier to resolve the dependency conflicts
> among
> the modules (usually by managing them upwards to the latest needed by any
> dependency)
> and tracking this via a plugin [1]
>
> Having such complicated class loader situations just leads to even more
> subtle and confusing bugs IMO.
>
> [1] https://github.com/ning/maven-dependency-versions-check-plugin
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Resolving classpath hell issues by allowing libraries to only use the version they request?
Posted by Steven Schlansker <st...@gmail.com>.
On Jan 5, 2015, at 1:03 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I spent a ton of time tonight in classpath hell.
>
> Basically, Apache Spark, Cassandra, and Cassandra Unit, and Guava, Jackson
> JSON, and Jetty have an INSANE dependency graph. They're all trampling on
> each other with broken
> dependencies. This results in a lot of <exclusion> work to get them to use
> sane bindings.
>
> Has anyone done any work with only allowing a library to use the version of
> a class it requested?
>
> So what would happen is every dependency would only be sourced from the
> version you require.
>
> This could be done via a hierarchical class loader. Of course this would
> probably end up burning up some memory but probably worth it.
>
> The main issue, is it WORTH it.. would this introduce new bugs?
You are basically talking about OSGi or some other such module system.
https://en.wikipedia.org/wiki/OSGi
In my opinion, it is actually easier to resolve the dependency conflicts among
the modules (usually by managing them upwards to the latest needed by any dependency)
and tracking this via a plugin [1]
Having such complicated class loader situations just leads to even more subtle and confusing bugs IMO.
[1] https://github.com/ning/maven-dependency-versions-check-plugin
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org