You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "Roman Shaposhnik (JIRA)" <ji...@apache.org> on 2011/09/16 01:50:08 UTC
[jira] [Created] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Ability to force ivy/maven version inter-dependency needs to be implemented
---------------------------------------------------------------------------
Key: BIGTOP-98
URL: https://issues.apache.org/jira/browse/BIGTOP-98
Project: Bigtop
Issue Type: Bug
Reporter: Roman Shaposhnik
We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Andrew Bayer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106759#comment-13106759 ]
Andrew Bayer commented on BIGTOP-98:
------------------------------------
Bruno - I think Roman's talking about being able to, say, do test builds where we build a pre-release version of Hadoop, and then set all the other components to depend on that pre-release version we just built.
I think working with the component projects to get overrideable dependency versions for those dependencies within Bigtop would be part of the right solution here - we've got that with Ant/Ivy already, and the HBase approach seems portable to the other Maven projects as well. It's not the best approach on a purely Maven level - I'm not a huge fan of the indirection of using property values for one-off dependency versions - but it's functional.
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Andrew Bayer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109853#comment-13109853 ]
Andrew Bayer commented on BIGTOP-98:
------------------------------------
Any chance that ZK patch is in a Github branch somewhere for me to play with?
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105902#comment-13105902 ]
Konstantin Boudnik commented on BIGTOP-98:
------------------------------------------
Such on-the fly parametrization of dependencies would be disastrous IMO for it will allow to create stacks with pretty much unknown set of dependencies.
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Bruno Mahé (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106743#comment-13106743 ]
Bruno Mahé commented on BIGTOP-98:
----------------------------------
{quote}Making sure that we can make freshly built bits available to downstream parts of the build as dependencies {quote}
I am not sure to follow. It's very likely an upstream project has already its artifacts in a maven repository. So they will already be available downstream. If a project does not publish artifacts for maven, maybe we should focus on helping projects do that
{quote}Making sure that we can override build dependencies of the downstream parts of the build to pick up versions that we specify
I'm about to investigate how easy/difficult it'll be to solve both of them. Feel free to chime in. {quote}
How far can we go? A patch would be perfect so we can separate the build responsibility from the version specification.
As for the political side, I don't disagree with you, but this is still something to clear up
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Andrew Bayer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105781#comment-13105781 ]
Andrew Bayer commented on BIGTOP-98:
------------------------------------
This is actually pretty tough to do without patching the underlying components - I think we may be able to get away with passing properties to Ant/Ivy builds to force new versions for dependencies without actually editing any files, but off the top of my head, I'm not sure there's a way to do that with Maven.
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108200#comment-13108200 ]
Roman Shaposhnik commented on BIGTOP-98:
----------------------------------------
Ok, here's a bit of investigation I've done as far as the current state of tweakability of deps is concerned:
1. ant projects (* marks the one that need to be made tweakable):
1.1. zookeeper -- no explicit dependencies on other stack components
1.2. hadoop -- no explicit dependencies on other stack components
1.3. flume -- no explicit dependencies on other stack components
1.4. sqoop -- depends on hadoop-core(*), hadoop-test(*), hbase
1.5. pig -- depends on hadoop-core, hadoop-test, zookeeper, hbase
1.6. hive -- quite broken :-(
2. maven projects
2.1. hbase -- hadoop-core, zookeeper
2.2. mahout -- hadoop-core(*)
2.3. oozie -- quite broken :-(
2.4. whirr -- hadoop-core
Overall, I think we're doing pretty well. I'll try to run a custom build once BIGTOP-100
gets integrated.
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109849#comment-13109849 ]
Patrick Hunt commented on BIGTOP-98:
------------------------------------
FYI ZK and Maven, patch pending: https://issues.apache.org/jira/browse/ZOOKEEPER-1078
I'd love it if someone were to jump in and help out with this effort. (it's on trunk so it would be for the ZK 3.5.0 release)
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112226#comment-13112226 ]
Patrick Hunt commented on BIGTOP-98:
------------------------------------
No, but it is attached to the the ZOOKEEPER-1078 jira. I'm planning to commit it next I have some free time.
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (BIGTOP-98) Ability to force
ivy/maven version inter-dependency needs to be implemented
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105902#comment-13105902 ]
Konstantin Boudnik edited comment on BIGTOP-98 at 9/16/11 6:36 AM:
-------------------------------------------------------------------
Such on-the fly parametrization of dependencies would be disastrous IMO for it will allow to create stacks with pretty much unknown set of dependent artifacts.
was (Author: cos):
Such on-the fly parametrization of dependencies would be disastrous IMO for it will allow to create stacks with pretty much unknown set of dependencies.
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Bruno Mahé (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105935#comment-13105935 ]
Bruno Mahé commented on BIGTOP-98:
----------------------------------
Andrew> What if we make sure each project does not hardcode versions but uses properties which define their versions for their dependencies (similar to what HBase does here https://github.com/apache/hbase/blob/trunk/pom.xml )? We could then override them.
Konstantin> I believe the goal would be more about ensuring a coherent set of dependencies and ensuring that inter-dependencies between projects are correct and respected. For instance, when I build flume packages, maven/ivy would pull the same versions of zookeeper we distribute and not some other arbitrary one which may or may not be compatible. Maybe Roman could clarify a little bit more?
We should also check with mentors and Apache people if it would be allowed. Because of what has been decided during the incubation process, we cannot patch for security or build issues. And overriding dependencies versions might make us create artifacts not based on what was voted on for each project (if we want to be strict about it, which I don't really want to).
Furthermore, we would loose all the testing done by each of these projects. So we would have the additional burden of proving these changes wouldn't break anything.
But this seems to be something helpful and worthy to pursue or at least to think about.
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-98) Ability to force ivy/maven version
inter-dependency needs to be implemented
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106687#comment-13106687 ]
Roman Shaposhnik commented on BIGTOP-98:
----------------------------------------
@Cos
I think I might not have been as clear as I needed to be in my description -- what I'm talking about is having a functionality in place that would ensure that what we build against and what we run against is the same set of versioned components. So far, we're actually dependent on the versioning of dependencies supplied by every upstream component. So lets take our .22 effort as an example -- we would like to bump a Hadoop version to .22 and that's easy. Yet, things like Pig and HBase will still pull the versions of hadoop-core artifacts that were recorded in their ivy/maven properties at the time of a component release (definitely NOT .22). All of this will lead to an unfortunate situation where we are going to build Pig against 0.20.2 and execute against .22.
@Bruno
1. On a technical side -- we have to problems to solve:
* Making sure that we can make freshly built bits available to downstream parts of the build as dependencies
* Making sure that we can override build dependencies of the downstream parts of the build to pick up versions that we specify
I'm about to investigate how easy/difficult it'll be to solve both of them. Feel free to chime in.
2. On a political side -- there's a legitimate problem to be solved here. The problem being -- we can NOT rely on the version
information recorded as part of the release of leaf-node components, since different customers will ALWAY have different
needs. E.g. Pig 0.9.0 is currently released as requiring 0.20.2. Yet there's plenty of customers who need it to be part
of the stack that has Hadoop 0.20.20X as the basis OR Hadoop 0.21 as the basis, etc.
> Ability to force ivy/maven version inter-dependency needs to be implemented
> ---------------------------------------------------------------------------
>
> Key: BIGTOP-98
> URL: https://issues.apache.org/jira/browse/BIGTOP-98
> Project: Bigtop
> Issue Type: Bug
> Reporter: Roman Shaposhnik
>
> We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira