You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2015/05/15 17:13:01 UTC
[jira] [Commented] (SPARK-7669) Builds against Hadoop 2.6+ get
inconsistent curator dependencies
[ https://issues.apache.org/jira/browse/SPARK-7669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545636#comment-14545636 ]
Steve Loughran commented on SPARK-7669:
---------------------------------------
snippet of the maven dependencies with {{-Phadoop-2.4 -Dhadoop.version=2.6.0 }}
{code}
[INFO] | | | +- org.apache.curator:curator-client:jar:2.6.0:compile
[INFO] | | | | +- (org.slf4j:slf4j-api:jar:1.7.10:compile - version managed from 1.7.6; omitted for duplicate)
[INFO] | | | | +- (org.apache.zookeeper:zookeeper:jar:3.4.5:compile - version managed from 3.4.6; omitted for duplicate)
[INFO] | | | | \- (com.google.guava:guava:jar:14.0.1:provided - version managed from 11.0.2; scope managed from compile; omitted for duplicate)
[INFO] | | | +- (org.apache.curator:curator-recipes:jar:2.4.0:compile - version managed from 2.6.0; omitted for duplicate)
{code}
What's happened is that the curator-recipes version is being set by spark, but no version of curator-client is set. Nor can you fix the versions on the CLI, as the curator version isn't yet property driven.
I have a patch to make things consistent, with a hadoop 2.6 profile which sets curator.version=2.6,0, zookeeper.version=3.4.6, and drives the curator-recipes version off that curator.version property. There's one possible enhancement to this: declare the version of curator-client &c directly, putting the spark build in control of which version gets picked up.
> Builds against Hadoop 2.6+ get inconsistent curator dependencies
> ----------------------------------------------------------------
>
> Key: SPARK-7669
> URL: https://issues.apache.org/jira/browse/SPARK-7669
> Project: Spark
> Issue Type: Bug
> Components: Build
> Affects Versions: 1.3.1, 1.4.0
> Environment: Hadoop 2.6
> Reporter: Steve Loughran
> Priority: Minor
>
> If you build spark against Hadoop 2.6 you end up with an inconsistent set of curator dependencies -curator-recipe 2.4.0 with curator 2.6.0.
> A dedicated hadoop-2.6 profile along with extraction of curator version into a property can keep the curator versions in sync, along with ZK.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org