You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/07/29 22:49:21 UTC
[jira] [Commented] (SPARK-1121) Only add avro if the build is for
Hadoop 0.23.X and SPARK_YARN is set
[ https://issues.apache.org/jira/browse/SPARK-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400167#comment-15400167 ]
Apache Spark commented on SPARK-1121:
-------------------------------------
User 'pwendell' has created a pull request for this issue:
https://github.com/apache/spark/pull/37
> Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
> ---------------------------------------------------------------------
>
> Key: SPARK-1121
> URL: https://issues.apache.org/jira/browse/SPARK-1121
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Patrick Cogan
> Assignee: prashant
> Fix For: 1.0.0
>
>
> The reason why this is needed is that in the 0.23.X versions of hadoop-client the avro dependency is fully excluded:
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/0.23.10/hadoop-client-0.23.10.pom
> In later versions 2.2.X the avro dependency is correctly inherited from hadoop-common:
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.2.0/hadoop-client-2.2.0.pom
> So as a workaround Spark currently depends on Avro directly in the sbt and scala builds. This is a bit ugly so I'd like to propose the following:
> 1. In the Maven build just remove avro and make a note on the building-with-maven page that they will need to manually add avro for this build.
> 2. On sbt only add the avro dependency if the version is 0.23.X and SPARK_YARN is true. Also we only need to add avro not both {avro, avro-ipc} like is there now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org