You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2019/08/13 13:46:00 UTC
[jira] [Commented] (SPARK-23151) Provide a distribution of Spark
with Hadoop 3.0
[ https://issues.apache.org/jira/browse/SPARK-23151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906218#comment-16906218 ]
Thomas Graves commented on SPARK-23151:
---------------------------------------
Hey so for the hadoop-3.2 profile, the dependencies are quite a bit different since we are using the hive version of orc. The things it pulls in are quite different then the nohive version of orc. It would be really nice if we had a different classifier or something so users can actually tell the difference in what is published on mvn. Right now all we know is its a spark jar with version X.x and scala version y.y but don't know which hadoop profile and thus dependencies got pulled in. Something to think about anyway
> Provide a distribution of Spark with Hadoop 3.0
> -----------------------------------------------
>
> Key: SPARK-23151
> URL: https://issues.apache.org/jira/browse/SPARK-23151
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Louis Burke
> Priority: Major
>
> Provide a Spark package that supports Hadoop 3.0.0. Currently the Spark package
> only supports Hadoop 2.7 i.e. spark-2.2.1-bin-hadoop2.7.tgz. The implication is
> that using up to date Kinesis libraries alongside s3 causes a clash w.r.t
> aws-java-sdk.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org