You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2019/09/22 04:25:00 UTC
[jira] [Comment Edited] (SPARK-23710) Upgrade the built-in Hive to
2.3.5 for hadoop-3.2
[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935207#comment-16935207 ]
Dongjoon Hyun edited comment on SPARK-23710 at 9/22/19 4:24 AM:
----------------------------------------------------------------
Hi, All. (Especially, [~yumwang])
Can we move on the next stage of this workitem?
The remaining two items (`Update Migration Guide` and `Update License/NOTICE`) are required of courses, but that's more UX or legal issue instead of technical issue. I'd like to recommend to resolve this issue as `DONE` and move those to another JIRA issue.
That will motivate the community to test this feature more and more.
was (Author: dongjoon):
Hi, All. (Especially, [~yumwang])
Can you move on the next stage of this workitem?
The remaining two items (`Update Migration Guide` and `Update License/NOTICE`) are required of courses, but that's more UX or legal issue instead of technical issue. I'd like to recommend to resolve this issue as `DONE` and move those to another JIRA issue.
That will motivate the community to test this feature more and more.
> Upgrade the built-in Hive to 2.3.5 for hadoop-3.2
> -------------------------------------------------
>
> Key: SPARK-23710
> URL: https://issues.apache.org/jira/browse/SPARK-23710
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Yuming Wang
> Assignee: Yuming Wang
> Priority: Critical
>
> Spark fail to run on Hadoop 3.x, because Hive's ShimLoader considers Hadoop 3.x to be an unknown Hadoop version. see SPARK-18673 and HIVE-16081 for more details. So we need to upgrade the built-in Hive for Hadoop-3.x. This is an umbrella JIRA to track this upgrade.
>
> *Upgrade Plan*:
> # SPARK-27054 Remove the Calcite dependency. This can avoid some jar conflicts.
> # SPARK-23749 Replace built-in Hive API (isSub/toKryo) and remove OrcProto.Type usage
> # SPARK-27158, SPARK-27130 Update dev/* to support dynamic change profiles when testing
> # Fix ORC dependency conflict to makes it test passed on Hive 1.2.1 and compile passed on Hive 2.3.4
> # Add an empty hive-thriftserverV2 module. then we could test all test cases in next step
> # Make Hadoop-3.1 with Hive 2.3.4 test passed
> # Adapted hive-thriftserverV2 from hive-thriftserver with Hive 2.3.4's [TCLIService.thrift|https://github.com/apache/hive/blob/rel/release-2.3.4/service-rpc/if/TCLIService.thrift]
>
> I have completed the [initial work|https://github.com/apache/spark/pull/24044] and plan to finish this upgrade step by step.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org