You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (JIRA)" <ji...@apache.org> on 2018/03/16 16:24:00 UTC
[jira] [Updated] (SPARK-23710) Upgrade Hive to 2.3.2
[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-23710:
--------------------------------
Description:
h1. Mainly changes
* Maven dependency:
hive.version from {{1.2.1.spark2}} to {{2.3.2}} and change {{hive.classifier}} to {{core}}
calcite.version from {{1.2.0-incubating}} to {{1.10.0}}
datanucleus-core.version from {{3.2.10}} to {{4.1.17}}
remove {{orc.classifier}}, it means orc use the {{hive.storage.api}}, see: ORC-174
add new dependency {{avatica}} and {{hive.storage.api}}
* ORC compatibility changes:
OrcColumnVector.java, OrcColumnarBatchReader.java, OrcDeserializer.scala, OrcFilters.scala, OrcSerializer.scala, OrcFilterSuite.scala
* hive-thriftserver java file update:
update {{sql/hive-thriftserver/if/TCLIService.thrift}} to hive 2.3.2
update {{sql/hive-thriftserver/src/main/java/org/apache/hive/service/*}} to hive 2.3.2
* TestSuite should update:
||TestSuite||Reson||
|StatisticsSuite|HIVE-16098|
|SessionCatalogSuite|[VersionsSuite.scala#L427|#L427]|
|CliSuite, HiveThriftServer2Suites, HiveSparkSubmitSuite, HiveQuerySuite, SQLQuerySuite|Update hive-hcatalog-core-0.13.1.jar to hive-hcatalog-core-2.3.2.jar|
|SparkExecuteStatementOperationSuite|Interface changed from org.apache.hive.service.cli.Type.NULL_TYPE to org.apache.hadoop.hive.serde2.thrift.Type.NULL_TYPE|
|ClasspathDependenciesSuite|org.apache.hive.com.esotericsoftware.kryo.Kryo change to com.esotericsoftware.kryo.Kryo|
|HiveMetastoreCatalogSuite|Result format changed from Seq("1.1\t1", "2.1\t2") to Seq("1.100\t1", "2.100\t2")|
|HiveOrcFilterSuite|Result format changed|
|HiveDDLSuite|Remove $ (This change needs to be reconsidered)|
was:
h1. Mainly changes
* Maven dependency:
hive.version from {{1.2.1.spark2}} to {{2.3.2}} and change {{hive.classifier}} to {{core}}
derby.version from {{10.12.1.1}} to {{10.10.2.0}}
calcite.version from {{1.2.0-incubating}} to {{1.10.0}}
datanucleus-core.version from {{3.2.10}} to {{4.1.17}}
remove {{orc.classifier}}, it means orc use the {{hive.storage.api}}, see: ORC-174
add new dependency {{avatica}} and {{hive.storage.api}}
* ORC compatibility changes:
OrcColumnVector.java, OrcColumnarBatchReader.java, OrcDeserializer.scala, OrcFilters.scala, OrcSerializer.scala, OrcFilterSuite.scala
* hive-thriftserver java file update:
update {{sql/hive-thriftserver/if/TCLIService.thrift}} to hive 2.3.2
update {{sql/hive-thriftserver/src/main/java/org/apache/hive/service/*}} to hive 2.3.2
* TestSuite should update:
||TestSuite||Reson||
|StatisticsSuite|HIVE-16098|
|SessionCatalogSuite|[VersionsSuite.scala#L427|Like this https://github.com/apache/spark/blob/ff1480189b827af0be38605d566a4ee71b4c36f6/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala#L427]|
|CliSuite, HiveThriftServer2Suites, HiveSparkSubmitSuite, HiveQuerySuite, SQLQuerySuite|Update hive-hcatalog-core-0.13.1.jar to hive-hcatalog-core-2.3.2.jar|
|SparkExecuteStatementOperationSuite|Interface changed from org.apache.hive.service.cli.Type.NULL_TYPE to org.apache.hadoop.hive.serde2.thrift.Type.NULL_TYPE|
|ClasspathDependenciesSuite|org.apache.hive.com.esotericsoftware.kryo.Kryo change to com.esotericsoftware.kryo.Kryo|
|HiveMetastoreCatalogSuite|Result format changed from Seq("1.1\t1", "2.1\t2") to Seq("1.100\t1", "2.100\t2")|
|HiveOrcFilterSuite|Result format changed|
|HiveDDLSuite|Remove $ (This change needs to be reconsidered)|
> Upgrade Hive to 2.3.2
> ---------------------
>
> Key: SPARK-23710
> URL: https://issues.apache.org/jira/browse/SPARK-23710
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Yuming Wang
> Priority: Major
>
> h1. Mainly changes
> * Maven dependency:
> hive.version from {{1.2.1.spark2}} to {{2.3.2}} and change {{hive.classifier}} to {{core}}
> calcite.version from {{1.2.0-incubating}} to {{1.10.0}}
> datanucleus-core.version from {{3.2.10}} to {{4.1.17}}
> remove {{orc.classifier}}, it means orc use the {{hive.storage.api}}, see: ORC-174
> add new dependency {{avatica}} and {{hive.storage.api}}
> * ORC compatibility changes:
> OrcColumnVector.java, OrcColumnarBatchReader.java, OrcDeserializer.scala, OrcFilters.scala, OrcSerializer.scala, OrcFilterSuite.scala
> * hive-thriftserver java file update:
> update {{sql/hive-thriftserver/if/TCLIService.thrift}} to hive 2.3.2
> update {{sql/hive-thriftserver/src/main/java/org/apache/hive/service/*}} to hive 2.3.2
> * TestSuite should update:
> ||TestSuite||Reson||
> |StatisticsSuite|HIVE-16098|
> |SessionCatalogSuite|[VersionsSuite.scala#L427|#L427]|
> |CliSuite, HiveThriftServer2Suites, HiveSparkSubmitSuite, HiveQuerySuite, SQLQuerySuite|Update hive-hcatalog-core-0.13.1.jar to hive-hcatalog-core-2.3.2.jar|
> |SparkExecuteStatementOperationSuite|Interface changed from org.apache.hive.service.cli.Type.NULL_TYPE to org.apache.hadoop.hive.serde2.thrift.Type.NULL_TYPE|
> |ClasspathDependenciesSuite|org.apache.hive.com.esotericsoftware.kryo.Kryo change to com.esotericsoftware.kryo.Kryo|
> |HiveMetastoreCatalogSuite|Result format changed from Seq("1.1\t1", "2.1\t2") to Seq("1.100\t1", "2.100\t2")|
> |HiveOrcFilterSuite|Result format changed|
> |HiveDDLSuite|Remove $ (This change needs to be reconsidered)|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org