You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/09/28 17:29:30 UTC
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc should not...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/19382
[SPARK-22158][SQL] convertMetastoreOrc should not ignore table property
## What changes were proposed in this pull request?
From the beginning, convertMetastoreOrc ignores table properties and use an empty map instead. This PR fixes that.
```scala
val options = Map[String, String]()
```
- [SPARK-14070: HiveMetastoreCatalog.scala](https://github.com/apache/spark/pull/11891/files#diff-ee66e11b56c21364760a5ed2b783f863R650)
- [Master branch: HiveStrategies.scala](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala#L197
)
## How was this patch tested?
Pass the Jenkins with an updated test suite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-22158
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19382.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19382
----
commit 09c621bae172562b9dba0da91a527465d0ad7702
Author: Dongjoon Hyun <do...@apache.org>
Date: 2017-09-28T17:23:52Z
[SPARK-22158][SQL] convertMetastoreOrc should not ignore table properties
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc should not ignore...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82283/testReport)** for PR 19382 at commit [`09c621b`](https://github.com/apache/spark/commit/09c621bae172562b9dba0da91a527465d0ad7702).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141953332
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1437,30 +1439,75 @@ class HiveDDLSuite
}
}
- test("create hive serde table with new syntax") {
+ test("create hive serde table with new syntax - orc") {
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
+ |LOCATION '${path.toURI}'
+ """.stripMargin)
+ val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
+ assert(DDLUtils.isHiveTable(table))
+ assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
+ assert(table.storage.properties.get("compression") == Some("Zlib"))
+ assert(spark.table("t").collect().isEmpty)
+
+ sql("INSERT INTO t SELECT 1")
+ checkAnswer(spark.table("t"), Row(1))
+ // Check if this is compressed as ZLIB.
+ val maybeOrcFile = path.listFiles().find(_.getName.startsWith("part"))
+ assert(maybeOrcFile.isDefined)
+ val orcFilePath = maybeOrcFile.get.toPath.toString
+ val expectedCompressionKind =
+ OrcFileOperator.getFileReader(orcFilePath).get.getCompression
+ assert("ZLIB" === expectedCompressionKind.name())
+
+ sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
+ val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(DDLUtils.isHiveTable(table2))
+ assert(
+ table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
+ checkAnswer(spark.table("t2"), Row(1, "a"))
+
+ sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
+ sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
+ checkAnswer(spark.table("t3"), Row(0, 1))
+ }
+ }
+ }
+ }
+ }
+
+ test("create hive serde table with new syntax - parquet") {
withTable("t", "t2", "t3") {
withTempPath { path =>
sql(
s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'parquet', compression 'gzip')
--- End diff --
Try your best to reduce duplicate codes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19382
Please submit a separate PR to 2.2. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82324/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19382
Retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19382
@gatorsmile .
Whiling testing, I noticed that Parquet ignores table properties in case of `convertMetastoreParquet=false`, too. For that case, we need to proceed in another JIRA issue. #19218 may cover that, too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc should not ignore...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82283/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141939086
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1437,30 +1439,75 @@ class HiveDDLSuite
}
}
- test("create hive serde table with new syntax") {
+ test("create hive serde table with new syntax - orc") {
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
+ |LOCATION '${path.toURI}'
+ """.stripMargin)
+ val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
+ assert(DDLUtils.isHiveTable(table))
+ assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
+ assert(table.storage.properties.get("compression") == Some("Zlib"))
+ assert(spark.table("t").collect().isEmpty)
+
+ sql("INSERT INTO t SELECT 1")
+ checkAnswer(spark.table("t"), Row(1))
+ // Check if this is compressed as ZLIB.
+ val maybeOrcFile = path.listFiles().find(_.getName.startsWith("part"))
+ assert(maybeOrcFile.isDefined)
+ val orcFilePath = maybeOrcFile.get.toPath.toString
+ val expectedCompressionKind =
+ OrcFileOperator.getFileReader(orcFilePath).get.getCompression
+ assert("ZLIB" === expectedCompressionKind.name())
+
+ sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
+ val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(DDLUtils.isHiveTable(table2))
+ assert(
+ table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
+ checkAnswer(spark.table("t2"), Row(1, "a"))
+
+ sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
+ sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
+ checkAnswer(spark.table("t3"), Row(0, 1))
+ }
+ }
+ }
+ }
+ }
+
+ test("create hive serde table with new syntax - parquet") {
withTable("t", "t2", "t3") {
withTempPath { path =>
sql(
s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'parquet', compression 'gzip')
--- End diff --
Unfortunately, it's not feasible. First of all,
- Old ORC test case covers `convertMetastoreOrc=false` only. So, I extended it to cover both.
- New Parquet test case covers `convertMetastoreParquet=true (default) only because Parquet has another bug when convertMetastoreParquet=false.
Also, the compression codec check is quite different between ORC and Parquet. Please see line 1510.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141953271
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1437,30 +1439,75 @@ class HiveDDLSuite
}
}
- test("create hive serde table with new syntax") {
+ test("create hive serde table with new syntax - orc") {
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
+ |LOCATION '${path.toURI}'
+ """.stripMargin)
+ val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
+ assert(DDLUtils.isHiveTable(table))
+ assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
+ assert(table.storage.properties.get("compression") == Some("Zlib"))
+ assert(spark.table("t").collect().isEmpty)
+
+ sql("INSERT INTO t SELECT 1")
+ checkAnswer(spark.table("t"), Row(1))
+ // Check if this is compressed as ZLIB.
+ val maybeOrcFile = path.listFiles().find(_.getName.startsWith("part"))
+ assert(maybeOrcFile.isDefined)
+ val orcFilePath = maybeOrcFile.get.toPath.toString
+ val expectedCompressionKind =
+ OrcFileOperator.getFileReader(orcFilePath).get.getCompression
+ assert("ZLIB" === expectedCompressionKind.name())
+
+ sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
+ val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(DDLUtils.isHiveTable(table2))
+ assert(
+ table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
+ checkAnswer(spark.table("t2"), Row(1, "a"))
+
+ sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
+ sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
+ checkAnswer(spark.table("t3"), Row(0, 1))
+ }
+ }
+ }
+ }
+ }
+
+ test("create hive serde table with new syntax - parquet") {
withTable("t", "t2", "t3") {
withTempPath { path =>
sql(
s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'parquet', compression 'gzip')
--- End diff --
Still can create a common helper function, right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82327 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82327/testReport)** for PR 19382 at commit [`d218c98`](https://github.com/apache/spark/commit/d218c98d7af75ffc13e9900a9370885f83f55dc6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastore should not ig...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19382
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82391/testReport)** for PR 19382 at commit [`452a003`](https://github.com/apache/spark/commit/452a00376e0cf6e743da18f80cc44900390df948).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141920280
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1437,30 +1439,75 @@ class HiveDDLSuite
}
}
- test("create hive serde table with new syntax") {
+ test("create hive serde table with new syntax - orc") {
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
+ |LOCATION '${path.toURI}'
+ """.stripMargin)
+ val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
+ assert(DDLUtils.isHiveTable(table))
+ assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
+ assert(table.storage.properties.get("compression") == Some("Zlib"))
+ assert(spark.table("t").collect().isEmpty)
+
+ sql("INSERT INTO t SELECT 1")
+ checkAnswer(spark.table("t"), Row(1))
+ // Check if this is compressed as ZLIB.
+ val maybeOrcFile = path.listFiles().find(_.getName.startsWith("part"))
+ assert(maybeOrcFile.isDefined)
+ val orcFilePath = maybeOrcFile.get.toPath.toString
+ val expectedCompressionKind =
+ OrcFileOperator.getFileReader(orcFilePath).get.getCompression
+ assert("ZLIB" === expectedCompressionKind.name())
+
+ sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
+ val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(DDLUtils.isHiveTable(table2))
+ assert(
+ table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
+ checkAnswer(spark.table("t2"), Row(1, "a"))
+
+ sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
+ sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
+ checkAnswer(spark.table("t3"), Row(0, 1))
+ }
+ }
+ }
+ }
+ }
+
+ test("create hive serde table with new syntax - parquet") {
withTable("t", "t2", "t3") {
withTempPath { path =>
sql(
s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'parquet', compression 'gzip')
+ |LOCATION '${path.toURI}'
+ """.stripMargin)
val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
assert(DDLUtils.isHiveTable(table))
- assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
- assert(table.storage.properties.get("compression") == Some("Zlib"))
+ assert(table.storage.serde ==
+ Some("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"))
+ assert(table.storage.properties.get("compression") == Some("gzip"))
assert(spark.table("t").collect().isEmpty)
sql("INSERT INTO t SELECT 1")
checkAnswer(spark.table("t"), Row(1))
- // Check if this is compressed as ZLIB.
- val maybeOrcFile = path.listFiles().find(!_.getName.endsWith(".crc"))
- assert(maybeOrcFile.isDefined)
- val orcFilePath = maybeOrcFile.get.toPath.toString
- val expectedCompressionKind =
- OrcFileOperator.getFileReader(orcFilePath).get.getCompression
- assert("ZLIB" === expectedCompressionKind.name())
+ val maybeParquetFile = path.listFiles().find(f => f.getName.startsWith("part"))
+ assert(maybeParquetFile.isDefined)
+
+ val footer = ParquetFileReader.readFooter(
+ sparkContext.hadoopConfiguration,
+ new Path(maybeParquetFile.get.getPath),
+ NO_FILTER)
+ assert("GZIP" === footer.getBlocks.get(0).getColumns().get(0).getCodec.toString)
--- End diff --
It's for getting compression codec.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82327/testReport)** for PR 19382 at commit [`d218c98`](https://github.com/apache/spark/commit/d218c98d7af75ffc13e9900a9370885f83f55dc6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19382
Retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc should not ignore...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82283/testReport)** for PR 19382 at commit [`09c621b`](https://github.com/apache/spark/commit/09c621bae172562b9dba0da91a527465d0ad7702).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r142184730
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1437,30 +1439,75 @@ class HiveDDLSuite
}
}
- test("create hive serde table with new syntax") {
+ test("create hive serde table with new syntax - orc") {
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
+ |LOCATION '${path.toURI}'
+ """.stripMargin)
+ val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
+ assert(DDLUtils.isHiveTable(table))
+ assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
+ assert(table.storage.properties.get("compression") == Some("Zlib"))
+ assert(spark.table("t").collect().isEmpty)
+
+ sql("INSERT INTO t SELECT 1")
+ checkAnswer(spark.table("t"), Row(1))
+ // Check if this is compressed as ZLIB.
+ val maybeOrcFile = path.listFiles().find(_.getName.startsWith("part"))
+ assert(maybeOrcFile.isDefined)
+ val orcFilePath = maybeOrcFile.get.toPath.toString
+ val expectedCompressionKind =
+ OrcFileOperator.getFileReader(orcFilePath).get.getCompression
+ assert("ZLIB" === expectedCompressionKind.name())
+
+ sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
+ val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(DDLUtils.isHiveTable(table2))
+ assert(
+ table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
+ checkAnswer(spark.table("t2"), Row(1, "a"))
+
+ sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
+ sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
+ checkAnswer(spark.table("t3"), Row(0, 1))
+ }
+ }
+ }
+ }
+ }
+
+ test("create hive serde table with new syntax - parquet") {
withTable("t", "t2", "t3") {
withTempPath { path =>
sql(
s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'parquet', compression 'gzip')
--- End diff --
Sure!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19382
I retriggered because I noticed that master branch is fixed after the previous triggering.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc should not...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141908151
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1438,39 +1438,44 @@ class HiveDDLSuite
}
test("create hive serde table with new syntax") {
- withTable("t", "t2", "t3") {
- withTempPath { path =>
- sql(
- s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
- val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
- assert(DDLUtils.isHiveTable(table))
- assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
- assert(table.storage.properties.get("compression") == Some("Zlib"))
- assert(spark.table("t").collect().isEmpty)
-
- sql("INSERT INTO t SELECT 1")
- checkAnswer(spark.table("t"), Row(1))
- // Check if this is compressed as ZLIB.
- val maybeOrcFile = path.listFiles().find(!_.getName.endsWith(".crc"))
- assert(maybeOrcFile.isDefined)
- val orcFilePath = maybeOrcFile.get.toPath.toString
- val expectedCompressionKind =
- OrcFileOperator.getFileReader(orcFilePath).get.getCompression
- assert("ZLIB" === expectedCompressionKind.name())
-
- sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
- val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
- assert(DDLUtils.isHiveTable(table2))
- assert(table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
- checkAnswer(spark.table("t2"), Row(1, "a"))
-
- sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
- sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
- checkAnswer(spark.table("t3"), Row(0, 1))
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
--- End diff --
Thank you for review, @gatorsmile . I'll check that, too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141919885
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1437,30 +1439,75 @@ class HiveDDLSuite
}
}
- test("create hive serde table with new syntax") {
+ test("create hive serde table with new syntax - orc") {
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
+ |LOCATION '${path.toURI}'
+ """.stripMargin)
+ val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
+ assert(DDLUtils.isHiveTable(table))
+ assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
+ assert(table.storage.properties.get("compression") == Some("Zlib"))
+ assert(spark.table("t").collect().isEmpty)
+
+ sql("INSERT INTO t SELECT 1")
+ checkAnswer(spark.table("t"), Row(1))
+ // Check if this is compressed as ZLIB.
+ val maybeOrcFile = path.listFiles().find(_.getName.startsWith("part"))
+ assert(maybeOrcFile.isDefined)
+ val orcFilePath = maybeOrcFile.get.toPath.toString
+ val expectedCompressionKind =
+ OrcFileOperator.getFileReader(orcFilePath).get.getCompression
+ assert("ZLIB" === expectedCompressionKind.name())
+
+ sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
+ val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(DDLUtils.isHiveTable(table2))
+ assert(
+ table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
+ checkAnswer(spark.table("t2"), Row(1, "a"))
+
+ sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
+ sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
+ checkAnswer(spark.table("t3"), Row(0, 1))
+ }
+ }
+ }
+ }
+ }
+
+ test("create hive serde table with new syntax - parquet") {
withTable("t", "t2", "t3") {
withTempPath { path =>
sql(
s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'parquet', compression 'gzip')
--- End diff --
parquet use `gzip`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82395/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82391/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82394/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82395/testReport)** for PR 19382 at commit [`17ccf9a`](https://github.com/apache/spark/commit/17ccf9aff9953af421b48262f8fd6af582bddf0a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82391/testReport)** for PR 19382 at commit [`452a003`](https://github.com/apache/spark/commit/452a00376e0cf6e743da18f80cc44900390df948).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r142217964
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -2009,4 +2012,48 @@ class HiveDDLSuite
}
}
}
+
+ private def assertCompression(maybeFile: Option[File], format: String, compression: String) = {
+ assert(maybeFile.isDefined)
+
+ val actualCompression = format match {
+ case "orc" =>
+ OrcFileOperator.getFileReader(maybeFile.get.toPath.toString).get.getCompression.name
+
+ case "parquet" =>
+ val footer = ParquetFileReader.readFooter(
+ sparkContext.hadoopConfiguration, new Path(maybeFile.get.getPath), NO_FILTER)
+ footer.getBlocks.get(0).getColumns.get(0).getCodec.toString
+ }
+
+ assert(compression === actualCompression)
+ }
+
+ // Since ORC uses 'ZLIB' and Parquet uses 'SNAPPY' by default, we test with different formats.
+ Seq(("orc", "SNAPPY"), ("parquet", "GZIP")).foreach { case (fileFormat, compression) =>
+ test(s"SPARK-22158 convertMetastore should not ignore table property - $fileFormat") {
--- End diff --
Hi, @gatorsmile .
Now, this test is focusing solely on `convertMetastore should not ignore table property` case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19382
Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82332/testReport)** for PR 19382 at commit [`d218c98`](https://github.com/apache/spark/commit/d218c98d7af75ffc13e9900a9370885f83f55dc6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82332/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc should not...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141801811
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1438,39 +1438,44 @@ class HiveDDLSuite
}
test("create hive serde table with new syntax") {
- withTable("t", "t2", "t3") {
- withTempPath { path =>
- sql(
- s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
- val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
- assert(DDLUtils.isHiveTable(table))
- assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
- assert(table.storage.properties.get("compression") == Some("Zlib"))
- assert(spark.table("t").collect().isEmpty)
-
- sql("INSERT INTO t SELECT 1")
- checkAnswer(spark.table("t"), Row(1))
- // Check if this is compressed as ZLIB.
- val maybeOrcFile = path.listFiles().find(!_.getName.endsWith(".crc"))
- assert(maybeOrcFile.isDefined)
- val orcFilePath = maybeOrcFile.get.toPath.toString
- val expectedCompressionKind =
- OrcFileOperator.getFileReader(orcFilePath).get.getCompression
- assert("ZLIB" === expectedCompressionKind.name())
-
- sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
- val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
- assert(DDLUtils.isHiveTable(table2))
- assert(table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
- checkAnswer(spark.table("t2"), Row(1, "a"))
-
- sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
- sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
- checkAnswer(spark.table("t3"), Row(0, 1))
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
--- End diff --
It sounds like we have the same issue for parquet?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82332/testReport)** for PR 19382 at commit [`d218c98`](https://github.com/apache/spark/commit/d218c98d7af75ffc13e9900a9370885f83f55dc6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82327/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82324/testReport)** for PR 19382 at commit [`d218c98`](https://github.com/apache/spark/commit/d218c98d7af75ffc13e9900a9370885f83f55dc6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19382
Thanks! Merged to master/2.2
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19382
It fails due to `flume dependency`.
```
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.spark#spark-streaming-flume_2.11;2.3.0-SNAPSHOT: not found
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141938103
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -1437,30 +1439,75 @@ class HiveDDLSuite
}
}
- test("create hive serde table with new syntax") {
+ test("create hive serde table with new syntax - orc") {
+ Seq("true", "false").foreach { value =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
+ withTable("t", "t2", "t3") {
+ withTempPath { path =>
+ sql(
+ s"""
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'orc', compression 'Zlib')
+ |LOCATION '${path.toURI}'
+ """.stripMargin)
+ val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
+ assert(DDLUtils.isHiveTable(table))
+ assert(table.storage.serde == Some("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))
+ assert(table.storage.properties.get("compression") == Some("Zlib"))
+ assert(spark.table("t").collect().isEmpty)
+
+ sql("INSERT INTO t SELECT 1")
+ checkAnswer(spark.table("t"), Row(1))
+ // Check if this is compressed as ZLIB.
+ val maybeOrcFile = path.listFiles().find(_.getName.startsWith("part"))
+ assert(maybeOrcFile.isDefined)
+ val orcFilePath = maybeOrcFile.get.toPath.toString
+ val expectedCompressionKind =
+ OrcFileOperator.getFileReader(orcFilePath).get.getCompression
+ assert("ZLIB" === expectedCompressionKind.name())
+
+ sql("CREATE TABLE t2 USING HIVE AS SELECT 1 AS c1, 'a' AS c2")
+ val table2 = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(DDLUtils.isHiveTable(table2))
+ assert(
+ table2.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
+ checkAnswer(spark.table("t2"), Row(1, "a"))
+
+ sql("CREATE TABLE t3(a int, p int) USING hive PARTITIONED BY (p)")
+ sql("INSERT INTO t3 PARTITION(p=1) SELECT 0")
+ checkAnswer(spark.table("t3"), Row(0, 1))
+ }
+ }
+ }
+ }
+ }
+
+ test("create hive serde table with new syntax - parquet") {
withTable("t", "t2", "t3") {
withTempPath { path =>
sql(
s"""
- |CREATE TABLE t(id int) USING hive
- |OPTIONS(fileFormat 'orc', compression 'Zlib')
- |LOCATION '${path.toURI}'
- """.stripMargin)
+ |CREATE TABLE t(id int) USING hive
+ |OPTIONS(fileFormat 'parquet', compression 'gzip')
--- End diff --
Could you combine these two test cases?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet sh...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19382#discussion_r141919805
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala ---
@@ -189,12 +189,12 @@ case class RelationConversions(
private def convert(relation: HiveTableRelation): LogicalRelation = {
val serde = relation.tableMeta.storage.serde.getOrElse("").toLowerCase(Locale.ROOT)
if (serde.contains("parquet")) {
- val options = Map(ParquetOptions.MERGE_SCHEMA ->
+ val options = relation.tableMeta.storage.properties + (ParquetOptions.MERGE_SCHEMA ->
--- End diff --
Hi, @gatorsmile . It's the same in the Paquet. Without this, the test case for parquet fails.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82395/testReport)** for PR 19382 at commit [`17ccf9a`](https://github.com/apache/spark/commit/17ccf9aff9953af421b48262f8fd6af582bddf0a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc/Parquet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19382
**[Test build #82324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82324/testReport)** for PR 19382 at commit [`d218c98`](https://github.com/apache/spark/commit/d218c98d7af75ffc13e9900a9370885f83f55dc6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastoreOrc should not ignore...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19382
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19382: [SPARK-22158][SQL] convertMetastore should not ignore ta...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19382
Thank you, @gatorsmile . I'll.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org