You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2017/01/18 19:17:48 UTC
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Block Creating a Hive Table W...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/16636
[SPARK-19279] [SQL] Block Creating a Hive Table With an Empty Schema
### What changes were proposed in this pull request?
So far, we allow users to create a table with an empty schema: `CREATE TABLE tab1`. This could break many code paths if we enable it. Thus, we should follow Hive to block it.
### How was this patch tested?
Added test cases for both Hive and data source tables
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark fixEmptyTableSchema
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16636.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16636
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16636
: ) Done. Found a solution to infer the schema of Hive Serde tables. Let me clean the code now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #71796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71796/testReport)** for PR 16636 at commit [`e3cc423`](https://github.com/apache/spark/commit/e3cc423e2ecf3e9128b8036905c044a3f658cd25).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99458936
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -1527,6 +1527,21 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
}
}
+ test("create a data source table without schema") {
+ import testImplicits._
+ withTempPath { tempDir =>
+ withTable("tab1", "tab2") {
+ (("a", "b") :: Nil).toDF().write.json(tempDir.getCanonicalPath)
+
+ val e = intercept[AnalysisException] { sql("CREATE TABLE tab1 USING json") }.getMessage
--- End diff --
Test case for LibSVM is added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99472089
--- Diff: mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala ---
@@ -110,4 +110,22 @@ class LibSVMRelationSuite extends SparkFunSuite with MLlibTestSparkContext {
df.select("features").rdd.map { case Row(d: Vector) => d }.first
df.select("features").collect
}
+
+ test("create libsvmTable table without schema") {
+ try {
+ spark.sql(
+ s"""
+ |CREATE TABLE libsvmTable
+ |USING libsvm
+ |OPTIONS (
+ | path '$path'
--- End diff --
It will output an error message `No input path specified for libsvm data`. Added a test case for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #71612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71612/testReport)** for PR 16636 at commit [`a8281b1`](https://github.com/apache/spark/commit/a8281b15676eaae43eed58f8dbb288d15e13294d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Block Creating a Hive Table W...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r98315149
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -1527,6 +1527,21 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
}
}
+ test("create a data source table without schema") {
+ import testImplicits._
+ withTempPath { tempDir =>
+ withTable("tab1", "tab2") {
+ (("a", "b") :: Nil).toDF().write.json(tempDir.getCanonicalPath)
+
+ val e = intercept[AnalysisException] { sql("CREATE TABLE tab1 USING json") }.getMessage
--- End diff --
This error message is not from the code added by this PR. It is from [the original logics](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L343-L345). The error message is right if our file-based data sources are unable to infer the schema.
Sure, I will add a test case for `LibSVM `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72101/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99647218
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -79,6 +79,25 @@ class HiveDDLSuite
}
}
+ test("create a hive table without schema") {
+ import testImplicits._
+ withTempPath { tempDir =>
+ withTable("tab1", "tab2") {
+ (("a", "b") :: Nil).toDF().write.json(tempDir.getCanonicalPath)
+
+ var e = intercept[AnalysisException] { sql("CREATE TABLE tab1 USING hive") }.getMessage
+ assert(e.contains("Unable to infer the schema. The schema specification is required to " +
+ "create the table `default`.`tab1`"))
+
+ e = intercept[AnalysisException] {
+ sql(s"CREATE TABLE tab2 location '${tempDir.getCanonicalPath}'")
--- End diff --
Found an interesting case. I am thinking we need to support what Hive did in `CREATE TABLE LIKE`. That is, allowing users to create another table with a different data source provider:
```
CREATE TABLE avro_test ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS AVRO TBLPROPERTIES ('avro.schema.url'='myHost/myAvroSchema.avsc');
CREATE EXTERNAL TABLE parquet_test LIKE avro_test STORED AS PARQUET LOCATION 'hdfs://myParquetFilesPath';
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99464898
--- Diff: mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala ---
@@ -110,4 +110,22 @@ class LibSVMRelationSuite extends SparkFunSuite with MLlibTestSparkContext {
df.select("features").rdd.map { case Row(d: Vector) => d }.first
df.select("features").collect
}
+
+ test("create libsvmTable table without schema") {
+ try {
+ spark.sql(
+ s"""
+ |CREATE TABLE libsvmTable
+ |USING libsvm
+ |OPTIONS (
+ | path '$path'
--- End diff --
what if no path specified?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #72101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72101/testReport)** for PR 16636 at commit [`c5cfa1a`](https://github.com/apache/spark/commit/c5cfa1afe3896ab92b34b833ba6c18cfce88b224).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #72358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72358/testReport)** for PR 16636 at commit [`d87676f`](https://github.com/apache/spark/commit/d87676f0a71edb93b06bdb3107be3cac519b1e81).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99645731
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -79,6 +79,25 @@ class HiveDDLSuite
}
}
+ test("create a hive table without schema") {
+ import testImplicits._
+ withTempPath { tempDir =>
+ withTable("tab1", "tab2") {
+ (("a", "b") :: Nil).toDF().write.json(tempDir.getCanonicalPath)
+
+ var e = intercept[AnalysisException] { sql("CREATE TABLE tab1 USING hive") }.getMessage
+ assert(e.contains("Unable to infer the schema. The schema specification is required to " +
+ "create the table `default`.`tab1`"))
+
+ e = intercept[AnalysisException] {
+ sql(s"CREATE TABLE tab2 location '${tempDir.getCanonicalPath}'")
--- End diff --
Based on the official Hive document about [Row Formats & SerDe](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormats&SerDe), users must specify the columns of the schema for the native SerDe. For the custom SerDe, Hive will query the SerDe to get the columns in the schema. It does not sounds Hive has such a capability to infer the schema from the data files.
Actually, there is two unresolved tickets for it. https://issues.apache.org/jira/browse/HIVE-8950 and https://issues.apache.org/jira/browse/HIVE-10593. It sounds like the Hive community prefers to adding a new DDL syntax to get the table schema from a Parquet file (or some other format), but it is not available in Hive yet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #72101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72101/testReport)** for PR 16636 at commit [`c5cfa1a`](https://github.com/apache/spark/commit/c5cfa1afe3896ab92b34b833ba6c18cfce88b224).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #71795 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71795/testReport)** for PR 16636 at commit [`f99e078`](https://github.com/apache/spark/commit/f99e078dd677798c8d9674ea5e08e9a95b43c065).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #71802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71802/testReport)** for PR 16636 at commit [`e3cc423`](https://github.com/apache/spark/commit/e3cc423e2ecf3e9128b8036905c044a3f658cd25).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #72387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72387/testReport)** for PR 16636 at commit [`f7e0264`](https://github.com/apache/spark/commit/f7e0264c6786fb69d313406beba066300a63f3b1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71796/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Block Creating a Hive Table W...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r98323927
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -455,4 +462,133 @@ private[spark] object HiveUtils extends Logging {
case (decimal, DecimalType()) => decimal.toString
case (other, tpe) if primitiveTypes contains tpe => other.toString
}
+
+ /** Converts the native StructField to Hive's FieldSchema. */
+ private def toHiveColumn(c: StructField): FieldSchema = {
+ val typeString = if (c.metadata.contains(HiveUtils.hiveTypeString)) {
+ c.metadata.getString(HiveUtils.hiveTypeString)
+ } else {
+ c.dataType.catalogString
+ }
+ new FieldSchema(c.name, typeString, c.getComment.orNull)
+ }
+
+ /** Builds the native StructField from Hive's FieldSchema. */
+ private def fromHiveColumn(hc: FieldSchema): StructField = {
+ val columnType = try {
+ CatalystSqlParser.parseDataType(hc.getType)
+ } catch {
+ case e: ParseException =>
+ throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
+ }
+
+ val metadata = new MetadataBuilder().putString(HiveUtils.hiveTypeString, hc.getType).build()
+ val field = StructField(
+ name = hc.getName,
+ dataType = columnType,
+ nullable = true,
+ metadata = metadata)
+ Option(hc.getComment).map(field.withComment).getOrElse(field)
+ }
+
+ // TODO: merge this with HiveClientImpl#toHiveTable
+ /** Converts the native table metadata representation format CatalogTable to Hive's Table. */
+ def toHiveTable(catalogTable: CatalogTable): HiveTable = {
+ // We start by constructing an API table as Hive performs several important transformations
+ // internally when converting an API table to a QL table.
+ val tTable = new org.apache.hadoop.hive.metastore.api.Table()
+ tTable.setTableName(catalogTable.identifier.table)
+ tTable.setDbName(catalogTable.database)
+
+ val tableParameters = new java.util.HashMap[String, String]()
+ tTable.setParameters(tableParameters)
+ catalogTable.properties.foreach { case (k, v) => tableParameters.put(k, v) }
+
+ tTable.setTableType(catalogTable.tableType match {
+ case CatalogTableType.EXTERNAL => HiveTableType.EXTERNAL_TABLE.toString
+ case CatalogTableType.MANAGED => HiveTableType.MANAGED_TABLE.toString
+ case CatalogTableType.VIEW => HiveTableType.VIRTUAL_VIEW.toString
+ })
+
+ val sd = new org.apache.hadoop.hive.metastore.api.StorageDescriptor()
+ tTable.setSd(sd)
+
+ // Note: In Hive the schema and partition columns must be disjoint sets
+ val (partCols, schema) = catalogTable.schema.map(toHiveColumn).partition { c =>
+ catalogTable.partitionColumnNames.contains(c.getName)
+ }
+ sd.setCols(schema.asJava)
+ tTable.setPartitionKeys(partCols.asJava)
+
+ catalogTable.storage.locationUri.foreach(sd.setLocation)
+ catalogTable.storage.inputFormat.foreach(sd.setInputFormat)
+ catalogTable.storage.outputFormat.foreach(sd.setOutputFormat)
+
+ val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
+ catalogTable.storage.serde.foreach(serdeInfo.setSerializationLib)
+ sd.setSerdeInfo(serdeInfo)
+
+ val serdeParameters = new java.util.HashMap[String, String]()
+ catalogTable.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ serdeInfo.setParameters(serdeParameters)
+
+ new HiveTable(tTable)
+ }
+
+ /**
+ * Converts the native partition metadata representation format CatalogTablePartition to
+ * Hive's Partition.
+ */
+ def toHivePartition(
+ catalogTable: CatalogTable,
+ hiveTable: HiveTable,
+ partition: CatalogTablePartition): HivePartition = {
+ val tPartition = new org.apache.hadoop.hive.metastore.api.Partition
+ tPartition.setDbName(catalogTable.database)
+ tPartition.setTableName(catalogTable.identifier.table)
+ tPartition.setValues(catalogTable.partitionColumnNames.map(partition.spec(_)).asJava)
+
+ val sd = new org.apache.hadoop.hive.metastore.api.StorageDescriptor()
+ tPartition.setSd(sd)
+
+ // Note: In Hive the schema and partition columns must be disjoint sets
+ val schema = catalogTable.schema.map(toHiveColumn).filter { c =>
+ !catalogTable.partitionColumnNames.contains(c.getName)
+ }
+ sd.setCols(schema.asJava)
+
+ partition.storage.locationUri.foreach(sd.setLocation)
+ partition.storage.inputFormat.foreach(sd.setInputFormat)
+ partition.storage.outputFormat.foreach(sd.setOutputFormat)
+
+ val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
+ sd.setSerdeInfo(serdeInfo)
+ // maps and lists should be set only after all elements are ready (see HIVE-7975)
+ partition.storage.serde.foreach(serdeInfo.setSerializationLib)
+
+ val serdeParameters = new java.util.HashMap[String, String]()
+ catalogTable.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ partition.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ serdeInfo.setParameters(serdeParameters)
+
+ new HivePartition(hiveTable, tPartition)
+ }
--- End diff --
Basically, the above two functions `toHivePartition`, `toHiveTable` and `toHiveColumn ` are moved from `MetastoreRelation.scala`.
`inferSchema ` and `fromHiveColumn ` are newly added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16636
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #3546 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3546/testReport)** for PR 16636 at commit [`e3cc423`](https://github.com/apache/spark/commit/e3cc423e2ecf3e9128b8036905c044a3f658cd25).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99326535
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -455,4 +462,133 @@ private[spark] object HiveUtils extends Logging {
case (decimal, DecimalType()) => decimal.toString
case (other, tpe) if primitiveTypes contains tpe => other.toString
}
+
+ /** Converts the native StructField to Hive's FieldSchema. */
+ private def toHiveColumn(c: StructField): FieldSchema = {
+ val typeString = if (c.metadata.contains(HiveUtils.hiveTypeString)) {
+ c.metadata.getString(HiveUtils.hiveTypeString)
+ } else {
+ c.dataType.catalogString
+ }
+ new FieldSchema(c.name, typeString, c.getComment.orNull)
+ }
+
+ /** Builds the native StructField from Hive's FieldSchema. */
+ private def fromHiveColumn(hc: FieldSchema): StructField = {
+ val columnType = try {
+ CatalystSqlParser.parseDataType(hc.getType)
+ } catch {
+ case e: ParseException =>
+ throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
+ }
+
+ val metadata = new MetadataBuilder().putString(HiveUtils.hiveTypeString, hc.getType).build()
+ val field = StructField(
+ name = hc.getName,
+ dataType = columnType,
+ nullable = true,
+ metadata = metadata)
+ Option(hc.getComment).map(field.withComment).getOrElse(field)
+ }
+
+ // TODO: merge this with HiveClientImpl#toHiveTable
+ /** Converts the native table metadata representation format CatalogTable to Hive's Table. */
+ def toHiveTable(catalogTable: CatalogTable): HiveTable = {
+ // We start by constructing an API table as Hive performs several important transformations
+ // internally when converting an API table to a QL table.
+ val tTable = new org.apache.hadoop.hive.metastore.api.Table()
+ tTable.setTableName(catalogTable.identifier.table)
+ tTable.setDbName(catalogTable.database)
+
+ val tableParameters = new java.util.HashMap[String, String]()
+ tTable.setParameters(tableParameters)
+ catalogTable.properties.foreach { case (k, v) => tableParameters.put(k, v) }
+
+ tTable.setTableType(catalogTable.tableType match {
+ case CatalogTableType.EXTERNAL => HiveTableType.EXTERNAL_TABLE.toString
+ case CatalogTableType.MANAGED => HiveTableType.MANAGED_TABLE.toString
+ case CatalogTableType.VIEW => HiveTableType.VIRTUAL_VIEW.toString
+ })
+
+ val sd = new org.apache.hadoop.hive.metastore.api.StorageDescriptor()
+ tTable.setSd(sd)
+
+ // Note: In Hive the schema and partition columns must be disjoint sets
+ val (partCols, schema) = catalogTable.schema.map(toHiveColumn).partition { c =>
+ catalogTable.partitionColumnNames.contains(c.getName)
+ }
+ sd.setCols(schema.asJava)
+ tTable.setPartitionKeys(partCols.asJava)
+
+ catalogTable.storage.locationUri.foreach(sd.setLocation)
+ catalogTable.storage.inputFormat.foreach(sd.setInputFormat)
+ catalogTable.storage.outputFormat.foreach(sd.setOutputFormat)
+
+ val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
+ catalogTable.storage.serde.foreach(serdeInfo.setSerializationLib)
+ sd.setSerdeInfo(serdeInfo)
+
+ val serdeParameters = new java.util.HashMap[String, String]()
+ catalogTable.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ serdeInfo.setParameters(serdeParameters)
+
+ new HiveTable(tTable)
+ }
+
+ /**
+ * Converts the native partition metadata representation format CatalogTablePartition to
+ * Hive's Partition.
+ */
+ def toHivePartition(
+ catalogTable: CatalogTable,
+ hiveTable: HiveTable,
+ partition: CatalogTablePartition): HivePartition = {
+ val tPartition = new org.apache.hadoop.hive.metastore.api.Partition
+ tPartition.setDbName(catalogTable.database)
+ tPartition.setTableName(catalogTable.identifier.table)
+ tPartition.setValues(catalogTable.partitionColumnNames.map(partition.spec(_)).asJava)
+
+ val sd = new org.apache.hadoop.hive.metastore.api.StorageDescriptor()
+ tPartition.setSd(sd)
+
+ // Note: In Hive the schema and partition columns must be disjoint sets
+ val schema = catalogTable.schema.map(toHiveColumn).filter { c =>
+ !catalogTable.partitionColumnNames.contains(c.getName)
+ }
+ sd.setCols(schema.asJava)
+
+ partition.storage.locationUri.foreach(sd.setLocation)
+ partition.storage.inputFormat.foreach(sd.setInputFormat)
+ partition.storage.outputFormat.foreach(sd.setOutputFormat)
+
+ val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
+ sd.setSerdeInfo(serdeInfo)
+ // maps and lists should be set only after all elements are ready (see HIVE-7975)
+ partition.storage.serde.foreach(serdeInfo.setSerializationLib)
+
+ val serdeParameters = new java.util.HashMap[String, String]()
+ catalogTable.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ partition.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ serdeInfo.setParameters(serdeParameters)
+
+ new HivePartition(hiveTable, tPartition)
+ }
+
+ /**
+ * Infers the schema for Hive serde tables and returns the CatalogTable with the inferred schema.
+ * When the tables are data source tables or the schema already exists, returns the original
+ * CatalogTable.
+ */
+ def inferSchema(table: CatalogTable): CatalogTable = {
+ if (DDLUtils.isDatasourceTable(table) || table.schema.nonEmpty) {
+ table
+ } else {
+ val hiveTable = toHiveTable(table)
+ // Note: Hive separates partition columns and the schema, but for us the
+ // partition columns are part of the schema
+ val partCols = hiveTable.getPartCols.asScala.map(fromHiveColumn)
+ val schema = StructType(hiveTable.getCols.asScala.map(fromHiveColumn) ++ partCols)
+ table.copy(schema = schema, partitionColumnNames = partCols.map(_.name))
--- End diff --
will `partitionColumnNames` change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99458457
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -455,4 +462,133 @@ private[spark] object HiveUtils extends Logging {
case (decimal, DecimalType()) => decimal.toString
case (other, tpe) if primitiveTypes contains tpe => other.toString
}
+
+ /** Converts the native StructField to Hive's FieldSchema. */
+ private def toHiveColumn(c: StructField): FieldSchema = {
+ val typeString = if (c.metadata.contains(HiveUtils.hiveTypeString)) {
+ c.metadata.getString(HiveUtils.hiveTypeString)
+ } else {
+ c.dataType.catalogString
+ }
+ new FieldSchema(c.name, typeString, c.getComment.orNull)
+ }
+
+ /** Builds the native StructField from Hive's FieldSchema. */
+ private def fromHiveColumn(hc: FieldSchema): StructField = {
+ val columnType = try {
+ CatalystSqlParser.parseDataType(hc.getType)
+ } catch {
+ case e: ParseException =>
+ throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
+ }
+
+ val metadata = new MetadataBuilder().putString(HiveUtils.hiveTypeString, hc.getType).build()
+ val field = StructField(
+ name = hc.getName,
+ dataType = columnType,
+ nullable = true,
+ metadata = metadata)
+ Option(hc.getComment).map(field.withComment).getOrElse(field)
+ }
+
+ // TODO: merge this with HiveClientImpl#toHiveTable
+ /** Converts the native table metadata representation format CatalogTable to Hive's Table. */
+ def toHiveTable(catalogTable: CatalogTable): HiveTable = {
+ // We start by constructing an API table as Hive performs several important transformations
+ // internally when converting an API table to a QL table.
+ val tTable = new org.apache.hadoop.hive.metastore.api.Table()
+ tTable.setTableName(catalogTable.identifier.table)
+ tTable.setDbName(catalogTable.database)
+
+ val tableParameters = new java.util.HashMap[String, String]()
+ tTable.setParameters(tableParameters)
+ catalogTable.properties.foreach { case (k, v) => tableParameters.put(k, v) }
+
+ tTable.setTableType(catalogTable.tableType match {
+ case CatalogTableType.EXTERNAL => HiveTableType.EXTERNAL_TABLE.toString
+ case CatalogTableType.MANAGED => HiveTableType.MANAGED_TABLE.toString
+ case CatalogTableType.VIEW => HiveTableType.VIRTUAL_VIEW.toString
+ })
+
+ val sd = new org.apache.hadoop.hive.metastore.api.StorageDescriptor()
+ tTable.setSd(sd)
+
+ // Note: In Hive the schema and partition columns must be disjoint sets
+ val (partCols, schema) = catalogTable.schema.map(toHiveColumn).partition { c =>
+ catalogTable.partitionColumnNames.contains(c.getName)
+ }
+ sd.setCols(schema.asJava)
+ tTable.setPartitionKeys(partCols.asJava)
+
+ catalogTable.storage.locationUri.foreach(sd.setLocation)
+ catalogTable.storage.inputFormat.foreach(sd.setInputFormat)
+ catalogTable.storage.outputFormat.foreach(sd.setOutputFormat)
+
+ val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
+ catalogTable.storage.serde.foreach(serdeInfo.setSerializationLib)
+ sd.setSerdeInfo(serdeInfo)
+
+ val serdeParameters = new java.util.HashMap[String, String]()
+ catalogTable.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ serdeInfo.setParameters(serdeParameters)
+
+ new HiveTable(tTable)
+ }
+
+ /**
+ * Converts the native partition metadata representation format CatalogTablePartition to
+ * Hive's Partition.
+ */
+ def toHivePartition(
+ catalogTable: CatalogTable,
+ hiveTable: HiveTable,
+ partition: CatalogTablePartition): HivePartition = {
+ val tPartition = new org.apache.hadoop.hive.metastore.api.Partition
+ tPartition.setDbName(catalogTable.database)
+ tPartition.setTableName(catalogTable.identifier.table)
+ tPartition.setValues(catalogTable.partitionColumnNames.map(partition.spec(_)).asJava)
+
+ val sd = new org.apache.hadoop.hive.metastore.api.StorageDescriptor()
+ tPartition.setSd(sd)
+
+ // Note: In Hive the schema and partition columns must be disjoint sets
+ val schema = catalogTable.schema.map(toHiveColumn).filter { c =>
+ !catalogTable.partitionColumnNames.contains(c.getName)
+ }
+ sd.setCols(schema.asJava)
+
+ partition.storage.locationUri.foreach(sd.setLocation)
+ partition.storage.inputFormat.foreach(sd.setInputFormat)
+ partition.storage.outputFormat.foreach(sd.setOutputFormat)
+
+ val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
+ sd.setSerdeInfo(serdeInfo)
+ // maps and lists should be set only after all elements are ready (see HIVE-7975)
+ partition.storage.serde.foreach(serdeInfo.setSerializationLib)
+
+ val serdeParameters = new java.util.HashMap[String, String]()
+ catalogTable.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ partition.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ serdeInfo.setParameters(serdeParameters)
+
+ new HivePartition(hiveTable, tPartition)
+ }
+
+ /**
+ * Infers the schema for Hive serde tables and returns the CatalogTable with the inferred schema.
+ * When the tables are data source tables or the schema already exists, returns the original
+ * CatalogTable.
+ */
+ def inferSchema(table: CatalogTable): CatalogTable = {
+ if (DDLUtils.isDatasourceTable(table) || table.schema.nonEmpty) {
+ table
+ } else {
+ val hiveTable = toHiveTable(table)
+ // Note: Hive separates partition columns and the schema, but for us the
+ // partition columns are part of the schema
+ val partCols = hiveTable.getPartCols.asScala.map(fromHiveColumn)
+ val schema = StructType(hiveTable.getCols.asScala.map(fromHiveColumn) ++ partCols)
+ table.copy(schema = schema, partitionColumnNames = partCols.map(_.name))
--- End diff --
I think it should not be changed. Let me remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #72359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72359/testReport)** for PR 16636 at commit [`1836887`](https://github.com/apache/spark/commit/18368874e16702d641131cd7c2a80af4e98d09e0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class VersionsSuite extends QueryTest with SQLTestUtils with TestHiveSingleton with Logging `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16636
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99459298
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -455,4 +462,133 @@ private[spark] object HiveUtils extends Logging {
case (decimal, DecimalType()) => decimal.toString
case (other, tpe) if primitiveTypes contains tpe => other.toString
}
+
+ /** Converts the native StructField to Hive's FieldSchema. */
+ private def toHiveColumn(c: StructField): FieldSchema = {
+ val typeString = if (c.metadata.contains(HiveUtils.hiveTypeString)) {
+ c.metadata.getString(HiveUtils.hiveTypeString)
+ } else {
+ c.dataType.catalogString
+ }
+ new FieldSchema(c.name, typeString, c.getComment.orNull)
+ }
+
+ /** Builds the native StructField from Hive's FieldSchema. */
+ private def fromHiveColumn(hc: FieldSchema): StructField = {
+ val columnType = try {
+ CatalystSqlParser.parseDataType(hc.getType)
+ } catch {
+ case e: ParseException =>
+ throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
+ }
+
+ val metadata = new MetadataBuilder().putString(HiveUtils.hiveTypeString, hc.getType).build()
+ val field = StructField(
+ name = hc.getName,
+ dataType = columnType,
+ nullable = true,
+ metadata = metadata)
+ Option(hc.getComment).map(field.withComment).getOrElse(field)
+ }
+
+ // TODO: merge this with HiveClientImpl#toHiveTable
--- End diff --
I moved a test case `SPARK-13709: reading partitioned Avro table with nested schema` to `VersionSuite.scala`. This can help us to know the above statement holds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #71795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71795/testReport)** for PR 16636 at commit [`f99e078`](https://github.com/apache/spark/commit/f99e078dd677798c8d9674ea5e08e9a95b43c065).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99519100
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -79,6 +79,25 @@ class HiveDDLSuite
}
}
+ test("create a hive table without schema") {
+ import testImplicits._
+ withTempPath { tempDir =>
+ withTable("tab1", "tab2") {
+ (("a", "b") :: Nil).toDF().write.json(tempDir.getCanonicalPath)
+
+ var e = intercept[AnalysisException] { sql("CREATE TABLE tab1 USING hive") }.getMessage
+ assert(e.contains("Unable to infer the schema. The schema specification is required to " +
+ "create the table `default`.`tab1`"))
+
+ e = intercept[AnalysisException] {
+ sql(s"CREATE TABLE tab2 location '${tempDir.getCanonicalPath}'")
--- End diff --
is there any hive serde that can infer schema from data files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16636
Ideally the table schema must be specified or inferred before saving to metastore, however, for hive serde tables, we have to save it to metastore first, and let the hive metastore to infer the schema. Is it possible we can extract the schema inference logic from hive metastore? so that we can make data source tables and hive serde tables more consistent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71802/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #72387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72387/testReport)** for PR 16636 at commit [`f7e0264`](https://github.com/apache/spark/commit/f7e0264c6786fb69d313406beba066300a63f3b1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #3546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3546/testReport)** for PR 16636 at commit [`e3cc423`](https://github.com/apache/spark/commit/e3cc423e2ecf3e9128b8036905c044a3f658cd25).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Block Creating a Hive Table W...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r97216875
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -129,6 +130,21 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
}
}
+ /**
+ * For Hive serde tables, some serde libraries require the specified schema and record it
+ * in the metastore. The main code is from the function hasMetastoreBasedSchema in Table.java
--- End diff --
To the reviewers: `hasMetastoreBasedSchema` in `Table.java` can be accessed via the link: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L943-L946
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r98324289
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -455,4 +462,133 @@ private[spark] object HiveUtils extends Logging {
case (decimal, DecimalType()) => decimal.toString
case (other, tpe) if primitiveTypes contains tpe => other.toString
}
+
+ /** Converts the native StructField to Hive's FieldSchema. */
+ private def toHiveColumn(c: StructField): FieldSchema = {
+ val typeString = if (c.metadata.contains(HiveUtils.hiveTypeString)) {
+ c.metadata.getString(HiveUtils.hiveTypeString)
+ } else {
+ c.dataType.catalogString
+ }
+ new FieldSchema(c.name, typeString, c.getComment.orNull)
+ }
+
+ /** Builds the native StructField from Hive's FieldSchema. */
+ private def fromHiveColumn(hc: FieldSchema): StructField = {
+ val columnType = try {
+ CatalystSqlParser.parseDataType(hc.getType)
+ } catch {
+ case e: ParseException =>
+ throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
+ }
+
+ val metadata = new MetadataBuilder().putString(HiveUtils.hiveTypeString, hc.getType).build()
+ val field = StructField(
+ name = hc.getName,
+ dataType = columnType,
+ nullable = true,
+ metadata = metadata)
+ Option(hc.getComment).map(field.withComment).getOrElse(field)
+ }
+
+ // TODO: merge this with HiveClientImpl#toHiveTable
--- End diff --
So far, it is a little bit tricky when merging them, because our execution is using 1.2.1, but Hive metadata APIs support the versions from 0.12 to 1.2. Thus, it does not make sense to do it.
So far, the schema inference is not using metadata Hive client. I checked the code. The changes between 0.12 and 1.2 look fine to me. Schema inference should work correctly. I think I need to add a test case to VersionSuite.scala.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #71802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71802/testReport)** for PR 16636 at commit [`e3cc423`](https://github.com/apache/spark/commit/e3cc423e2ecf3e9128b8036905c044a3f658cd25).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99459314
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/QueryPartitionSuite.scala ---
@@ -68,82 +68,4 @@ class QueryPartitionSuite extends QueryTest with SQLTestUtils with TestHiveSingl
sql("DROP TABLE IF EXISTS createAndInsertTest")
}
}
-
- test("SPARK-13709: reading partitioned Avro table with nested schema") {
--- End diff --
Moved it to `VersionSuite` to ensure all the Hive metastore versions can correctly infer the schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Block Creating a Hive Table W...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r98324106
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -455,4 +462,133 @@ private[spark] object HiveUtils extends Logging {
case (decimal, DecimalType()) => decimal.toString
case (other, tpe) if primitiveTypes contains tpe => other.toString
}
+
+ /** Converts the native StructField to Hive's FieldSchema. */
+ private def toHiveColumn(c: StructField): FieldSchema = {
+ val typeString = if (c.metadata.contains(HiveUtils.hiveTypeString)) {
+ c.metadata.getString(HiveUtils.hiveTypeString)
+ } else {
+ c.dataType.catalogString
+ }
+ new FieldSchema(c.name, typeString, c.getComment.orNull)
+ }
+
+ /** Builds the native StructField from Hive's FieldSchema. */
+ private def fromHiveColumn(hc: FieldSchema): StructField = {
+ val columnType = try {
+ CatalystSqlParser.parseDataType(hc.getType)
+ } catch {
+ case e: ParseException =>
+ throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
+ }
+
+ val metadata = new MetadataBuilder().putString(HiveUtils.hiveTypeString, hc.getType).build()
+ val field = StructField(
+ name = hc.getName,
+ dataType = columnType,
+ nullable = true,
+ metadata = metadata)
+ Option(hc.getComment).map(field.withComment).getOrElse(field)
+ }
+
+ // TODO: merge this with HiveClientImpl#toHiveTable
+ /** Converts the native table metadata representation format CatalogTable to Hive's Table. */
+ def toHiveTable(catalogTable: CatalogTable): HiveTable = {
+ // We start by constructing an API table as Hive performs several important transformations
+ // internally when converting an API table to a QL table.
+ val tTable = new org.apache.hadoop.hive.metastore.api.Table()
+ tTable.setTableName(catalogTable.identifier.table)
+ tTable.setDbName(catalogTable.database)
+
+ val tableParameters = new java.util.HashMap[String, String]()
+ tTable.setParameters(tableParameters)
+ catalogTable.properties.foreach { case (k, v) => tableParameters.put(k, v) }
+
+ tTable.setTableType(catalogTable.tableType match {
+ case CatalogTableType.EXTERNAL => HiveTableType.EXTERNAL_TABLE.toString
+ case CatalogTableType.MANAGED => HiveTableType.MANAGED_TABLE.toString
+ case CatalogTableType.VIEW => HiveTableType.VIRTUAL_VIEW.toString
+ })
+
+ val sd = new org.apache.hadoop.hive.metastore.api.StorageDescriptor()
+ tTable.setSd(sd)
+
+ // Note: In Hive the schema and partition columns must be disjoint sets
+ val (partCols, schema) = catalogTable.schema.map(toHiveColumn).partition { c =>
+ catalogTable.partitionColumnNames.contains(c.getName)
+ }
+ sd.setCols(schema.asJava)
+ tTable.setPartitionKeys(partCols.asJava)
+
+ catalogTable.storage.locationUri.foreach(sd.setLocation)
+ catalogTable.storage.inputFormat.foreach(sd.setInputFormat)
+ catalogTable.storage.outputFormat.foreach(sd.setOutputFormat)
+
+ val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
+ catalogTable.storage.serde.foreach(serdeInfo.setSerializationLib)
+ sd.setSerdeInfo(serdeInfo)
+
+ val serdeParameters = new java.util.HashMap[String, String]()
+ catalogTable.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ serdeInfo.setParameters(serdeParameters)
+
+ new HiveTable(tTable)
+ }
+
+ /**
+ * Converts the native partition metadata representation format CatalogTablePartition to
+ * Hive's Partition.
+ */
+ def toHivePartition(
+ catalogTable: CatalogTable,
+ hiveTable: HiveTable,
+ partition: CatalogTablePartition): HivePartition = {
+ val tPartition = new org.apache.hadoop.hive.metastore.api.Partition
+ tPartition.setDbName(catalogTable.database)
+ tPartition.setTableName(catalogTable.identifier.table)
+ tPartition.setValues(catalogTable.partitionColumnNames.map(partition.spec(_)).asJava)
+
+ val sd = new org.apache.hadoop.hive.metastore.api.StorageDescriptor()
+ tPartition.setSd(sd)
+
+ // Note: In Hive the schema and partition columns must be disjoint sets
+ val schema = catalogTable.schema.map(toHiveColumn).filter { c =>
+ !catalogTable.partitionColumnNames.contains(c.getName)
+ }
+ sd.setCols(schema.asJava)
+
+ partition.storage.locationUri.foreach(sd.setLocation)
+ partition.storage.inputFormat.foreach(sd.setInputFormat)
+ partition.storage.outputFormat.foreach(sd.setOutputFormat)
+
+ val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
+ sd.setSerdeInfo(serdeInfo)
+ // maps and lists should be set only after all elements are ready (see HIVE-7975)
+ partition.storage.serde.foreach(serdeInfo.setSerializationLib)
+
+ val serdeParameters = new java.util.HashMap[String, String]()
+ catalogTable.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ partition.storage.properties.foreach { case (k, v) => serdeParameters.put(k, v) }
+ serdeInfo.setParameters(serdeParameters)
+
+ new HivePartition(hiveTable, tPartition)
+ }
+
+ /**
+ * Infers the schema for Hive serde tables and returns the CatalogTable with the inferred schema.
+ * When the tables are data source tables or the schema already exists, returns the original
+ * CatalogTable.
+ */
+ def inferSchema(table: CatalogTable): CatalogTable = {
+ if (DDLUtils.isDatasourceTable(table) || table.schema.nonEmpty) {
+ table
+ } else {
+ val hiveTable = toHiveTable(table)
+ // Note: Hive separates partition columns and the schema, but for us the
+ // partition columns are part of the schema
+ val partCols = hiveTable.getPartCols.asScala.map(fromHiveColumn)
+ val schema = StructType(hiveTable.getCols.asScala.map(fromHiveColumn) ++ partCols)
+ table.copy(schema = schema, partitionColumnNames = partCols.map(_.name))
--- End diff --
To infer the schema of Hive serde tables is simple. We just need to construct the Hive's Table struct and then call `getCols` to get the inferred schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #72358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72358/testReport)** for PR 16636 at commit [`d87676f`](https://github.com/apache/spark/commit/d87676f0a71edb93b06bdb3107be3cac519b1e81).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde T...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r99326044
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -26,6 +26,7 @@ import scala.util.control.NonFatal
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hive.conf.HiveConf
--- End diff --
unnecessary change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72387/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71795/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16636
LGTM, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72359/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16636
cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72358/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16636
uh, not all the serde need the schema. We need to check `hive.serdes.using.metastore.for.schema`, which contains a list of serdes that require user-specified schema:
```
"org.apache.hadoop.hive.ql.io.orc.OrcSerde," +
"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe," +
"org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe," +
"org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe," +
"org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe," +
"org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe," +
"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe," +
"org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe"
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16636
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71612/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16636: [SPARK-19279] [SQL] Block Creating a Hive Table W...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16636#discussion_r97247351
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -1527,6 +1527,21 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
}
}
+ test("create a data source table without schema") {
+ import testImplicits._
+ withTempPath { tempDir =>
+ withTable("tab1", "tab2") {
+ (("a", "b") :: Nil).toDF().write.json(tempDir.getCanonicalPath)
+
+ val e = intercept[AnalysisException] { sql("CREATE TABLE tab1 USING json") }.getMessage
--- End diff --
we should also test a data source that can infer schema without files(e.g. LibSVM data source has fixed schema). Ideally we should only fail if the given data source can't infer schema without files.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Infer Schema for Hive Serde Tables a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #72359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72359/testReport)** for PR 16636 at commit [`1836887`](https://github.com/apache/spark/commit/18368874e16702d641131cd7c2a80af4e98d09e0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16636: [SPARK-19279] [SQL] Block Creating a Hive Table With an ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16636
**[Test build #71612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71612/testReport)** for PR 16636 at commit [`a8281b1`](https://github.com/apache/spark/commit/a8281b15676eaae43eed58f8dbb288d15e13294d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org