You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rajeshbalamohan <gi...@git.apache.org> on 2016/08/08 08:30:46 UTC

[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

GitHub user rajeshbalamohan opened a pull request:

    https://github.com/apache/spark/pull/14537

    [SPARK-16948][SQL] Querying empty partitioned orc tables throws excep\u2026

    ## What changes were proposed in this pull request?
    Querying empty partitioned ORC tables from spark-sql throws exception with `spark.sql.hive.convertMetastoreOrc=true`.  This is due to the fact that the inferschema() would end up throwing `FileNotFoundException` when no files are present in partitioned orc tables.  Patch attempts to fix it, wherein it would fall back to metastore based schema information. 
    
    ## How was this patch tested?
    Included unit tests and also tested it in small scale cluster.
    
    
    (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
    
    
    \u2026tion

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rajeshbalamohan/spark SPARK-16948

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/14537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #14537
    
----
commit 5721b88c7c816f57ef39374ac9b335d870543628
Author: Rajesh Balamohan <rb...@apache.org>
Date:   2016-08-08T08:28:23Z

    [SPARK-16948][SQL] Querying empty partitioned orc tables throws exception

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64314/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Thanks @gatorsmile . Removed the changes related to OrcFileFormat


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76170206
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +373,42 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    +        withTable("empty_orc_partitioned") {
    +          spark.sql(
    +            s"""
    +                |CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +                |PARTITIONED BY (p INT) STORED AS ORC
    +              """.stripMargin)
    +
    +          val emptyDF = Seq.empty[(Int, String)].toDF("key", "value").coalesce(1)
    +
    +          // Query empty table
    +          checkAnswer(
    +            sql("SELECT key, value FROM empty_orc_partitioned WHERE key > 10"),
    +            emptyDF)
    +        }
    +
    +        withTable("empty_orc") {
    +          spark.sql(
    +            s"""
    +               |CREATE TABLE empty_orc(key INT, value STRING)
    +               |STORED AS ORC
    +              """.stripMargin)
    --- End diff --
    
    Nit: 
    ```
              sql(
                """
                  |CREATE TABLE empty_orc(key INT, value STRING)
                  |STORED AS ORC
                """.stripMargin)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r74092780
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -287,14 +287,14 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    +        val schema =
    +          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
             val inferredSchema = if (fileType.equals("parquet")) {
    --- End diff --
    
    I just noticed the boolean expression should be `fileType == "parquet"` to make it idiomatic Scala. Can you make that change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r79972251
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,27 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +
    +            // For Parquet, get correct schema by merging Metastore schema data types
    --- End diff --
    
    Sure. Will change to return metastoreSchema for parq as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #65755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65755/consoleFull)** for PR 14537 at commit [`fa71370`](https://github.com/apache/spark/commit/fa713700f853e78053ac0be5db49250951aaa715).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    The schema inferring is replaced with metastore schema completely in #14690. I think we can close this now? cc @cloud-fan @liancheng 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64384/consoleFull)** for PR 14537 at commit [`5f7e5ad`](https://github.com/apache/spark/commit/5f7e5ad5eac524ca2b153748d7a7611cadb43736).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @tejasapatil, @mallman - Can you please review when you find time?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #63474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63474/consoleFull)** for PR 14537 at commit [`75bca1b`](https://github.com/apache/spark/commit/75bca1b822e4c3fc6b894a9bc7fb6067b36900b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76189706
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +373,40 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc tables in ORC") {
    --- End diff --
    
    how about `support empty orc table when converting hive serde table to data source table`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64397/consoleFull)** for PR 14537 at commit [`0772a96`](https://github.com/apache/spark/commit/0772a9630d56ecbe6ab0c5d9d850e55ca7b1b8bf).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64314/consoleFull)** for PR 14537 at commit [`eb8a955`](https://github.com/apache/spark/commit/eb8a955e9fd39b2c8b9b88cbc1f01b1937af1189).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @rajeshbalamohan So for Orc 2.x files, would schema inference be unnecessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @cloud-fan . Failure is related to the parquet changes introduced for returning metastoreSchema (it has issues with complex types). I am not very comfortable with the Parquet codepath. For time being, I would revert back the last change. We can create subsequent jira if needed for parq related changes; Alternatively I am fine with someone who is comfortable with parq code taking over this as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76170101
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +373,42 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    +        withTable("empty_orc_partitioned") {
    +          spark.sql(
    +            s"""
    +                |CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +                |PARTITIONED BY (p INT) STORED AS ORC
    +              """.stripMargin)
    +
    +          val emptyDF = Seq.empty[(Int, String)].toDF("key", "value").coalesce(1)
    +
    +          // Query empty table
    +          checkAnswer(
    +            sql("SELECT key, value FROM empty_orc_partitioned WHERE key > 10"),
    +            emptyDF)
    +        }
    +
    +        withTable("empty_orc") {
    +          spark.sql(
    +            s"""
    +               |CREATE TABLE empty_orc(key INT, value STRING)
    +               |STORED AS ORC
    +              """.stripMargin)
    +
    +          val emptyDF = Seq.empty[(Int, String)].toDF("key", "value").coalesce(1)
    +
    +          // Query empty table
    +          checkAnswer(
    +            sql("SELECT key, value FROM empty_orc WHERE key > 10"),
    --- End diff --
    
    The same here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #63441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63441/consoleFull)** for PR 14537 at commit [`4ae92d8`](https://github.com/apache/spark/commit/4ae92d810720ab56081a03f1c222903af51ce751).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    For non-partitioned ORC, it is currently using the metastore schema and is not inferring the schema currently in HiveMetastoreCatalog, and hence not an issue. But the problem of wrong mapping (i.e physical col name in file being different than that of metastore) still exists.  The more I see it, it would be easier to club the patches in this JIRA itself.  If so, HiveMetastoreCatalog could just rely on metastoreSchema and ORCFileFormat can later do the mapping if the mappings are different. I will revise the patch to include this scenario and post it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64341/consoleFull)** for PR 14537 at commit [`c9d677b`](https://github.com/apache/spark/commit/c9d677b0629b39e28bf6944cd3cf38457fcfe319).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64285/consoleFull)** for PR 14537 at commit [`9a8838a`](https://github.com/apache/spark/commit/9a8838a962a347db7303944c0d911f1417fb80e7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64385/consoleFull)** for PR 14537 at commit [`e8e2d70`](https://github.com/apache/spark/commit/e8e2d707828715aa026970f3cb745439b8617f2d).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/14537


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64449/consoleFull)** for PR 14537 at commit [`9ecb2ed`](https://github.com/apache/spark/commit/9ecb2ed01db1daa19dfe837745d5468cc4990703).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64446/consoleFull)** for PR 14537 at commit [`fc14e2d`](https://github.com/apache/spark/commit/fc14e2d95cb95becf90a38e91e7725e483bae835).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64449 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64449/consoleFull)** for PR 14537 at commit [`9ecb2ed`](https://github.com/apache/spark/commit/9ecb2ed01db1daa19dfe837745d5468cc4990703).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #63352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63352/consoleFull)** for PR 14537 at commit [`5721b88`](https://github.com/apache/spark/commit/5721b88c7c816f57ef39374ac9b335d870543628).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #65755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65755/consoleFull)** for PR 14537 at commit [`fa71370`](https://github.com/apache/spark/commit/fa713700f853e78053ac0be5db49250951aaa715).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @rajeshbalamohan, the changes to `HiveMetastoreCatalog.scala` look reasonable. This mirrors the behavior of this method before the `if (fileType.equals("parquet"))` expression was introduced in 1e886159849e3918445d3fdc3c4cef86c6c1a236.
    
    @tejasapatil, can you help review this PR? I ask because you're the author of 1e886159849e3918445d3fdc3c4cef86c6c1a236, which is where the code in question in `HiveMetastoreCatalog.scala` was written.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r80639203
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,24 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +            inferredSchema.map { inferred =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            metastoreSchema
    --- End diff --
    
    @cloud-fan As we discussed offline yesterday, this is probably fine since ORC supports column pruning. Therefore, when reading an ORC file in a partitioned table, the reader always ignores partition columns stored inside the physical file and uses the value encoded in partition directory path.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #63441 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63441/consoleFull)** for PR 14537 at commit [`4ae92d8`](https://github.com/apache/spark/commit/4ae92d810720ab56081a03f1c222903af51ce751).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76179877
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
    @@ -54,10 +57,12 @@ class OrcFileFormat extends FileFormat with DataSourceRegister with Serializable
           sparkSession: SparkSession,
           options: Map[String, String],
           files: Seq[FileStatus]): Option[StructType] = {
    -    OrcFileOperator.readSchema(
    -      files.map(_.getPath.toUri.toString),
    -      Some(sparkSession.sessionState.newHadoopConf())
    -    )
    +    // Safe to ignore FileNotFoundException in case no files are found.
    +    val schema = Try(OrcFileOperator.readSchema(
    --- End diff --
    
    Yes, in case this is referred anytime later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #63474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63474/consoleFull)** for PR 14537 at commit [`75bca1b`](https://github.com/apache/spark/commit/75bca1b822e4c3fc6b894a9bc7fb6067b36900b7).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76189247
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,27 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +
    +            // For Parquet, get correct schema by merging Metastore schema data types
    --- End diff --
    
    Do we have a test for this feature? I think we should make them consistent, i.e. parquet conversions should also use the metastore schema.
    
    cc @yhuai @liancheng 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #65765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65765/consoleFull)** for PR 14537 at commit [`e39715e`](https://github.com/apache/spark/commit/e39715e853e4f87e1657fcb491dd7ed47e9da686).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64386/consoleFull)** for PR 14537 at commit [`0f901aa`](https://github.com/apache/spark/commit/0f901aa48aea8c8a575578c30142eb057defe09b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65755/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76189474
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
    @@ -54,10 +57,12 @@ class OrcFileFormat extends FileFormat with DataSourceRegister with Serializable
           sparkSession: SparkSession,
           options: Map[String, String],
           files: Seq[FileStatus]): Option[StructType] = {
    -    OrcFileOperator.readSchema(
    -      files.map(_.getPath.toUri.toString),
    -      Some(sparkSession.sessionState.newHadoopConf())
    -    )
    +    // Safe to ignore FileNotFoundException in case no files are found.
    +    val schema = Try(OrcFileOperator.readSchema(
    --- End diff --
    
    @rajeshbalamohan is this change unnecessary for this PR? If so, I'd like to revert it to make the PR as small as possible.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75884267
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,26 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val inferredSchema =
    +          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +        val schema = fileType match {
    +          case "parquet" =>
    +            // For Parquet, get correct schema by merging Metastore schema data types
    +            // and Parquet schema field names.
    +            inferredSchema.map { schema =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, schema)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            inferredSchema.getOrElse(metastoreSchema)
    +          case _ =>
    +            inferredSchema.get
    --- End diff --
    
    @gatorsmile has asserted that we shouldn't get here. So shouldn't we throw some kind of exception here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75902767
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,26 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val inferredSchema =
    +          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +        val schema = fileType match {
    +          case "parquet" =>
    +            // For Parquet, get correct schema by merging Metastore schema data types
    +            // and Parquet schema field names.
    +            inferredSchema.map { schema =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, schema)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            inferredSchema.getOrElse(metastoreSchema)
    +          case _ =>
    +            inferredSchema.get
    --- End diff --
    
    Not sure if exception has to be thrown in this case. Or just return null in this case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @cloud-fan 
    >>
    For branch 2.0, we should open another PR to fix the OrcFileFormat.inferSchema, to not throw FileNotFoundException for empty table.
    >>
    Code for not throwing FileNotFoundException in OrcFileFormat.inferSchema was removed from this patch. I can create separate JIRA for that; plz let me know if that is blocking this patch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64399/consoleFull)** for PR 14537 at commit [`6ff7e5d`](https://github.com/apache/spark/commit/6ff7e5d50de530a71df5c4a4b220a8119ca3a3f6).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75988035
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +372,29 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc partitioned tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    --- End diff --
    
    Could you remove this line?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64446/consoleFull)** for PR 14537 at commit [`fc14e2d`](https://github.com/apache/spark/commit/fc14e2d95cb95becf90a38e91e7725e483bae835).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Thanks @mallman . Fixed review comments in latest commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64385/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75987946
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +372,29 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc partitioned tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    +        withTable("empty_orc_partitioned") {
    +          spark.sql(
    +            s"""CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +                | PARTITIONED BY (p INT) STORED AS ORC
    +              """.stripMargin)
    --- End diff --
    
    A comment about the style
    ```Scala
              sql(
                """
                  |CREATE TABLE empty_orc_partitioned(key INT, value STRING)
                  |PARTITIONED BY (p INT) STORED AS ORC
                """.stripMargin)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75987870
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +372,29 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc partitioned tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    +        withTable("empty_orc_partitioned") {
    +          spark.sql(
    +            s"""CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +                | PARTITIONED BY (p INT) STORED AS ORC
    +              """.stripMargin)
    +
    +          val emptyDF = Seq.empty[(Int, String)].toDF("key", "value").coalesce(1)
    +          emptyDF.createOrReplaceTempView("empty")
    +
    +          // Query empty table
    +          val df = spark.sql(
    +            s"""SELECT key, value FROM empty_orc_partitioned
    +                | WHERE key > 10
    +                      """.stripMargin)
    +          checkAnswer(df, emptyDF)
    --- End diff --
    
    A comment about the style. 
    ```Scala
              checkAnswer(
                sql("SELECT key, value FROM empty_orc_partitioned WHERE key > 10"),
                emptyDF)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65765/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    What's the progress on this one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    ok, reverted the changes related to physical schema changes. In both cases, it returns metastoreschema, and mismatches can be handled separately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    uh, I missed this ping. Will review it tonight. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    You might forget this comment https://github.com/apache/spark/pull/14537#discussion_r76189474


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64183 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64183/consoleFull)** for PR 14537 at commit [`4004c0a`](https://github.com/apache/spark/commit/4004c0a559a435ae2c4ef6a18c7b03408e6283fb).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76305079
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,27 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +
    +            // For Parquet, get correct schema by merging Metastore schema data types
    --- End diff --
    
    I think we have a test. @liancheng should have more info. But, one clarification is that this merging is based on column name (we do take care the case sensitivity issue though). So, if you want to name a column with another name, I think it is not doable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Sorry about the delay. Updated the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @rajeshbalamohan We'll need a committer to review your patch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64341/consoleFull)** for PR 14537 at commit [`c9d677b`](https://github.com/apache/spark/commit/c9d677b0629b39e28bf6944cd3cf38457fcfe319).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @gatorsmile Thanks for cc'ing me.
    
    As `spark.sql.hive.convertMetastoreOrc` is set to `false` by default, this change looks fine. However, if setting the config to `true`, and hitting with inconsistent schema between metastore and Orc files, I remember it will cause failure when reading the files.
    
    I've implemented two approaches to this issue, #14282 is simply disabling Orc conversion if the case happens, #14365 is doing complicated schema mapping. Once this is merged, I think we should fix the schema inconsistency soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64384/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Right, for Parquet this could be part of initial codebase (from Spark-1251 I believe) which merges any metastore conflicts with parq files.  But in the case of ORC, this inference is still valid as the column names stored  in old ORC format could be different from that of Hive Metastore (e.g HIVE-4243). There is a separate PR:https://github.com/apache/spark/pull/14471 which track the ORC compatibility issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    LGTM. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64446/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75067639
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
    @@ -56,10 +59,11 @@ private[sql] class OrcFileFormat
           sparkSession: SparkSession,
           options: Map[String, String],
           files: Seq[FileStatus]): Option[StructType] = {
    -    OrcFileOperator.readSchema(
    -      files.map(_.getPath.toUri.toString),
    -      Some(sparkSession.sessionState.newHadoopConf())
    -    )
    +    val schema = Try(OrcFileOperator.readSchema(
    +        files.map(_.getPath.toUri.toString),
    +        Some(sparkSession.sessionState.newHadoopConf())))
    +      .recover { case _: FileNotFoundException => None }
    --- End diff --
    
    why are we ignoring file not found exception here?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Support empty orc table when c...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76355262
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,27 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +
    +            // For Parquet, get correct schema by merging Metastore schema data types
    --- End diff --
    
    To follow the decision we made in https://github.com/apache/spark/pull/14207 , I think we should always use the metastore schema and not infer it again.
    
    For branch 2.0, we should open another PR to fix the `OrcFileFormat.inferSchema`, to not throw `FileNotFoundException` for empty table.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r80686780
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,24 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +            inferredSchema.map { inferred =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    --- End diff --
    
    I'm a little worried here. If the table is partitioned, `metastoreSchema` will always contain partition columns, and thus the merged schema will contain partition columns too. This means, we always read parquet files with partition columns, I think we may have a hidden bug.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Thanks @gatorsmile, it would be good to retain the change in OrcFileInputFormat's inferschema (just in case it is referenced later).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r74092457
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -287,14 +287,14 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    +        val schema =
    --- End diff --
    
    Thanks for refactoring this.
    
    I think it makes more sense if
    
        defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    
    is called `inferredSchema` and the value of the `if (fileType.equals("parquet"))` expression is called `schema`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63441/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75067717
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -287,21 +287,21 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    +        val inferredSchema =
    +          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    --- End diff --
    
    can we be more specific here? e.g. doing it only if it is parquet or orc.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76170085
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +373,42 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    +        withTable("empty_orc_partitioned") {
    +          spark.sql(
    +            s"""
    +                |CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +                |PARTITIONED BY (p INT) STORED AS ORC
    +              """.stripMargin)
    +
    +          val emptyDF = Seq.empty[(Int, String)].toDF("key", "value").coalesce(1)
    +
    +          // Query empty table
    +          checkAnswer(
    +            sql("SELECT key, value FROM empty_orc_partitioned WHERE key > 10"),
    --- End diff --
    
    Do we need the where clause?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75812470
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +372,48 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc partitioned tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    +        withTable("empty_orc_partitioned") {
    +          spark.sql(
    +            s"""CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +                | PARTITIONED BY (p INT) STORED AS ORC
    +              """.stripMargin)
    +
    +          val emptyDF = Seq.empty[(Int, String)].toDF("key", "value").coalesce(1)
    +          emptyDF.createOrReplaceTempView("empty")
    +
    +          // Query empty table
    +          val df = spark.sql(
    +            s"""SELECT key, value FROM empty_orc_partitioned
    +                | WHERE key > 10
    +                      """.stripMargin)
    +          checkAnswer(df, emptyDF)
    +        }
    +      }
    +
    +      withTempPath { dir =>
    +        withTable("empty_text_partitioned") {
    +          spark.sql(
    +            s"""CREATE TABLE empty_text_partitioned(key INT, value STRING)
    +                | PARTITIONED BY (p INT) STORED AS TEXTFILE
    --- End diff --
    
    Testing the textfile format sounds useless. We do not convert it to `LogicalRelation`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64449/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76170033
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +373,42 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    --- End diff --
    
    Do we still need this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r79788532
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,27 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +
    +            // For Parquet, get correct schema by merging Metastore schema data types
    --- End diff --
    
    can you address this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75812657
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,24 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    +        val inferredSchema =
    +          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +        val schema = if (fileType == "parquet") {
    +          // For Parquet, get correct schema by merging Metastore schema data types
    +          // and Parquet schema field names.
    +          inferredSchema.map { schema =>
    +            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, schema)
               }.getOrElse(metastoreSchema)
             } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +          // For others (e.g orc), fall back to metastore schema if needed.
    --- End diff --
    
    If the fileType is not parquet, the only possible type is orc. To ensure this is always true, we should check `fileType == "orc"`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64285/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64399/consoleFull)** for PR 14537 at commit [`6ff7e5d`](https://github.com/apache/spark/commit/6ff7e5d50de530a71df5c4a4b220a8119ca3a3f6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64183 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64183/consoleFull)** for PR 14537 at commit [`4004c0a`](https://github.com/apache/spark/commit/4004c0a559a435ae2c4ef6a18c7b03408e6283fb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r80640012
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,24 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +            inferredSchema.map { inferred =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            metastoreSchema
    --- End diff --
    
    We already have a test case for this case [here][1].
    
    [1]: https://github.com/apache/spark/blob/7f16affa262b059580ed2775a7b05a767aa72315/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcPartitionDiscoverySuite.scala#L129


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r80217010
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,24 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +            inferredSchema.map { inferred =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            metastoreSchema
    --- End diff --
    
    cc @yhuai @gatorsmile 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64285/consoleFull)** for PR 14537 at commit [`9a8838a`](https://github.com/apache/spark/commit/9a8838a962a347db7303944c0d911f1417fb80e7).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75904621
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,26 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val inferredSchema =
    +          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +        val schema = fileType match {
    +          case "parquet" =>
    +            // For Parquet, get correct schema by merging Metastore schema data types
    +            // and Parquet schema field names.
    +            inferredSchema.map { schema =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, schema)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            inferredSchema.getOrElse(metastoreSchema)
    +          case _ =>
    +            inferredSchema.get
    --- End diff --
    
    IMHO returning `null` here would be worse than `inferredSchema.get`. Nobody likes debugging a `NullPointerException`. \U0001f604
    
    If we're going to assert that `fileType` must be either `parquet` or `orc`, I think throwing an exception like
    
        throw new RuntimeException(s"Cannot convert a $fileType to a HadoopFsRelation")
    
    here would be most appropriate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #63352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63352/consoleFull)** for PR 14537 at commit [`5721b88`](https://github.com/apache/spark/commit/5721b88c7c816f57ef39374ac9b335d870543628).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @rajeshbalamohan do you have time to update it? thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64341/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Thank you thejas and @mallman


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Thanks @rxin .  Incorporated review comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64386/consoleFull)** for PR 14537 at commit [`0f901aa`](https://github.com/apache/spark/commit/0f901aa48aea8c8a575578c30142eb057defe09b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64386/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64399/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r80216989
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,24 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +            inferredSchema.map { inferred =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            metastoreSchema
    --- End diff --
    
    I went through the code path again, seems we must infer the schema here.
    
    In metastore, we store the table schema and partition columns. `HadoopFsRelation` need a `dataSchema` which is the real schema of the data files. Normally it's just the table schema exclude partition columns, however, Spark SQL supports a special case: partition columns can also exist in data files. (see the doc for `HadoopFsRelation.dataSchema`). This information is not preserved in metastore, so we have to infer the data schema based on data files here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Fixed the test case name. I haven't changed the parquet code path as I wasn't sure on whether it would break any backward compatibility.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    For latest ORC, if the data was written out by Hive, it would have the same mapping. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by mallman <gi...@git.apache.org>.
Github user mallman commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r74003788
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -294,7 +294,9 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
                 ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
               }.getOrElse(metastoreSchema)
             } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +          val inferredSchema =
    --- End diff --
    
    There's some code duplicated in both branches of this `if` expression. Can you refactor it to remove the duplication, please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    can you address this https://github.com/apache/spark/pull/14537/files#r76355262? thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Use metastore schema instead o...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r80639483
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +373,40 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("support empty orc table when converting hive serde table to data source table") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTable("empty_orc_partitioned") {
    +        sql(
    +          """
    +            |CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +            |PARTITIONED BY (p INT) STORED AS ORC
    +          """.stripMargin)
    +
    +        val emptyDF = Seq.empty[(Int, String)].toDF("key", "value").coalesce(1)
    --- End diff --
    
    You don't really need `.coalesce(1)` since the create DataFrame wraps a `LocalRelation`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76180471
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +373,40 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTable("empty_orc_partitioned") {
    +        spark.sql(
    --- End diff --
    
    Nit: `spark.sql` -> `sql`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76170264
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +373,42 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    +        withTable("empty_orc_partitioned") {
    +          spark.sql(
    +            s"""
    +                |CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +                |PARTITIONED BY (p INT) STORED AS ORC
    +              """.stripMargin)
    --- End diff --
    
    Nit: 
    ```Scala
              sql(
                """
                  |CREATE TABLE empty_orc_partitioned(key INT, value STRING)
                  |PARTITIONED BY (p INT) STORED AS ORC
                """.stripMargin)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #65765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65765/consoleFull)** for PR 14537 at commit [`e39715e`](https://github.com/apache/spark/commit/e39715e853e4f87e1657fcb491dd7ed47e9da686).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76170526
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,27 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val schema = fileType match {
    +          case "parquet" =>
    +            val inferredSchema =
    +              defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +
    +            // For Parquet, get correct schema by merging Metastore schema data types
    +            // and Parquet schema field names.
    +            inferredSchema.map { schema =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, schema)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            metastoreSchema
    +          case _ =>
    +            throw new RuntimeException(s"Cannot convert a $fileType to a HadoopFsRelation")
    --- End diff --
    
    Spark SQL users do not know what is a `HadoopFsRelation`. 
    
    `to a data source table`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    cc @cloud-fan  @gatorsmile can you also take a look at this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63474/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Support empty orc table when converti...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    BTW, @rajeshbalamohan as you directly use metastore schema now, the PR description looks not correct anymore, can you also update it? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76180414
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
    @@ -54,10 +57,12 @@ class OrcFileFormat extends FileFormat with DataSourceRegister with Serializable
           sparkSession: SparkSession,
           options: Map[String, String],
           files: Seq[FileStatus]): Option[StructType] = {
    -    OrcFileOperator.readSchema(
    -      files.map(_.getPath.toUri.toString),
    -      Some(sparkSession.sessionState.newHadoopConf())
    -    )
    +    // Safe to ignore FileNotFoundException in case no files are found.
    +    val schema = Try(OrcFileOperator.readSchema(
    --- End diff --
    
    You know, the changes could impact the behaviors of the callers of `inferSchema`. 
    
    Could you write a test case to cover this scenario? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75987730
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
    @@ -372,6 +372,29 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
         }
       }
     
    +  test("SPARK-16948. Check empty orc partitioned tables in ORC") {
    +    withSQLConf((HiveUtils.CONVERT_METASTORE_ORC.key, "true")) {
    +      withTempPath { dir =>
    +        withTable("empty_orc_partitioned") {
    +          spark.sql(
    +            s"""CREATE TABLE empty_orc_partitioned(key INT, value STRING)
    +                | PARTITIONED BY (p INT) STORED AS ORC
    +              """.stripMargin)
    +
    +          val emptyDF = Seq.empty[(Int, String)].toDF("key", "value").coalesce(1)
    +          emptyDF.createOrReplaceTempView("empty")
    --- End diff --
    
    Could you remove this line?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64314/consoleFull)** for PR 14537 at commit [`eb8a955`](https://github.com/apache/spark/commit/eb8a955e9fd39b2c8b9b88cbc1f01b1937af1189).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63352/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    why do we infer schema for tables? Table schema should be persisted to metastore when it was created.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64384/consoleFull)** for PR 14537 at commit [`5f7e5ad`](https://github.com/apache/spark/commit/5f7e5ad5eac524ca2b153748d7a7611cadb43736).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Based on what you replied to @cloud-fan 's question, my follow-up question is:
    
    How about the non-partitioned empty ORC table? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64385/consoleFull)** for PR 14537 at commit [`e8e2d70`](https://github.com/apache/spark/commit/e8e2d707828715aa026970f3cb745439b8617f2d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64397/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    I quickly checked the latest fixes. It is based on a few assumptions. For example, the column order and length of metastore schema are the same as the physical schema. I am not sure whether we can make such an assumption. 
    
    I saw the previously comment by @cloud-fan https://github.com/apache/spark/pull/14365#issuecomment-236875668 in another PR https://github.com/apache/spark/pull/14365. 
    
    Maybe we do not need to infer the schema in both cases. Just use the metastore schema. If it does not match with the physical schema, we do not convert Hive orc tables to data source tables.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Thanks @gatorsmile. Addressed review comments


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    @rxin Can you please review when you find time? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64183/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    **[Test build #64397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64397/consoleFull)** for PR 14537 at commit [`0772a96`](https://github.com/apache/spark/commit/0772a9630d56ecbe6ab0c5d9d850e55ca7b1b8bf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    LGTM, pending jenkins


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r76170021
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
    @@ -54,10 +57,12 @@ class OrcFileFormat extends FileFormat with DataSourceRegister with Serializable
           sparkSession: SparkSession,
           options: Map[String, String],
           files: Seq[FileStatus]): Option[StructType] = {
    -    OrcFileOperator.readSchema(
    -      files.map(_.getPath.toUri.toString),
    -      Some(sparkSession.sessionState.newHadoopConf())
    -    )
    +    // Safe to ignore FileNotFoundException in case no files are found.
    +    val schema = Try(OrcFileOperator.readSchema(
    --- End diff --
    
    Do we still need this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14537: [SPARK-16948][SQL] Use metastore schema instead of infer...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14537
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14537#discussion_r75967137
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
    @@ -237,21 +237,26 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
               new Path(metastoreRelation.catalogTable.storage.locationUri.get),
               partitionSpec)
     
    -        val inferredSchema = if (fileType.equals("parquet")) {
    -          val inferredSchema =
    -            defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    -          inferredSchema.map { inferred =>
    -            ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred)
    -          }.getOrElse(metastoreSchema)
    -        } else {
    -          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()).get
    +        val inferredSchema =
    +          defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles())
    +        val schema = fileType match {
    +          case "parquet" =>
    +            // For Parquet, get correct schema by merging Metastore schema data types
    +            // and Parquet schema field names.
    +            inferredSchema.map { schema =>
    +              ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, schema)
    +            }.getOrElse(metastoreSchema)
    +          case "orc" =>
    +            inferredSchema.getOrElse(metastoreSchema)
    +          case _ =>
    +            inferredSchema.get
    --- End diff --
    
    Thanks @mallman . Addressed this in latest commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org