You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/12 05:28:05 UTC

[GitHub] [spark] linhongliu-db opened a new pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

linhongliu-db opened a new pull request #34254:
URL: https://github.com/apache/spark/pull/34254


   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943127310


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144231/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940726869


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144128/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940833297


   **[Test build #144124 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144124/testReport)** for PR 34254 at commit [`927eae6`](https://github.com/apache/spark/commit/927eae68bbac706a575aa4059a26cf677812d985).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941157553


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48633/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r728607439



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]
     val viewDDL = buildViewDDL(metadata, isTempView)
 
-    val projectList = viewColumnNames.zip(metadata.schema).map { case (name, field) =>
-      val normalizedName = normalizeColName(name)
-      val count = nameToCounts(normalizedName)
-      val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
-      nameToCurrentOrdinal(normalizedName) = ordinal + 1
-      val col = GetViewColumnByNameAndOrdinal(
-        metadata.identifier.toString, name, ordinal, count, viewDDL)
-      Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+    val projectList = if (!isHiveCreatedView(metadata)) {
+      viewColumnNames.zip(metadata.schema).map { case (name, field) =>
+        val normalizedName = normalizeColName(name)
+        val count = nameToCounts(normalizedName)
+        val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
+        nameToCurrentOrdinal(normalizedName) = ordinal + 1
+        val col = GetViewColumnByNameAndOrdinal(
+          metadata.identifier.toString, name, ordinal, count, viewDDL)
+        Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+      }
+    } else {
+      // For view created by hive, the parsed view plan may have different output columns with
+      // the schema stored in metadata. For example: `CREATE VIEW v AS SELECT 1 FROM t`
+      // the schema in metadata will be `_c0` while the parsed view plan has column named `1`
+      metadata.schema.zipWithIndex.map { case (field, index) =>
+        val col = GetColumnByOrdinal(index, field.dataType)

Review comment:
       make sense.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] shardulm94 commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727415203



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]

Review comment:
       [These lines](https://github.com/apache/spark/blob/1af7072fc27193f65caeb630f8229c8be89b57d3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L873-L899) are not used if `isHiveCreatedView(metadata)` is true. Should these lines be moved into the `if-else` branch for non-Hive views?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943392053


   **[Test build #144249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144249/testReport)** for PR 34254 at commit [`9854030`](https://github.com/apache/spark/commit/9854030c62db1f002a46a3c8e43b5e2c2bb2fafa).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940813225


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48612/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941050390






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940771454


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48606/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727177130



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -862,6 +862,13 @@ class SessionCatalog(
     }
   }
 
+  private def isHiveCreatedView(metadata: CatalogTable): Boolean = {
+    // For views created by hive without explicit column names, there will be auto-generated
+    // column names like "_c0", "_c1", "_c2"...
+    metadata.viewQueryColumnNames.isEmpty &&
+      metadata.schema.fieldNames.exists(_.matches("_c[0-9]+"))

Review comment:
       ```suggestion
         metadata.schema.fieldNames.forall(_.matches("_c[0-9]+"))
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940769438


   **[Test build #144135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144135/testReport)** for PR 34254 at commit [`87f749f`](https://github.com/apache/spark/commit/87f749f8f59d1ee29b24ce459d25781460200ed1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940771480


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48606/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942913429


   **[Test build #144231 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144231/testReport)** for PR 34254 at commit [`42dba17`](https://github.com/apache/spark/commit/42dba17e2f536b89ab6c647b2f555d4cd6aba958).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r728786416



##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2632,6 +2632,16 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
       }
     }
   }
+
+  test("SPARK-36905: read hive views without without explicit column names") {
+      withTable("t1") {
+        withView("test_view") {
+          hiveClient.runSqlHive("create table t1 stored as avro as select 2 as id")
+          hiveClient.runSqlHive("create view test_view as select 1, id + 1 from t1")
+          checkAnswer(sql("select * from test_view"), Seq(Row(1, 3)))
+        }
+      }

Review comment:
       ```suggestion
       withTable("t1") {
         withView("test_view") {
           hiveClient.runSqlHive("create table t1 stored as avro as select 2 as id")
           hiveClient.runSqlHive("create view test_view as select 1, id + 1 from t1")
           checkAnswer(sql("select * from test_view"), Seq(Row(1, 3)))
         }
       }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943402473


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144249/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941941194


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48656/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941050390


   **[Test build #144155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144155/testReport)** for PR 34254 at commit [`35be0ee`](https://github.com/apache/spark/commit/35be0eecfc92f1c27f52013d2c1fc99cd9576cc1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727672882



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]
     val viewDDL = buildViewDDL(metadata, isTempView)
 
-    val projectList = viewColumnNames.zip(metadata.schema).map { case (name, field) =>
-      val normalizedName = normalizeColName(name)
-      val count = nameToCounts(normalizedName)
-      val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
-      nameToCurrentOrdinal(normalizedName) = ordinal + 1
-      val col = GetViewColumnByNameAndOrdinal(
-        metadata.identifier.toString, name, ordinal, count, viewDDL)
-      Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+    val projectList = if (!isHiveCreatedView(metadata)) {
+      viewColumnNames.zip(metadata.schema).map { case (name, field) =>
+        val normalizedName = normalizeColName(name)
+        val count = nameToCounts(normalizedName)
+        val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
+        nameToCurrentOrdinal(normalizedName) = ordinal + 1
+        val col = GetViewColumnByNameAndOrdinal(
+          metadata.identifier.toString, name, ordinal, count, viewDDL)
+        Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+      }
+    } else {
+      // For view created by hive, the parsed view plan may have different output columns with
+      // the schema stored in metadata. For example: `CREATE VIEW v AS SELECT 1 FROM t`
+      // the schema in metadata will be `_c0` while the parsed view plan has column named `1`
+      metadata.schema.zipWithIndex.map { case (field, index) =>
+        val col = GetColumnByOrdinal(index, field.dataType)

Review comment:
       `UpCast` is used to handle the data type change when the underlying table column is altered. In this case, most time, we need to use `GetViewColumnByNameAndOrdinal` as well.
   But for the hive, since we can only use `GetColumnByOrdinal`, so the `UpCast` couldn't bring many benefits. And if the underlying table is not changed, we don't need to cast the data type.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941922529


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48656/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941902859


   **[Test build #144178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144178/testReport)** for PR 34254 at commit [`8bd5916`](https://github.com/apache/spark/commit/8bd5916c475c1d50f4b744efbd3f48e421c258b6).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941157553






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943262342


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48728/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940769438


   **[Test build #144135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144135/testReport)** for PR 34254 at commit [`87f749f`](https://github.com/apache/spark/commit/87f749f8f59d1ee29b24ce459d25781460200ed1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942089734


   **[Test build #144178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144178/testReport)** for PR 34254 at commit [`8bd5916`](https://github.com/apache/spark/commit/8bd5916c475c1d50f4b744efbd3f48e421c258b6).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941143982


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48633/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943173330


   **[Test build #144249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144249/testReport)** for PR 34254 at commit [`9854030`](https://github.com/apache/spark/commit/9854030c62db1f002a46a3c8e43b5e2c2bb2fafa).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942965939


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48711/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727860800



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]
     val viewDDL = buildViewDDL(metadata, isTempView)
 
-    val projectList = viewColumnNames.zip(metadata.schema).map { case (name, field) =>
-      val normalizedName = normalizeColName(name)
-      val count = nameToCounts(normalizedName)
-      val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
-      nameToCurrentOrdinal(normalizedName) = ordinal + 1
-      val col = GetViewColumnByNameAndOrdinal(
-        metadata.identifier.toString, name, ordinal, count, viewDDL)
-      Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+    val projectList = if (!isHiveCreatedView(metadata)) {
+      viewColumnNames.zip(metadata.schema).map { case (name, field) =>
+        val normalizedName = normalizeColName(name)
+        val count = nameToCounts(normalizedName)
+        val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
+        nameToCurrentOrdinal(normalizedName) = ordinal + 1
+        val col = GetViewColumnByNameAndOrdinal(
+          metadata.identifier.toString, name, ordinal, count, viewDDL)
+        Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+      }
+    } else {
+      // For view created by hive, the parsed view plan may have different output columns with
+      // the schema stored in metadata. For example: `CREATE VIEW v AS SELECT 1 FROM t`
+      // the schema in metadata will be `_c0` while the parsed view plan has column named `1`
+      metadata.schema.zipWithIndex.map { case (field, index) =>
+        val col = GetColumnByOrdinal(index, field.dataType)

Review comment:
       Adding `UpCast` does not hurt. If the column type didn't change, the optimizer will remove useless casts.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727173051



##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2632,6 +2632,21 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
       }
     }
   }
+
+  test("SPARK-36905: read hive views without without explicit column names") {
+    withTempDir { dir =>
+      val t1Loc = s"file:///$dir/t1"
+      withTable("t1") {
+        withView("test_view") {
+          hiveClient.runSqlHive(
+            s"create table t1(id int) stored as avro location '$t1Loc'")

Review comment:
       fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942095130


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144178/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940732200


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48605/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941050390


   **[Test build #144155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144155/testReport)** for PR 34254 at commit [`35be0ee`](https://github.com/apache/spark/commit/35be0eecfc92f1c27f52013d2c1fc99cd9576cc1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940700374


   **[Test build #144128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144128/testReport)** for PR 34254 at commit [`95506bc`](https://github.com/apache/spark/commit/95506bc1f3ab48faf8ee86e4e42c170da10a7368).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940766946


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48605/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943262342


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48728/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940730049


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48606/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940866857


   **[Test build #144144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144144/testReport)** for PR 34254 at commit [`95c00b9`](https://github.com/apache/spark/commit/95c00b9ec64aa14eeac9f8d6dca7a8142b64b308).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941955264


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48656/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940726869


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144128/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940745086


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48601/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727860800



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]
     val viewDDL = buildViewDDL(metadata, isTempView)
 
-    val projectList = viewColumnNames.zip(metadata.schema).map { case (name, field) =>
-      val normalizedName = normalizeColName(name)
-      val count = nameToCounts(normalizedName)
-      val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
-      nameToCurrentOrdinal(normalizedName) = ordinal + 1
-      val col = GetViewColumnByNameAndOrdinal(
-        metadata.identifier.toString, name, ordinal, count, viewDDL)
-      Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+    val projectList = if (!isHiveCreatedView(metadata)) {
+      viewColumnNames.zip(metadata.schema).map { case (name, field) =>
+        val normalizedName = normalizeColName(name)
+        val count = nameToCounts(normalizedName)
+        val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
+        nameToCurrentOrdinal(normalizedName) = ordinal + 1
+        val col = GetViewColumnByNameAndOrdinal(
+          metadata.identifier.toString, name, ordinal, count, viewDDL)
+        Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+      }
+    } else {
+      // For view created by hive, the parsed view plan may have different output columns with
+      // the schema stored in metadata. For example: `CREATE VIEW v AS SELECT 1 FROM t`
+      // the schema in metadata will be `_c0` while the parsed view plan has column named `1`
+      metadata.schema.zipWithIndex.map { case (field, index) =>
+        val col = GetColumnByOrdinal(index, field.dataType)

Review comment:
       Adding `UpCast` does not hurt. If the column type didn't change, the optimizer will remove useless casts.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34254:
URL: https://github.com/apache/spark/pull/34254


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943127310


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144231/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940694342


   **[Test build #144124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144124/testReport)** for PR 34254 at commit [`927eae6`](https://github.com/apache/spark/commit/927eae68bbac706a575aa4059a26cf677812d985).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940702210


   **[Test build #144129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144129/testReport)** for PR 34254 at commit [`7bfcaa3`](https://github.com/apache/spark/commit/7bfcaa34d35ded171d4d27893669a7a436382f62).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
linhongliu-db removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940685120


   For now, this PR is only to demonstrate the root cause, it's not a formal fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940866857


   **[Test build #144144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144144/testReport)** for PR 34254 at commit [`95c00b9`](https://github.com/apache/spark/commit/95c00b9ec64aa14eeac9f8d6dca7a8142b64b308).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] shardulm94 commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727359054



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]
     val viewDDL = buildViewDDL(metadata, isTempView)
 
-    val projectList = viewColumnNames.zip(metadata.schema).map { case (name, field) =>
-      val normalizedName = normalizeColName(name)
-      val count = nameToCounts(normalizedName)
-      val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
-      nameToCurrentOrdinal(normalizedName) = ordinal + 1
-      val col = GetViewColumnByNameAndOrdinal(
-        metadata.identifier.toString, name, ordinal, count, viewDDL)
-      Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+    val projectList = if (!isHiveCreatedView(metadata)) {
+      viewColumnNames.zip(metadata.schema).map { case (name, field) =>
+        val normalizedName = normalizeColName(name)
+        val count = nameToCounts(normalizedName)
+        val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
+        nameToCurrentOrdinal(normalizedName) = ordinal + 1
+        val col = GetViewColumnByNameAndOrdinal(
+          metadata.identifier.toString, name, ordinal, count, viewDDL)
+        Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+      }
+    } else {
+      // For view created by hive, the parsed view plan may have different output columns with
+      // the schema stored in metadata. For example: `CREATE VIEW v AS SELECT 1 FROM t`
+      // the schema in metadata will be `_c0` while the parsed view plan has column named `1`
+      metadata.schema.zipWithIndex.map { case (field, index) =>
+        val col = GetColumnByOrdinal(index, field.dataType)

Review comment:
       I see that the `dataType` parameter of `GetColumnByOrdinal` is ignored by the [Analyzer](https://github.com/apache/spark/blob/1af7072fc27193f65caeb630f8229c8be89b57d3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L1774). Should we keep the `UpCast` like in the branch above?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]

Review comment:
       [These lines](https://github.com/apache/spark/blob/1af7072fc27193f65caeb630f8229c8be89b57d3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L873-L899) are not used if `isHiveCreatedView(metadata)` is true. Should these lines be moved into the `if-else` branch for non-Hive views?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941955264


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48656/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940897426


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48621/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942913429


   **[Test build #144231 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144231/testReport)** for PR 34254 at commit [`42dba17`](https://github.com/apache/spark/commit/42dba17e2f536b89ab6c647b2f555d4cd6aba958).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943125578


   **[Test build #144231 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144231/testReport)** for PR 34254 at commit [`42dba17`](https://github.com/apache/spark/commit/42dba17e2f536b89ab6c647b2f555d4cd6aba958).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940771480


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48606/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940768050






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940702210


   **[Test build #144129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144129/testReport)** for PR 34254 at commit [`7bfcaa3`](https://github.com/apache/spark/commit/7bfcaa34d35ded171d4d27893669a7a436382f62).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940719000


   **[Test build #144128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144128/testReport)** for PR 34254 at commit [`95506bc`](https://github.com/apache/spark/commit/95506bc1f3ab48faf8ee86e4e42c170da10a7368).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940845690


   **[Test build #144135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144135/testReport)** for PR 34254 at commit [`87f749f`](https://github.com/apache/spark/commit/87f749f8f59d1ee29b24ce459d25781460200ed1).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941326364


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144155/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727672882



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]
     val viewDDL = buildViewDDL(metadata, isTempView)
 
-    val projectList = viewColumnNames.zip(metadata.schema).map { case (name, field) =>
-      val normalizedName = normalizeColName(name)
-      val count = nameToCounts(normalizedName)
-      val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
-      nameToCurrentOrdinal(normalizedName) = ordinal + 1
-      val col = GetViewColumnByNameAndOrdinal(
-        metadata.identifier.toString, name, ordinal, count, viewDDL)
-      Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+    val projectList = if (!isHiveCreatedView(metadata)) {
+      viewColumnNames.zip(metadata.schema).map { case (name, field) =>
+        val normalizedName = normalizeColName(name)
+        val count = nameToCounts(normalizedName)
+        val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
+        nameToCurrentOrdinal(normalizedName) = ordinal + 1
+        val col = GetViewColumnByNameAndOrdinal(
+          metadata.identifier.toString, name, ordinal, count, viewDDL)
+        Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+      }
+    } else {
+      // For view created by hive, the parsed view plan may have different output columns with
+      // the schema stored in metadata. For example: `CREATE VIEW v AS SELECT 1 FROM t`
+      // the schema in metadata will be `_c0` while the parsed view plan has column named `1`
+      metadata.schema.zipWithIndex.map { case (field, index) =>
+        val col = GetColumnByOrdinal(index, field.dataType)

Review comment:
       `UpCast` is used to handle the data type change when the underlying table column is altered. In this case, most time, we need to use `GetViewColumnByNameAndOrdinal` as well.
   But for the hive, since we can only use `GetColumnByOrdinal`, so the `UpCast` couldn't bring many benefits. And if the underlying table is not changed, we don't need to cast the data type.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]

Review comment:
       @shardulm94, thanks for reviewing. I can do this. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942965939


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48711/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940718040


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48601/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941326364


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144155/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940920237


   **[Test build #144144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144144/testReport)** for PR 34254 at commit [`95c00b9`](https://github.com/apache/spark/commit/95c00b9ec64aa14eeac9f8d6dca7a8142b64b308).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941091381


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48633/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940700374


   **[Test build #144128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144128/testReport)** for PR 34254 at commit [`95506bc`](https://github.com/apache/spark/commit/95506bc1f3ab48faf8ee86e4e42c170da10a7368).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941157553


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48633/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941902859


   **[Test build #144178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144178/testReport)** for PR 34254 at commit [`8bd5916`](https://github.com/apache/spark/commit/8bd5916c475c1d50f4b744efbd3f48e421c258b6).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943262297


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48728/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943173330


   **[Test build #144249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144249/testReport)** for PR 34254 at commit [`9854030`](https://github.com/apache/spark/commit/9854030c62db1f002a46a3c8e43b5e2c2bb2fafa).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942095130


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144178/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940768050






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727123835



##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2632,6 +2632,21 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
       }
     }
   }
+
+  test("SPARK-36905: read hive views without without explicit column names") {
+    withTempDir { dir =>
+      val t1Loc = s"file:///$dir/t1"
+      withTable("t1") {
+        withView("test_view") {
+          hiveClient.runSqlHive(
+            s"create table t1(id int) stored as avro location '$t1Loc'")

Review comment:
       do we have to provide a location to reproduce the bug?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940951700






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940864107






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940685120


   For now, this PR is only to demonstrate the root cause, it's not a formal fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940864111






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727179997



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -862,6 +862,13 @@ class SessionCatalog(
     }
   }
 
+  private def isHiveCreatedView(metadata: CatalogTable): Boolean = {
+    // For views created by hive without explicit column names, there will be auto-generated
+    // column names like "_c0", "_c1", "_c2"...
+    metadata.viewQueryColumnNames.isEmpty &&
+      metadata.schema.fieldNames.exists(_.matches("_c[0-9]+"))

Review comment:
       we may have `create view v as select 1, 2, t1.a from t1`, then the schema will be "c0, c1, a"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940762468


   **[Test build #144129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144129/testReport)** for PR 34254 at commit [`7bfcaa3`](https://github.com/apache/spark/commit/7bfcaa34d35ded171d4d27893669a7a436382f62).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [WIP][SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940694342


   **[Test build #144124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144124/testReport)** for PR 34254 at commit [`927eae6`](https://github.com/apache/spark/commit/927eae68bbac706a575aa4059a26cf677812d985).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727673118



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]

Review comment:
       @shardulm94, thanks for reviewing. I can do this. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943407813


   thanks, merging to master/3.2!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] shardulm94 commented on a change in pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #34254:
URL: https://github.com/apache/spark/pull/34254#discussion_r727359054



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -898,14 +905,24 @@ class SessionCatalog(
     val nameToCurrentOrdinal = scala.collection.mutable.HashMap.empty[String, Int]
     val viewDDL = buildViewDDL(metadata, isTempView)
 
-    val projectList = viewColumnNames.zip(metadata.schema).map { case (name, field) =>
-      val normalizedName = normalizeColName(name)
-      val count = nameToCounts(normalizedName)
-      val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
-      nameToCurrentOrdinal(normalizedName) = ordinal + 1
-      val col = GetViewColumnByNameAndOrdinal(
-        metadata.identifier.toString, name, ordinal, count, viewDDL)
-      Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+    val projectList = if (!isHiveCreatedView(metadata)) {
+      viewColumnNames.zip(metadata.schema).map { case (name, field) =>
+        val normalizedName = normalizeColName(name)
+        val count = nameToCounts(normalizedName)
+        val ordinal = nameToCurrentOrdinal.getOrElse(normalizedName, 0)
+        nameToCurrentOrdinal(normalizedName) = ordinal + 1
+        val col = GetViewColumnByNameAndOrdinal(
+          metadata.identifier.toString, name, ordinal, count, viewDDL)
+        Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata))
+      }
+    } else {
+      // For view created by hive, the parsed view plan may have different output columns with
+      // the schema stored in metadata. For example: `CREATE VIEW v AS SELECT 1 FROM t`
+      // the schema in metadata will be `_c0` while the parsed view plan has column named `1`
+      metadata.schema.zipWithIndex.map { case (field, index) =>
+        val col = GetColumnByOrdinal(index, field.dataType)

Review comment:
       I see that the `dataType` parameter of `GetColumnByOrdinal` is ignored by the [Analyzer](https://github.com/apache/spark/blob/1af7072fc27193f65caeb630f8229c8be89b57d3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L1774). Should we keep the `UpCast` like in the branch above?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940859931


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48612/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941322352


   **[Test build #144155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144155/testReport)** for PR 34254 at commit [`35be0ee`](https://github.com/apache/spark/commit/35be0eecfc92f1c27f52013d2c1fc99cd9576cc1).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941157553






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940951700






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943402473


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144249/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-943216286


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48728/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-941143982






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942965864


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48711/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-940929139


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48621/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34254: [SPARK-36905] Fix reading hive views without explicit column names

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34254:
URL: https://github.com/apache/spark/pull/34254#issuecomment-942937275


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48711/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org