You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/19 06:23:55 UTC

[GitHub] [spark] AngersZhuuuu opened a new pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

AngersZhuuuu opened a new pull request #30421:
URL: https://github.com/apache/spark/pull/30421


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   Hive support type constructed value as partition spec value, spark should support too.
   
   
   ### Why are the changes needed?
    Support TypeConstructed partition spec value keep same with hive
   
   ### Does this PR introduce _any_ user-facing change?
   Yes, user can use TypeConstruct value as partition spec value such as 
   ``
   
   ### How was this patch tested?
   Added UT
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730450478






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787968968


   **[Test build #135592 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135592/testReport)** for PR 30421 at commit [`adc3984`](https://github.com/apache/spark/commit/adc3984d73683d8eebbc723c7f6cf8b1fb007bc9).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `public class JavaModelSelectionViaRandomHyperparametersExample `
     * `class GangliaSink(`
     * `case class Limits[T: Numeric](x: T, y: T)`
     * `abstract class Generator[T: Numeric] `
     * `class ParamRandomBuilder extends ParamGridBuilder `
     * `class ParamRandomBuilder(ParamGridBuilder):`
     * `sealed trait PartitionSpec extends LeafExpression with Unevaluable `
     * `trait V2PartitionCommand extends Command `
     * `case class AnalyzeTables(`
     * `case class TruncateTable(table: LogicalPlan) extends Command `
     * `case class TruncatePartition(`
     * `case class AnalyzeTablesCommand(`
     * `case class RepairTableCommand(`
     * `case class TruncatePartitionExec(`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r584844688



##########
File path: docs/sql-migration-guide.md
##########
@@ -65,6 +65,8 @@ license: |
  
   - In Spark 3.2, the output schema of `SHOW TBLPROPERTIES` becomes `key: string, value: string` whether you specify the table property key or not. In Spark 3.1 and earlier, the output schema of `SHOW TBLPROPERTIES` is `value: string` when you specify the table property key. To restore the old schema with the builtin catalog, you can set `spark.sql.legacy.keepCommandOutputSchema` to `true`.
 
+  - In Spark 3.2, we support a typed literal for a partition constant value in a INSERT clause. For example, a right-side constant value in `PARTITION (dt = date'2020-01-01')` is parsed as a date-typed literal in the partition spec. In Spark 3.1 and earlier, the partition value will be treated as string value `date '2020-01-01'` and it's a illegal date type string value and will been converted to `__HIVE_DEFAULT_PARTITION__`.

Review comment:
       > let's update the migration guide as it's not only INSERT now.
   
   How about `we support a typed literal for a partition constant value in a partition spec clause`? it's more general.

##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -41,7 +41,7 @@ INSERT INTO [ TABLE ] table_identifier [ partition_spec ] [ ( column_list ) ]
 * **partition_spec**
 
     An optional parameter that specifies a comma-separated list of key and value pairs
-    for partitions.
+    for partitions. Note that one can use a typed literal (e.g., date'2019-01-02') for a partition column value.

Review comment:
       Sure, done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786574829


   **[Test build #135508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135508/testReport)** for PR 30421 at commit [`4cdb4a7`](https://github.com/apache/spark/commit/4cdb4a7378c0cb5ba40c1b4cc5e9943cd55a321c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r584886041



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLInsertTestSuite.scala
##########
@@ -209,6 +211,72 @@ trait SQLInsertTestSuite extends QueryTest with SQLTestUtils {
     }
   }
 
+  test("SPARK-33474: Support typed literals as partition spec values") {
+    withTable("t1", "t3") {

Review comment:
       did we create t3?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786527744


   **[Test build #135504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135504/testReport)** for PR 30421 at commit [`5e05adb`](https://github.com/apache/spark/commit/5e05adbde1c86938d9ada0dd89b9fbd3a5cfb3eb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788534570


   **[Test build #135620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135620/testReport)** for PR 30421 at commit [`63a4fb4`](https://github.com/apache/spark/commit/63a4fb47412bf236f677bf20f980719e782b9bba).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r583528725



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLInsertTestSuite.scala
##########
@@ -208,6 +212,53 @@ trait SQLInsertTestSuite extends QueryTest with SQLTestUtils {
       checkAnswer(spark.table("t"), Row("1", null))
     }
   }
+
+  test("SPARK-33474: Support typed literals as partition spec values") {
+    withTable("t1", "t2", "t4", "t5") {
+      val binaryStr = "Spark SQL"
+      val binaryHexStr = Hex.hex(UTF8String.fromString(binaryStr).getBytes).toString
+      sql("CREATE TABLE t1(name STRING, part DATE) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t1 PARTITION(part = date'2019-01-02') VALUES('a')")
+      checkAnswer(sql("SELECT name, CAST(part AS STRING) FROM t1"), Row("a", "2019-01-02"))
+
+      sql("CREATE TABLE t2(name STRING, part TIMESTAMP) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t2 PARTITION(part = timestamp'2019-01-02 11:11:11') VALUES('a')")
+      checkAnswer(sql("SELECT name, CAST(part AS STRING) FROM t2"), Row("a", "2019-01-02 11:11:11"))
+
+      val e = intercept[AnalysisException] {
+        sql("CREATE TABLE t3(name STRING, part INTERVAL) USING PARQUET PARTITIONED BY (part)")
+      }.getMessage
+      assert(e.contains("Cannot use interval for partition column"))
+
+      sql("CREATE TABLE t4(name STRING, part BINARY) USING CSV PARTITIONED BY (part)")
+      sql(s"INSERT INTO t4 PARTITION(part = X'$binaryHexStr') VALUES('a')")
+      checkAnswer(sql("SELECT name, cast(part as string) FROM t4"), Row("a", binaryStr))
+
+      // test type conversion
+      sql("CREATE TABLE t5(name STRING, part STRING) USING CSV PARTITIONED BY (part)")
+      sql(s"INSERT INTO t5 PARTITION(part = X'$binaryHexStr') VALUES('a')")
+      sql("INSERT INTO t5 PARTITION(part = date'2019-01-02') VALUES('a')")
+      sql("INSERT INTO t5 PARTITION(part = timestamp'2019-01-02 11:11:11') VALUES('a')")
+      checkAnswer(sql("SELECT name, cast(part as string) FROM t5"),
+        Row("a", binaryStr) :: Row("a", "2019-01-02") ::
+          Row("a", "2019-01-02 11:11:11") :: Nil)
+
+      // test insert timestamp literal partition value to date partition
+      sql("INSERT INTO t1 PARTITION(part = timestamp'2019-01-02 11:11:11') VALUES('b')")
+      checkAnswer(sql("SELECT name, cast(part as string) FROM t1"),
+        Row("a", "2019-01-02") :: Row("b", "2019-01-02") :: Nil)
+      // test insert invalid binary string partition value to date partition
+      val e2 = intercept[DateTimeException] {
+        sql(s"INSERT INTO t2 PARTITION(part = X'$binaryHexStr') VALUES('a')")
+      }.getMessage
+      assert(e2 == s"Cannot cast $binaryStr to TimestampType.")
+
+      // test change work for ALTER TABLE ... ADD PARTITION(part = date'2020-01-01')

Review comment:
       > We have dedicated test suits for ADD/DROP/RENAME PARTITION, can we add tests there?
   
   yea, updated




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787855877


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40173/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788156722


   **[Test build #135602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135602/testReport)** for PR 30421 at commit [`fe5095a`](https://github.com/apache/spark/commit/fe5095a257dd9d07723e87d78882c2527778add8).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785086034


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40005/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788814092


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40234/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-732802365


   **[Test build #131646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131646/testReport)** for PR 30421 at commit [`6adefa7`](https://github.com/apache/spark/commit/6adefa76eb31c82dc44e826014a16e656da5a3b1).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r583466821



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -505,10 +505,14 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
   protected def visitStringConstant(
       ctx: ConstantContext,
       legacyNullAsString: Boolean): String = withOrigin(ctx) {
-    ctx match {
-      case _: NullLiteralContext if !legacyNullAsString => null
-      case s: StringLiteralContext => createString(s)
-      case o => o.getText
+    expression(ctx) match {
+      case Literal(null, _) if !legacyNullAsString => null
+      case l @ Literal(null, _) => l.toString
+      case l: Literal =>

Review comment:
       can we add a TODO here? For v2 commands, we will cast the string back to its actual value, which is a waste and can be improved in the future.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788322873


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135600/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r532187408



##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2583,6 +2583,27 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
       }
     }
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {

Review comment:
       > Why did you place the test in `sql/hive`? Are your changes hive specific?
   
   No , moved to sql module




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782593669


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39885/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786527744


   **[Test build #135504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135504/testReport)** for PR 30421 at commit [`5e05adb`](https://github.com/apache/spark/commit/5e05adbde1c86938d9ada0dd89b9fbd3a5cfb3eb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-732862083


   **[Test build #131646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131646/testReport)** for PR 30421 at commit [`6adefa7`](https://github.com/apache/spark/commit/6adefa76eb31c82dc44e826014a16e656da5a3b1).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741690052


   **[Test build #132489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132489/testReport)** for PR 30421 at commit [`e6092c1`](https://github.com/apache/spark/commit/e6092c1856c637e9790140531d4c13947d148796).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730181255


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35928/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787812741


   **[Test build #135592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135592/testReport)** for PR 30421 at commit [`adc3984`](https://github.com/apache/spark/commit/adc3984d73683d8eebbc723c7f6cf8b1fb007bc9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735401701


   **[Test build #131928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131928/testReport)** for PR 30421 at commit [`4898fb4`](https://github.com/apache/spark/commit/4898fb42a9400b122c8873aafb280723ac8e1a71).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733425083


   cc @MaxGekk 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-748975830


   **[Test build #133161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133161/testReport)** for PR 30421 at commit [`bc3e347`](https://github.com/apache/spark/commit/bc3e347e8bb399eb438d79748db8d9a89f176025).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734538571


   **[Test build #131856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131856/testReport)** for PR 30421 at commit [`713da66`](https://github.com/apache/spark/commit/713da66339e74ed850b314a959866479103fd8ac).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735523052


   **[Test build #131952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131952/testReport)** for PR 30421 at commit [`0228292`](https://github.com/apache/spark/commit/0228292d6c505ff2b7ab7fb02e8c084b7fad957f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733467504


   **[Test build #131735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131735/testReport)** for PR 30421 at commit [`e2749c3`](https://github.com/apache/spark/commit/e2749c3c62a22731a8659d89beb68270a679242d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734718358


   **[Test build #131861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131861/testReport)** for PR 30421 at commit [`713da66`](https://github.com/apache/spark/commit/713da66339e74ed850b314a959866479103fd8ac).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735553227






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735526287






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r584796590



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLInsertTestSuite.scala
##########
@@ -209,6 +211,43 @@ trait SQLInsertTestSuite extends QueryTest with SQLTestUtils {
     }
   }
 
+  test("SPARK-33474: Support typed literals as partition spec values") {
+    withTable("t1", "t2", "t4", "t5") {
+      val binaryStr = "Spark SQL"
+      val binaryHexStr = Hex.hex(UTF8String.fromString(binaryStr).getBytes).toString
+      sql("CREATE TABLE t1(name STRING, part DATE) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t1 PARTITION(part = date'2019-01-02') VALUES('a')")

Review comment:
       Can we create one table with many partition columns of different types, and test the positive normal cases together?
   ```
   CREATE TABLE t1(name STRING, p1 DATE, p2 TIMESTAMP, p3 ...) USING PARQUET PARTITIONED BY (p1, p2, ...)
   INSERT ...
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r579618193



##########
File path: docs/sql-migration-guide.md
##########
@@ -101,6 +101,8 @@ license: |
 
   - Since Spark 3.1, CHAR/CHARACTER and VARCHAR types are supported in the table schema. Table scan/insertion will respect the char/varchar semantic. If char/varchar is used in places other than table schema, an exception will be thrown (CAST is an exception that simply treats char/varchar as string like before). To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`.
 
+  - In Spark 3.1, we support using corresponding typed literal of partition column value type as partition column value in SQL, such as if we have a partition table with partition column of date type, we can use typed date literal `date '2020-01-01'` as partition spec `PARTITION (dt = date '2020-01-01')`, it will be treated as partition column value `2020-01-01`. In Spark 3.0 the partition value will be treated as string value `date '2020-01-01'` and it's a illegal date type string value and will been converted to `__HIVE_DEFAULT_PARTITION__`.

Review comment:
       > Should be `In Spark 3.2`?
   
   Hmm, yea, updated.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786698116


   **[Test build #135508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135508/testReport)** for PR 30421 at commit [`4cdb4a7`](https://github.com/apache/spark/commit/4cdb4a7378c0cb5ba40c1b4cc5e9943cd55a321c).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r532183247



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(

Review comment:
       Could you clarify why do you need the conversion: String -> UTF8String -> String 

##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2583,6 +2583,27 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
       }
     }
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {

Review comment:
       Why did you place the test in `sql/hive`? Are your changes hive specific?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, _: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString

Review comment:
       ```suggestion
       case Literal(data: Int, _: DateType) =>
         DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone)).format(data)
       case Literal(data: Long, _: TimestampType) =>
         TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
           .format(data)
       case Literal(data: CalendarInterval, _: CalendarIntervalType) => data.toString
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788580886


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40199/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785153242






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-732802953


   > > Yes, user can use TypeConstruct value as partition spec value such as
   > > ``
   > 
   > Is the description above incomplete? Btw, could you put an example query that this PR intends to support in the PR description?
   
   Yea... seem I forgot to save it， updated.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782561704






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r582018299



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -4021,6 +4021,27 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
       }
     }
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {
+    withTable("t1", "t2", "t4") {
+      sql("CREATE TABLE t1(name STRING, part DATE) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t1 PARTITION(part = date'2019-01-02') VALUES('a')")
+      checkAnswer(sql("SELECT name, CAST(part AS STRING) FROM t1"), Row("a", "2019-01-02"))
+
+      sql("CREATE TABLE t2(name STRING, part TIMESTAMP) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t2 PARTITION(part = timestamp'2019-01-02 11:11:11') VALUES('a')")
+      checkAnswer(sql("SELECT name, CAST(part AS STRING) FROM t2"), Row("a", "2019-01-02 11:11:11"))
+
+      val e = intercept[AnalysisException] {
+        sql("CREATE TABLE t3(name STRING, part INTERVAL) USING PARQUET PARTITIONED BY (part)")
+      }.getMessage
+      assert(e.contains("Cannot use interval for partition column"))
+
+      sql("CREATE TABLE t4(name STRING, part BINARY) USING CSV PARTITIONED BY (part)")
+      sql(s"INSERT INTO t4 PARTITION(part = X'537061726B2053514C') VALUES('a')")
+      checkAnswer(sql("SELECT name, cast(part as string) FROM t4"), Row("a", "Spark SQL"))
+    }

Review comment:
       Hmm Add some UT case. Got the concern that since now we treat partition spec use string, it's not  type safe.
   I have chat with @cloud-fan  that we can make partition sepc's value as literal. Then it will be more type safe.
   
   So should we still continue this pr or just start work on treat partition value as Literal?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786602978


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40089/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733539495






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733180683






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733419305


   **[Test build #131722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131722/testReport)** for PR 30421 at commit [`a8f26a1`](https://github.com/apache/spark/commit/a8f26a13b9b11cb591686c30e261c4a591d942da).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744266471


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37355/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735526287






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-732914432






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741694117


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132489/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730185856


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733539495






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785226292


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40009/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782683809


   **[Test build #135311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135311/testReport)** for PR 30421 at commit [`d3b1960`](https://github.com/apache/spark/commit/d3b196008a769e09fd1ef621df21f68af4e2e221).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787007693


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40111/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785157116


   **[Test build #135429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135429/testReport)** for PR 30421 at commit [`9f5d569`](https://github.com/apache/spark/commit/9f5d569021ccd2ddb4161529bf0a34af4fe97cc3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-789005658


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40244/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r583469842



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##########
@@ -2488,4 +2488,27 @@ class DDLParserSuite extends AnalysisTest {
 
     testCreateOrReplaceDdl(sql, expectedTableSpec, expectedIfNotExists = false)
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {
+    def insertPartitionPlan(part: String): InsertIntoStatement = {
+      InsertIntoStatement(
+        UnresolvedRelation(Seq("t")),
+        Map("part" -> Some(part)),
+        Seq.empty[String],
+        UnresolvedInlineTable(Seq("col1"), Seq(Seq(Literal("a")))),
+        overwrite = false, ifPartitionNotExists = false)
+    }
+
+    val dateTypeSql = "INSERT INTO t PARTITION(part = date'2019-01-02') VALUES('a')"
+    val interval = new CalendarInterval(7, 1, 1000).toString
+    val intervalTypeSql = s"INSERT INTO t PARTITION(part = interval'$interval') VALUES('a')"
+    val timestamp = "2019-01-02 11:11:11"
+    val timestampTypeSql = s"INSERT INTO t PARTITION(part = timestamp'$timestamp') VALUES('a')"
+    val binaryTypeSql = s"INSERT INTO t PARTITION(part = X'537061726B2053514C') VALUES('a')"

Review comment:
       Can we generate the binary of `537061726B2053514C` instead of hardcode it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r529210974



##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2584,6 +2584,14 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
       }
     }
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {
+    withTable("t") {
+      sql("CREATE TABLE t(name STRING) PARTITIONED BY (part DATE) STORED AS ORC")

Review comment:
       btw, why did you use `stored as orc` here for this test?

##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2584,6 +2584,14 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
       }
     }
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {
+    withTable("t") {
+      sql("CREATE TABLE t(name STRING) PARTITIONED BY (part DATE) STORED AS ORC")

Review comment:
       btw, why did you use `stored as orc` explicitly for this test?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788797059


   **[Test build #135654 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135654/testReport)** for PR 30421 at commit [`4f61e60`](https://github.com/apache/spark/commit/4f61e60f9543d7c29547dca7395ab8f9f66d30f7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744220772


   **[Test build #132753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132753/testReport)** for PR 30421 at commit [`06a9321`](https://github.com/apache/spark/commit/06a9321ca7202c8bd266514e082bac7e5a3abeb5).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788166073


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40180/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733604649






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r531335405



##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -198,6 +198,21 @@ SELECT * FROM students;
 +-------------+-------------------------+----------+
 ```
 
+#### Insert Using a Type Constructed Partition Column Value

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785239922


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40009/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788354558


   **[Test build #135602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135602/testReport)** for PR 30421 at commit [`fe5095a`](https://github.com/apache/spark/commit/fe5095a257dd9d07723e87d78882c2527778add8).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734650163






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782607578


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135305/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741690052


   **[Test build #132489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132489/testReport)** for PR 30421 at commit [`e6092c1`](https://github.com/apache/spark/commit/e6092c1856c637e9790140531d4c13947d148796).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730185862


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35926/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785351205


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135429/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-750857994


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37941/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741779013


   **[Test build #132495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132495/testReport)** for PR 30421 at commit [`06a9321`](https://github.com/apache/spark/commit/06a9321ca7202c8bd266514e082bac7e5a3abeb5).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r585199108



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLInsertTestSuite.scala
##########
@@ -209,6 +211,72 @@ trait SQLInsertTestSuite extends QueryTest with SQLTestUtils {
     }
   }
 
+  test("SPARK-33474: Support typed literals as partition spec values") {
+    withTable("t1", "t3") {

Review comment:
       > did we create t3?
   
   pushed failed yesterday, updated now

##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -126,7 +126,7 @@ ALTER TABLE table_identifier ADD [IF NOT EXISTS]
 
 * **partition_spec**
 
-    Partition to be added.
+    Partition to be added. Note that one can use a typed literal (e.g., date'2019-01-02') for a partition column value.

Review comment:
       Done

##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -49,7 +49,7 @@ ALTER TABLE table_identifier partition_spec RENAME TO partition_spec
 
 * **partition_spec**
 
-    Partition to be renamed.
+    Partition to be renamed. Note that one can use a typed literal (e.g., date'2019-01-02') for a partition column value.

Review comment:
       > Note that one can use types literals (e.g. ...) in the partition spec.
   
   Done

##########
File path: docs/sql-migration-guide.md
##########
@@ -65,6 +65,8 @@ license: |
  
   - In Spark 3.2, the output schema of `SHOW TBLPROPERTIES` becomes `key: string, value: string` whether you specify the table property key or not. In Spark 3.1 and earlier, the output schema of `SHOW TBLPROPERTIES` is `value: string` when you specify the table property key. To restore the old schema with the builtin catalog, you can set `spark.sql.legacy.keepCommandOutputSchema` to `true`.
 
+  - In Spark 3.2, we support a typed literal for a partition constant value in a partition spec clause. For example, a right-side constant value in `PARTITION (dt = date'2020-01-01')` is parsed as a date-typed literal in the partition spec. In Spark 3.1 and earlier, the partition value will be treated as string value `date '2020-01-01'` and it's a illegal date type string value and will be converted to `__HIVE_DEFAULT_PARTITION__`.

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735508672


   **[Test build #131943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131943/testReport)** for PR 30421 at commit [`e875bcf`](https://github.com/apache/spark/commit/e875bcfe3ff487a5cd7669ed924c1313638d4c88).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744266507


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37355/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734635388


   **[Test build #131861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131861/testReport)** for PR 30421 at commit [`713da66`](https://github.com/apache/spark/commit/713da66339e74ed850b314a959866479103fd8ac).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r584881851



##########
File path: docs/sql-migration-guide.md
##########
@@ -65,6 +65,8 @@ license: |
  
   - In Spark 3.2, the output schema of `SHOW TBLPROPERTIES` becomes `key: string, value: string` whether you specify the table property key or not. In Spark 3.1 and earlier, the output schema of `SHOW TBLPROPERTIES` is `value: string` when you specify the table property key. To restore the old schema with the builtin catalog, you can set `spark.sql.legacy.keepCommandOutputSchema` to `true`.
 
+  - In Spark 3.2, we support a typed literal for a partition constant value in a partition spec clause. For example, a right-side constant value in `PARTITION (dt = date'2020-01-01')` is parsed as a date-typed literal in the partition spec. In Spark 3.1 and earlier, the partition value will be treated as string value `date '2020-01-01'` and it's a illegal date type string value and will be converted to `__HIVE_DEFAULT_PARTITION__`.

Review comment:
       ```
   In Spark 3.2, we support typed literals in the partition spec of INSERT and ADD/DROP/RENAME PARTITION.
   For example, `ADD PARTITION(dt = date'2020-01-01')` adds a partition with date value `2020-01-01`.
   In Spark 3.1 and earlier, the partition value will be parsed as string value `date '2020-01-01', which is
   an illegal date value and we add a partition with null value at the end.
   ```
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733282467






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782568973


   **[Test build #135298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135298/testReport)** for PR 30421 at commit [`b68ee81`](https://github.com/apache/spark/commit/b68ee8196f072cedb151ffc623a7c4e8c7755d1d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735607652






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735574727


   **[Test build #131943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131943/testReport)** for PR 30421 at commit [`e875bcf`](https://github.com/apache/spark/commit/e875bcfe3ff487a5cd7669ed924c1313638d4c88).
    * This patch **fails SparkR unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-748975830


   **[Test build #133161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133161/testReport)** for PR 30421 at commit [`bc3e347`](https://github.com/apache/spark/commit/bc3e347e8bb399eb438d79748db8d9a89f176025).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r530064053



##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -41,7 +41,7 @@ INSERT INTO [ TABLE ] table_identifier [ partition_spec ]
 * **partition_spec**
 
     An optional parameter that specifies a comma separated list of key and value pairs
-    for partitions.
+    for partitions. Here we support use type constructed value as partition col value.

Review comment:
       Use  `typed literal` to keep  same with 
   ```
     /**
      * Create a typed Literal expression. A typed literal has the following SQL syntax:
      * {{{
      *   [TYPE] '[VALUE]'
      * }}}
      * Currently Date, Timestamp, Interval and Binary typed literals are supported.
      */
     override def visitTypeConstructor(ctx: TypeConstructorContext): Literal = withOrigin(ctx) {
    
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741693204


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37091/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782594834


   **[Test build #135298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135298/testReport)** for PR 30421 at commit [`b68ee81`](https://github.com/apache/spark/commit/b68ee8196f072cedb151ffc623a7c4e8c7755d1d).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786717938


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135508/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785041889


   **[Test build #135425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135425/testReport)** for PR 30421 at commit [`2e9058a`](https://github.com/apache/spark/commit/2e9058a2512aa2aa6b87b4ef6cdca0c80c2c4f36).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-750842261


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133350/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788150446


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40181/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787812318


   > > if partition column is date, if we pass a partition value as Spark SQL, DSV2 will treat it as null, but others will throw exception that Spark SQL can't been convert to TimestampType.
   > 
   > Can you create a JIRA for this bug?
   
   Sure, will raise a JIRA with how to reproduce.
   
   > Do we allow it anywhere? I don't think Spark allows to write out interval values.
   
   Maybe it's just a different way to show error make me misunderstood. 
   
   Hmmm, any more suggestion about current PR?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-789513768


   Thanks! Merged to master.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-750857994


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37941/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788534570


   **[Test build #135620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135620/testReport)** for PR 30421 at commit [`63a4fb4`](https://github.com/apache/spark/commit/63a4fb47412bf236f677bf20f980719e782b9bba).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787009802


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40111/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r529206500



##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2584,6 +2584,14 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
       }
     }
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {
+    withTable("t") {
+      sql("CREATE TABLE t(name STRING) PARTITIONED BY (part DATE) STORED AS ORC")

Review comment:
       Please add tests for all the types and add tests in `DDLParserSuite`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733031728


   **[Test build #131672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131672/testReport)** for PR 30421 at commit [`e312697`](https://github.com/apache/spark/commit/e312697c7e6e9feebed0be15cd2bec0f829bbf49).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r529504680



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypeConstructedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, dataType: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, dataType: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, dataType: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString
+    case Literal(data: Array[Byte], dataType: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(data, dataType) =>
+      UTF8String.fromString(data.toString).toString

Review comment:
       > We need this entry? What's an example to match this case?
   
   Normally this won't happen, just remove this match case?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744380137


   **[Test build #132753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132753/testReport)** for PR 30421 at commit [`06a9321`](https://github.com/apache/spark/commit/06a9321ca7202c8bd266514e082bac7e5a3abeb5).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730160635


   **[Test build #131324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131324/testReport)** for PR 30421 at commit [`c9a97d0`](https://github.com/apache/spark/commit/c9a97d06897014f25a2a0be4c085c5cc6a008608).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744220772


   **[Test build #132753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132753/testReport)** for PR 30421 at commit [`06a9321`](https://github.com/apache/spark/commit/06a9321ca7202c8bd266514e082bac7e5a3abeb5).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733031728


   **[Test build #131672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131672/testReport)** for PR 30421 at commit [`e312697`](https://github.com/apache/spark/commit/e312697c7e6e9feebed0be15cd2bec0f829bbf49).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735523052


   **[Test build #131952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131952/testReport)** for PR 30421 at commit [`0228292`](https://github.com/apache/spark/commit/0228292d6c505ff2b7ab7fb02e8c084b7fad957f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730194576






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r531283037



##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -41,7 +41,7 @@ INSERT INTO [ TABLE ] table_identifier [ partition_spec ]
 * **partition_spec**
 
     An optional parameter that specifies a comma separated list of key and value pairs
-    for partitions.
+    for partitions. Note that one can use a typed value (e.g., date'2019-01-02') for a partition column value.

Review comment:
       nit: `a typed value` -> `a typed literal`?

##########
File path: docs/sql-migration-guide.md
##########
@@ -52,6 +52,8 @@ license: |
 
   - In Spark 3.1, refreshing a table will trigger an uncache operation for all other caches that reference the table, even if the table itself is not cached. In Spark 3.0 the operation will only be triggered if the table itself is cached.
 
+  - In Spark 3.1, we support typed literal as partition column value, such as `PARTITION (dt = date '2020-01-01')`, it will be treated as partition column value `2020-01-01`. In Spark 3.0 the partition value will be treated as `date '2020-01-01'`.

Review comment:
       > we support typed literal as partition column value
   
   Do you mean `we support a typed literal for binary, date, timestamp, and interval in the INSERT INTO statement`?
   
   > it will be treated as partition column value `2020-01-01`. In Spark 3.0 the partition value will be treated as `date '2020-01-01'`.
   
   What does the statement above mean? For instance, the partition value of the example query you added in the doc page is null;
   ```
   scala> sql("""CREATE TABLE students (name STRING, address  STRING) PARTITIONED BY (birthday DATE);""")
   scala> sql("""INSERT INTO students PARTITION (birthday = date'2019-01-02') VALUES ('Amy Smith', '123 Park Ave, San Jose');""")
   scala> sql("""SELECT * FROM students;""").show()
   +---------+--------------------+--------+
   |     name|             address|birthday|
   +---------+--------------------+--------+
   |Amy Smith|123 Park Ave, San...|    null|
   +---------+--------------------+--------+
   ```
   Could you make it clearer so that users can understand smoothly?

##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -198,6 +198,21 @@ SELECT * FROM students;
 +-------------+-------------------------+----------+
 ```
 
+#### Insert Using a Type Constructed Partition Column Value

Review comment:
       `Insert Using a Typed Date Literal for a Partition Column Value`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782607578


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135305/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741815127


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132495/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r532186917



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, _: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString

Review comment:
       Updated




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r581935782



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -4021,6 +4021,27 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
       }
     }
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {
+    withTable("t1", "t2", "t4") {
+      sql("CREATE TABLE t1(name STRING, part DATE) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t1 PARTITION(part = date'2019-01-02') VALUES('a')")
+      checkAnswer(sql("SELECT name, CAST(part AS STRING) FROM t1"), Row("a", "2019-01-02"))
+
+      sql("CREATE TABLE t2(name STRING, part TIMESTAMP) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t2 PARTITION(part = timestamp'2019-01-02 11:11:11') VALUES('a')")
+      checkAnswer(sql("SELECT name, CAST(part AS STRING) FROM t2"), Row("a", "2019-01-02 11:11:11"))
+
+      val e = intercept[AnalysisException] {
+        sql("CREATE TABLE t3(name STRING, part INTERVAL) USING PARQUET PARTITIONED BY (part)")
+      }.getMessage
+      assert(e.contains("Cannot use interval for partition column"))
+
+      sql("CREATE TABLE t4(name STRING, part BINARY) USING CSV PARTITIONED BY (part)")
+      sql(s"INSERT INTO t4 PARTITION(part = X'537061726B2053514C') VALUES('a')")
+      checkAnswer(sql("SELECT name, cast(part as string) FROM t4"), Row("a", "Spark SQL"))
+    }

Review comment:
       For  these three concern
   >  All the literals are supported. Non-literals are forbidden. e.g. part_col=array(1) does not create a string value "array(1)".
   
   Now type constructor only support `DATE` `TIMESTAMP`  `INTERVAL` `X`, so that's not a concern.
   
    >  Null literal is supported. We should use null instead of "null" to represent it.
   
   It's solved  since #30538 
   
    >  If the literal data type doesn't match the partition column data type, we should do type check and cast like normal table insertion.
   
   Seems there is auto type conversion. I will check this with UT
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730228128


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35935/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786614218


   **[Test build #135504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135504/testReport)** for PR 30421 at commit [`5e05adb`](https://github.com/apache/spark/commit/5e05adbde1c86938d9ada0dd89b9fbd3a5cfb3eb).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r533436952



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -509,13 +509,28 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone)).format(data)
+    case Literal(data: Long, _: TimestampType) =>
+      TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+        .format(data)
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) => data.toString
+    case Literal(data: Array[Byte], _: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(_, dataType) =>
+      throw new IllegalArgumentException(s"Literals of type '$dataType' are not" +
+        " supported when use typed literal as partition col value.")
+  }
+
   /**
    * Convert a constant of any type into a string. This is typically used in DDL commands, and its
    * main purpose is to prevent slight differences due to back to back conversions i.e.:
    * String -> Literal -> String.
    */
   protected def visitStringConstant(ctx: ConstantContext): String = withOrigin(ctx) {

Review comment:
       > ... but Cast.eval can't run here.but Cast.eval can't run here.
   
   Why? What's the error?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782695615


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135311/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786999972


   Hmmm， seems DSV2 have many different behaviors.
   1. can't use interval as table schema?
   2. if partition column is date, if we pass a partition value as `Spark SQL`, DSV2 will treat it as null, but others will throw exception that `Spark SQL` can't convert to TimestampType.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782557674


   **[Test build #135294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135294/testReport)** for PR 30421 at commit [`0b4b211`](https://github.com/apache/spark/commit/0b4b211ad03a5c60fde46bd4fa43c6cbcff90b94).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r581885977



##########
File path: docs/sql-migration-guide.md
##########
@@ -63,6 +63,8 @@ license: |
  
   - In Spark 3.2, the output schema of `SHOW TBLPROPERTIES` becomes `key: string, value: string` whether you specify the table property key or not. In Spark 3.1 and earlier, the output schema of `SHOW TBLPROPERTIES` is `value: string` when you specify the table property key. To restore the old schema with the builtin catalog, you can set `spark.sql.legacy.keepCommandOutputSchema` to `true`.
 
+  - In Spark 3.2, we support using corresponding typed literal of partition column value type as partition column value in SQL, such as if we have a partition table with partition column of date type, we can use typed date literal `date '2020-01-01'` as partition spec `PARTITION (dt = date '2020-01-01')`, it will be treated as partition column value `2020-01-01`. In Spark 3.1 and earlier, the partition value will be treated as string value `date '2020-01-01'` and it's a illegal date type string value and will been converted to `__HIVE_DEFAULT_PARTITION__`.

Review comment:
       How about simply saying it like this?
   ```
   In Spark 3.2, we support a typed literal for a partition constant value in a INSERT clause. 
   For example, a right-side constant value in `PARTITION (dt = date'2020-01-01')` is parsed
   as a date-typed literal in the partition spec. In Spark 3.1 and earlier, ...
   ```

##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -40,8 +40,8 @@ INSERT INTO [ TABLE ] table_identifier [ partition_spec ] [ ( column_list ) ]
 
 * **partition_spec**
 
-    An optional parameter that specifies a comma-separated list of key and value pairs
-    for partitions.
+    An optional parameter that specifies a comma separated list of key and value pairs

Review comment:
       nit: why did you changed `comma-separated` => `comma separated`?

##########
File path: docs/sql-ref-syntax-dml-insert-overwrite-table.md
##########
@@ -40,8 +40,8 @@ INSERT OVERWRITE [ TABLE ] table_identifier [ partition_spec [ IF NOT EXISTS ] ]
 
 * **partition_spec**
 
-    An optional parameter that specifies a comma-separated list of key and value pairs
-    for partitions.
+    An optional parameter that specifies a comma separated list of key and value pairs

Review comment:
       ditto

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -4021,6 +4021,27 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
       }
     }
   }
+
+  test("SPARK-33474: Support TypeConstructed partition spec value") {
+    withTable("t1", "t2", "t4") {
+      sql("CREATE TABLE t1(name STRING, part DATE) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t1 PARTITION(part = date'2019-01-02') VALUES('a')")
+      checkAnswer(sql("SELECT name, CAST(part AS STRING) FROM t1"), Row("a", "2019-01-02"))
+
+      sql("CREATE TABLE t2(name STRING, part TIMESTAMP) USING PARQUET PARTITIONED BY (part)")
+      sql("INSERT INTO t2 PARTITION(part = timestamp'2019-01-02 11:11:11') VALUES('a')")
+      checkAnswer(sql("SELECT name, CAST(part AS STRING) FROM t2"), Row("a", "2019-01-02 11:11:11"))
+
+      val e = intercept[AnalysisException] {
+        sql("CREATE TABLE t3(name STRING, part INTERVAL) USING PARQUET PARTITIONED BY (part)")
+      }.getMessage
+      assert(e.contains("Cannot use interval for partition column"))
+
+      sql("CREATE TABLE t4(name STRING, part BINARY) USING CSV PARTITIONED BY (part)")
+      sql(s"INSERT INTO t4 PARTITION(part = X'537061726B2053514C') VALUES('a')")
+      checkAnswer(sql("SELECT name, cast(part as string) FROM t4"), Row("a", "Spark SQL"))
+    }

Review comment:
       The last thing that I'm concerned about is whether we already have tests corresponding to the @cloud-fan 's last comment.
   
   https://github.com/apache/spark/pull/30421#issuecomment-734049999
   ```
   Let's make sure this feature works correctly:
   
    - All the literals are supported. Non-literals are forbidden. e.g. part_col=array(1) does not create a string value "array(1)".
    - Null literal is supported. We should use null instead of "null" to represent it.
    - If the literal data type doesn't match the partition column data type, we should do type check and cast like normal table insertion.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733035708






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-738683038


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36808/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r529206798



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  def convertTypeConstructedLiteralToString(literal: Literal): String = literal match {

Review comment:
       private?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733419305


   **[Test build #131722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131722/testReport)** for PR 30421 at commit [`a8f26a1`](https://github.com/apache/spark/commit/a8f26a13b9b11cb591686c30e261c4a591d942da).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788116748


   **[Test build #135599 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135599/testReport)** for PR 30421 at commit [`2ef0fd0`](https://github.com/apache/spark/commit/2ef0fd0e70de5a4856fafb88bd42aa61afbad702).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787042588


   **[Test build #135530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135530/testReport)** for PR 30421 at commit [`093b062`](https://github.com/apache/spark/commit/093b0628b42f237eb1f91b5fbbba6fafb8986508).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788156722


   **[Test build #135602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135602/testReport)** for PR 30421 at commit [`fe5095a`](https://github.com/apache/spark/commit/fe5095a257dd9d07723e87d78882c2527778add8).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r530099993



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, _: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString
+    case Literal(data: Array[Byte], _: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(_, dataType) =>
+      throw new IllegalArgumentException(s"Literals of type '$dataType' are not" +
+        " supported when use typed literal as partition col value.")
+  }
+
   /**
    * Convert a constant of any type into a string. This is typically used in DDL commands, and its
    * main purpose is to prevent slight differences due to back to back conversions i.e.:
    * String -> Literal -> String.
    */
   protected def visitStringConstant(ctx: ConstantContext): String = withOrigin(ctx) {
     ctx match {
+      case l: TypeConstructorContext => convertTypedLiteralToString(visitTypeConstructor(l))
       case s: StringLiteralContext => createString(s)
       case o => o.getText

Review comment:
       > previous, `date'xxx'` is just a string `"date'xxx'"`? then we need a migration guide for this change.
   
   Updated




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786622468


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40089/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733174299






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r583471240



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -4022,6 +4023,46 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     }
   }
 
+  test("SPARK-33474: Support TypeConstructed partition spec value") {

Review comment:
       let's move the test to `SQLInsertTestSuite`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734631335


   **[Test build #131856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131856/testReport)** for PR 30421 at commit [`713da66`](https://github.com/apache/spark/commit/713da66339e74ed850b314a959866479103fd8ac).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r530088471



##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -41,7 +41,7 @@ INSERT INTO [ TABLE ] table_identifier [ partition_spec ]
 * **partition_spec**
 
     An optional parameter that specifies a comma separated list of key and value pairs
-    for partitions.
+    for partitions. Here we support use type constructed value as partition col value.

Review comment:
       Ah, I see. Do you know what's called in Hive?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730372131






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735414019


   **[Test build #131932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131932/testReport)** for PR 30421 at commit [`e875bcf`](https://github.com/apache/spark/commit/e875bcfe3ff487a5cd7669ed924c1313638d4c88).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782558191


   **[Test build #135294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135294/testReport)** for PR 30421 at commit [`0b4b211`](https://github.com/apache/spark/commit/0b4b211ad03a5c60fde46bd4fa43c6cbcff90b94).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787812741


   **[Test build #135592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135592/testReport)** for PR 30421 at commit [`adc3984`](https://github.com/apache/spark/commit/adc3984d73683d8eebbc723c7f6cf8b1fb007bc9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730203617


   **[Test build #131330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131330/testReport)** for PR 30421 at commit [`c9a97d0`](https://github.com/apache/spark/commit/c9a97d06897014f25a2a0be4c085c5cc6a008608).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788175913


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40181/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-738754519


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132208/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733419965






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735402658






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r533410197



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -509,13 +509,28 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone)).format(data)
+    case Literal(data: Long, _: TimestampType) =>
+      TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+        .format(data)
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) => data.toString
+    case Literal(data: Array[Byte], _: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(_, dataType) =>
+      throw new IllegalArgumentException(s"Literals of type '$dataType' are not" +
+        " supported when use typed literal as partition col value.")
+  }
+
   /**
    * Convert a constant of any type into a string. This is typically used in DDL commands, and its
    * main purpose is to prevent slight differences due to back to back conversions i.e.:
    * String -> Literal -> String.
    */
   protected def visitStringConstant(ctx: ConstantContext): String = withOrigin(ctx) {

Review comment:
       > can we support all the literals here? e.g.
   > 
   > ```
   > expression(ctx) match {
   >   case l: Litral => if (l.value == null) null else Cast(l, StringType, timezone).eval()
   >   case _ => fail... // this should never happen
   > }
   > ```
   
   your mean support all `constant` here
   ```
   constant
       : NULL                                                                                     #nullLiteral
       | interval                                                                                 #intervalLiteral
       | identifier STRING                                                                        #typeConstructor
       | number                                                                                   #numericLiteral
       | booleanValue                                                                             #booleanLiteral
       | STRING+                                                                                  #stringLiteral
       ;
   ```
   
   and use `expression` to visit this constant then we don't need so many convert logic.
   In this way we should add a new method such as :
   ```
     protected def visitConstant(ctx: ConstantContext): String = withOrigin(ctx) {
       expression(ctx) match {
         case l: Literal => if (l.value == null) {
           null
         } else {
           Cast(l, StringType, Some(SQLConf.get.sessionLocalTimeZone)).eval()
         }
         case _ => 
           throw IllegalArgumentException("")
       }
     }
   ```
   I have tried similar way  before, but `Cast.eval` can't run here.
   I have tried change `CastBase` code and extract `cast` functions as Half-life object class `object CastBase`. Then call  `CastBase.castToString(literal)`.
   
   IMO, we don't need to expand the scope of the modification, but your suggestion is pretty good when we change partition spec to `Map[String, Literal]`.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733324368






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735414019


   **[Test build #131932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131932/testReport)** for PR 30421 at commit [`e875bcf`](https://github.com/apache/spark/commit/e875bcfe3ff487a5cd7669ed924c1313638d4c88).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782695615


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135311/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r584882458



##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -49,7 +49,7 @@ ALTER TABLE table_identifier partition_spec RENAME TO partition_spec
 
 * **partition_spec**
 
-    Partition to be renamed.
+    Partition to be renamed. Note that one can use a typed literal (e.g., date'2019-01-02') for a partition column value.

Review comment:
       Note that one can use types literals (e.g. ...) in the partition spec.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r530070612



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, _: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString
+    case Literal(data: Array[Byte], _: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(_, dataType) =>
+      throw new IllegalArgumentException(s"Literals of type '$dataType' are not" +
+        " supported when use typed literal as partition col value.")
+  }
+
   /**
    * Convert a constant of any type into a string. This is typically used in DDL commands, and its
    * main purpose is to prevent slight differences due to back to back conversions i.e.:
    * String -> Literal -> String.
    */
   protected def visitStringConstant(ctx: ConstantContext): String = withOrigin(ctx) {
     ctx match {
+      case l: TypeConstructorContext => convertTypedLiteralToString(visitTypeConstructor(l))

Review comment:
       > how about null literal?
   
   Null literal seems won't be supported by `visitTypeConstructor`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730201327






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733035708






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741779013


   **[Test build #132495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132495/testReport)** for PR 30421 at commit [`06a9321`](https://github.com/apache/spark/commit/06a9321ca7202c8bd266514e082bac7e5a3abeb5).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788118950


   **[Test build #135600 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135600/testReport)** for PR 30421 at commit [`4023041`](https://github.com/apache/spark/commit/4023041f421c6c9297ffb1cbf227d213ac776a10).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788319191


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135599/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788009788


   > 2. if partition column is date, if we pass a partition value as `Spark SQL`, DSV2 will treat it as null, but others will throw exception that `Spark SQL` can't been convert to TimestampType.
   
   @cloud-fan Raise a PR with UT demo https://issues.apache.org/jira/browse/SPARK-34584


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735441808


   **[Test build #131932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131932/testReport)** for PR 30421 at commit [`e875bcf`](https://github.com/apache/spark/commit/e875bcfe3ff487a5cd7669ed924c1313638d4c88).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782596071


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39885/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733536615






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788187240


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40183/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785281765


   **[Test build #135426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135426/testReport)** for PR 30421 at commit [`31c1092`](https://github.com/apache/spark/commit/31c1092103d6d83b4b8ca471f10ddaaaa053aba0).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r530021740



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypeConstructedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, dataType: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, dataType: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, dataType: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString
+    case Literal(data: Array[Byte], dataType: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(data, dataType) =>
+      UTF8String.fromString(data.toString).toString

Review comment:
       Yea, that's the effect of the recent compiler update: https://github.com/apache/spark/pull/30455




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788814092


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40234/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r532307721



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, _: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString

Review comment:
       @MaxGekk  Use BinaryType as partition column in DataSource API have bug so I remove test case of binary type. I will raise a issue later  about this bug.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730450448


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35956/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730200930


   **[Test build #131324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131324/testReport)** for PR 30421 at commit [`c9a97d0`](https://github.com/apache/spark/commit/c9a97d06897014f25a2a0be4c085c5cc6a008608).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-732802365


   **[Test build #131646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131646/testReport)** for PR 30421 at commit [`6adefa7`](https://github.com/apache/spark/commit/6adefa76eb31c82dc44e826014a16e656da5a3b1).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-749114691


   **[Test build #133161 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133161/testReport)** for PR 30421 at commit [`bc3e347`](https://github.com/apache/spark/commit/bc3e347e8bb399eb438d79748db8d9a89f176025).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r529597136



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypeConstructedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, dataType: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, dataType: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, dataType: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString
+    case Literal(data: Array[Byte], dataType: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(data, dataType) =>
+      UTF8String.fromString(data.toString).toString

Review comment:
       > We need this entry? What's an example to match this case?
   
   removed this line




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744215910


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741824016


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37097/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785110434


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40005/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782599904


   **[Test build #135311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135311/testReport)** for PR 30421 at commit [`d3b1960`](https://github.com/apache/spark/commit/d3b196008a769e09fd1ef621df21f68af4e2e221).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782604237


   **[Test build #135305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135305/testReport)** for PR 30421 at commit [`dc0783a`](https://github.com/apache/spark/commit/dc0783afd4be103750cf3918c51e728a46ff54ad).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734049999


   Let's do it step by step, and support typed literal first. We can figure out how to eliminate the string <-> actual value roundtrip in v2 commands later.
   
   Let's make sure this feature works correctly:
   1. All the literals are supported. Non-literals are forbidden. e.g. `part_col=array(1)` does not create a string value "array(1)".
   2. Null literal is supported. We should use null instead of "null" to represent it.
   3. If the literal data type doesn't match the partition column data type, we should do type check and cast like normal table insertion.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785327667


   **[Test build #135429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135429/testReport)** for PR 30421 at commit [`9f5d569`](https://github.com/apache/spark/commit/9f5d569021ccd2ddb4161529bf0a34af4fe97cc3).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734729713






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788965356


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135654/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735441927






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735597050


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782558456


   **[Test build #135295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135295/testReport)** for PR 30421 at commit [`7264a3d`](https://github.com/apache/spark/commit/7264a3d19e9e55b15bfcd3d1b902154d537c04cc).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733017916


   > Could you add some descriptions about this syntax and add examples in the `INSERT` document?: https://spark.apache.org/docs/3.0.1/sql-ref-syntax-dml-insert-into.html#parameters
   
   Updated


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744248284


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37355/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-738666071


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36808/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735607652






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-738618950


   **[Test build #132208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132208/testReport)** for PR 30421 at commit [`055a903`](https://github.com/apache/spark/commit/055a903d148f91d74ac60b8d319425a1f5c2f7ee).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788116748


   **[Test build #135599 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135599/testReport)** for PR 30421 at commit [`2ef0fd0`](https://github.com/apache/spark/commit/2ef0fd0e70de5a4856fafb88bd42aa61afbad702).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787855912


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40173/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785090224


   **[Test build #135426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135426/testReport)** for PR 30421 at commit [`31c1092`](https://github.com/apache/spark/commit/31c1092103d6d83b4b8ca471f10ddaaaa053aba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734632234


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734634980






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734094780


   Any more comment on current PR? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-749126122


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133161/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730370667


   **[Test build #131330 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131330/testReport)** for PR 30421 at commit [`c9a97d0`](https://github.com/apache/spark/commit/c9a97d06897014f25a2a0be4c085c5cc6a008608).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788001446


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135592/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788362262


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135602/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787044010


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135530/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786622468


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40089/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730201343


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131324/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735427100






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741694069


   **[Test build #132489 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132489/testReport)** for PR 30421 at commit [`e6092c1`](https://github.com/apache/spark/commit/e6092c1856c637e9790140531d4c13947d148796).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `            logInfo(s\"Adding $url to class loader\")`
     * `class OneVsRest(Estimator, _OneVsRestParams, HasParallelism, MLReadable, MLWritable):`
     * `class OneVsRestReader(MLReader):`
     * `class OneVsRestWriter(MLWriter):`
     * `class OneVsRestModel(Model, _OneVsRestParams, MLReadable, MLWritable):`
     * `class OneVsRestModelReader(MLReader):`
     * `class OneVsRestModelWriter(MLWriter):`
     * `class CrossValidatorReader(MLReader):`
     * `class CrossValidatorWriter(MLWriter):`
     * `class CrossValidatorModelReader(MLReader):`
     * `class CrossValidatorModelWriter(MLWriter):`
     * `class TrainValidationSplitReader(MLReader):`
     * `class TrainValidationSplitWriter(MLWriter):`
     * `class TrainValidationSplitModelReader(MLReader):`
     * `class TrainValidationSplitModelWriter(MLWriter):`
     * `trait CheckAnalysis extends PredicateHelper with LookupCatalog `
     * `case class UnresolvedView(`
     * `case class TemporaryViewRelation(tableMeta: CatalogTable) extends LeafNode `
     * `case class CreateNamedStruct(children: Seq[Expression]) extends Expression with NoThrow `
     * `case class DateFromUnixDate(child: Expression) extends UnaryExpression`
     * `case class UnixDate(child: Expression) extends UnaryExpression`
     * `case class UnixSeconds(child: Expression) extends TimestampToLongBase `
     * `case class UnixMillis(child: Expression) extends TimestampToLongBase `
     * `case class UnixMicros(child: Expression) extends TimestampToLongBase `
     * `case class MakeDate(`
     * `abstract class LikeAllBase extends MultiLikeBase `
     * `abstract class LikeAnyBase extends MultiLikeBase `
     * `case class LikeAny(child: Expression, patterns: Seq[UTF8String]) extends LikeAnyBase `
     * `case class NotLikeAny(child: Expression, patterns: Seq[UTF8String]) extends LikeAnyBase `
     * `case class NoopCommand(`
     * `case class ShowTableExtended(`
     * `case class DropView(`
     * `case class RepairTable(child: LogicalPlan) extends Command `
     * `class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging `
     * `class StreamingQueryListenerBus(sparkListenerBus: Option[LiveListenerBus])`
     * `case class StateSchemaNotCompatible(message: String) extends Exception(message)`
     * `class StateSchemaCompatibilityChecker(`
     * `class StreamingQueryHistoryServerPlugin extends AppHistoryServerPlugin `
     * `class StreamingQueryStatusStore(store: KVStore) `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733419950


   **[Test build #131722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131722/testReport)** for PR 30421 at commit [`a8f26a1`](https://github.com/apache/spark/commit/a8f26a13b9b11cb591686c30e261c4a591d942da).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r529498575



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypeConstructedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, dataType: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, dataType: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, dataType: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString
+    case Literal(data: Array[Byte], dataType: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(data, dataType) =>
+      UTF8String.fromString(data.toString).toString

Review comment:
       We need this entry? What's an example to match this case?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788319191


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135599/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r579617798



##########
File path: docs/sql-migration-guide.md
##########
@@ -101,6 +101,8 @@ license: |
 
   - Since Spark 3.1, CHAR/CHARACTER and VARCHAR types are supported in the table schema. Table scan/insertion will respect the char/varchar semantic. If char/varchar is used in places other than table schema, an exception will be thrown (CAST is an exception that simply treats char/varchar as string like before). To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`.
 
+  - In Spark 3.1, we support using corresponding typed literal of partition column value type as partition column value in SQL, such as if we have a partition table with partition column of date type, we can use typed date literal `date '2020-01-01'` as partition spec `PARTITION (dt = date '2020-01-01')`, it will be treated as partition column value `2020-01-01`. In Spark 3.0 the partition value will be treated as string value `date '2020-01-01'` and it's a illegal date type string value and will been converted to `__HIVE_DEFAULT_PARTITION__`.

Review comment:
       Should be `In Spark 3.2`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785041889


   **[Test build #135425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135425/testReport)** for PR 30421 at commit [`2e9058a`](https://github.com/apache/spark/commit/2e9058a2512aa2aa6b87b4ef6cdca0c80c2c4f36).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] rdblue commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733979208


   In general, I support the idea of moving away from tracking partition values using String. I'm not convinced that using a Literal would work much more easily because literal values would need to be cast to the correct type when constructing an `InternalRow` as well. But the idea seems like it's going in the right direction.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786717938


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135508/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787005890


   **[Test build #135530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135530/testReport)** for PR 30421 at commit [`093b062`](https://github.com/apache/spark/commit/093b0628b42f237eb1f91b5fbbba6fafb8986508).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r531347720



##########
File path: docs/sql-migration-guide.md
##########
@@ -52,6 +52,8 @@ license: |
 
   - In Spark 3.1, refreshing a table will trigger an uncache operation for all other caches that reference the table, even if the table itself is not cached. In Spark 3.0 the operation will only be triggered if the table itself is cached.
 
+  - In Spark 3.1, we support typed literal as partition column value, such as `PARTITION (dt = date '2020-01-01')`, it will be treated as partition column value `2020-01-01`. In Spark 3.0 the partition value will be treated as `date '2020-01-01'`.

Review comment:
       Updated, now maybe more clear?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735577041






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744394112


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132753/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735500581


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-748982141


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37760/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-741693204


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37091/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-750848242


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37941/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-734634982






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735608543


   **[Test build #131967 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131967/testReport)** for PR 30421 at commit [`0228292`](https://github.com/apache/spark/commit/0228292d6c505ff2b7ab7fb02e8c084b7fad957f).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730246400






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733099339


   **[Test build #131675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131675/testReport)** for PR 30421 at commit [`edd270e`](https://github.com/apache/spark/commit/edd270eaf1725939ecaee384c57bfa6b9036e521).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-744266507






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730185837


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35926/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730246400






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r532187359



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(

Review comment:
       > Could you clarify why do you need the conversion: String -> UTF8String -> String
   
   Forgot where I see similar logical, emmm, so this place don't need this convert to keep standard charset UTF8




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730194556


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35928/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787005890


   **[Test build #135530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135530/testReport)** for PR 30421 at commit [`093b062`](https://github.com/apache/spark/commit/093b0628b42f237eb1f91b5fbbba6fafb8986508).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-786557343


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40085/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r584844133



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -509,10 +509,16 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
   protected def visitStringConstant(
       ctx: ConstantContext,
       legacyNullAsString: Boolean): String = withOrigin(ctx) {
-    ctx match {
-      case _: NullLiteralContext if !legacyNullAsString => null
-      case s: StringLiteralContext => createString(s)
-      case o => o.getText
+    expression(ctx) match {
+      case Literal(null, _) if !legacyNullAsString => null
+      case l @ Literal(null, _) => l.toString
+      case l: Literal =>
+        // TODO For v2 commands, we will cast the string back to its actual value,
+        //  which is a waste and can be improved in the future.
+        Cast(l, StringType, Some(SQLConf.get.sessionLocalTimeZone)).eval().toString
+      case _ =>
+        throw new IllegalArgumentException("Only support convert Literal to string when visit" +

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785282925


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135426/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r530057299



##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -198,6 +198,21 @@ SELECT * FROM students;
 +-------------+-------------------------+----------+
 ```
 
+#### Insert Using a type constructed partition col value

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r584797630



##########
File path: docs/sql-ref-syntax-dml-insert-into.md
##########
@@ -41,7 +41,7 @@ INSERT INTO [ TABLE ] table_identifier [ partition_spec ] [ ( column_list ) ]
 * **partition_spec**
 
     An optional parameter that specifies a comma-separated list of key and value pairs
-    for partitions.
+    for partitions. Note that one can use a typed literal (e.g., date'2019-01-02') for a partition column value.

Review comment:
       shall we update the sql doc for ADD/DROP/RENAME PARTITION as well?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-750839195


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37941/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735402658






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-732827738


   **[Test build #131650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131650/testReport)** for PR 30421 at commit [`05f1962`](https://github.com/apache/spark/commit/05f196254fb2054aa3926e830db8889b60f4fbac).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733604649






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782599594






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730435665


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35956/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782574525


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39877/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782568973


   **[Test build #135298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135298/testReport)** for PR 30421 at commit [`b68ee81`](https://github.com/apache/spark/commit/b68ee8196f072cedb151ffc623a7c4e8c7755d1d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782558008


   **[Test build #135295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135295/testReport)** for PR 30421 at commit [`7264a3d`](https://github.com/apache/spark/commit/7264a3d19e9e55b15bfcd3d1b902154d537c04cc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735553240






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-750827197


   **[Test build #133350 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133350/testReport)** for PR 30421 at commit [`ca69533`](https://github.com/apache/spark/commit/ca6953368069a4ddcfea21cba0094466fc53d480).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-735373560


   **[Test build #131928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131928/testReport)** for PR 30421 at commit [`4898fb4`](https://github.com/apache/spark/commit/4898fb42a9400b122c8873aafb280723ac8e1a71).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-732662307


   > Yes, user can use TypeConstruct value as partition spec value such as
   ``
   
   Is the description above incomplete? Btw, could you put an example query that this PR intends to support in the PR description?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730450478






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788181289


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40183/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-733073179


   **[Test build #131675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131675/testReport)** for PR 30421 at commit [`edd270e`](https://github.com/apache/spark/commit/edd270eaf1725939ecaee384c57bfa6b9036e521).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-730203617


   **[Test build #131330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131330/testReport)** for PR 30421 at commit [`c9a97d0`](https://github.com/apache/spark/commit/c9a97d06897014f25a2a0be4c085c5cc6a008608).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782577362


   Gentle ping @maropu @cloud-fan @MaxGekk  Since #30538  is Done, I think we can restart this pr. Any more suggestion?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-785110434


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40005/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788965356


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135654/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-787044010


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135530/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782583569


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39877/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782583569


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39877/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-788308646


   **[Test build #135599 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135599/testReport)** for PR 30421 at commit [`2ef0fd0`](https://github.com/apache/spark/commit/2ef0fd0e70de5a4856fafb88bd42aa61afbad702).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30421:
URL: https://github.com/apache/spark/pull/30421#issuecomment-782586281


   **[Test build #135305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135305/testReport)** for PR 30421 at commit [`dc0783a`](https://github.com/apache/spark/commit/dc0783afd4be103750cf3918c51e728a46ff54ad).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r530070129



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, _: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString
+    case Literal(data: Array[Byte], _: BinaryType) =>
+      UTF8String.fromBytes(data).toString
+    case Literal(_, dataType) =>
+      throw new IllegalArgumentException(s"Literals of type '$dataType' are not" +
+        " supported when use typed literal as partition col value.")
+  }
+
   /**
    * Convert a constant of any type into a string. This is typically used in DDL commands, and its
    * main purpose is to prevent slight differences due to back to back conversions i.e.:
    * String -> Literal -> String.
    */
   protected def visitStringConstant(ctx: ConstantContext): String = withOrigin(ctx) {
     ctx match {
+      case l: TypeConstructorContext => convertTypedLiteralToString(visitTypeConstructor(l))

Review comment:
       how about null literal?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30421:
URL: https://github.com/apache/spark/pull/30421#discussion_r532325245



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -503,13 +503,32 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  private def convertTypedLiteralToString(literal: Literal): String = literal match {
+    case Literal(data: Int, _: DateType) =>
+      UTF8String.fromString(
+        DateFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: Long, _: TimestampType) =>
+      UTF8String.fromString(
+        TimestampFormatter.getFractionFormatter(getZoneId(SQLConf.get.sessionLocalTimeZone))
+          .format(data)).toString
+    case Literal(data: CalendarInterval, _: CalendarIntervalType) =>
+      UTF8String.fromString(data.toString).toString

Review comment:
       Add. back using csv instead now.
   also cc @maropu @cloud-fan 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org