You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "aokolnychyi (via GitHub)" <gi...@apache.org> on 2025/05/05 20:41:55 UTC

[PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

aokolnychyi opened a new pull request, #50792:
URL: https://github.com/apache/spark/pull/50792

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   This PR allows connectors to expose expression-based defaults on write.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   These changes are needed to avoid the requirement of producing Spark SQL dialect in connectors.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
   
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   No.
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   
   This PR comes with tests.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2074192772


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -49,7 +49,7 @@ import org.apache.spark.util.ArrayImplicits._
  */
 abstract class InMemoryBaseTable(
     val name: String,
-    val schema: StructType,
+    override val columns: Array[Column],

Review Comment:
   I had to change this to ensure we expose passed columns directly, rather than going through a round of conversion.



##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -49,7 +49,7 @@ import org.apache.spark.util.ArrayImplicits._
  */
 abstract class InMemoryBaseTable(
     val name: String,
-    val schema: StructType,
+    override val columns: Array[Column],

Review Comment:
   I had to change this to expose passed columns directly, rather than going through a round of conversion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2076237611


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -49,7 +49,7 @@ import org.apache.spark.util.ArrayImplicits._
  */
 abstract class InMemoryBaseTable(
     val name: String,
-    val schema: StructType,
+    override val columns: Array[Column],

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #50792:
URL: https://github.com/apache/spark/pull/50792#issuecomment-2855902805

   cc @huaxingao , too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi closed pull request #50792: [SPARK-51987][SQL] DSv2 expressions in column defaults on write
URL: https://github.com/apache/spark/pull/50792


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi closed pull request #50792: [SPARK-51987][SQL] DSv2 expressions in column defaults on write
URL: https://github.com/apache/spark/pull/50792


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2077970161


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {

Review Comment:
   `DefaultValue` is a user-facing DS v2 API. We should put these internal util functions in internal classes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078340204


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))

Review Comment:
   i mean the method javadoc:
   
   ```
   // For built-in file sources, we encode the default value in StructField metadata. An analyzer
   // rule will check the special metadata and change the DML input plan to fill the default value.
   ```
   do we need to update it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078846754


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   Ah, I misinterpreted the promise behind `.sql`. It was my understanding it is actually reliable. Let me think more about this tomorrow. If `.sql` is not reliable, then it does change things a bit, I guess. The problem that if we prefer the string over the expression it kind of breaks the promise behind `DefaultValue`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2076236417


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala:
##########
@@ -43,10 +43,22 @@ class InMemoryTable(
     advisoryPartitionSize: Option[Long] = None,
     isDistributionStrictlyRequired: Boolean = true,
     override val numRowsPerSplit: Int = Int.MaxValue)
-  extends InMemoryBaseTable(name, schema, partitioning, properties, distribution,
+  extends InMemoryBaseTable(name, columns, partitioning, properties, distribution,
     ordering, numPartitions, advisoryPartitionSize, isDistributionStrictlyRequired,
     numRowsPerSplit) with SupportsDelete {
 
+  def this(
+      name: String,
+      schema: StructType,

Review Comment:
   Thank you for adding this constructor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2076321571


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {
+      V2ExpressionUtils.toCatalyst(defaultValue.getExpression) match {
+        case Some(catalystExpr) =>
+          catalystExpr.sql
+        case None if defaultValue.getSql != null =>
+          defaultValue.getSql
+        case _ =>
+          throw SparkException.internalError(

Review Comment:
   Shall we add a test for this branch?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2076430871


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   iiuc, this is prioritizing expression over sql , if we get both.  Is that desired?  We could put a comment somewhere if so



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))

Review Comment:
   is this method comment outdated (its about built-in file sources)?



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {

Review Comment:
   style question: did we consider putting this logic in DefaultValue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2077968761


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))

Review Comment:
   It's for all data sources that can add column default values with existing data



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078340204


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))

Review Comment:
   i mean the method javadoc:
   
   ```// For built-in file sources, we encode the default value in StructField metadata. An analyzer
     // rule will check the special metadata and change the DML input plan to fill the default value.```
   do we need to update it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078351646


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   The problem is that whatever we encode here will be picked up and used on write. Spark doesn't currently use expressions to represent default values. We persist the expression SQL in struct field metadata. What this logic does is saying if I a connector provides an expression, I am going to convert it to Catalyst and trust Spark to generate an accurate SQL representation. If we flip this, it means we prioritize the connector-provided SQL string over our own SQL generation for Catalyst expressions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2076410633


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {
+      V2ExpressionUtils.toCatalyst(defaultValue.getExpression) match {

Review Comment:
   Hmm, when it is possible that a default expression cannot be converted back to catalyst one?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2077971050


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   Yea I have the same question. It seems more reasonable to use the original SQL string first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2084773935


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   If it's just for the legacy v1 code path (e.g. built-in file sources), I'd say let's keep the old behavior and save the original SQL string only. We can refactor them to use `ColumnDefinition` instead of `StructType`, but that is non trival and we can leave it for the future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #50792:
URL: https://github.com/apache/spark/pull/50792#issuecomment-2855882644

   cc @cloud-fan @dongjoon-hyun @viirya @szehon-ho @gengliangwang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2076411731


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {
+      V2ExpressionUtils.toCatalyst(defaultValue.getExpression) match {

Review Comment:
   Dynamic expressions other than literals?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078329249


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {
+      V2ExpressionUtils.toCatalyst(defaultValue.getExpression) match {

Review Comment:
   V2 expressions are quite flexible, users can represent a function with an arbitrary name. In practice, I expect it to be rare but you never know.
   
   I'll add a test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078846754


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   Ah, I misinterpreted the promise behind `.sql`. It was my understanding it is actually reliable. Let me think more about this tomorrow. If `.sql` is not reliable, then it does change things a bit, I guess. The problem that if we prefer the string over the expression it kind of breaks the promise behind `DefaultValue`, which states the expression takes precedence.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078170501


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   When we need to evaluate, it makes sense to use expression first and fallback to SQL string if the expression cannot be converted. But here we just need to get the SQL string.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2076432178


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {
+      V2ExpressionUtils.toCatalyst(defaultValue.getExpression) match {

Review Comment:
   Can we also add a test case that falls back to `defaultValue.getSql` when `toCatalyst` returns `None`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078337138


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {

Review Comment:
   Correct, the idea is to keep `DefaultValue` as simple as possible.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078411477


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))

Review Comment:
   That still seems accurate even after this PR. Spark always converts `DefaultValue` to `StructField` metadata. It is a good question whether that's strictly required or desirable but reconsidering that will be a big effort.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2079231341


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   There is a way to produce reliable SQL strings from v2 expressions: `V2ExpressionSQLBuilder`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2082562046


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   @cloud-fan, that could be an option. However, it doesn't produce Spark SQL dialect... for things like DAY_OF_WEEK and similar expressions. It seems to generate a user-friendly representation.
   
   The fundamental problem is that we have a Catalyst expression and we need to persist it in the struct field metadata. I wonder if we can avoid the need to put the expression in the metadata. It is going to be challenging for sure, as it requires changing `StructField`, which is a very stable API.
   
   Any thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078776399


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   But Spark is not capable of generating an executable SQL string from a query plan/expression. The `.sql` only generates a pretty string for display. We don't even have a test to parse back the `.sql` result.
   
   On the other hand, the `DefaultValue.sql` is usually the original user-specified SQL, or the connector is responsible to give an executable SQL if it generates it by itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078336303


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))

Review Comment:
   It seems to still apply as it describes the concept of "exists" default that we don't change here. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078427702


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {
+      V2ExpressionUtils.toCatalyst(defaultValue.getExpression) match {
+        case Some(catalystExpr) =>
+          catalystExpr.sql
+        case None if defaultValue.getSql != null =>
+          defaultValue.getSql
+        case _ =>
+          throw SparkException.internalError(

Review Comment:
   Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078427376


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {
+      V2ExpressionUtils.toCatalyst(defaultValue.getExpression) match {

Review Comment:
   Added more tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2076251885


##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala:
##########
@@ -483,6 +488,32 @@ class DataSourceV2DataFrameSuite
     }
   }
 
+  test("write with expression-based default values") {

Review Comment:
   This test would previously fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi closed pull request #50792: [SPARK-51987][SQL] DSv2 expressions in column defaults on write
URL: https://github.com/apache/spark/pull/50792


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078009535


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   Based on `DefaultValue` document, it uses expression first and fallbacks to SQL string if the expression cannot be converted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2078352104


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   Does this make sense, @szehon-ho @viirya @cloud-fan?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2082562046


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   @cloud-fan, that could be an option. However, it doesn't produce Spark SQL dialect... for things like DAY_OF_WEEK and similar expressions. It seems to generate a user-friendly representation.
   
   The fundamental problem is that we have a Catalyst expression and we need to persist it in the struct field metadata. I wonder if we can avoid the need to put the expression in the metadata. It is going to be challenging for sure, as it requires changing `StructField`, which is a very stable API.
   
   Any thoughts? Any ways to avoid converting Catalyst expression to string just parse it back soon after?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51987][SQL] DSv2 expressions in column defaults on write [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50792:
URL: https://github.com/apache/spark/pull/50792#discussion_r2082562046


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:
##########
@@ -598,10 +598,28 @@ private[sql] object CatalogV2Util {
       //       data unchanged and let the data reader to return "exist default" for missing
       //       columns.
       val existingDefault = Literal(default.getValue.value(), default.getValue.dataType()).sql
-      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(default.getSql)
+      f.withExistenceDefaultValue(existingDefault).withCurrentDefaultValue(toSql(defaultValue))
     }.getOrElse(f)
   }
 
+  private def toSql(defaultValue: DefaultValue): String = {
+    if (defaultValue.getExpression != null) {

Review Comment:
   @cloud-fan, that could be an option. However, it doesn't produce Spark SQL dialect... for things like DAY_OF_WEEK and similar expressions.
   
   The fundamental problem is that we have a Catalyst expression and we need to persist it in the struct field metadata. I wonder if we can avoid the need to put the expression in the metadata. It is going to be challenging for sure, as it requires changing `StructField`, which is a very stable API.
   
   Any thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org