You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "szehon-ho (via GitHub)" <gi...@apache.org> on 2025/07/16 00:55:56 UTC

[PR] [MINOR][SQL] Add unit test for V2 write schema evolution [spark]

szehon-ho opened a new pull request, #51506:
URL: https://github.com/apache/spark/pull/51506

   
   
   ### What changes were proposed in this pull request?
   
   Add test for DSV2 Write schema evolution for InMemoryTable
   
   ### Why are the changes needed?
   There is a test on the analysis phase in V2WriteAnalysisSuite to make sure it skips the output resolution.  But it is not end to end and not obvious its for this feature.  For authors of DSV2 connector, its better to have an end to end test using the InMemoryTable to show to how to use this feature.
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   Ran the new test
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [MINOR][SQL][Test] Support V2 write schema evolution in InMemoryTable [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #51506:
URL: https://github.com/apache/spark/pull/51506#issuecomment-3085830374

   @cloud-fan @dongjoon-hyun @huaxingao can one of you take a look?  Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52860][SQL][Test] Support V2 write schema evolution in InMemoryTable [spark]

Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #51506:
URL: https://github.com/apache/spark/pull/51506#issuecomment-3102340780

   All tests passed.
   Merged to `master` (4.1.0).
   Thanks @szehon-ho and @dongjoon-hyun.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [MINOR][SQL][Test] Support V2 write schema evolution in InMemoryTable [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #51506:
URL: https://github.com/apache/spark/pull/51506#discussion_r2214597949


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -571,6 +591,17 @@ abstract class InMemoryBaseTable(
       override def reportDriverMetrics(): Array[CustomTaskMetric] = {
         Array(new InMemoryCustomDriverTaskMetric(rows.size))
       }
+
+      def mergeSchema(oldType: StructType, newType: StructType): StructType = {
+        val (oldFields, newFields) = (oldType.fields, newType.fields)
+
+        // this does not override the old field with the new field with same name for now
+        val nameToFieldMap = oldFields.map (f => f.name -> f).toMap
+        val remainingNewFields = newFields.filterNot (f => nameToFieldMap.contains (f.name) )

Review Comment:
   OK i added this support as well in InMemoryTable, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52860][SQL][Test] Support V2 write schema evolution in InMemoryTable [spark]

Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth closed pull request #51506: [SPARK-52860][SQL][Test] Support V2 write schema evolution in InMemoryTable
URL: https://github.com/apache/spark/pull/51506


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [MINOR][SQL][Test] Support V2 write schema evolution in InMemoryTable [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #51506:
URL: https://github.com/apache/spark/pull/51506#discussion_r2214498347


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -571,6 +591,17 @@ abstract class InMemoryBaseTable(
       override def reportDriverMetrics(): Array[CustomTaskMetric] = {
         Array(new InMemoryCustomDriverTaskMetric(rows.size))
       }
+
+      def mergeSchema(oldType: StructType, newType: StructType): StructType = {
+        val (oldFields, newFields) = (oldType.fields, newType.fields)
+
+        // this does not override the old field with the new field with same name for now
+        val nameToFieldMap = oldFields.map (f => f.name -> f).toMap
+        val remainingNewFields = newFields.filterNot (f => nameToFieldMap.contains (f.name) )

Review Comment:
   According to the code, do we assume case-sensitive column names always?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [MINOR][SQL][Test] Support V2 write schema evolution in InMemoryTable [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #51506:
URL: https://github.com/apache/spark/pull/51506#discussion_r2214561309


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -571,6 +591,17 @@ abstract class InMemoryBaseTable(
       override def reportDriverMetrics(): Array[CustomTaskMetric] = {
         Array(new InMemoryCustomDriverTaskMetric(rows.size))
       }
+
+      def mergeSchema(oldType: StructType, newType: StructType): StructType = {
+        val (oldFields, newFields) = (oldType.fields, newType.fields)
+
+        // this does not override the old field with the new field with same name for now
+        val nameToFieldMap = oldFields.map (f => f.name -> f).toMap
+        val remainingNewFields = newFields.filterNot (f => nameToFieldMap.contains (f.name) )

Review Comment:
   Yea its up to the DSV2 implementation, for example Iceberg uses a config [1], which actually is the spark config `spark.sql.caseSensitive` [2].   ref: 
   
   1. https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkWriteBuilder.java#L199
   2. https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java#L456
   
   I was just making a simple example for InMemoryTable, should I can make it more complex and take this property into account?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52860][SQL][Test] Support V2 write schema evolution in InMemoryTable [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #51506:
URL: https://github.com/apache/spark/pull/51506#issuecomment-3104007016

   Thanks a lot @peter-toth , @dongjoon-hyun !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org