You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "xupefei (via GitHub)" <gi...@apache.org> on 2024/03/28 09:52:41 UTC
[PR] [WIP][SPARK-47627] Add SQL MERGE syntax to enable schema evolution [spark]
xupefei opened a new pull request, #45748:
URL: https://github.com/apache/spark/pull/45748
### Why are the changes needed?
This PR introduces a syntax `WITH SCHEMA EVOLUTION` to the SQL MERGE command, which allows the user to specify automatic schema evolution for a specific operation.
```sql
MERGE WITH SCHEMA EVOLUTION
INTO tgt
USING src
ON ...
WHEN ...
```
### Does this PR introduce _any_ user-facing change?
Yes, see the previous section.
### How was this patch tested?
New tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-47627] Add SQL MERGE syntax to enable schema evolution [spark]
Posted by "n-young-db (via GitHub)" <gi...@apache.org>.
n-young-db commented on code in PR #45748:
URL: https://github.com/apache/spark/pull/45748#discussion_r1543365738
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala:
##########
@@ -1663,11 +1667,13 @@ class PlanResolutionSuite extends AnalysisTest {
mergeCondition,
Seq(UpdateAction(None, updateAssigns)),
Seq(InsertAction(None, insertAssigns)),
- Seq()) =>
+ Seq(),
+ withSchemaEvolution) =>
Review Comment:
Noticed that the tests in this file didn't have the added `WITH SCHEMA EVOLUTION` like in the internal PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47627][SQL] Add SQL MERGE syntax to enable schema evolution [spark]
Posted by "n-young-db (via GitHub)" <gi...@apache.org>.
n-young-db commented on code in PR #45748:
URL: https://github.com/apache/spark/pull/45748#discussion_r1543365738
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala:
##########
@@ -1663,11 +1667,13 @@ class PlanResolutionSuite extends AnalysisTest {
mergeCondition,
Seq(UpdateAction(None, updateAssigns)),
Seq(InsertAction(None, insertAssigns)),
- Seq()) =>
+ Seq(),
+ withSchemaEvolution) =>
Review Comment:
Noticed that the tests in this file didn't have the added `WITH SCHEMA EVOLUTION` like in the internal PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47627][SQL] Add SQL MERGE syntax to enable schema evolution [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45748:
URL: https://github.com/apache/spark/pull/45748#issuecomment-2061909354
cc @huaxingao , @RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47627][SQL] Add SQL MERGE syntax to enable schema evolution [spark]
Posted by "xupefei (via GitHub)" <gi...@apache.org>.
xupefei commented on PR #45748:
URL: https://github.com/apache/spark/pull/45748#issuecomment-2059899197
> @xupefei could you provide more details in the PR description? For example, what is the difference with/without `WITH SCHEMA EVOLUTION`
Hi @gengliangwang, I added to the PR description as you advised. Please have a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47627][SQL] Add SQL MERGE syntax to enable schema evolution [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #45748:
URL: https://github.com/apache/spark/pull/45748#issuecomment-2059645090
@xupefei could you provide more details in the PR description? For example, what is the difference with/without `WITH SCHEMA EVOLUTION`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47627][SQL] Add SQL MERGE syntax to enable schema evolution [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #45748: [SPARK-47627][SQL] Add SQL MERGE syntax to enable schema evolution
URL: https://github.com/apache/spark/pull/45748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47627][SQL] Add SQL MERGE syntax to enable schema evolution [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #45748:
URL: https://github.com/apache/spark/pull/45748#issuecomment-2061886298
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org