You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/15 16:03:59 UTC
[GitHub] [spark] jackylee-ch opened a new pull request #35862: [SPARK-38521][SQL][WIP] Insert overwrite partitioned datasource table with dynamic mode
jackylee-ch opened a new pull request #35862:
URL: https://github.com/apache/spark/pull/35862
### Why are the changes needed?
We have supported partition overwrite with static mode by default while partition manged by catalog for a long time. It looks weird to me as `static mode` behaves catastrophically and deletes all data in table. Maybe setting partition overwrite with dynamic mode by default while partition manged by catalog, and user can set `partitionOverwriteMode` in table properties to support old behaviors. And also, this change can also bring a better experience to users when migrating from the original hive table to the datasource table, rather than the old data being deleted when the user does not know the meaning of `partitionOverwriteMode`.
### Does this PR introduce _any_ user-facing change?
Yes. We will no longer delete old partition data by default when user insert overwrite partitioned datasource tables.
### How was this patch tested?
Origin UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jackylee-ch commented on pull request #35862: [SPARK-38521][SQL][WIP] Insert overwrite partitioned datasource table with dynamic mode
Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on pull request #35862:
URL: https://github.com/apache/spark/pull/35862#issuecomment-1068174365
@cloud-fan @HyukjinKwon I wan't to bring a discuss about this, as setting `partitionOvermode` with static by default is realy not a good way. It may result in the loss of unrecoverable data when user use datasource table without knowing this config, such as migrating from the original hive table to the datasource table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35862: [SPARK-38521][SQL][WIP] Insert overwrite partitioned datasource table with dynamic mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35862:
URL: https://github.com/apache/spark/pull/35862#issuecomment-1071270850
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jackylee-ch commented on pull request #35862: [SPARK-38521][SQL][WIP] Insert overwrite partitioned datasource table with dynamic mode
Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on pull request #35862:
URL: https://github.com/apache/spark/pull/35862#issuecomment-1068164767
cc @cloud-fan @HyukjinKwon @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35862: [SPARK-38521][SQL][WIP] Insert overwrite partitioned datasource table with dynamic mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35862:
URL: https://github.com/apache/spark/pull/35862#issuecomment-1071270850
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jackylee-ch removed a comment on pull request #35862: [SPARK-38521][SQL][WIP] Insert overwrite partitioned datasource table with dynamic mode
Posted by GitBox <gi...@apache.org>.
jackylee-ch removed a comment on pull request #35862:
URL: https://github.com/apache/spark/pull/35862#issuecomment-1068164767
cc @cloud-fan @HyukjinKwon @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org