You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/14 04:18:02 UTC
[GitHub] [spark] cxzl25 opened a new pull request #34889: [SPARK-37217][SQL][3.2] The number of dynamic partitions should early check when writing to external tables
cxzl25 opened a new pull request #34889:
URL: https://github.com/apache/spark/pull/34889
### What changes were proposed in this pull request?
SPARK-29295 introduces a mechanism that writes to external tables is a dynamic partition method, and the data in the target partition will be deleted first.
Assuming that 1001 partitions are written, the data of 10001 partitions will be deleted first, but because `hive.exec.max.dynamic.partitions` is 1000 by default, loadDynamicPartitions will fail at this time, but the data of 1001 partitions has been deleted.
So we can check whether the number of dynamic partitions is greater than `hive.exec.max.dynamic.partitions` before deleting, it should fail quickly at this time.
### Why are the changes needed?
Avoid data that cannot be recovered when the job fails.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
add UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34889: [SPARK-37217][SQL][3.2] The number of dynamic partitions should early check when writing to external tables
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34889:
URL: https://github.com/apache/spark/pull/34889#issuecomment-993856356
Merged to `branch-3.2`. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34889: [SPARK-37217][SQL][3.2] The number of dynamic partitions should early check when writing to external tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34889:
URL: https://github.com/apache/spark/pull/34889#issuecomment-993143752
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao closed pull request #34889: [SPARK-37217][SQL][3.2] The number of dynamic partitions should early check when writing to external tables
Posted by GitBox <gi...@apache.org>.
sunchao closed pull request #34889:
URL: https://github.com/apache/spark/pull/34889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org