You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "juliuszsompolski (via GitHub)" <gi...@apache.org> on 2023/10/06 10:19:11 UTC
[PR] [SPARK-45435] Document that lazy checkpoint may not be a consistent snapshot [spark]
juliuszsompolski opened a new pull request, #43247:
URL: https://github.com/apache/spark/pull/43247
### What changes were proposed in this pull request?
Some may want to use checkpoint to get a consistent snapshot of the Dataset / RDD. Warn that this is not the case with lazy checkpoint, because checkpoint is computed only at the end of the first action, and the data used during the first action may be different because of non-determinism and retries.
`doCheckpoint` is only called at the end of [SparkContext.runJob](https://github.com/apache/spark/blob/5446f548bbc8a93414f1c773a8daf714b57b7d1a/core/src/main/scala/org/apache/spark/SparkContext.scala#L2426). This may cause recomputation both of data of [local checkpoint data](https://github.com/apache/spark/blob/5446f548bbc8a93414f1c773a8daf714b57b7d1a/core/src/main/scala/org/apache/spark/rdd/LocalRDDCheckpointData.scala#L54) and [reliable checkpoint data](https://github.com/apache/spark/blob/5446f548bbc8a93414f1c773a8daf714b57b7d1a/core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala#L166) before it is finalized.
### Why are the changes needed?
Document a gnarly edge case.
### Does this PR introduce _any_ user-facing change?
Yes, change to documentation of public APIs.
### How was this patch tested?
Doc only change.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #43247: [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot
URL: https://github.com/apache/spark/pull/43247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43247:
URL: https://github.com/apache/spark/pull/43247#issuecomment-1891127875
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #43247:
URL: https://github.com/apache/spark/pull/43247#discussion_r1452024364
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -684,6 +684,14 @@ class Dataset[T] private[sql](
* plan may grow exponentially. It will be saved to files inside the checkpoint
* directory set with `SparkContext#setCheckpointDir`.
*
+ * @param eager Whether to checkpoint this dataframe immediately
+ *
+ * @note When checkpoint is used with eager = false, the final data that is checkpointed after
+ * the first action may be different from the data that was used during the job due to non
+ * deteminism of the underlying operation and retries. If checkpoint is used to achieve
Review Comment:
typo `deteminism`: fix in https://github.com/apache/spark/pull/44733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #43247:
URL: https://github.com/apache/spark/pull/43247#discussion_r1451988375
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -706,6 +714,14 @@ class Dataset[T] private[sql](
* plan may grow exponentially. Local checkpoints are written to executor storage and despite
* potentially faster they are unreliable and may compromise job completion.
*
+ * @param eager Whether to checkpoint this dataframe immediately
+ *
+ * @note When checkpoint is used with eager = false, the final data that is checkpointed after
Review Comment:
no problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #43247:
URL: https://github.com/apache/spark/pull/43247#issuecomment-1750351795
@po1o @gatorsmile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #43247:
URL: https://github.com/apache/spark/pull/43247#discussion_r1451991958
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -706,6 +714,14 @@ class Dataset[T] private[sql](
* plan may grow exponentially. Local checkpoints are written to executor storage and despite
* potentially faster they are unreliable and may compromise job completion.
*
+ * @param eager Whether to checkpoint this dataframe immediately
+ *
+ * @note When checkpoint is used with eager = false, the final data that is checkpointed after
Review Comment:
not supported in Python Client either:
https://github.com/apache/spark/blob/f4e41e0e318ea1269de5991f4635637e6e5233f3/python/pyspark/sql/connect/dataframe.py#L1691-L1698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #43247:
URL: https://github.com/apache/spark/pull/43247#discussion_r1451992951
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -706,6 +714,14 @@ class Dataset[T] private[sql](
* plan may grow exponentially. Local checkpoints are written to executor storage and despite
* potentially faster they are unreliable and may compromise job completion.
*
+ * @param eager Whether to checkpoint this dataframe immediately
+ *
+ * @note When checkpoint is used with eager = false, the final data that is checkpointed after
Review Comment:
ok, we are fine then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #43247:
URL: https://github.com/apache/spark/pull/43247#discussion_r1451984367
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -706,6 +714,14 @@ class Dataset[T] private[sql](
* plan may grow exponentially. Local checkpoints are written to executor storage and despite
* potentially faster they are unreliable and may compromise job completion.
*
+ * @param eager Whether to checkpoint this dataframe immediately
+ *
+ * @note When checkpoint is used with eager = false, the final data that is checkpointed after
Review Comment:
I think scala client need this doc as well, @zhengruifeng can you followup on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #43247:
URL: https://github.com/apache/spark/pull/43247#issuecomment-1891421647
This a doc only change, thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #43247:
URL: https://github.com/apache/spark/pull/43247#discussion_r1451991434
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -706,6 +714,14 @@ class Dataset[T] private[sql](
* plan may grow exponentially. Local checkpoints are written to executor storage and despite
* potentially faster they are unreliable and may compromise job completion.
*
+ * @param eager Whether to checkpoint this dataframe immediately
+ *
+ * @note When checkpoint is used with eager = false, the final data that is checkpointed after
Review Comment:
oh, Spark Connect doesn't support checkpoint
https://github.com/apache/spark/blob/8132e1700c81eff2e814756291b4bf853e764e9a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala#L3333-L3347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org