You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/23 20:36:19 UTC
[GitHub] [spark] agrawaldevesh opened a new pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
agrawaldevesh opened a new pull request #35005:
URL: https://github.com/apache/spark/pull/35005
Run checkpoint job only once when asked to do so eagerly.
### What changes were proposed in this pull request?
The flow is like so:
```
- df.checkpoint(eager = true, reliable = true)
- rdd = get rdd from this df's physical plan
- rdd.checkpoint (just marks checkpointData)
- rdd.count (if eager = true)
- SparkContext.runJob for all the partitions
- DAGScheduler.runJob
- rdd.doCheckpoint
- ReliableCheckpointRDD#writeRDDToCheckpointDirectory
- SparkContext.runJob for all the partitions
- DAGScheduler.runJob (<-- This is the repeat job)
```
The local checkpointing case is better because there it will just
recompute the missing partitions.
We tried a fix where we just replaced `rdd.count` above with
`rdd.doCheckpoint` and it seemed to work and pass the unit tests.
So the new flow is simply:
```
- df.checkpoint(eager = true, reliable = true)
- rdd = get rdd from this df's physical plan
- rdd.checkpoint (just marks checkpointData)
- rdd.doCheckpoint (if eager = true)
- ReliableCheckpointRDD#writeRDDToCheckpointDirectory
- SparkContext.runJob for all the partitions
- DAGScheduler.runJob (<-- Only one job is run)
```
### Why are the changes needed?
This simple fix drastically improves spark jobs that make heavy use of
Dataframe.checkpoint.
### Does this PR introduce _any_ user-facing change?
Yes, it would make eager checkpointing jobs supposedly faster by doing half as many spark jobs.
### How was this patch tested?
Customer spark apps using checkpoint with this fix see half as many
spark jobs launched, seeing upto 50% less runtime in some cases.
Also, added one more unit test to check that only job is created.
This patch may have some interactions with the Spark-Streaming, since it
touches the codepaths enabled via the config
spark.checkpoint.checkpointAllMarkedAncestors, so would be happy to add
more testing there if pointed in the right direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000553758
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/51012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000554683
**[Test build #146537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146537/testReport)** for PR 35005 at commit [`7fc5172`](https://github.com/apache/spark/commit/7fc517250a5fb367dcb3ab4b4b66a2bbb4031d76).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000554798
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146537/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1003445114
> To confirm: If people do
>
> ```
> rdd.checkpoint()
> rdd.count
> ```
>
> Spark will run the job twice? This looks like an existing bug in spark core. I'm fine with this PR as a workaround at the SQL side.
It does not run the complete job, just the suffix required to perform the action.
Typically only the last result stage or (in case DAG involves shuffles) - or when persisted, the suffix after the persist to materialize the files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1002400388
I think you cc'ed related people properly. I think we just would have to wait for their reviews. cc @Ngone51 too FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000731944
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/51029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1002395717
Thanks @HyukjinKwon !. I am wondering how can I get some eyes on this PR. There haven't been many recent editors to checkpointing logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000707610
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000810609
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146554/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000810609
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146554/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1003444483
Thanks for the feedback @cloud-fan. This PR is ready for another round of review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000554798
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146537/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000687379
**[Test build #146554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146554/testReport)** for PR 35005 at commit [`6ca9b74`](https://github.com/apache/spark/commit/6ca9b7477c3a0cdc38afbfbcbfb868e067ef9790).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh edited a comment on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
agrawaldevesh edited a comment on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1002395717
Thanks @HyukjinKwon !. I am wondering how can I get some eyes on this PR ?. There haven't been many recent editors to checkpointing logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1003279326
> ```
> rdd.checkpoint()
> rdd.count
> ```
>
> Spark will run the job twice? This looks like an existing bug in spark core. I'm fine with this PR as a workaround at the SQL side.
Indeed. The first `rdd.checkpoint` will merely mark the RDD for checkpointing. If we open up the actual "RDD.doCheckpoint" call as a public API then users can call that and it shall eagerly checkpoint.
But otherwise: Rdd.count will run the spark job twice: Once for the actual "count" and then SparkContext.runJob shall call "RDD.doCheckpoint" which will finally trigger SparkContext.runJob again for the "writing of the checkpoint" output.
Do you think we should open up the RDD.doCheckpoint API ? I am not sure about the ramifications of that so I tried to do something scoped to the Dataframe/SQL side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1004372726
Hi @mridulm , Thanks for pointing out that it is not necessary for checkpointing to rerun the whole job. But in practice, we have noticed that it often does end up re-running the job for the following reasons:
- Many user jobs don't have a shuffle, for example simple ETL or ML training jobs.
- Users sometimes don't cache/persist the RDD prior to calling checkpoint. They do this sometimes out of oversight, sometimes because they notice that Dynamic Executor Allocation does not play well with .cache/.persist. In this case as you note, the whole non-shuffle would be rerun.
We have noticed that while caching/persisting can be faster than checkpointing to HDFS, it is a bit less robust in our environment, and thus we see users going for the latter approach.
We tried this PR in a couple of internal heavy ML training jobs that use Checkpointing and it results in about 40% improvement.
Do you have any concerns about the approach in this PR -- or perhaps you think there is an alternative way to implement this ?
Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000520482
**[Test build #146537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146537/testReport)** for PR 35005 at commit [`7fc5172`](https://github.com/apache/spark/commit/7fc517250a5fb367dcb3ab4b4b66a2bbb4031d76).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000727391
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1003445114
> To confirm: If people do
>
> ```
> rdd.checkpoint()
> rdd.count
> ```
>
> Spark will run the job twice? This looks like an existing bug in spark core. I'm fine with this PR as a workaround at the SQL side.
It does not run the complete job, just the suffix required to perform the action.
Typically only the last result stage (in case DAG involves shuffles) - or when persisted, the suffix after the persist to materialize the files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35005:
URL: https://github.com/apache/spark/pull/35005#discussion_r776156951
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
##########
@@ -1457,8 +1458,29 @@ class DatasetSuite extends QueryTest
}
testCheckpointing("basic") {
- val ds = spark.range(10).repartition($"id" % 2).filter($"id" > 5).orderBy($"id".desc)
- val cp = if (reliable) ds.checkpoint(eager) else ds.localCheckpoint(eager)
+ val ds = spark
+ .range(10)
+ // Num partitions is set to 1 to avoid a RangePartitioner in the orderBy below
+ .repartition(1, $"id" % 2)
+ .filter($"id" > 5)
+ .orderBy($"id".desc)
+ @volatile var jobCounter = 0
+ spark.sparkContext.addSparkListener(new SparkListener {
Review comment:
we should remove this listener at the end of test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000520482
**[Test build #146537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146537/testReport)** for PR 35005 at commit [`7fc5172`](https://github.com/apache/spark/commit/7fc517250a5fb367dcb3ab4b4b66a2bbb4031d76).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000553758
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/51012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1005459355
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1002402205
To confirm: If people do
```
rdd.checkpoint()
rdd.count
```
Spark will run the job twice? This looks like an existing bug in spark core. I'm fine with this PR as a workaround at the SQL side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000808868
**[Test build #146554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146554/testReport)** for PR 35005 at commit [`6ca9b74`](https://github.com/apache/spark/commit/6ca9b7477c3a0cdc38afbfbcbfb868e067ef9790).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000550207
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000731944
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/51029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1005374908
Gentle ping for a review @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #35005:
URL: https://github.com/apache/spark/pull/35005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #35005:
URL: https://github.com/apache/spark/pull/35005#discussion_r776932868
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
##########
@@ -1457,8 +1458,29 @@ class DatasetSuite extends QueryTest
}
testCheckpointing("basic") {
- val ds = spark.range(10).repartition($"id" % 2).filter($"id" > 5).orderBy($"id".desc)
- val cp = if (reliable) ds.checkpoint(eager) else ds.localCheckpoint(eager)
+ val ds = spark
+ .range(10)
+ // Num partitions is set to 1 to avoid a RangePartitioner in the orderBy below
+ .repartition(1, $"id" % 2)
+ .filter($"id" > 5)
+ .orderBy($"id".desc)
+ @volatile var jobCounter = 0
+ spark.sparkContext.addSparkListener(new SparkListener {
Review comment:
Good call. I will use the withListener helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000687379
**[Test build #146554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146554/testReport)** for PR 35005 at commit [`6ca9b74`](https://github.com/apache/spark/commit/6ca9b7477c3a0cdc38afbfbcbfb868e067ef9790).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000917675
cc: @zsxwing , @andrewor14, @cloud-fan for review please. Thank you !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #35005: [SPARK-8582] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1000534747
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1005410963
I'm fine with it. @mridulm do you have any other concerns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35005: [SPARK-8582][CORE] Checkpoint eagerly when asked to do so for real
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35005:
URL: https://github.com/apache/spark/pull/35005#issuecomment-1005448514
Looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org