You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2015/01/14 06:44:14 UTC
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/4032
[SPARK-5233][Streaming] Fix error replaying of WAL introduced bug
Because of lacking of `BlockAllocationEvent` in WAL recovery, the dangled event will mix into the new batch, which will lead to the wrong result. Details can be seen in [SPARK-5233](https://issues.apache.org/jira/browse/SPARK-5233).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-5233
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4032.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4032
----
commit 622cbbca5b7e169dcae5f08b001723eebfbbc4cd
Author: jerryshao <sa...@intel.com>
Date: 2015-01-14T05:24:21Z
Correctly replay the WAL log when recovering from failure
commit 4c82be8e41383a4e4cb6b1e53e47928b3f245f2a
Author: jerryshao <sa...@intel.com>
Date: 2015-01-14T05:41:20Z
Fix bug in unit test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71151116
LGTM. Merging this. Thanks!!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23361754
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala ---
@@ -106,6 +106,12 @@ private[streaming] class ReceivedBlockTracker(
timeToAllocatedBlocks(batchTime) = allocatedBlocks
lastAllocatedBatchTime = batchTime
allocatedBlocks
+ } else if (batchTime == lastAllocatedBatchTime) {
+ // This situation occurs when WAL is ended with BatchAllocationEvent,
+ // but without BatchCleanupEvent, possibly processed batch job or half-processed batch
+ // job need to process again, so the batchTime will be equal to lastAllocatedBatchTime.
+ // This situation will only occurs in recovery time.
+ logWarning(s"Possibly processed batch $batchTime need to be processed again in WAL recovery")
} else {
--- End diff --
Actually lets remove this exception completely. Instead any attempts to allocate blocks to a batch such that batch <= lastAllocatedBatchTime, is completely ignored.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-69876526
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25514/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71030745
[Test build #25966 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25966/consoleFull) for PR 4032 at commit [`f0b0c0b`](https://github.com/apache/spark/commit/f0b0c0bcc8db551c57c1590c8a12b01f5f0ae2d0).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-70990301
OK, will do, thanks a lot for your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-70997210
[Test build #25964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25964/consoleFull) for PR 4032 at commit [`a237c75`](https://github.com/apache/spark/commit/a237c75f87518b26245fd688de9bbae4bc151ae2).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4032
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23366552
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala ---
@@ -107,8 +107,14 @@ private[streaming] class ReceivedBlockTracker(
lastAllocatedBatchTime = batchTime
allocatedBlocks
} else {
- throw new SparkException(s"Unexpected allocation of blocks, " +
- s"last batch = $lastAllocatedBatchTime, batch time to allocate = $batchTime ")
+ // This situation occurs when:
+ // 1. WAL is ended with BatchAllocationEvent, but without BatchCleanupEvent,
+ // possibly processed batch job or half-processed batch job need to be processed again,
+ // so the batchTime will be equal to lastAllocatedBatchTime.
+ // 2. Slow checkpointing makes recovered batch time older than WAL recovered
+ // lastAllocatedBatchTime.
+ // This situation will only occurs in recovery time.
+ logWarning(s"Possibly processed batch $batchTime need to be processed again in WAL recovery")
--- End diff --
i am not sure we even should throw a warning. Warnings are thrown when something bad may have happened. In this case, if the code reaches here, nothing bad or unexpected has happened.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71020806
[Test build #25966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25966/consoleFull) for PR 4032 at commit [`f0b0c0b`](https://github.com/apache/spark/commit/f0b0c0bcc8db551c57c1590c8a12b01f5f0ae2d0).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23361948
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala ---
@@ -106,6 +106,12 @@ private[streaming] class ReceivedBlockTracker(
timeToAllocatedBlocks(batchTime) = allocatedBlocks
lastAllocatedBatchTime = batchTime
allocatedBlocks
+ } else if (batchTime == lastAllocatedBatchTime) {
--- End diff --
Update to this comment. See other comment about removing this check altogether. Please update unit test to verify this new behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-70989477
I understand this patch now. Please update it based on my comments, and test it in your harness to make sure that it addresses the exception problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-69876522
[Test build #25514 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25514/consoleFull) for PR 4032 at commit [`4c82be8`](https://github.com/apache/spark/commit/4c82be8e41383a4e4cb6b1e53e47928b3f245f2a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71151653
Thanks for your reviewing :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23361989
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockTrackerSuite.scala ---
@@ -82,11 +82,6 @@ class ReceivedBlockTrackerSuite
receivedBlockTracker.allocateBlocksToBatch(2)
receivedBlockTracker.getBlocksOfBatchAndStream(2, streamId) shouldBe empty
- // Verify that batch 2 cannot be allocated again
--- End diff --
Please add test to verify that `allocateBlocksToBatch(x)` where x<=2 does nothing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23360371
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala ---
@@ -106,6 +106,12 @@ private[streaming] class ReceivedBlockTracker(
timeToAllocatedBlocks(batchTime) = allocatedBlocks
lastAllocatedBatchTime = batchTime
allocatedBlocks
+ } else if (batchTime == lastAllocatedBatchTime) {
--- End diff --
Could you improve the unit test to actually verify the behavior. The unit test line you removed can be modified to verify that calling `batchTime` = `lastAllocatedBatchTime` is a no-op and does no allocation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71000469
looks almost good. some minor comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23366414
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockTrackerSuite.scala ---
@@ -82,15 +82,13 @@ class ReceivedBlockTrackerSuite
receivedBlockTracker.allocateBlocksToBatch(2)
receivedBlockTracker.getBlocksOfBatchAndStream(2, streamId) shouldBe empty
- // Verify that batch 2 cannot be allocated again
- intercept[SparkException] {
- receivedBlockTracker.allocateBlocksToBatch(2)
- }
+ // Verify that older batches have no operation on batch allocation,
+ // will return the same blocks as previously allocated.
+ receivedBlockTracker.allocateBlocksToBatch(1)
+ receivedBlockTracker.getBlocksOfBatchAndStream(1, streamId) shouldEqual blockInfos
- // Verify that older batches cannot be allocated again
- intercept[SparkException] {
- receivedBlockTracker.allocateBlocksToBatch(1)
- }
+ receivedBlockTracker.allocateBlocksToBatch(2)
+ receivedBlockTracker.getBlocksOfBatchAndStream(2, streamId) shouldBe empty
--- End diff --
Could you also test that even if there are new unallocated blocks, calling `receivedBlockTracker.allocateBlocksToBatch(2)` does not allocate those blocks? I dont think that is covered in this test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-69872152
[Test build #25514 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25514/consoleFull) for PR 4032 at commit [`4c82be8`](https://github.com/apache/spark/commit/4c82be8e41383a4e4cb6b1e53e47928b3f245f2a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71151138
LGTM. Merging this. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71004957
[Test build #25964 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25964/consoleFull) for PR 4032 at commit [`a237c75`](https://github.com/apache/spark/commit/a237c75f87518b26245fd688de9bbae4bc151ae2).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71030753
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25966/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23366311
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala ---
@@ -206,9 +208,13 @@ class JobGenerator(jobScheduler: JobScheduler) extends Logging {
val timesToReschedule = (pendingTimes ++ downTimes).distinct.sorted(Time.ordering)
logInfo("Batches to reschedule (" + timesToReschedule.size + " batches): " +
timesToReschedule.mkString(", "))
- timesToReschedule.foreach(time =>
+ timesToReschedule.foreach { time =>
+ // Allocate the related blocks when recovering from failure, because some added but not
+ // allocated block is dangled in the queue after recovering, we have to insert some block
--- End diff --
Grammar fix
"..some blocks that were added but not allocated, are dangling in the queue after recovering. We have to allocate thoe blocks to the next batch, which is the batch they were supposed to go to."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23362757
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala ---
@@ -106,6 +106,12 @@ private[streaming] class ReceivedBlockTracker(
timeToAllocatedBlocks(batchTime) = allocatedBlocks
lastAllocatedBatchTime = batchTime
allocatedBlocks
+ } else if (batchTime == lastAllocatedBatchTime) {
+ // This situation occurs when WAL is ended with BatchAllocationEvent,
+ // but without BatchCleanupEvent, possibly processed batch job or half-processed batch
+ // job need to process again, so the batchTime will be equal to lastAllocatedBatchTime.
+ // This situation will only occurs in recovery time.
+ logWarning(s"Possibly processed batch $batchTime need to be processed again in WAL recovery")
} else {
--- End diff --
IIUC, I think this might not be happened normally, unless the checkpoint operation is so slow. Anyway we should consider this to make the code more robust.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-70997541
Hey @tdas , I've updated the code and rebased the branch according to your comments, also with several rounds of test in my local set, the previous exception I reported is gone :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4032#issuecomment-71004965
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25964/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23373392
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala ---
@@ -107,8 +107,14 @@ private[streaming] class ReceivedBlockTracker(
lastAllocatedBatchTime = batchTime
allocatedBlocks
} else {
- throw new SparkException(s"Unexpected allocation of blocks, " +
- s"last batch = $lastAllocatedBatchTime, batch time to allocate = $batchTime ")
+ // This situation occurs when:
+ // 1. WAL is ended with BatchAllocationEvent, but without BatchCleanupEvent,
+ // possibly processed batch job or half-processed batch job need to be processed again,
+ // so the batchTime will be equal to lastAllocatedBatchTime.
+ // 2. Slow checkpointing makes recovered batch time older than WAL recovered
+ // lastAllocatedBatchTime.
+ // This situation will only occurs in recovery time.
+ logWarning(s"Possibly processed batch $batchTime need to be processed again in WAL recovery")
--- End diff --
At least I think we should inform user this batch is processed or half-processed before, so user may pay attention to this, whether to clean or overwrite the previously processed result. I'm downgrading to logInfo, what's your opinion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5233][Streaming] Fix error replaying of...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/4032#discussion_r23360875
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala ---
@@ -106,6 +106,12 @@ private[streaming] class ReceivedBlockTracker(
timeToAllocatedBlocks(batchTime) = allocatedBlocks
lastAllocatedBatchTime = batchTime
allocatedBlocks
+ } else if (batchTime == lastAllocatedBatchTime) {
--- End diff --
OK, I will improve this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org