You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2016/04/19 02:08:15 UTC
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/12484
[SPARK-14719] WriteAheadLogBasedBlockHandler should ignore BlockManager put errors
WriteAheadLogBasedBlockHandler will currently throw exceptions if its BlockManager `put()` calls fail, even though those calls are only performed as a performance optimization. Instead, it should log and ignore exceptions during that `put()`.
This is a longstanding issue that was masked by an incorrect test case. I think that we haven't noticed this in production because
1. most people probably use a `MEMORY_AND_DISK` storage level, and
2. typically, individual blocks may be small enough relative to the total storage memory such that they're able to evict blocks from previous batches, so `put()` failures here may be rare in practice.
This patch adds a regression test and fixes the bug.
/cc @tdas
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark received-block-hadndler-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12484.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12484
----
commit 621b20917dcce720fde4e1973a6af33184639a07
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-18T23:57:07Z
Add regression test.
commit 1832158879c4e02f1328968f2cc33d87ad057687
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-19T00:00:00Z
Don't throw exception if block store fails in WAL received block handler.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211698479
**[Test build #56171 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56171/consoleFull)** for PR 12484 at commit [`2bfa11e`](https://github.com/apache/spark/commit/2bfa11e53120e3bf3951aef3096583db3fced82e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211758183
@dibbhatt,
Are you suggesting that this pull request introduces a bug? If so, are there any regression tests that will demonstrate it? I'm still unclear on precisely what the problem is from Spark Streaming's point of view, since your linked PR only adds unit tests for BlockManager functionality and doesn't have end-to-end application-level tests which exhibit how the old BlockManager behavior caused problems for streaming.
The PR discussion that you linked to is really long and has a somewhat unclear resolution. If there was a bug which motivated that PR, do you know whether it was previously resolved through another patch / other fixes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211658614
Whoops, looks like I have to update this test, too:
```
[info] - WriteAheadLogBasedBlockHandler - handle errors in storing block *** FAILED *** (25 milliseconds)
[info] Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown. (ReceivedBlockHandlerSuite.scala:358)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by dibbhatt <gi...@git.apache.org>.
Github user dibbhatt commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211808725
@JoshRosen you are right. This test case you referred seems to be doing wrong thing. If block not able to put in memory (in WAL case), there is no point writing it to WAL.
I guess the issue here is little different. If Block not written to Primary then ReceivedBlockHandler will get the block id, but Block id wont be there in BlockManagerMaster. And hence no point writing the block to WAL. @tdas is that correct ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211676340
**[Test build #56171 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56171/consoleFull)** for PR 12484 at commit [`2bfa11e`](https://github.com/apache/spark/commit/2bfa11e53120e3bf3951aef3096583db3fced82e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211658039
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211698737
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56171/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211658042
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56159/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by dibbhatt <gi...@git.apache.org>.
Github user dibbhatt commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211734038
Hi @JoshRosen , Isn't this fix is somehow related to the issue discussed here https://github.com/apache/spark/pull/6990.
You can refer to the final comments from @andrewor14
https://github.com/apache/spark/pull/6990#issuecomment-120515683
The issue here is , If a block fails to unroll, the ReceivedBlockHandler wont be getting the block id and will never know about the block and will not include it in a future computation. the problem is that if you can't store a block locally, the receiver thinks the block has not been stored anywhere -- even if it has been successfully written to WAL . isn't it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12484
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-212166811
Given the confusion here, I'm going to revert this patch. I'm not confident in the test coverage of this component and therefore don't want to make any changes to it now. I'll investigate and deal with the test problems that affected me in a separate PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211698736
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211642970
**[Test build #56159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56159/consoleFull)** for PR 12484 at commit [`1832158`](https://github.com/apache/spark/commit/1832158879c4e02f1328968f2cc33d87ad057687).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211699961
LGTM. Merging this to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12484#discussion_r60180310
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockHandlerSuite.scala ---
@@ -204,26 +222,26 @@ class ReceivedBlockHandlerSuite
sparkConf.set("spark.storage.unrollFraction", "0.4")
// Block Manager with 12000 * 0.4 = 4800 bytes of free space for unroll
blockManager = createBlockManager(12000, sparkConf)
+ // This block is way too large to possibly be cached in memory:
+ def hugeBlock: IteratorBlock = IteratorBlock(List.fill(100)(new Array[Byte](1000)).iterator)
// there is not enough space to store this block in MEMORY,
// But BlockManager will be able to serialize this block to WAL
// and hence count returns correct value.
--- End diff --
@dibbhatt, I'm confused because it seems like your comment says that we should fail a job if blocks cannot be persisted because without that persistence the job will not work correctly even if the WAL is enabled. However, that claim seems to be contradicted by the comment describing this test case, which seems to suggest that this job should succeed despite the block being far too large to be successfully stored. In the old test case, however, the block appeared to be too small and actually _was_ being stored in memory, meaning that this comment wasn't describing the actual behavior of the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14719] WriteAheadLogBasedBlockHandler s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12484#issuecomment-211657966
**[Test build #56159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56159/consoleFull)** for PR 12484 at commit [`1832158`](https://github.com/apache/spark/commit/1832158879c4e02f1328968f2cc33d87ad057687).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org