You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by srowen <gi...@git.apache.org> on 2016/07/12 12:01:41 UTC
[GitHub] spark pull request #14152: [SPARK-16395] [STREAMING] Fail if too many Checkp...
GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/14152
[SPARK-16395] [STREAMING] Fail if too many CheckpointWriteHandlers are queued up in the fixed thread pool
## What changes were proposed in this pull request?
Begin failing if checkpoint writes will likely keep up with storage's ability to write them, to fail fast instead of slowly filling memory
## How was this patch tested?
Jenkins tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-16395
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14152.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14152
----
commit a6fce665b4de00d37b13d12ace4eebad364c1a3d
Author: Sean Owen <so...@cloudera.com>
Date: 2016-07-12T12:00:14Z
Begin failing if checkpoint writes will likely keep up with storage's ability to write them, to fail fast instead of slowly filling memory
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14152
**[Test build #62164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62164/consoleFull)** for PR 14152 at commit [`a6fce66`](https://github.com/apache/spark/commit/a6fce665b4de00d37b13d12ace4eebad364c1a3d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14152: [SPARK-16395] [STREAMING] Fail if too many Checkp...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14152#discussion_r70575075
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala ---
@@ -18,8 +18,8 @@
package org.apache.spark.streaming
import java.io._
-import java.util.concurrent.Executors
-import java.util.concurrent.RejectedExecutionException
+import java.util.concurrent.{ArrayBlockingQueue, RejectedExecutionException,
--- End diff --
Yeah it's the style I see elsewhere, like https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L39 Arguably it's time for a _ import at this stage; I'm indifferent here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14152: [SPARK-16395] [STREAMING] Fail if too many Checkp...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/14152#discussion_r70541473
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala ---
@@ -18,8 +18,8 @@
package org.apache.spark.streaming
import java.io._
-import java.util.concurrent.Executors
-import java.util.concurrent.RejectedExecutionException
+import java.util.concurrent.{ArrayBlockingQueue, RejectedExecutionException,
--- End diff --
do we usually break up the import lines this way? i dont think so, but I may be wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14152
**[Test build #62164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62164/consoleFull)** for PR 14152 at commit [`a6fce66`](https://github.com/apache/spark/commit/a6fce665b4de00d37b13d12ace4eebad364c1a3d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14152
I'm gonna go for this if there are no objections. I'd even put it into 2.0, but unless someone seconds that I won't.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14152
@tdas how does the substance of the change strike you? it's an improvement over the current situation where millions of checkpoints would stack up, but the cap of 1000 was arbitrary. I didn't necessarily think it worth making configurable, just "large", but YMMV
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14152: [SPARK-16395] [STREAMING] Fail if too many Checkp...
Posted by srowen <gi...@git.apache.org>.
Github user srowen closed the pull request at:
https://github.com/apache/spark/pull/14152
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14152
CC @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14152
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62164/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14152
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org