You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2015/04/07 08:36:42 UTC
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/5385
[SPARK-6691][Streaming][WIP] Add a dynamic RateLimiter for Spark Streaming
This proposal add a dynamic RateLimiter for Spark Streaming's both receiver and direct based input streams. The details can be seen in [SPARK-6691](https://issues.apache.org/jira/browse/SPARK-6691).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark dynamic-rate-limiter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5385.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5385
----
commit 293958ebc91ad4b0374714463a4eb84fc5d7240a
Author: jerryshao <sa...@intel.com>
Date: 2015-04-02T09:59:56Z
Initial commit
commit b8b0d6b2762657049da2b85ce5a1b79d98003a50
Author: jerryshao <sa...@intel.com>
Date: 2015-04-03T07:10:17Z
Add the slow start configuration and improve the code
commit 8891e4e24754a9d11216ecfe368bc22d87e18a63
Author: jerryshao <sa...@intel.com>
Date: 2015-04-03T09:32:00Z
Code polish
commit 9235598b5fcacbaa4f7ea1ca6bbc16ff1011d6d7
Author: jerryshao <sa...@intel.com>
Date: 2015-04-07T06:01:56Z
Add unit test
commit d2382dc8dd301422530e1433687b8d9954cbd25f
Author: jerryshao <sa...@intel.com>
Date: 2015-04-07T06:20:28Z
Fix some bugs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-136940195
OK, thanks a lot. Will port if still make sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-136939299
I'm afraid not. The corresponding Jira ticket has been closed as a duplicated by @tdas on July 27, so we should close this PR as well. @jerryshao do you think there's something in your approach that would make sense to be ported to the existing backpressure implementation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-91149527
[Test build #29927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29927/consoleFull) for PR 5385 at commit [`b576d9e`](https://github.com/apache/spark/commit/b576d9e95c89515ed6977b46feeccf42ca2c607a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90397047
[Test build #29781 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29781/consoleFull) for PR 5385 at commit [`d2382dc`](https://github.com/apache/spark/commit/d2382dc8dd301422530e1433687b8d9954cbd25f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao closed the pull request at:
https://github.com/apache/spark/pull/5385
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90398330
[Test build #29781 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29781/consoleFull) for PR 5385 at commit [`d2382dc`](https://github.com/apache/spark/commit/d2382dc8dd301422530e1433687b8d9954cbd25f).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90449387
[Test build #29787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29787/consoleFull) for PR 5385 at commit [`180dfdf`](https://github.com/apache/spark/commit/180dfdf39821087936650dbc3f5d6302e0a98b36).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-136913368
@tdas @dragos is this patch still relevant after the latest back pressure work?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90398338
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29781/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90455304
[Test build #29787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29787/consoleFull) for PR 5385 at commit [`180dfdf`](https://github.com/apache/spark/commit/180dfdf39821087936650dbc3f5d6302e0a98b36).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-91126900
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-91128782
[Test build #29926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29926/consoleFull) for PR 5385 at commit [`82ae607`](https://github.com/apache/spark/commit/82ae60741611675d023dd7129da6159b71c1ba0d).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-91127152
[Test build #29926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29926/consoleFull) for PR 5385 at commit [`82ae607`](https://github.com/apache/spark/commit/82ae60741611675d023dd7129da6159b71c1ba0d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90455313
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29787/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90829596
[Test build #29843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29843/consoleFull) for PR 5385 at commit [`82ae607`](https://github.com/apache/spark/commit/82ae60741611675d023dd7129da6159b71c1ba0d).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-91149538
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29927/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-91135071
[Test build #29927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29927/consoleFull) for PR 5385 at commit [`b576d9e`](https://github.com/apache/spark/commit/b576d9e95c89515ed6977b46feeccf42ca2c607a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90828096
[Test build #29843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29843/consoleFull) for PR 5385 at commit [`82ae607`](https://github.com/apache/spark/commit/82ae60741611675d023dd7129da6159b71c1ba0d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-91128786
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29926/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5385#issuecomment-90829603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29843/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org