You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tcondie <gi...@git.apache.org> on 2017/03/10 00:11:47 UTC
[GitHub] spark pull request #17231: [SPARK-19891][SS] Await Batch Lock notified on st...
GitHub user tcondie opened a pull request:
https://github.com/apache/spark/pull/17231
[SPARK-19891][SS] Await Batch Lock notified on stream execution exit
## What changes were proposed in this pull request?
We need to notify the await batch lock when the stream exits early e.g., when an exception has been thrown.
## How was this patch tested?
Current tests that throw exceptions at runtime will finish faster as a result of this update.
@zsxwing
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tcondie/spark kafka-writer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17231.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17231
----
commit d371758538f659cbcf604e591110665cfca4f216
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-20T01:53:03Z
add kafka relation and refactor kafka source
commit b6c3055f2c21050cb9e203213651e02779d724bc
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-20T02:29:27Z
update
commit 4c81812e93907219af69dd57e0f4f5ebaf92b262
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-20T18:24:07Z
update
commit ab02a4c631f9fc0ecd8528d85c61fe3c5de64040
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-20T19:53:24Z
single kafka provider for both stream and batch
commit e6b57edb0958649062749cb9a0f7cda74f0b2829
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-24T00:32:56Z
added uninterruptible thread version of kafka offset reader
commit ff94ed803474448f6bb388f8933e6ec091fc24a1
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-24T00:44:31Z
added uninterruptible thread version of kafka offset reader
commit f8fd34cf0c2da4f1b9c793e4e021c23a923a0285
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-24T01:13:16Z
update tests
commit 41271e25896387262c2c3a5b4fad6ba48feb9121
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-24T18:32:26Z
resolve conflicts in KafakSource
commit 74d96fc9049a0a0fb6de6d011eb896b7d7c32b30
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-24T18:45:12Z
update comments
commit d31fc8104bf94c82bc6fa1c099def9ca16fec93a
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-25T00:53:11Z
address comments from @zsxwing
commit 1db1649201361bcede52997ec8c2f0610a55da8b
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-25T01:04:06Z
update
commit 3b0d48b53c1b7a992557c6a42910a4681a84865e
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-25T18:05:55Z
Merge branch 'master' of https://github.com/apache/spark into SPARK-18682
commit a5b02691ddbaef6c8092881ea15b582d9137be71
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-26T21:32:00Z
address comments from @zsxwing
commit c08c01fd21fc53d3db3504a257ab6bb6115cd462
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-27T16:33:29Z
late binding offsets
commit 79d335e697ba5af3f02f524d39c16e61d2cc73d9
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-27T18:44:50Z
update to late binding logic
commit a44b365bba4bcdb7cd56d16c31d0447de9ed5aa3
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-27T19:00:50Z
Merge branch 'SPARK-18682' into kafka-writer
commit 51291e36fad5f3511ce1d3afc17e2e714dffe2d3
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-27T20:21:36Z
remove kafka log4j debug
commit b597cf1bf2135659de1e25bb7d41aeeac26f87f9
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-27T20:22:33Z
remove kafka log4j debug
commit 84b32c55787290415016186d0c53543661b12851
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-30T21:12:48Z
Merge branch 'SPARK-18682' into kafka-writer
commit f5ae3012ec37be0c4a6208ba00438592cd3aa791
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-31T20:17:18Z
update
commit 2487a7260ec11496aed0135a723e156376c6ff31
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-31T22:29:24Z
address comments from @zsxwing
commit 789d3afc0163b93aa0f859354221ea7e2374f74a
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-01-31T23:12:43Z
update
commit 56a06e7dc75a72148fca498b41d7adf7c41e4cfe
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-02-01T22:02:18Z
Merge branch 'SPARK-18682' into kafka-writer
commit e74473b28b160633ab6545fae208981096fe367c
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-02-02T00:58:01Z
update
commit 73df054e87eb755bc0b25ee15e7fdd501cd0c10c
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-02-02T22:43:39Z
update
commit 5b48fc65ac08e8ed4a09edd0d346990d40d042e0
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-02-03T01:20:03Z
address comments from @tdas
commit 57760094591439525d3a53d0c5845a0da7e9b8eb
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-02-03T19:46:00Z
address feedback from @tdas and @sxwing
commit 63d453f6078897d4049640d009aa0722b1f93908
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-02-03T20:00:13Z
update merge
commit 3c4eecf3cd529c45c7b7a1bf053c7ea92e4caa50
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-02-03T23:16:03Z
update
commit b0611e48f71be67e1481cb0754e7f49f06f73dc2
Author: Tyson Condie <tc...@gmail.com>
Date: 2017-02-16T23:28:02Z
update
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17231
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17231
**[Test build #74294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74294/testReport)** for PR 17231 at commit [`8170792`](https://github.com/apache/spark/commit/817079234203dc428032c5a110a03f43ec7a813f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17231
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17231
**[Test build #74292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74292/testReport)** for PR 17231 at commit [`a23162a`](https://github.com/apache/spark/commit/a23162ac7322af2b662e870ea4f889f19d4a8b2c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17231
**[Test build #74287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74287/testReport)** for PR 17231 at commit [`b6f1bb0`](https://github.com/apache/spark/commit/b6f1bb0f314e73ef2a94f01b4c24696eea164849).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17231: [SPARK-19891][SS] Await Batch Lock notified on st...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17231
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17231
**[Test build #74287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74287/testReport)** for PR 17231 at commit [`b6f1bb0`](https://github.com/apache/spark/commit/b6f1bb0f314e73ef2a94f01b4c24696eea164849).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17231
**[Test build #74292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74292/testReport)** for PR 17231 at commit [`a23162a`](https://github.com/apache/spark/commit/a23162ac7322af2b662e870ea4f889f19d4a8b2c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17231
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17231
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74287/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17231
**[Test build #74293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74293/testReport)** for PR 17231 at commit [`c49e183`](https://github.com/apache/spark/commit/c49e183bf391ad0d9352400b19a2e191a3f0be11).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17231
**[Test build #74293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74293/testReport)** for PR 17231 at commit [`c49e183`](https://github.com/apache/spark/commit/c49e183bf391ad0d9352400b19a2e191a3f0be11).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17231
LGTM. Merging to master and 2.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17231
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74293/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17231
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74292/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17231
**[Test build #74294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74294/testReport)** for PR 17231 at commit [`8170792`](https://github.com/apache/spark/commit/817079234203dc428032c5a110a03f43ec7a813f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17231
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17231
Could you just submit one commit, please? The PR commit history will be a part of the commit message after merging. That's totally not necessary for such minor fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17231: [SPARK-19891][SS] Await Batch Lock notified on stream ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17231
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74294/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org