You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by YuvalItzchakov <gi...@git.apache.org> on 2018/08/03 09:50:08 UTC
[GitHub] spark pull request #21983: SPARK-24987 - Fix Kafka consumer leak when no new...
GitHub user YuvalItzchakov opened a pull request:
https://github.com/apache/spark/pull/21983
SPARK-24987 - Fix Kafka consumer leak when no new offsets for TopicPartition
## What changes were proposed in this pull request?
This small fix adds a `consumer.release()` call to `KafkaSourceRDD` in the case where we've retrieved offsets from Kafka, but the `fromOffset` is equal to the `lastOffset`, meaning there is no new data to read for a particular topic. Up until now, we'd just return an empty iterator without closing the consumer which would cause a FD leak.
If accepted, this pull request should be merged into master as well.
## How was this patch tested?
Haven't ran any specific tests, would love help on how to test methods running inside `RDD.compute`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/YuvalItzchakov/spark branch-2.3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21983.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21983
----
commit 4e366f8be24373de98908465070f718cbabd3790
Author: Yuval Itzchakov <yu...@...>
Date: 2018-08-02T18:12:08Z
Fixes SPARK-24987. Kafka consumer wasn't released when `fromOffset` was equal to `toOffset`.
commit e5db69f291bd099bee38d3b555b0c040ef942f29
Author: Yuval Itzchakov <yu...@...>
Date: 2018-08-03T09:41:47Z
Merge remote-tracking branch 'upstream/branch-2.3' into branch-2.3
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21983
**[Test build #94133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94133/testReport)** for PR 21983 at commit [`e5db69f`](https://github.com/apache/spark/commit/e5db69f291bd099bee38d3b555b0c040ef942f29).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21983
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when ...
Posted by YuvalItzchakov <gi...@git.apache.org>.
Github user YuvalItzchakov closed the pull request at:
https://github.com/apache/spark/pull/21983
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/21983
Jenkins, ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21983
**[Test build #94133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94133/testReport)** for PR 21983 at commit [`e5db69f`](https://github.com/apache/spark/commit/e5db69f291bd099bee38d3b555b0c040ef942f29).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21983
@YuvalItzchakov you should open the PR against master - it can be picked to release branch (eg. 2.3) when merged.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by YuvalItzchakov <gi...@git.apache.org>.
Github user YuvalItzchakov commented on the issue:
https://github.com/apache/spark/pull/21983
@felixcheung Thanks, will do.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21983
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21983
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21983
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94133/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21983
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21983: [SPARK-24987][SS] - Fix Kafka consumer leak when no new ...
Posted by YuvalItzchakov <gi...@git.apache.org>.
Github user YuvalItzchakov commented on the issue:
https://github.com/apache/spark/pull/21983
Should I create a separate PR for the master branch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org