You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by akonopko <gi...@git.apache.org> on 2017/10/04 18:10:23 UTC
[GitHub] spark pull request #19430: Spark 18580
GitHub user akonopko opened a pull request:
https://github.com/apache/spark/pull/19430
Spark 18580
## What changes were proposed in this pull request?
Add `spark.streaming.backpressure.initialRate` to direct Kafka Streams for Kafka 0.8 and 0.10
This is required in order to be able to use backpressure with huge lags, which cannot be processed at once. Without this parameter `DirectKafkaInputDStream` with backpressure enabled would try to get all the possible data from Kafka before adjusting consumption rate
## How was this patch tested?
- Tests added to `org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala` and `org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala`
- Manual tests on YARN cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/akonopko/spark SPARK-18580
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19430.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19430
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19430: Spark 18580
Posted by akonopko <gi...@git.apache.org>.
Github user akonopko commented on the issue:
https://github.com/apache/spark/pull/19430
@vanzin plz take a look at https://github.com/apache/spark/pull/19431
I am deleting this PR cause it was mistakenly created against branch-2.2
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19430: Spark 18580
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19430
Why isn't this PR against master? Is the change not needed there?
The title also doesn't follow the Spark convention.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19430: Spark 18580
Posted by akonopko <gi...@git.apache.org>.
GitHub user akonopko reopened a pull request:
https://github.com/apache/spark/pull/19430
Spark 18580
## What changes were proposed in this pull request?
Add `spark.streaming.backpressure.initialRate` to direct Kafka Streams for Kafka 0.8 and 0.10
This is required in order to be able to use backpressure with huge lags, which cannot be processed at once. Without this parameter `DirectKafkaInputDStream` with backpressure enabled would try to get all the possible data from Kafka before adjusting consumption rate
## How was this patch tested?
- Tests added to `org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala` and `org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala`
- Manual tests on YARN cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/akonopko/spark SPARK-18580
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19430.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19430
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19430: Spark 18580
Posted by akonopko <gi...@git.apache.org>.
Github user akonopko closed the pull request at:
https://github.com/apache/spark/pull/19430
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19430: Spark 18580
Posted by akonopko <gi...@git.apache.org>.
Github user akonopko closed the pull request at:
https://github.com/apache/spark/pull/19430
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19430: Spark 18580
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19430
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org