You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by omuravskiy <gi...@git.apache.org> on 2016/11/24 16:36:38 UTC
[GitHub] spark pull request #16006: Use spark.streaming.backpressure.initialRate in D...
GitHub user omuravskiy opened a pull request:
https://github.com/apache/spark/pull/16006
Use spark.streaming.backpressure.initialRate in DirectKafkaInputDStream
## What changes were proposed in this pull request?
Use spark.streaming.backpressure.initialRate in DirectKafkaInputDStream
## How was this patch tested?
tested manually
This is intended as a PR for SPARK-18580
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/omuravskiy/spark SPARK-18580
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16006.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16006
----
commit 6b086a33a052b24c868f1e7373d59eb74c037125
Author: Oleg Muravskiy <ol...@ripe.net>
Date: 2016-11-24T16:23:23Z
Use spark.streaming.backpressure.initialRate in DirectKafkaInputDStream
This is intended as a PR for SPARK-18580
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/16006#discussion_r105720383
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala ---
@@ -143,9 +147,16 @@ private[spark] class DirectKafkaInputDStream[K, V](
lagPerPartition.map { case (tp, lag) =>
val maxRateLimitPerPartition = ppc.maxRatePerPartition(tp)
- val backpressureRate = Math.round(lag / totalLag.toFloat * rate)
- tp -> (if (maxRateLimitPerPartition > 0) {
- Math.min(backpressureRate, maxRateLimitPerPartition)} else backpressureRate)
+ val effectiveRate = if (rate >= 0) rate else backpressureInitialRate
--- End diff --
This can be moved before the map, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/16006
Any interest in picking this back up if you can get a committer's attention?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/16006#discussion_r91118458
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala ---
@@ -67,6 +67,10 @@ private[spark] class DirectKafkaInputDStream[K, V](
ekp
}
+ val backpressureInitialRate: Long =
+ _ssc.sparkContext.conf.getLong("spark.streaming.backpressure.initialRate",
+ _ssc.sparkContext.conf.getDouble("spark.streaming.backpressure.pid.minRate", 100).round)
--- End diff --
Shouldn't these be using ssc instead of _ssc for consistency?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16006
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/16006
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by omuravskiy <gi...@git.apache.org>.
Github user omuravskiy commented on the issue:
https://github.com/apache/spark/pull/16006
I added new commit, the default value of `backpressureInitialRate` if `spark.streaming.backpressure.initialRate` is now taken from `spark.streaming.backpressure.pid.minRate`, or 100 if that is also not set (which is the same as in `org.apache.spark.streaming.scheduler.rate.RateEstimator`).
And `backpressureInitialRate`, when applied, is now shared proportionally among all partitions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/16006#discussion_r91118893
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala ---
@@ -143,9 +147,14 @@ private[spark] class DirectKafkaInputDStream[K, V](
lagPerPartition.map { case (tp, lag) =>
val maxRateLimitPerPartition = ppc.maxRatePerPartition(tp)
- val backpressureRate = Math.round(lag / totalLag.toFloat * rate)
- tp -> (if (maxRateLimitPerPartition > 0) {
- Math.min(backpressureRate, maxRateLimitPerPartition)} else backpressureRate)
+ val effectiveRate = if (rate >= 0) rate else backpressureInitialRate
+ val estimateRate = Math.round(lag / totalLag.toFloat * effectiveRate)
+ val backpressureRate =
+ if (estimateRate > maxRateLimitPerPartition && maxRateLimitPerPartition > 0) {
+ maxRateLimitPerPartition
+ }
+ else estimateRate
--- End diff --
http://spark.apache.org/contributing.html
use braces around the else clause unless it's all on one line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16006
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by omuravskiy <gi...@git.apache.org>.
Github user omuravskiy closed the pull request at:
https://github.com/apache/spark/pull/16006
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/16006#discussion_r91001763
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala ---
@@ -67,6 +67,10 @@ private[spark] class DirectKafkaInputDStream[K, V](
ekp
}
+ val backpressureInitialRate: Long =
+ _ssc.sparkContext.conf.getLong("spark.streaming.backpressure.initialRate",
+ 1 + _ssc.sparkContext.conf.getLong("spark.streaming.kafka.maxRatePerPartition", 1000L) / 10)
+
--- End diff --
Where is this divide by 10 coming from?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by omuravskiy <gi...@git.apache.org>.
Github user omuravskiy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16006#discussion_r91035292
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala ---
@@ -67,6 +67,10 @@ private[spark] class DirectKafkaInputDStream[K, V](
ekp
}
+ val backpressureInitialRate: Long =
+ _ssc.sparkContext.conf.getLong("spark.streaming.backpressure.initialRate",
+ 1 + _ssc.sparkContext.conf.getLong("spark.streaming.kafka.maxRatePerPartition", 1000L) / 10)
+
--- End diff --
It is to supply just a 10% of the maximum rate at the start of execution, while workers are still warming up. At least on my system this starting rate works nicely, gradually increasing the rate without building a backlog of stages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/16006
@zsxwing you or anyone else have time to look at this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: Use spark.streaming.backpressure.initialRate in DirectKa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16006
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by omuravskiy <gi...@git.apache.org>.
Github user omuravskiy commented on the issue:
https://github.com/apache/spark/pull/16006
Yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/16006
#19431 was merged, thanks for your work. This PR should probably be closed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by omuravskiy <gi...@git.apache.org>.
Github user omuravskiy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16006#discussion_r105780496
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala ---
@@ -143,9 +147,16 @@ private[spark] class DirectKafkaInputDStream[K, V](
lagPerPartition.map { case (tp, lag) =>
val maxRateLimitPerPartition = ppc.maxRatePerPartition(tp)
- val backpressureRate = Math.round(lag / totalLag.toFloat * rate)
- tp -> (if (maxRateLimitPerPartition > 0) {
- Math.min(backpressureRate, maxRateLimitPerPartition)} else backpressureRate)
+ val effectiveRate = if (rate >= 0) rate else backpressureInitialRate
--- End diff --
Yes, good catch!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by omuravskiy <gi...@git.apache.org>.
Github user omuravskiy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16006#discussion_r91038777
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala ---
@@ -143,9 +147,14 @@ private[spark] class DirectKafkaInputDStream[K, V](
lagPerPartition.map { case (tp, lag) =>
val maxRateLimitPerPartition = ppc.maxRatePerPartition(tp)
- val backpressureRate = Math.round(lag / totalLag.toFloat * rate)
- tp -> (if (maxRateLimitPerPartition > 0) {
- Math.min(backpressureRate, maxRateLimitPerPartition)} else backpressureRate)
+ val backpressureRate =
+ if (rate >= 0) {
+ val estimateRate = Math.round(lag / totalLag.toFloat * rate)
+ if (estimateRate > maxRateLimitPerPartition && maxRateLimitPerPartition > 0) {
+ maxRateLimitPerPartition
+ } else estimateRate
+ } else math.min(backpressureInitialRate, maxRateLimitPerPartition)
--- End diff --
I was using `backpressureInitialRate` as a per-partition value. But I agree, that's not clear. I'll submit an update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Us...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/16006#discussion_r91002372
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala ---
@@ -143,9 +147,14 @@ private[spark] class DirectKafkaInputDStream[K, V](
lagPerPartition.map { case (tp, lag) =>
val maxRateLimitPerPartition = ppc.maxRatePerPartition(tp)
- val backpressureRate = Math.round(lag / totalLag.toFloat * rate)
- tp -> (if (maxRateLimitPerPartition > 0) {
- Math.min(backpressureRate, maxRateLimitPerPartition)} else backpressureRate)
+ val backpressureRate =
+ if (rate >= 0) {
+ val estimateRate = Math.round(lag / totalLag.toFloat * rate)
+ if (estimateRate > maxRateLimitPerPartition && maxRateLimitPerPartition > 0) {
+ maxRateLimitPerPartition
+ } else estimateRate
+ } else math.min(backpressureInitialRate, maxRateLimitPerPartition)
--- End diff --
I don't think these units are comparable, one is records / sec, and the other is records / sec / partition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by omuravskiy <gi...@git.apache.org>.
Github user omuravskiy commented on the issue:
https://github.com/apache/spark/pull/16006
Tests added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/16006
@omuravskiy can you comment on
https://github.com/apache/spark/pull/19431
since it appears to be based on your PR
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org