You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:04:48 UTC
[jira] [Updated] (SPARK-19114) Backpressure could support
non-integral rates (< 1)
[ https://issues.apache.org/jira/browse/SPARK-19114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-19114:
---------------------------------
Labels: bulk-closed (was: )
> Backpressure could support non-integral rates (< 1)
> ---------------------------------------------------
>
> Key: SPARK-19114
> URL: https://issues.apache.org/jira/browse/SPARK-19114
> Project: Spark
> Issue Type: Improvement
> Reporter: Tony Novak
> Priority: Minor
> Labels: bulk-closed
>
> We have a Spark streaming job where each record takes well over a second to execute, so the stable rate is under 1 element/second. We set spark.streaming.backpressure.enabled=true and spark.streaming.backpressure.pid.minRate=0.1, but backpressure did not appear to be effective, even though the TRACE level logs from PIDRateEstimator showed that the new rate was 0.1.
> As it turns out, even though the minRate parameter is a Double, and the rate estimate generated by PIDRateEstimator is a Double as well, RateController casts the new rate to a Long. As a result, if the computed rate is less than 1, it's truncated to 0, which ends up being interpreted as "no limit".
> What's particularly confusing is that the Guava RateLimiter class takes a rate limit as a double, so the long value ends up being cast back to a double.
> Is there any reason not to keep the rate limit as a double all the way through? I'm happy to create a pull request if this makes sense.
> We encountered the bug on Spark 1.6.2, but it looks like the code in the master branch is still affected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org