[jira] [Commented] (SPARK-14531) Flume streaming should respect maxRate (and backpressure)

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235794#comment-15235794 ] 

Apache Spark commented on SPARK-14531:
--------------------------------------

User 'yongtang' has created a pull request for this issue:
https://github.com/apache/spark/pull/12305

> Flume streaming should respect maxRate (and backpressure)
> ---------------------------------------------------------
>
>                 Key: SPARK-14531
>                 URL: https://issues.apache.org/jira/browse/SPARK-14531
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.6.1
>            Reporter: Herman Schistad
>            Priority: Minor
>
> As far as I can understand the FlumeUtils.createPollingStream(...) ignores key spark streaming configuration options such as:
> spark.streaming.backpressure.enabled
> spark.streaming.receiver.maxRate
> ...
> I might just be mistaken, but since the DEFAULT_POLLING_BATCH_SIZE is set to 1000 in the source code itself, then I presume it should use the variables above instead.
> *Relevant code:*  https://github.com/apache/spark/blob/master/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeUtils.scala



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org