[jira] [Commented] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

["Cody Koeninger (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489102#comment-15489102 ] 

Cody Koeninger commented on SPARK-17510:
----------------------------------------

This would require a constructor change and another overload, which I'm personally not necessarily against, but I'd want to understand why it was necessary.

I know there was some list discussion, but I never heard back on your testing of the direct stream only with backpressure on.  There had been some work done already on making sure backpressure didn't starve particular topicpartitions.  WIth backpressure, max rate should only be necessary for making sure the initial rdd doesn't blow out any limits.

> Set Streaming MaxRate Independently For Multiple Streams
> --------------------------------------------------------
>
>                 Key: SPARK-17510
>                 URL: https://issues.apache.org/jira/browse/SPARK-17510
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 2.0.0
>            Reporter: Jeff Nadler
>
> We use multiple DStreams coming from different Kafka topics in a Streaming application.
> Some settings like maxrate and backpressure enabled/disabled would be better passed as config to KafkaUtils.createStream and KafkaUtils.createDirectStream, instead of setting them in SparkConf.
> Being able to set a different maxrate for different streams is an important requirement for us; we currently work-around the problem by using one receiver-based stream and one direct stream.   
> We would like to be able to turn on backpressure for only one of the streams as well.   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org