You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Reza Safi (JIRA)" <ji...@apache.org> on 2018/08/24 16:44:00 UTC

[jira] [Commented] (SPARK-25233) Give the user the option of specifying a fixed minimum message per partition per batch when using kafka direct API with backpressure

    [ https://issues.apache.org/jira/browse/SPARK-25233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591889#comment-16591889 ] 

Reza Safi commented on SPARK-25233:
-----------------------------------

I will send a PR shortly for this.

> Give the user the option of specifying a fixed minimum message per partition per batch when using kafka direct API with backpressure
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-25233
>                 URL: https://issues.apache.org/jira/browse/SPARK-25233
>             Project: Spark
>          Issue Type: Improvement
>          Components: DStreams
>    Affects Versions: 2.4.0
>            Reporter: Reza Safi
>            Priority: Major
>
> After SPARK-18371, it is guaranteed that there would be at least *one* message per partition per batch using direct kafka API with backpressure when new messages exist in the topics. It would be better if the user has the option of setting the minimum instead of just a hard coded 1 limit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org