You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Cody Koeninger (JIRA)" <ji...@apache.org> on 2018/08/30 18:28:00 UTC

[jira] [Resolved] (SPARK-25233) Give the user the option of specifying a fixed minimum message per partition per batch when using kafka direct API with backpressure

     [ https://issues.apache.org/jira/browse/SPARK-25233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cody Koeninger resolved SPARK-25233.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 2.4.0

Issue resolved by pull request 22223
[https://github.com/apache/spark/pull/22223]

> Give the user the option of specifying a fixed minimum message per partition per batch when using kafka direct API with backpressure
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-25233
>                 URL: https://issues.apache.org/jira/browse/SPARK-25233
>             Project: Spark
>          Issue Type: Improvement
>          Components: DStreams
>    Affects Versions: 2.4.0
>            Reporter: Reza Safi
>            Assignee: Reza Safi
>            Priority: Major
>             Fix For: 2.4.0
>
>
> After SPARK-18371, it is guaranteed that there would be at least *one* message per partition per batch using direct kafka API with backpressure when new messages exist in the topics. It would be better if the user has the option of setting the minimum instead of just a hard coded 1 limit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org