You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Subham Singhal (Jira)" <ji...@apache.org> on 2025/04/03 05:27:00 UTC

[jira] [Resolved] (SPARK-51706) Ratelimit kafka publish in spark

     [ https://issues.apache.org/jira/browse/SPARK-51706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Subham Singhal resolved SPARK-51706.
------------------------------------
    Resolution: Not A Problem

> Ratelimit kafka publish in spark
> --------------------------------
>
>                 Key: SPARK-51706
>                 URL: https://issues.apache.org/jira/browse/SPARK-51706
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.5.5
>            Reporter: Subham Singhal
>            Priority: Minor
>
> Currently there is no rate limiting applied on kafka publish. While reading from streaming source we can definitely control fetch bytes and write smaller batches to kafka but if we are reading in batch and batch size is huge we will end up publishing too many messages to kafka at once and this may cause issues in kafka. It would be better to have a rate limiting feature on kafka publisher.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org