[jira] [Created] (SPARK-49442) Complete Metadata requests on each micro batch causing Kafka issues

["vipin Kumar (Jira)" <ji...@apache.org>]

vipin Kumar created SPARK-49442:
-----------------------------------

             Summary: Complete Metadata requests on each micro batch causing Kafka issues
                 Key: SPARK-49442
                 URL: https://issues.apache.org/jira/browse/SPARK-49442
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 3.3.2
            Reporter: vipin Kumar


We have noticed that spark does complete metadata requests on each micro batch and this is causing high metadata requests on small micro batch intervals .

 

For example Kafka with 1900 partitions and 10 sec micro batch we are seeing order of 
~{*}360K{*} metadata requests / sec 

Same with job with 60 sec micro batch we are observing *~60K* meta data requests.

 

Metadata requests are controlled by *metadata.max.age.ms* but these config have no effect on spark consumers by default its 5 mins still we are seeing these huge requests.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org