[jira] [Created] (SPARK-33606) Delay in batch processing when creating Spark metadata compact file

["Stela Stefanova (Jira)" <ji...@apache.org>]

Stela Stefanova created SPARK-33606:
---------------------------------------

             Summary: Delay in batch processing when creating Spark metadata compact file
                 Key: SPARK-33606
                 URL: https://issues.apache.org/jira/browse/SPARK-33606
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.4.3
            Reporter: Stela Stefanova


Hi,

We are using Spark structured streaming to consume data from a Kafka cluster, transform it, and write it in an S3 Objectstore Service. Currently our Spark metadata compact files have 4,046,119 lines (1.2GB), and when a compact file gets written there is a delay with the batch processing - the batches are processing 1.8-2 minutes, when it usually takes 10-15 seconds. This delay in the processing generates lag in Kafka, which is a significant problem for us. Is this a known issue and if it is, are there any plans for fixing it in the future?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org