You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Iqbal Singh (JIRA)" <ji...@apache.org> on 2018/05/16 11:53:00 UTC

[jira] [Created] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

Iqbal Singh created SPARK-24295:
-----------------------------------

             Summary: Purge Structured streaming FileStreamSinkLog metadata compact file data.
                 Key: SPARK-24295
                 URL: https://issues.apache.org/jira/browse/SPARK-24295
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.3.0
            Reporter: Iqbal Singh


FileStreamSinkLog metadata logs are concatenated to a single compact file after defined compact interval.

For long running jobs, compact file size can grow up to 10's of GB's, Causing slowness  while reading the data from FileStreamSinkLog dir as spark is defaulting to the "__spark__metadata" dir for the read.

We need a functionality to purge the compact file size.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org