You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2019/02/21 04:45:00 UTC

[jira] [Assigned] (SPARK-26949) Prevent "purge" to remove needed batch files in CompactibleFileStreamLog

     [ https://issues.apache.org/jira/browse/SPARK-26949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-26949:
------------------------------------

    Assignee: Apache Spark

> Prevent "purge" to remove needed batch files in CompactibleFileStreamLog
> ------------------------------------------------------------------------
>
>                 Key: SPARK-26949
>                 URL: https://issues.apache.org/jira/browse/SPARK-26949
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.0.0
>            Reporter: Jungtaek Lim
>            Assignee: Apache Spark
>            Priority: Minor
>
> I've seen couple of trials (in opened PRs, even I've also tried) which calls purge() in CompactibleFileStreamLog, but after looking at the codebase of CompactibleFileStreamLog, I've realized that purging latest compaction batch would break internal of CompactibleFileStreamLog and it throws IllegalStateException.
> Given that CompactibleFileStreamLog maintains the batches and purges according to its configuration, it would be safer to just rely on CompactibleFileStreamLog to purge and prevent calling `purge` outside of CompactibleFileStreamLog.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org