You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2019/06/12 17:07:00 UTC
[jira] [Assigned] (SPARK-26949) Prevent "purge" to remove needed
batch files in CompactibleFileStreamLog
[ https://issues.apache.org/jira/browse/SPARK-26949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun reassigned SPARK-26949:
-------------------------------------
Assignee: Jungtaek Lim
> Prevent "purge" to remove needed batch files in CompactibleFileStreamLog
> ------------------------------------------------------------------------
>
> Key: SPARK-26949
> URL: https://issues.apache.org/jira/browse/SPARK-26949
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Minor
>
> I've seen couple of trials (in opened PRs, even I've also tried) which calls purge() in CompactibleFileStreamLog, but after looking at the codebase of CompactibleFileStreamLog, I've realized that purging latest compaction batch would break internal of CompactibleFileStreamLog and it throws IllegalStateException.
> Given that CompactibleFileStreamLog maintains the batches and purges according to its configuration, it would be safer to just rely on CompactibleFileStreamLog to purge and prevent calling `purge` outside of CompactibleFileStreamLog.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org