[GitHub] HeartSaVioR commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog

[GitBox <gi...@apache.org>]

HeartSaVioR commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
URL: https://github.com/apache/spark/pull/23850#issuecomment-465861957
 
 
   While I think it is safest way to only let CompactibleFileStreamLog to maintain logs, we have alternative options here:
   
   1) Only purge batches which are safe to delete and don't throw exception
   
   Suppose it has batch 0, 1, 2, 3, 4 which batch 2 is compacted. If 4 is given as parameter, purge will try to remove 0, 1, 2, 3 which removing batch 2 (latest compaction batch) would break the internal state. Instead of this, this method could be overridden to remove only batch 0 and 1 and silently ignore removing 2 and 3.
   
   2) Throw exception (maybe IllegalArgumentException, or IllegalStateException?) only when batches to purge contain the latest compaction batch
   
   This would selectively throw exception - when it can break internal state of CompactibleFileStreamLog.
   
   Please let me know if alternative would make more sense. Thanks in advance!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org