[GitHub] alfredo-gimenez edited a comment on issue #23840: [WIP][DISCUSSION_NEEDED][SPARK-24295][SS] Add option to retain only last batch in file stream sink metadata

[GitBox <gi...@apache.org>]

alfredo-gimenez edited a comment on issue #23840: [WIP][DISCUSSION_NEEDED][SPARK-24295][SS] Add option to retain only last batch in file stream sink metadata
URL: https://github.com/apache/spark/pull/23840#issuecomment-467142031
 
 
   Taking a look at this, as it could fix a longstanding critical performance bug in our ingestion pipeline. Couple comments:
   
   1. When retaining only the last metadata log, we should probably also disable log file compaction, since it will just copy the last log.
   
   2. Can this be used in conjunction with checkpoints? If checkpoints are present, they depend on the metadata log contents. If they are guaranteed to only use the last metadata log, this is fine, otherwise this case will have to be handled separately.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org