[GitHub] [spark] HeartSaVioR opened a new pull request #24128: [SPARK-27188][SS] FileStreamSink: provide a new option to disable metadata log

[GitBox <gi...@apache.org>]

HeartSaVioR opened a new pull request #24128: [SPARK-27188][SS] FileStreamSink: provide a new option to disable metadata log
URL: https://github.com/apache/spark/pull/24128
 
 
   ## What changes were proposed in this pull request?
   
   This patch proposes to provide a new option to disable metadata log in FileStreamSink, which is only leveraged when Spark reads the output directory. 
   
   The metadata log greatly helps to easily achieve exactly-once but given the output path is open to arbitrary readers, there's no way to compact the metadata log, which ends up growing the metadata file as query runs for long time, especially for compacted batch.
   (There're some reports from end users which include their workarounds: [SPARK-24295](https://issues.apache.org/jira/browse/SPARK-24295))
   
   This patch takes a different commit protocol which doesn't leverage metadata if disable metadata option is true, as well as leverages no-op metadata log to ensure the code is not much diverged for each mode.
   
   ## How was this patch tested?
   
   Added unit tests.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org