[GitHub] [spark] HeartSaVioR commented on a change in pull request #24128: [SPARK-27188][SS] FileStreamSink: provide a new option to disable metadata log

[GitBox <gi...@apache.org>]

HeartSaVioR commented on a change in pull request #24128: [SPARK-27188][SS] FileStreamSink: provide a new option to disable metadata log
URL: https://github.com/apache/spark/pull/24128#discussion_r266374734
 
 

 ##########
 File path: docs/structured-streaming-programming-guide.md
 ##########
 @@ -1807,13 +1807,19 @@ Here are the details of all the sinks in Spark.
     <td>Append</td>
     <td>
         <code>path</code>: path to the output directory, must be specified.
+        <br/>
+        <code>disableMetadata</code>: whether to disable metadata log files or not (default: false)
+        Metadata log is growing incrementally while running streaming query which affect query execution time as well as disk space.
+        Disabling metadata log greatly helps on remedying the impact, but it changes fault-tolerance guarantee of FileStreamSink to
 
 Review comment:
   This might be an arguable topic: if the default implementation of batch query (`SQLHadoopMapReduceCommitProtocol`) is taken for commit protocol, which fault-tolerance semantic it can guarantee? I'm not sure so safely mention 'at-least-once' here, but we can change it if we feel it's still 'exactly-once'.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org