[GitHub] HeartSaVioR commented on issue #23840: [WIP][DISCUSSION_NEEDED][SPARK-24295][SS] Add option to retain only last batch in file stream sink metadata

[GitBox <gi...@apache.org>]

HeartSaVioR commented on issue #23840: [WIP][DISCUSSION_NEEDED][SPARK-24295][SS] Add option to retain only last batch in file stream sink metadata
URL: https://github.com/apache/spark/pull/23840#issuecomment-467160556
 
 
   FileStreamSink only reads the last batch of metadata to determine which batch sink wrote successfully. The huge metadata on FileStreamSink is actually not for FileStreamSink (it just effectively takes `the last successful batch ID`, for File(Stream)Source.
   
   Now I think the ideal way to address is incorporating last successful batch ID to query checkpoint and only writing sink metadata when option is enabled (maybe we should make it default though since it breaks backward compatibility).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org