[GitHub] [spark] steveloughran commented on a diff in pull request #37474: [SPARK-40039][SS] Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

[GitBox <gi...@apache.org>]

steveloughran commented on code in PR #37474:
URL: https://github.com/apache/spark/pull/37474#discussion_r953592317


##########
docs/cloud-integration.md:
##########
@@ -231,9 +231,15 @@ The size of the window needs to be set to handle this.
 is no need for a workflow of write-then-rename to ensure that files aren't picked up
 while they are still being written. Applications can write straight to the monitored directory.
 
-1. Streams should only be checkpointed to a store implementing a fast and
-atomic `rename()` operation.
-Otherwise the checkpointing may be slow and potentially unreliable.
+1. In case of the default checkpoint file manager called `FileContextBasedCheckpointFileManager`
+streams should only be checkpointed to a store implementing a fast and
+atomic `rename()` operation. Otherwise the checkpointing may be slow and potentially unreliable.
+On AWS S3 with Hadoop 3.3.1 or later the abortable stream based checkpoint file manager

Review Comment:
   need to be specific about s3a connector, as EMR doesn't have it (yet). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org