[GitHub] addisonj opened a new pull request #7460: [FLINK-11187] [s3] Use file over stream for writes

[GitBox <gi...@apache.org>]

addisonj opened a new pull request #7460: [FLINK-11187] [s3] Use file over stream for writes
URL: https://github.com/apache/flink/pull/7460
 
 
   
   
   ## What is the purpose of the change
   This changes the S3AccessHelper API to take a file instead of an input
   stream.
   
   This allows s3 client to properly reset a file instead of an iUse file
   over stream for writes.
   
   This fixes an issue where the underlying s3 implementation has an
   intermittent failure, tries to reset the stream, fails to do so, and
   results in hung requests with delayed errors.
   
   ## Brief change log
   
   - Improves the ability to handle intermittent errors when using the StreamingFileSink against s3 filesystem
   
   
   ## Verifying this change
   
   Currently, because this only happens with intermittent s3 issues, it is very difficult to reproduce, however, it does seem to fix the problem for multiple people, see the JIRA issue for details
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: (yes / no / don't know)
     - The runtime per-record code paths (performance sensitive): no 
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
     - The S3 file system connector: yes
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? n/a
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services