[GitHub] kl0u opened a new pull request #7160: S3 sink cleanup

[GitBox <gi...@apache.org>]

kl0u opened a new pull request #7160: S3 sink cleanup
URL: https://github.com/apache/flink/pull/7160
 
 
   ## What is the purpose of the change
   
   The S3 `RecoverableWriter` uses the Multipart Upload (MPU) Feature of S3 in order to upload the different part files. This means that a large part is split in chunks of at least 5MB which are uploaded independently, whenever each one of them is ready.
   
   This 5MB minimum size requires special handling of parts that are less than 5MB when a checkpoint barrier arrives. These small files are uploaded as independent objects (not associated with an active MPU). This way, when Flink needs to restore, it simply downloads them and resumes writing to them.
   
   These small objects are currently not cleaned up, thus leading to wasted space on S3.
   
   This PR changes this by cleaning these files upon restoring and also when the "next" checkpoint is successfully acknowledged. 
   
   LIMITATION: Upon a failure, still there may be objects that were created after the last successful checkpoint but before the failure. These do not belong to the checkpointed state and they are not cleaned up. But this is a more general limitation which will be covered by another JIRA.
   
   ## Brief change log
   
   This PR adds the following methods to the `RecoverableWriter`:
   
   ```
   boolean cleanupRecoverableState(ResumeRecoverable resumable) throws IOException;
   
   boolean requiresCleanupOfRecoverableState();
   ```
   
   The second one is just for optimization purposes.
   
   ## Verifying this change
   
   Added the `BucketTest` class and 2 tests in the `HadoopS3RecoverableWriterITCase`.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (**yes** / no / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / **no**)
     - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services