[jira] [Commented] (FLINK-39778) Recoverable writer silently loses the in-flight tail on resume

["Roman Khachatryan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-39778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089048#comment-18089048 ] 

Roman Khachatryan commented on FLINK-39778:
-------------------------------------------

Merged https://github.com/apache/flink/commit/757749336eafba19220b9d2223128018078d42a8

> Recoverable writer silently loses the in-flight tail on resume
> --------------------------------------------------------------
>
>                 Key: FLINK-39778
>                 URL: https://issues.apache.org/jira/browse/FLINK-39778
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Connectors / FileSystem
>            Reporter: Samrat Deb
>            Assignee: Samrat Deb
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.4.0
>
>
> persist() uploads the un-parted tail to a side object <key>/.incomplete/<uploadId>/<uuid> and records incompleteObjectName in the NativeS3Recoverable. But recover() rebuilds
> the stream from parts() + numBytesInParts() only — s3recoverable.incompleteObjectName() is never read:
> return new NativeS3RecoverableFsDataOutputStream(
> s3AccessHelper, s3recoverable.getObjectName(), s3recoverable.uploadId(),
> localTmpDir, userDefinedMinPartSize,
> s3recoverable.parts(), s3recoverable.numBytesInParts());
> After resume from a checkpoint with a partial part, those bytes are silently dropped — exactly-once degrades to "at-most-once minus tail." cleanupRecoverableState() deletes
> the side object, confirming the intent; recover() just forgot to read it back.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)