[I] [Bug] ResolvingFileIO does not forward newTwoPhaseOutputStream, losing native multipart commit [paimon]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang opened a new issue, #9650:
URL: https://github.com/apache/paimon/issues/9650

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   ### Paimon version
   
   master, `475be566f` (2.1-SNAPSHOT).
   
   ### Compute Engine
   
   Any engine writing a format table whose `FileIO` is a `ResolvingFileIO`, which is what `RESTCatalog.fileIOFromOptions` builds when `data-token.enabled` is off, and what `resolving-file-io.enabled` selects elsewhere.
   
   ### Minimal reproduce step
   
   Write a format table on OSS or S3 through a `ResolvingFileIO`. `FormatTableSingleFileWriter` asks for a two-phase output stream:
   
   ```java
   TwoPhaseOutputStream out = fileIO.newTwoPhaseOutputStream(path, false);
   ```
   
   `ResolvingFileIO` overrides `newOutputStream` and `tryToWriteAtomic` to forward to the resolved FileIO, but not `newTwoPhaseOutputStream`, so this one falls through to the interface default:
   
   ```java
   default TwoPhaseOutputStream newTwoPhaseOutputStream(Path path, boolean overwrite)
           throws IOException {
       return new RenamingTwoPhaseOutputStream(this, path, overwrite);
   }
   ```
   
   That writes `<dir>/_temporary/.tmp.<uuid>` and commits by renaming. `OSSFileIO`, `S3FileIO` and `JindoFileIO` all override the method with a native multipart-upload commit, and `RESTTokenFileIO` forwards to the resolved FileIO the way the other `ResolvingFileIO` methods do, so this is the one wrapper that loses it.
   
   On an object store, rename is a server-side copy plus delete: the whole file is copied a second time at commit, the commit is not atomic, and a `_temporary` directory is left behind.
   
   ### What doesn't meet your expectations?
   
   `ResolvingFileIO` exists to delegate to the FileIO that owns the scheme, and the sibling wrapper `RESTTokenFileIO` already forwards this method. The missing override quietly downgrades every format-table write on object storage.
   
   ### Anything else?
   
   Adding the forwarding on its own is not enough, which is why I am filing this rather than sending that patch. The native committers cast the FileIO they are handed at commit time:
   
   ```java
   // OSSMultiPartUploadCommitter.multiPartUploadStore
   OSSFileIO ossFileIO = (OSSFileIO) fileIO;
   ```
   
   `FormatTableCommit` commits with the FileIO it was constructed with, `message.getCommitter().commit(fileIO)`, which is the table's `ResolvingFileIO`. So forwarding the stream creation hands out an OSS-native committer that then receives a `ResolvingFileIO` and fails with a `ClassCastException` at commit, trading a slow commit for a broken one. `S3MultiPartUploadCommitter` and the Jindo one have the same cast.
   
   Making this work needs a decision about who resolves at commit time: the committers unwrapping a `ResolvingFileIO` themselves, `FormatTableCommit` passing an already-resolved FileIO, or `BaseMultiPartUploadCommitter` resolving before it casts. I did not want to pick one of those on your behalf inside a bug fix. Happy to implement whichever you prefer.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] ResolvingFileIO does not forward newTwoPhaseOutputStream, losing native multipart commit [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed issue #9650: [Bug] ResolvingFileIO does not forward newTwoPhaseOutputStream, losing native multipart commit
URL: https://github.com/apache/paimon/issues/9650


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org