Re: [I] Spark Iceberg streaming - checkpoint leverages S3fileIo signer path instead of hadoop's S3AFileSystem [iceberg]

["c2zwdjnlcg (via GitHub)" <gi...@apache.org>]

c2zwdjnlcg commented on issue #14762:
URL: https://github.com/apache/iceberg/issues/14762#issuecomment-3849224374

   Ran into this as well using s3 vended credential from a rest catalog.
   
   > Yes I think it's correct that the checkpoint uses the Table's FileIO, otherwise I don't see what other options we have and what other FileIO it would use. It sounds like it's possibly a permission issue on the Lakekeeper side?
   
   @nastra Not sure I agree, why would the catalog know anything about the checkpoint location for the spark job to vend credentials for it? The checkpoint location would be unique to each spark job and not at all related to the load table call. 
   
   I'd say we should fail back to the configured defaults rather than reusing the table's S3FileIo object. If you agree I can contribute a patch for this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org