Re: [I] S3 client for storage path not available [iceberg]

["nastra (via GitHub)" <gi...@apache.org>]

nastra commented on issue #13174:
URL: https://github.com/apache/iceberg/issues/13174#issuecomment-2929951166

   > Hey [@and124578963](https://github.com/and124578963) ,
   > 
   > To fix this S3FileIO issue, add the missing catalog implementation:
   > 
   > ```
   > spark.sql.catalog.iceberg.catalog-impl=org.apache.iceberg.spark.SparkCatalog
   > ```
   > 
   > Also, change your S3A paths from `s3a://` to `s3://` in your data locations. S3FileIO doesn't support the `s3a://` scheme, it only works with `s3://`.
   > 
   > The error occurs because S3FileIO can't find a registered client for the `s3a://` storage path.
   
   Setting `spark.sql.catalog.iceberg.catalog-impl=org.apache.iceberg.spark.SparkCatalog` is not a valid setting. 
   With `catalog-impl` you would configure an actual Iceberg catalog (anything that implements `org.apache.iceberg.catalog.Catalog`). Instead of `catalog-impl` you can also use `type` to refer to the catalog type instead of having to specify the fully-qualified name of the Iceberg catalog. 
   
   
   In your case I'm assuming you're using 
   ```
   spark.sql.catalog.iceberg: org.apache.iceberg.spark.SparkCatalog
   spark.sql.catalog.iceberg.type: hive
   ```
   which means you're using Iceberg's `HiveCatalog`.
   
   Using `s3a://` with `S3FileIO` is a perfectly valid use case, so there's no need to change data locations. 
   
   However, I'm still surprised why my changes from https://github.com/apache/iceberg/pull/12799 wouldn't work with `s3a://`, since we have some tests there that make sure  this works. 
   
   @and124578963 could you please check the logs whether setting up the s3 client actually failed (which can happen if certain settings are missing) and the `[BUG] S3 client for storage path not available` is actually just the result you're seeing?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org