You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "risey-yimu (via GitHub)" <gi...@apache.org> on 2025/08/11 10:01:05 UTC
[I] REST catalog use spark remove_orphan_files Failed [iceberg]
risey-yimu opened a new issue, #13781:
URL: https://github.com/apache/iceberg/issues/13781
### Apache Iceberg version
1.8.1
### Query engine
Spark
### Please describe the bug 🐞
use spark3.5.3 remove_orphan_files Failed!spark conf are flows:
```scala
sparkConf
.set("spark.ui.enabled", "true")
.set("spark.task.maxFailures", ls0Config.sparkTaskMaxFailures)
.set("spark.rpc.message.maxSize", ls0Config.sparkRpcMessageMaxSize)
.set("spark.sql.iceberg.handle-timestamp-without-timezone", "true")
.set("spark.hadoop.fs.s3a.access.key",ls0Config.jdbcCatalogS3AccessKey)
.set("spark.hadoop.fs.s3a.secret.key", ls0Config.jdbcCatalogS3SecretKey)
.set("spark.hadoop.fs.s3a.endpoint", ls0Config.jdbcCatalogS3Endpoint)
.set("spark.hadoop.fs.s3a.path.style.access", "true")
.set("spark.hadoop.fs.s3a.region", "cn-east-1")
.set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
.set("spark.hadoop.fs.defaultFS", "s3a://warehouse")
.set("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
.set("spark.hadoop.fs.AbstractFileSystem.s3.impl","org.apache.hadoop.fs.s3a.S3A")
.set("spark.sql.extensions", "org.projectnessie.spark.extensions.NessieSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.set(s"spark.sql.catalog.${ls0Config.restCatalogName}", "org.apache.iceberg.spark.SparkCatalog")
.set(s"spark.sql.catalog.${ls0Config.restCatalogName}.type", "rest")
.set(s"spark.sql.catalog.${ls0Config.restCatalogName}.uri", ls0Config.restCatalogURL)
.set(s"spark.sql.catalogImplementation", "in-memory")
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}", "org.apache.iceberg.spark.SparkCatalog")
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.type", "jdbc")
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.uri", ls0Config.jdbcCatalogJDBCURL)
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.jdbc.user", ls0Config.jdbcCatalogJDBCUser)
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.jdbc.password", ls0Config.jdbcCatalogJDBCPwd)
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.warehouse", ls0Config.jdbcCatalogWarehousePath)
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.s3.endpoint", ls0Config.jdbcCatalogS3Endpoint)
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.s3.access-key-id", ls0Config.jdbcCatalogS3AccessKey)
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.s3.secret-access-key", ls0Config.jdbcCatalogS3SecretKey)
.set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.client.region", ls0Config.jdbcCatalogClientRegion)
.set("spark.sql.defaultCatalog", ls0Config.restCatalogName)
SparkSession.builder.config(sparkConf).getOrCreate
```
I use spark-sql function remove_orphan_files to delete orphan_files,code:
```scala
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
spark.sparkContext.setLogLevel("INFO")
spark.sql("ALTER TABLE nessie.demo.ods_source_nome_t4 SET TBLPROPERTIES ('gc.enabled' = 'true')")
spark.sql(
s"""
|CALL nessie.system.remove_orphan_files(table => 'nessie.demo.ods_source_nome_test' ,location => 's3a://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/',older_than => TIMESTAMP '2025-08-03 00:00:00.000')
|""".stripMargin)
```
but l found logs:
```
java.lang.IllegalArgumentException: Invalid S3 URI: 'http://los.uisee.com/warehouse?delete'
at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:206) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:188) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:224) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:308) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
at org.apache.iceberg.rest.BaseHTTPClient.post(BaseHTTPClient.java:100) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
at org.apache.iceberg.aws.s3.signer.S3V4RestSignerClient.sign(S3V4RestSignerClient.java:351) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.lambda$signRequest$4(SigningStage.java:154) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:63) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.signRequest(SigningStage.java:153) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.execute(SigningStage.java:72) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.execute(SigningStage.java:50) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) ~[iceberg-aws-bundle-1.8.1.jar:?]
at software.amazon.awssdk.services.s3.DefaultS3Client.deleteObjects(DefaultS3Client.java:3626) ~[iceberg-aws-bundle-1.8.1.jar:?]
at org.apache.iceberg.aws.s3.S3FileIO.deleteBatch(S3FileIO.java:281) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
at org.apache.iceberg.aws.s3.S3FileIO.lambda$deleteFiles$3(S3FileIO.java:219) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00177.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00191.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00186.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00186.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00190.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00176.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00195.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00182.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00177.parquet
17:52:36.791 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00173.parquet
17:52:36.792 [main] WARN org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00182.parquet
17:52:36.792 [main] WARN org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction - Deleted only 0 of 11 files using bulk deletes
17:52:36.809 [main] INFO org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - Code generated in 3.935872 ms
17:52:36.826 [main] INFO org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - Code generated in 9.080008 ms
[s3a://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00173.parquet]
[s3a://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00177.parquet]
[s3a://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00182.parquet]
```
why delete orphan_files failed,How to solve this problem
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177891721
I couldn't tell as some of it is based on variables `ls0Config`, I would recommend comparing it with your JDBC config and check. @nastra: Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177425546
> > java.lang.IllegalArgumentException: Invalid S3 URI: 'http://los.uisee.com/warehouse?delete'
>
> Also, the error is not from Nessie. It is from AWS SDK. You are using the invalid configs? Even if you change catalogs, I assume you would get the same errors.
>
> It is using `S3FileIO` and reporting `Invalid S3 URI: 'http://los.uisee.com/warehouse?delete'`, check how that URI came from.
I debugged this program.This error occurs where the image is displayed.
<img width="1585" height="617" alt="Image" src="https://github.com/user-attachments/assets/6bac7587-62fa-427d-aa3e-c28f02eb2e13" />
I saw this URI in the S3 client. However, the configuration in the program all uses the IP address, not http://los.uisee.com. I'm not sure where this URI is obtained from.
<img width="1631" height="931" alt="Image" src="https://github.com/user-attachments/assets/39eb5be8-672a-42cd-ae9a-a42b987a7c03" />
orphan files list is correct
<img width="1675" height="439" alt="Image" src="https://github.com/user-attachments/assets/2b40510b-0373-47dc-9fe1-51d4a7f913b2" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177773091
> maybe it is this configuration?
>
> > .set("spark.hadoop.fs.s3a.endpoint", ls0Config.jdbcCatalogS3Endpoint)
I changed the configuration to this,but still got this error.Is there anything wrong with these configurations?
```
sparkConf
.set("spark.ui.enabled", "true")
.set("spark.task.maxFailures", ls0Config.sparkTaskMaxFailures)
.set("spark.rpc.message.maxSize", ls0Config.sparkRpcMessageMaxSize)
.set("spark.sql.iceberg.handle-timestamp-without-timezone", "true")
.set("spark.hadoop.fs.s3a.access.key",ls0Config.jdbcCatalogS3AccessKey)
.set("spark.hadoop.fs.s3a.secret.key", ls0Config.jdbcCatalogS3SecretKey)
.set("spark.hadoop.fs.s3a.endpoint", "http://10.0.162.28:9000")
.set("spark.hadoop.fs.s3a.path.style.access", "true")
.set("spark.hadoop.fs.s3a.region", "cn-east-1")
.set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
.set("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
.set("spark.hadoop.fs.AbstractFileSystem.s3.impl","org.apache.hadoop.fs.s3a.S3A")
.set("spark.sql.extensions", "org.projectnessie.spark.extensions.NessieSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.set(s"spark.sql.catalog.nessie", "org.apache.iceberg.spark.SparkCatalog")
.set(s"spark.sql.catalog.nessie.type", "rest")
.set(s"spark.sql.catalog.nessie.uri", "http://10.0.160.110:31107/iceberg/main")
.set(s"spark.sql.catalog.nessie.warehouse", "s3://warehouse/")
.set(s"spark.sql.catalog.nessie.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
.set(s"spark.sql.catalog.nessie.s3.endpoint", "http://10.0.162.28:9000")
.set(s"spark.sql.catalog.nessie.s3.access-key-id", ls0Config.jdbcCatalogS3AccessKey)
.set(s"spark.sql.catalog.nessie.s3.secret-access-key", ls0Config.jdbcCatalogS3SecretKey)
.set(s"spark.sql.catalog.nessie.client.region", ls0Config.jdbcCatalogClientRegion)
.set(s"spark.sql.catalogImplementation", "in-memory")
.set(s"spark.sql.catalog.jdbc", "org.apache.iceberg.spark.SparkCatalog")
.set(s"spark.sql.catalog.jdbc.type", "jdbc")
.set(s"spark.sql.catalog.jdbc.uri", ls0Config.jdbcCatalogJDBCURL)
.set(s"spark.sql.catalog.jdbc.jdbc.user", ls0Config.jdbcCatalogJDBCUser)
.set(s"spark.sql.catalog.jdbc.jdbc.password", ls0Config.jdbcCatalogJDBCPwd)
.set(s"spark.sql.catalog.jdbc.warehouse", "s3://warehouse/")
.set(s"spark.sql.catalog.jdbc.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
.set(s"spark.sql.catalog.jdbc.s3.endpoint", "http://10.0.162.28:9000")
.set(s"spark.sql.catalog.jdbc.s3.access-key-id", ls0Config.jdbcCatalogS3AccessKey)
.set(s"spark.sql.catalog.jdbc.s3.secret-access-key", ls0Config.jdbcCatalogS3SecretKey)
.set(s"spark.sql.catalog.jdbc.client.region", ls0Config.jdbcCatalogClientRegion)
.set("spark.sql.defaultCatalog", "nessie")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177421999
I debugged this program.This error occurs where the image is displayed.
<img width="1585" height="617" alt="Image" src="https://github.com/user-attachments/assets/6bac7587-62fa-427d-aa3e-c28f02eb2e13" />
I saw this URI in the S3 client. However, the configuration in the program all uses the IP address, not http://los.uisee.com. I'm not sure where this URI is obtained from.
<img width="1631" height="931" alt="Image" src="https://github.com/user-attachments/assets/39eb5be8-672a-42cd-ae9a-a42b987a7c03" />
orphan files list is correct
<img width="1675" height="439" alt="Image" src="https://github.com/user-attachments/assets/2b40510b-0373-47dc-9fe1-51d4a7f913b2" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3178205052
> [@ajantha-bhat](https://github.com/ajantha-bhat) the error comes from S3 remote signing. I'm guessing Nessie server is sending back the `s3.remote-signing-enabled` flag (indicating that it supports remote signing) with an additional `s3.signer.endpoint` that causes the issue.
>
> [@risey-yimu](https://github.com/risey-yimu) in your debugging, can you check the values for the Signer configuration and share a screenshot?
After debugging again, I found that the error occurred in the deleteFiles method of the DeleteOrphanFilesSparkAction class.

The IO here is from table.io. So could this problem be related to nessie's configuration?The configuration is provided by the partner above
<img width="1154" height="1065" alt="Image" src="https://github.com/user-attachments/assets/92c3d9c4-2db1-4c5f-a7de-10ff5746b823" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3235894126
@nastra @ajantha-bhat @Fokko @aokolnychyi Please help me find a way to solve this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "aka-peter (via GitHub)" <gi...@apache.org>.
aka-peter commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3292322512
I ran into the exact same problem. It was solved by adding this config to nessie:
`nessie.catalog.service.s3.default-options.request-signing-enabled=false
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3174552507
For Nessie, we recommend using Nessie gc tool from the nessie repo (not the Iceberg procedures). I recall we blocked it with `gc.enabled = false` (cannot override from alter table properties, at runtime it always used to set to false). So, I wonder how you executed it. Maybe something changed in the remove orphan files that doesn't check gc flag?
> java.lang.IllegalArgumentException: Invalid S3 URI: 'http://los.uisee.com/warehouse?delete'
Also, the error is not from Nessie. It is from AWS SDK. You are using the invalid configs?
Even if you change catalogs, I assume you would get the same errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177716527
maybe it is this configuration?
> .set("spark.hadoop.fs.s3a.endpoint", ls0Config.jdbcCatalogS3Endpoint)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3178549679
> [@risey-yimu](https://github.com/risey-yimu) can you please share the debugging info of the S3 client's configuration (basically what you shared in [#13781 (comment)](https://github.com/apache/iceberg/issues/13781#issuecomment-3177425546)) where one can see the Signer configuration that the server sent back
<img width="1096" height="566" alt="Image" src="https://github.com/user-attachments/assets/92ca0960-b55f-420c-8567-fa0d18e3b83d" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3186452908
> [@risey-yimu](https://github.com/risey-yimu) can you please share the debugging info of the S3 client's configuration (basically what you shared in [#13781 (comment)](https://github.com/apache/iceberg/issues/13781#issuecomment-3177425546)) where one can see the Signer configuration that the server sent back
Thanks @nastra and @ajantha-bhat Do you need any additional debug information or configuration? If need, please let us know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177928639
@ajantha-bhat the error comes from S3 remote signing. I'm guessing Nessie server is sending back the `s3.remote-signing-enabled` flag (indicating that it supports remote signing) with an additional `s3.signer.endpoint` that causes the issue.
@risey-yimu in your debugging, can you check the values for the Signer configuration and share a screenshot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3178251422
@risey-yimu can you please share the debugging info of the S3 client's configuration (basically what you shared in https://github.com/apache/iceberg/issues/13781#issuecomment-3177425546) where one can see the Signer configuration that the server sent back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3174213392
@ajantha-bhat could you take a look at this please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3178106163
> [@ajantha-bhat](https://github.com/ajantha-bhat) the error comes from S3 remote signing. I'm guessing Nessie server is sending back the `s3.remote-signing-enabled` flag (indicating that it supports remote signing) with an additional `s3.signer.endpoint` that causes the issue.
>
> [@risey-yimu](https://github.com/risey-yimu) in your debugging, can you check the values for the Signer configuration and share a screenshot?
@nastra
<img width="1056" height="758" alt="Image" src="https://github.com/user-attachments/assets/d7aca0f0-ba66-4db0-b6e4-3cd5740d4f0d" />
nessie server with restcatalog config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "TayfunBugraBas (via GitHub)" <gi...@apache.org>.
TayfunBugraBas commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3636515452
> I ran into the exact same problem. It was solved by adding this config to nessie:
>
> `nessie.catalog.service.s3.default-options.request-signing-enabled=false `
I have the same error but when i try this solution i get different error,
error is as follows :
25/12/09 12:07:33 WARN TaskSetManager: Lost task 0.0 in stage 6.0 (TID 4) (10.42.7.116 executor 1): software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentia
lsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.,
InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
credentials were predefined and working
nessie: 0.104.3
spark: 3.5.3
iceberg:1.9.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]
Posted by "TayfunBugraBas (via GitHub)" <gi...@apache.org>.
TayfunBugraBas commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3640571891
if you are using the nessie catalog type rest, try to change it with org.apache.iceberg.nessie.NessieCatalog
Not Working(REST):
`spark.sql.catalog.nessie.type=rest`
Working:
`spark.sql.catalog.nessie.catalog-impl=org.apache.iceberg.nessie.NessieCatalog`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org