You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "risey-yimu (via GitHub)" <gi...@apache.org> on 2025/08/11 10:01:05 UTC

[I] REST catalog use spark remove_orphan_files Failed [iceberg]

risey-yimu opened a new issue, #13781:
URL: https://github.com/apache/iceberg/issues/13781

   ### Apache Iceberg version
   
   1.8.1
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   use spark3.5.3 remove_orphan_files Failed!spark conf are flows:
   ```scala
   sparkConf
         .set("spark.ui.enabled", "true")
         .set("spark.task.maxFailures", ls0Config.sparkTaskMaxFailures)
         .set("spark.rpc.message.maxSize", ls0Config.sparkRpcMessageMaxSize)
         .set("spark.sql.iceberg.handle-timestamp-without-timezone", "true")
   
         .set("spark.hadoop.fs.s3a.access.key",ls0Config.jdbcCatalogS3AccessKey)
         .set("spark.hadoop.fs.s3a.secret.key", ls0Config.jdbcCatalogS3SecretKey)
         .set("spark.hadoop.fs.s3a.endpoint", ls0Config.jdbcCatalogS3Endpoint)
         .set("spark.hadoop.fs.s3a.path.style.access", "true")
         .set("spark.hadoop.fs.s3a.region", "cn-east-1")
         .set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
         .set("spark.hadoop.fs.defaultFS", "s3a://warehouse")
         .set("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
         .set("spark.hadoop.fs.AbstractFileSystem.s3.impl","org.apache.hadoop.fs.s3a.S3A")
   
   
         .set("spark.sql.extensions", "org.projectnessie.spark.extensions.NessieSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
         .set(s"spark.sql.catalog.${ls0Config.restCatalogName}", "org.apache.iceberg.spark.SparkCatalog")
         .set(s"spark.sql.catalog.${ls0Config.restCatalogName}.type", "rest")
         .set(s"spark.sql.catalog.${ls0Config.restCatalogName}.uri", ls0Config.restCatalogURL)
         .set(s"spark.sql.catalogImplementation", "in-memory")
   
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}", "org.apache.iceberg.spark.SparkCatalog")
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.type", "jdbc")
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.uri", ls0Config.jdbcCatalogJDBCURL)
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.jdbc.user", ls0Config.jdbcCatalogJDBCUser)
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.jdbc.password", ls0Config.jdbcCatalogJDBCPwd)
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.warehouse", ls0Config.jdbcCatalogWarehousePath)
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.s3.endpoint", ls0Config.jdbcCatalogS3Endpoint)
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.s3.access-key-id", ls0Config.jdbcCatalogS3AccessKey)
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.s3.secret-access-key", ls0Config.jdbcCatalogS3SecretKey)
         .set(s"spark.sql.catalog.${ls0Config.jdbcCatalogName}.client.region", ls0Config.jdbcCatalogClientRegion)
   
         .set("spark.sql.defaultCatalog", ls0Config.restCatalogName)
   
   
       SparkSession.builder.config(sparkConf).getOrCreate
   ``` 
   I use spark-sql function remove_orphan_files to delete orphan_files,code:
   ```scala
       spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
       spark.sparkContext.setLogLevel("INFO")
       spark.sql("ALTER TABLE nessie.demo.ods_source_nome_t4 SET TBLPROPERTIES ('gc.enabled' = 'true')")
       spark.sql(
         s"""
            |CALL nessie.system.remove_orphan_files(table => 'nessie.demo.ods_source_nome_test' ,location => 's3a://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/',older_than => TIMESTAMP '2025-08-03 00:00:00.000')
            |""".stripMargin)
   ``` 
   but l found logs:
   ```
   java.lang.IllegalArgumentException: Invalid S3 URI: 'http://los.uisee.com/warehouse?delete'
   	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:206) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
   	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:188) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
   	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:224) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
   	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:308) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
   	at org.apache.iceberg.rest.BaseHTTPClient.post(BaseHTTPClient.java:100) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
   	at org.apache.iceberg.aws.s3.signer.S3V4RestSignerClient.sign(S3V4RestSignerClient.java:351) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.lambda$signRequest$4(SigningStage.java:154) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:63) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.signRequest(SigningStage.java:153) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.execute(SigningStage.java:72) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.execute(SigningStage.java:50) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at software.amazon.awssdk.services.s3.DefaultS3Client.deleteObjects(DefaultS3Client.java:3626) ~[iceberg-aws-bundle-1.8.1.jar:?]
   	at org.apache.iceberg.aws.s3.S3FileIO.deleteBatch(S3FileIO.java:281) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
   	at org.apache.iceberg.aws.s3.S3FileIO.lambda$deleteFiles$3(S3FileIO.java:219) ~[iceberg-spark-runtime-3.5_2.12-1.8.1.jar:?]
   	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   	at java.lang.Thread.run(Thread.java:829) [?:?]
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00177.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00191.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00186.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00186.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00190.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00176.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00195.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00182.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00177.parquet
   17:52:36.791 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00173.parquet
   17:52:36.792 [main] WARN  org.apache.iceberg.aws.s3.S3FileIO - Failed to delete object at path s3://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00001-0-baf52e17-063f-4a42-a02a-18b1286b8a77-00182.parquet
   17:52:36.792 [main] WARN  org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction - Deleted only 0 of 11 files using bulk deletes
   17:52:36.809 [main] INFO  org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - Code generated in 3.935872 ms
   17:52:36.826 [main] INFO  org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - Code generated in 9.080008 ms
   [s3a://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00173.parquet]
   [s3a://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00177.parquet]
   [s3a://warehouse/demo/ods_source_nome_test_e0341727-b242-434a-ad20-d6b120d6fd59/data/dt=2025-07-31/00000-0-cd989951-6115-4da4-bb6e-b8942a69ba31-00182.parquet]
   
   ``` 
   why delete orphan_files failed,How to solve this problem
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [x] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177891721

   I couldn't tell as some of it is based on variables `ls0Config`, I would recommend comparing it with your JDBC config and check. @nastra: Thoughts?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177425546

   > > java.lang.IllegalArgumentException: Invalid S3 URI: 'http://los.uisee.com/warehouse?delete'
   > 
   > Also, the error is not from Nessie. It is from AWS SDK. You are using the invalid configs? Even if you change catalogs, I assume you would get the same errors.
   > 
   > It is using `S3FileIO` and reporting `Invalid S3 URI: 'http://los.uisee.com/warehouse?delete'`, check how that URI came from.
   
   I debugged this program.This error occurs where the image is displayed.
   
   <img width="1585" height="617" alt="Image" src="https://github.com/user-attachments/assets/6bac7587-62fa-427d-aa3e-c28f02eb2e13" />
   I saw this URI in the S3 client. However, the configuration in the program all uses the IP address, not http://los.uisee.com. I'm not sure where this URI is obtained from.
   
   <img width="1631" height="931" alt="Image" src="https://github.com/user-attachments/assets/39eb5be8-672a-42cd-ae9a-a42b987a7c03" />
   
   orphan files list is correct
   <img width="1675" height="439" alt="Image" src="https://github.com/user-attachments/assets/2b40510b-0373-47dc-9fe1-51d4a7f913b2" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177773091

   > maybe it is this configuration?
   > 
   > > .set("spark.hadoop.fs.s3a.endpoint", ls0Config.jdbcCatalogS3Endpoint)
   
   
   I changed the configuration to this,but still got this error.Is there anything wrong with these configurations?
   ```
   sparkConf
         .set("spark.ui.enabled", "true")
         .set("spark.task.maxFailures", ls0Config.sparkTaskMaxFailures)
         .set("spark.rpc.message.maxSize", ls0Config.sparkRpcMessageMaxSize)
         .set("spark.sql.iceberg.handle-timestamp-without-timezone", "true")
   
   
         .set("spark.hadoop.fs.s3a.access.key",ls0Config.jdbcCatalogS3AccessKey)
         .set("spark.hadoop.fs.s3a.secret.key", ls0Config.jdbcCatalogS3SecretKey)
         .set("spark.hadoop.fs.s3a.endpoint", "http://10.0.162.28:9000")
         .set("spark.hadoop.fs.s3a.path.style.access", "true")
         .set("spark.hadoop.fs.s3a.region", "cn-east-1")
         .set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
         .set("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
         .set("spark.hadoop.fs.AbstractFileSystem.s3.impl","org.apache.hadoop.fs.s3a.S3A")
   
   
         .set("spark.sql.extensions", "org.projectnessie.spark.extensions.NessieSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
         .set(s"spark.sql.catalog.nessie", "org.apache.iceberg.spark.SparkCatalog")
         .set(s"spark.sql.catalog.nessie.type", "rest")
         .set(s"spark.sql.catalog.nessie.uri", "http://10.0.160.110:31107/iceberg/main")
   
         .set(s"spark.sql.catalog.nessie.warehouse", "s3://warehouse/")
         .set(s"spark.sql.catalog.nessie.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
         .set(s"spark.sql.catalog.nessie.s3.endpoint", "http://10.0.162.28:9000")
         .set(s"spark.sql.catalog.nessie.s3.access-key-id", ls0Config.jdbcCatalogS3AccessKey)
         .set(s"spark.sql.catalog.nessie.s3.secret-access-key", ls0Config.jdbcCatalogS3SecretKey)
         .set(s"spark.sql.catalog.nessie.client.region", ls0Config.jdbcCatalogClientRegion)
         .set(s"spark.sql.catalogImplementation", "in-memory")
   
         .set(s"spark.sql.catalog.jdbc", "org.apache.iceberg.spark.SparkCatalog")
         .set(s"spark.sql.catalog.jdbc.type", "jdbc")
         .set(s"spark.sql.catalog.jdbc.uri", ls0Config.jdbcCatalogJDBCURL)
         .set(s"spark.sql.catalog.jdbc.jdbc.user", ls0Config.jdbcCatalogJDBCUser)
         .set(s"spark.sql.catalog.jdbc.jdbc.password", ls0Config.jdbcCatalogJDBCPwd)
   
         .set(s"spark.sql.catalog.jdbc.warehouse", "s3://warehouse/")
         .set(s"spark.sql.catalog.jdbc.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
         .set(s"spark.sql.catalog.jdbc.s3.endpoint", "http://10.0.162.28:9000")
         .set(s"spark.sql.catalog.jdbc.s3.access-key-id", ls0Config.jdbcCatalogS3AccessKey)
         .set(s"spark.sql.catalog.jdbc.s3.secret-access-key", ls0Config.jdbcCatalogS3SecretKey)
         .set(s"spark.sql.catalog.jdbc.client.region", ls0Config.jdbcCatalogClientRegion)
   
         .set("spark.sql.defaultCatalog", "nessie")
   ``` 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177421999

   I debugged this program.This error occurs where the image is displayed.
   
   <img width="1585" height="617" alt="Image" src="https://github.com/user-attachments/assets/6bac7587-62fa-427d-aa3e-c28f02eb2e13" />
   I saw this URI in the S3 client. However, the configuration in the program all uses the IP address, not http://los.uisee.com. I'm not sure where this URI is obtained from.
   
   <img width="1631" height="931" alt="Image" src="https://github.com/user-attachments/assets/39eb5be8-672a-42cd-ae9a-a42b987a7c03" />
   
   orphan files list is correct
   <img width="1675" height="439" alt="Image" src="https://github.com/user-attachments/assets/2b40510b-0373-47dc-9fe1-51d4a7f913b2" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3178205052

   > [@ajantha-bhat](https://github.com/ajantha-bhat) the error comes from S3 remote signing. I'm guessing Nessie server is sending back the `s3.remote-signing-enabled` flag (indicating that it supports remote signing) with an additional `s3.signer.endpoint` that causes the issue.
   > 
   > [@risey-yimu](https://github.com/risey-yimu) in your debugging, can you check the values for the Signer configuration and share a screenshot?
   
   After debugging again, I found that the error occurred in the deleteFiles method of the DeleteOrphanFilesSparkAction class.
   
   ![Image](https://github.com/user-attachments/assets/d3d6ba3d-51c2-41aa-b216-2916f3252d83)
   The IO here is from table.io. So could this problem be related to nessie's configuration?The configuration is provided by the partner above
   
   <img width="1154" height="1065" alt="Image" src="https://github.com/user-attachments/assets/92c3d9c4-2db1-4c5f-a7de-10ff5746b823" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3235894126

   @nastra @ajantha-bhat @Fokko @aokolnychyi Please help me find a way to solve this problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "aka-peter (via GitHub)" <gi...@apache.org>.
aka-peter commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3292322512

   I ran into the exact same problem. It was solved by adding this config to nessie:
   
   `nessie.catalog.service.s3.default-options.request-signing-enabled=false
   `


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3174552507

   For Nessie, we recommend using Nessie gc tool from the nessie repo (not the Iceberg procedures). I recall we blocked it with `gc.enabled = false` (cannot override from alter table properties, at runtime it always used to set to false).  So, I wonder how you executed it.  Maybe something changed in the remove orphan files that doesn't check gc flag? 
   
   > java.lang.IllegalArgumentException: Invalid S3 URI: 'http://los.uisee.com/warehouse?delete'
   
   Also, the error is not from Nessie. It is from AWS SDK. You are using the invalid configs? 
   Even if you change catalogs, I assume you would get the same errors. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177716527

   maybe it is this configuration? 
   
   > .set("spark.hadoop.fs.s3a.endpoint", ls0Config.jdbcCatalogS3Endpoint)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "risey-yimu (via GitHub)" <gi...@apache.org>.
risey-yimu commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3178549679

   > [@risey-yimu](https://github.com/risey-yimu) can you please share the debugging info of the S3 client's configuration (basically what you shared in [#13781 (comment)](https://github.com/apache/iceberg/issues/13781#issuecomment-3177425546)) where one can see the Signer configuration that the server sent back
   
   <img width="1096" height="566" alt="Image" src="https://github.com/user-attachments/assets/92ca0960-b55f-420c-8567-fa0d18e3b83d" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3186452908

   > [@risey-yimu](https://github.com/risey-yimu) can you please share the debugging info of the S3 client's configuration (basically what you shared in [#13781 (comment)](https://github.com/apache/iceberg/issues/13781#issuecomment-3177425546)) where one can see the Signer configuration that the server sent back
   
   Thanks @nastra and @ajantha-bhat  Do you need any additional debug information or configuration? If need, please let us know.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3177928639

   @ajantha-bhat the error comes from S3 remote signing. I'm guessing Nessie server is sending back the `s3.remote-signing-enabled` flag (indicating that it supports remote signing) with an additional `s3.signer.endpoint` that causes the issue.
   
   @risey-yimu in your debugging, can you check the values for the Signer configuration and share a screenshot?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3178251422

   @risey-yimu can you please share the debugging info of the S3 client's configuration (basically what you shared in https://github.com/apache/iceberg/issues/13781#issuecomment-3177425546) where one can see the Signer configuration that the server sent back


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3174213392

   @ajantha-bhat could you take a look at this please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "ming12713 (via GitHub)" <gi...@apache.org>.
ming12713 commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3178106163

   > [@ajantha-bhat](https://github.com/ajantha-bhat) the error comes from S3 remote signing. I'm guessing Nessie server is sending back the `s3.remote-signing-enabled` flag (indicating that it supports remote signing) with an additional `s3.signer.endpoint` that causes the issue.
   > 
   > [@risey-yimu](https://github.com/risey-yimu) in your debugging, can you check the values for the Signer configuration and share a screenshot?
   @nastra 
   
   <img width="1056" height="758" alt="Image" src="https://github.com/user-attachments/assets/d7aca0f0-ba66-4db0-b6e4-3cd5740d4f0d" />
   
   nessie server with restcatalog config 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "TayfunBugraBas (via GitHub)" <gi...@apache.org>.
TayfunBugraBas commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3636515452

   > I ran into the exact same problem. It was solved by adding this config to nessie:
   > 
   > `nessie.catalog.service.s3.default-options.request-signing-enabled=false `
   
   I have the same error but when i try  this solution i get different error, 
   
   error is as follows : 
   
   25/12/09 12:07:33 WARN TaskSetManager: Lost task 0.0 in stage 6.0 (TID 4) (10.42.7.116 executor 1): software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentia
 lsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.,
   InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
   
   
   credentials were predefined and working
   
   nessie: 0.104.3
   spark: 3.5.3
   iceberg:1.9.2
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Nessie catalog use spark remove_orphan_files Failed [iceberg]

Posted by "TayfunBugraBas (via GitHub)" <gi...@apache.org>.
TayfunBugraBas commented on issue #13781:
URL: https://github.com/apache/iceberg/issues/13781#issuecomment-3640571891

   if you are using the nessie catalog type rest, try to change it with org.apache.iceberg.nessie.NessieCatalog
   
   Not Working(REST):
   `spark.sql.catalog.nessie.type=rest`
   
   Working:
   `spark.sql.catalog.nessie.catalog-impl=org.apache.iceberg.nessie.NessieCatalog`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org