You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "cccs-eric (via GitHub)" <gi...@apache.org> on 2025/11/03 16:58:22 UTC
[I] Dropping Tables with PURGE in Spark 3.5 and Iceberg 1.9.1 Fails with Vended Credentials [iceberg]
cccs-eric opened a new issue, #14491:
URL: https://github.com/apache/iceberg/issues/14491
### Apache Iceberg version
1.9.1
### Query engine
Spark
### Please describe the bug 🐞
I am using a Polaris catalog with vended credentials enabled and encountering an issue when dropping tables with the `PURGE` option enabled. This happens when using Spark 3.5 and Iceberg 1.9.1.
The SQL command I run is as follows:
```sql
-- Using Spark SQL
DROP TABLE xyz PURGE
```
When examining the Spark HTTP requests sent to Polaris during this operation, I see the following sequence:
1. `DELETE /api/catalog/v1/my_catalog/namespaces/my_ns/tables/my_tbl HTTP/1.1` returns `HTTP/1.1 204 No Content`
2. `GET /api/catalog/v1/my_catalog/namespaces/my_ns/tables/my_tbl/credentials HTTP/1.1` returns `HTTP/1.1 404 Not Found` and the following body:
```json
{"error":{"message":"Table does not exist: [my_ns.my](http://my_ns.my/)_tbl","type":"NoSuchTableException","code":404}}
```
The table is successfully deleted from Polaris, but Spark then fails to delete the table's files because the credentials for the table cannot be retrieved. Since the table has already been deleted from Polaris, the GET request for its credentials results in an `HTTP 404 Not Found` error.
Below is a truncated stack trace from my logs:
```
ERROR 2025-10-29T17:48:55,975 154966 com.azure.storage.common.policy.RequestRetryPolicy [Thread-4] Unable to process: Table does not exist: my_ns.my_tbl
org.apache.iceberg.exceptions.RESTException: Unable to process: Table does not exist: my_ns.my_tbl
at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:248) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:212) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:215) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:299) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.rest.BaseHTTPClient.get(BaseHTTPClient.java:77) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.azure.adlsv2.VendedAdlsCredentialProvider.fetchCredentials(VendedAdlsCredentialProvider.java:139) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.azure.adlsv2.VendedAdlsCredentialProvider.sasTokenForAccount(VendedAdlsCredentialProvider.java:82) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.azure.adlsv2.VendedAdlsCredentialProvider.lambda$credentialForAccount$0(VendedAdlsCredentialProvider.java:75) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
...
at org.apache.iceberg.azure.adlsv2.VendedAdlsCredentialProvider.credentialForAccount(VendedAdlsCredentialProvider.java:78) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.azure.adlsv2.VendedAzureSasCredentialPolicy.maybeUpdateCredential(VendedAzureSasCredentialPolicy.java:58) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.azure.adlsv2.VendedAzureSasCredentialPolicy.processSync(VendedAzureSasCredentialPolicy.java:53) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53) ~[iceberg-azure-bundle-1.9.1.jar:?]
at com.azure.storage.common.policy.MetadataValidationPolicy.processSync(MetadataValidationPolicy.java:42) ~[iceberg-azure-bundle-1.9.1.jar:?]
...
at com.azure.storage.file.datalake.DataLakePathClient.lambda$existsWithResponse$10(DataLakePathClient.java:1593) ~[iceberg-azure-bundle-1.9.1.jar:?]
at com.azure.storage.file.datalake.implementation.util.DataLakeImplUtils.returnOrConvertException(DataLakeImplUtils.java:51) ~[iceberg-azure-bundle-1.9.1.jar:?]
at com.azure.storage.file.datalake.DataLakePathClient.existsWithResponse(DataLakePathClient.java:1593) ~[iceberg-azure-bundle-1.9.1.jar:?]
at com.azure.storage.file.datalake.DataLakePathClient.exists(DataLakePathClient.java:1570) ~[iceberg-azure-bundle-1.9.1.jar:?]
at org.apache.iceberg.azure.adlsv2.BaseADLSFile.exists(BaseADLSFile.java:66) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
at org.apache.iceberg.spark.SparkCatalog.purgeTable(SparkCatalog.java:372) ~[iceberg-spark-runtime-3.5_2.12-1.9.1.0-SNAPSHOT.jar:?]
```
The code responsible for this trace is [here](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.1/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java#L372).
Is this issue due to a fundamental limitation when using vended credentials? To me, this behavior appears broken for vended credentials because credential retrieval happens after the table is already dropped from the catalog. The `HTTP 404` response seems legitimate in this case.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Dropping Tables with PURGE in Spark 3.5 and Iceberg 1.9.1 Fails with Vended Credentials [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14491: Dropping Tables with PURGE in Spark 3.5 and Iceberg 1.9.1 Fails with Vended Credentials
URL: https://github.com/apache/iceberg/issues/14491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Dropping Tables with PURGE in Spark 3.5 and Iceberg 1.9.1 Fails with Vended Credentials [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14491:
URL: https://github.com/apache/iceberg/issues/14491#issuecomment-4393221480
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Dropping Tables with PURGE in Spark 3.5 and Iceberg 1.9.1 Fails with Vended Credentials [iceberg]
Posted by "cccs-eric (via GitHub)" <gi...@apache.org>.
cccs-eric commented on issue #14491:
URL: https://github.com/apache/iceberg/issues/14491#issuecomment-3505019843
This issue is linked to the following: https://github.com/apache/iceberg/issues/11754. In there, there is a reference to a community sync that talks about the issue I am facing: https://youtu.be/hcn2E8P7naU?si=tpiq98XN0qGCKXmC&t=775. At 17:55, a person describes a potential issue when credential vending is involved and it is exactly that issue I am facing.
I understand the problem with the current reference implementation and the support for purge/soft-delete, and I also understand what is being proposed for 2.0, but in the meantime, is there something that can be done? I am planning to modify my fork to simply delegate to the catalog when purge is requested by Spark and not having Spark do the delete, but a better solution would be to get credentials BEFORE removing the table from the catalog and let Spark do its thing.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Dropping Tables with PURGE in Spark 3.5 and Iceberg 1.9.1 Fails with Vended Credentials [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14491:
URL: https://github.com/apache/iceberg/issues/14491#issuecomment-4503804153
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org