You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mderoy (via GitHub)" <gi...@apache.org> on 2025/02/27 22:39:47 UTC
[I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
mderoy opened a new issue, #12420:
URL: https://github.com/apache/iceberg/issues/12420
### Apache Iceberg version
1.8.0 (latest release)
### Query engine
None
### Please describe the bug 🐞
Due to a failure using the latest AWS sdk caused by - https://github.com/apache/iceberg/issues/12237 https://github.com/apache/iceberg/pull/12264 the deleteObjects api began failing.
This means that the dropTable api's purge option stopped working.
This caused thousands of objects to build up on our minio s3 filesystem without our immediate knowledge, as the exception was not being raised to the application. Expected behavior would be that failing to purge files should raise some kind of error.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #12420:
URL: https://github.com/apache/iceberg/issues/12420#issuecomment-2721152332
looking a bulk delete with the v2 sdk, as well as the 2.30.x regression, there are two other issues
* 500 failures of individual files are not reported; if an s3 shard behind the load balancer fails, that is not picked up and retried
* 503 failures of individual files are not escalated to a 503 reponse to the entire message
In both cases, the HTTP request returns 200, and the SDK itself doesn't do retries/throttling. It's left to the caller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #12420:
URL: https://github.com/apache/iceberg/issues/12420#issuecomment-2690285929
@mderoy we just released Iceberg 1.8.1 where the AWS SDK version has been downgraded to 2.29.52. Can you please check if that fixes your issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12420:
URL: https://github.com/apache/iceberg/issues/12420#issuecomment-3325967973
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12420:
URL: https://github.com/apache/iceberg/issues/12420#issuecomment-3272742137
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
Posted by "mderoy (via GitHub)" <gi...@apache.org>.
mderoy commented on issue #12420:
URL: https://github.com/apache/iceberg/issues/12420#issuecomment-2690590053
@nastra I have already verified that downgrading to 2.29.52 does in fact fix the issue dropping the objects from S3....however what I've filed this bug for is the lack of any exception or error being raised if such a condition occurs when dropping the table (either due to this bug or any other issue).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12420: S3FileIO does not raise exception when failing to drop/purge data
URL: https://github.com/apache/iceberg/issues/12420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #12420:
URL: https://github.com/apache/iceberg/issues/12420#issuecomment-2690863243
@mderoy I've updated the description, since the issue is caused by the AWS SDK making changes to the default integrity protection (see https://github.com/aws/aws-sdk-java-v2/discussions/5802), which negatively affects S3-compliant object storage solutions. It isn't caused by the issues you had linked earlier (https://github.com/apache/iceberg/issues/12237, https://github.com/apache/iceberg/pull/12264).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] S3FileIO does not raise exception when failing to drop/purge data [iceberg]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #12420:
URL: https://github.com/apache/iceberg/issues/12420#issuecomment-2701093250
AWS SDK team do not test the SDK with third party stores, it is left to downstream projects. S3a project does has a [qualifying an SDK update](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update) doc, but I'm revising it to require long-haul connections and at least one third party store (which must be named). Google GCS is the obvious target, as it is easy to setup.
One part of the new doc, along with the "don't make this last minute" will be "pick a release two weeks old, so you can see what bugs have surfaced". Maybe it should become "four weeks old"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org