You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ragnard (via GitHub)" <gi...@apache.org> on 2026/09/25 19:10:26 UTC

[PR] AWS: Delete S3 batches with the client of each credential prefix [iceberg]

ragnard opened a new pull request, #18265:
URL: https://github.com/apache/iceberg/pull/18265

   S3FileIO.deleteFiles grouped keys by bucket only. A full batch was sent with the client of whichever path completed it, even when the batch held keys from other storage credential prefixes in the same bucket, and the remainder was sent with the client for s3://<bucket>, which matches no credential prefix and falls back to the root client. With vended credentials the root client has none, so a delete of fewer files than the batch size deleted nothing.
   
   Group keys by the client of their storage prefix as well as by bucket, and send every batch, full or remainder, with its group's client.
   
   Generated-by: Claude Code (Claude Opus 5.5)
   
   
   Claude-Session: https://claude.ai/code/session_01PaUGfnt9z6FX8dFBwLRp5Q


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] AWS: Delete S3 batches with the client of each credential prefix [iceberg]

Posted by "raviston (via GitHub) via issues" <is...@iceberg.apache.org>.
raviston commented on PR #18265:
URL: https://github.com/apache/iceberg/pull/18265#issuecomment-6948821926

   This was closed by the stale bot, but #18263 still reproduces on `main`, and this fix still applies cleanly. @ragnard, could you reopen it?
   
   I checked it with a unit test that has one mock `S3Client` per prefix: credentials for `s3://bucket/table1` and `s3://bucket/table2`, and `s3.delete.batch-size=5`. Then I compared which client each key was sent to.
   
   | Files deleted | `main` | With this PR |
   |---|---|---|
   | 7 in table1 | the table1 client gets 5 keys; the **root client** (no vended credentials) gets the last 2 | the table1 client gets all 7 |
   | 7 in table1 + 3 in table2 | the table1 client gets 5 keys; the **table2 client** gets table1/f5, table1/f6 and the 3 table2 keys | each client gets only its own table's keys |
   
   So there's a second case besides the remainder going to the root client. On `main`, batches are grouped only by bucket, so one batch can hold keys from several prefixes. It is then sent with the client of whichever path filled it. So any `deleteFiles` call whose paths span more than one credential prefix can send some keys with another prefix's credentials. Grouping by client as well, as this PR does, fixes both cases.
   
   The rest of `:iceberg-aws:test` passes with the PR (241 tests). I couldn't run the new `TestS3FileIO` integration test locally (no Docker).
   
   @uros-b approved this. cc @nastra, could you take a look when you have a chance?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] AWS: Delete S3 batches with the client of each credential prefix [iceberg]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] closed pull request #18265: AWS: Delete S3 batches with the client of each credential prefix
URL: https://github.com/apache/iceberg/pull/18265


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] AWS: Delete S3 batches with the client of each credential prefix [iceberg]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #18265:
URL: https://github.com/apache/iceberg/pull/18265#issuecomment-6868296249

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] AWS: Delete S3 batches with the client of each credential prefix [iceberg]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #18265:
URL: https://github.com/apache/iceberg/pull/18265#issuecomment-6016970970

   Let's ping relevant committers for further review here, fix looks good to me
   
   @ragnard can we just please make sure to link the issue (#18263) here in PR body


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] AWS: Delete S3 batches with the client of each credential prefix [iceberg]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #18265:
URL: https://github.com/apache/iceberg/pull/18265#issuecomment-6673056959

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org