You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wombatu-kun (via GitHub)" <gi...@apache.org> on 2026/05/21 03:47:51 UTC
[PR] GCP: Route GCS batch deletes per credential prefix [iceberg]
wombatu-kun opened a new pull request, #16499:
URL: https://github.com/apache/iceberg/pull/16499
## Summary
Closes #16480
`GCSFileIO.internalDeleteFiles` partitioned `BlobId`s into fixed-size batches and then selected the GCS `Storage` client once per batch from only the first object's path. When a single `GCSFileIO` is configured with multiple per-prefix `StorageCredential`s (vended-credentials flow), a batch that crossed prefix boundaries was issued in full through whichever client matched the first object — sending the rest of the batch through the wrong credentials. The fix groups `BlobId`s by their `PrefixedStorage` client (via the existing longest-prefix-match `clientForStoragePath` helper) before partitioning into batches, and uses each client's own `deleteBatchSize`. No public API or exception contract changes.
This mirrors how `S3FileIO.deleteFiles` already groups by bucket before batching.
## Tests
Two new unit tests in `TestGCSFileIO`:
- `deleteFilesRoutesToCorrectClientPerPrefix` — interleaves objects across two credential-prefixed buckets and asserts that each per-prefix `Storage` client receives only its own `BlobId`s.
- `deleteFilesBatchesPerClient` — sets a small `gcs.delete.batch-size` and asserts that batches stay per-client and never mix `BlobId`s from two prefixes.
Confirmed locally with `./gradlew :iceberg-gcp:test --tests "org.apache.iceberg.gcp.gcs.TestGCSFileIO"` (26 tests, 0 failures). The new tests also fail when run against the previous code, confirming they catch the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] GCP: Route GCS batch deletes per credential prefix [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16499:
URL: https://github.com/apache/iceberg/pull/16499#issuecomment-5610710157
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] GCP: Route GCS batch deletes per credential prefix [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on PR #16499:
URL: https://github.com/apache/iceberg/pull/16499#issuecomment-5905083117
@amogh-jahagirdar you merged #12799 (multi-prefix storage credentials for S3) and #13276 in `GCSFileIO`, so this looks like your area - would you have a moment for this one? `GCSFileIO` picks one client per delete batch from the batch's first object, so a batch that spans two credential prefixes sends the rest of its deletes through the wrong credentials (#16480). #18263 reports the same class of bug on the S3 side. The red CVE Scan jobs are the repo-wide jackson-databind finding that #18318 fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] GCP: Route GCS batch deletes per credential prefix [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16499:
URL: https://github.com/apache/iceberg/pull/16499#issuecomment-4763859176
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] GCP: Route GCS batch deletes per credential prefix [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on PR #16499:
URL: https://github.com/apache/iceberg/pull/16499#issuecomment-4766615177
not stale
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] GCP: Route GCS batch deletes per credential prefix [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on PR #16499:
URL: https://github.com/apache/iceberg/pull/16499#issuecomment-4504758341
PR isn't the cause of Kafka Connect CI failure. I've fixed Kafka Connect integration test flakiness in separate PR https://github.com/apache/iceberg/pull/16438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] GCP: Route GCS batch deletes per credential prefix [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16499:
URL: https://github.com/apache/iceberg/pull/16499#issuecomment-5234608024
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] GCP: Route GCS batch deletes per credential prefix [iceberg]
Posted by "wombatu-kun (via GitHub) via issues" <is...@iceberg.apache.org>.
wombatu-kun closed pull request #16499: GCP: Route GCS batch deletes per credential prefix
URL: https://github.com/apache/iceberg/pull/16499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org