You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "GayathriSrividya (via GitHub)" <gi...@apache.org> on 2026/05/26 10:53:44 UTC
[PR] Lazy pagination iterator for catalog list methods [iceberg-python]
GayathriSrividya opened a new pull request, #3416:
URL: https://github.com/apache/iceberg-python/pull/3416
Closes #3365
## Changes
- Changed `list_tables`, `list_namespaces`, `list_views` return types from `list[Identifier]` to `Iterator[Identifier]`
- REST catalog uses true generators with per-page HTTP fetch helpers (decorated with `@retry`) to correctly support auth retry logic
- Other catalogs (SQL, DynamoDB, Glue, Hive, BigQuery, NoOp) return `iter()` of their results
- CLI and output updated to work with iterators
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "GayathriSrividya (via GitHub)" <gi...@apache.org>.
GayathriSrividya commented on PR #3416:
URL: https://github.com/apache/iceberg-python/pull/3416#issuecomment-4560438407
Hi @rambleraptor and @Fokko — yes, PR #3416 was based on @jayceslesar's work from #2172. I've restructured the branch to credit them properly:
- **Commit 1** (authored by @jayceslesar): Changes abstract method signatures and adds `iter()` wrapping for all non-REST catalogs (BigQuery, DynamoDB, Glue, Hive, NoOp, SQL)
- **Commit 2** (authored by me): Implements proper lazy pagination generators for the REST catalog with per-page `@retry`-decorated helpers
All checks are passing on the latest push. Ready for review. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "GayathriSrividya (via GitHub)" <gi...@apache.org>.
GayathriSrividya commented on PR #3416:
URL: https://github.com/apache/iceberg-python/pull/3416#issuecomment-4582445564
> @GayathriSrividya was this based off of #2172? If it was, we should cherry-pick in the commits so that the original author shows up in this PR as well.
Hi @rambleraptor and @Fokko — yes, PR https://github.com/apache/iceberg-python/pull/3416 was based on @jayceslesar's work from https://github.com/apache/iceberg-python/pull/2172. I've restructured the branch to credit them properly:
Commit 1 (authored by @jayceslesar): Changes abstract method signatures and adds iter() wrapping for all non-REST catalogs (BigQuery, DynamoDB, Glue, Hive, NoOp, SQL)
Commit 2 (authored by me): Implements proper lazy pagination generators for the REST catalog with per-page @retry-decorated helpers
All checks are passing on the latest push. Ready for review. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "GayathriSrividya (via GitHub)" <gi...@apache.org>.
GayathriSrividya commented on PR #3416:
URL: https://github.com/apache/iceberg-python/pull/3416#issuecomment-4558562416
Addressed the CI failures with follow-up commits and pushed fixes to this PR branch.\n\nAll required checks are now passing on the latest commit (aa88377f).\n\nReady for review. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3416:
URL: https://github.com/apache/iceberg-python/pull/3416#issuecomment-4910303140
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "GayathriSrividya (via GitHub)" <gi...@apache.org>.
GayathriSrividya commented on PR #3416:
URL: https://github.com/apache/iceberg-python/pull/3416#issuecomment-4582442212
> @GayathriSrividya To credit the original author, it would be good to cherry-pick his work
Thanks for checking on this. Yes, this PR is based on @jayceslesar’s earlier work in #2172.
I restructured the branch to preserve attribution clearly:
- The original implementation commit is authored by @jayceslesar.
- My follow-up commit contains the REST lazy pagination changes and updates needed to bring the work forward.
I also updated the PR description to explicitly mention that this builds on @jayceslesar’s work. Let me know if there’s a better way you’d prefer me to handle attribution here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #3416:
URL: https://github.com/apache/iceberg-python/pull/3416#issuecomment-4548507093
@GayathriSrividya To credit the original author, it would be good to cherry-pick his work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #3416:
URL: https://github.com/apache/iceberg-python/pull/3416#issuecomment-4558966971
@GayathriSrividya was this based off of #2172? If it was, we should cherry-pick in the commits so that the original author shows up in this PR as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3416: Lazy pagination iterator for catalog list methods
URL: https://github.com/apache/iceberg-python/pull/3416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Lazy pagination iterator for catalog list methods [iceberg-python]
Posted by "GayathriSrividya (via GitHub)" <gi...@apache.org>.
GayathriSrividya commented on PR #3416:
URL: https://github.com/apache/iceberg-python/pull/3416#issuecomment-4582444124
> Hi @rambleraptor and @Fokko — yes, PR #3416 was based on @jayceslesar's work from #2172. I've restructured the branch to credit them properly:
>
> * **Commit 1** (authored by @jayceslesar): Changes abstract method signatures and adds `iter()` wrapping for all non-REST catalogs (BigQuery, DynamoDB, Glue, Hive, NoOp, SQL)
> * **Commit 2** (authored by me): Implements proper lazy pagination generators for the REST catalog with per-page `@retry`-decorated helpers
>
> All checks are passing on the latest push. Ready for review. Thanks!
Hi @rambleraptor and @Fokko — yes, PR https://github.com/apache/iceberg-python/pull/3416 was based on @jayceslesar's work from https://github.com/apache/iceberg-python/pull/2172. I've restructured the branch to credit them properly:
Commit 1 (authored by @jayceslesar): Changes abstract method signatures and adds iter() wrapping for all non-REST catalogs (BigQuery, DynamoDB, Glue, Hive, NoOp, SQL)
Commit 2 (authored by me): Implements proper lazy pagination generators for the REST catalog with per-page @retry-decorated helpers
All checks are passing on the latest push. Ready for review. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org