You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "colinre (via GitHub)" <gi...@apache.org> on 2026/05/26 19:48:59 UTC
[PR] Spark: Support long-valued streaming max rows per micro-batch [iceberg]
colinre opened a new pull request, #16571:
URL: https://github.com/apache/iceberg/pull/16571
### Problem
Spark Structured Streaming row-based micro-batch planning was effectively capped at `Integer.MAX_VALUE` rows. This made very large initial backfills impractical because streams over multi-trillion-row tables could require thousands of micro-batches before reaching the live tail.
### Root Cause
`streaming-max-rows-per-micro-batch` was parsed and stored as an `int`, and planner defaults initialized the effective row limit to `Integer.MAX_VALUE` even when no row limit was configured.
### Change
Parse and propagate the streaming row soft limit as `long`, use `Long.MAX_VALUE` as the unconfigured row-limit sentinel, and preserve complete-file soft-limit behavior. File-count rate limiting is unchanged. This is a Codex change; I'm generally unfamiliar with this codebase.
### Tests
Added coverage for long-valued option parsing, unconfigured multi-trillion-row planning, explicit long-valued soft limits, planner default unpacking, and existing small row-limit behavior. The structured streaming planner tests cover both sync and async planning through existing parameterization.
### Compatibility
Existing option names, offsets, checkpoint compatibility, file-count limits, and soft-limit semantics are unchanged. Existing values at or below `Integer.MAX_VALUE` keep their behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Support long-valued streaming max rows per micro-batch [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16571:
URL: https://github.com/apache/iceberg/pull/16571#issuecomment-4888197019
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Support long-valued streaming max rows per micro-batch [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #16571: Spark: Support long-valued streaming max rows per micro-batch
URL: https://github.com/apache/iceberg/pull/16571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Support long-valued streaming max rows per micro-batch [iceberg]
Posted by "colinre (via GitHub)" <gi...@apache.org>.
colinre commented on PR #16571:
URL: https://github.com/apache/iceberg/pull/16571#issuecomment-4566598658
@karuppayya @huaxingao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Support long-valued streaming max rows per micro-batch [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16571:
URL: https://github.com/apache/iceberg/pull/16571#issuecomment-4823482618
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Support long-valued streaming max rows per micro-batch [iceberg]
Posted by "colinre (via GitHub)" <gi...@apache.org>.
colinre commented on PR #16571:
URL: https://github.com/apache/iceberg/pull/16571#issuecomment-4566640850
@jackye1995 for earlier PR 👋
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org