You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "szehon-ho (via GitHub)" <gi...@apache.org> on 2026/06/30 18:09:41 UTC
[PR] Spark: Document broadcast size-map scaling in rewrite table action [iceberg]
szehon-ho opened a new pull request, #17023:
URL: https://github.com/apache/iceberg/pull/17023
## What changes are included in this PR?
Follow-up to #15470. That PR introduced a per-delete-file size map that is collected to the driver and broadcast to the manifest-rewrite tasks of `RewriteTablePathSparkAction`. During review I asked whether the broadcast map's memory footprint had been considered for delete-heavy tables ([review comment](https://github.com/apache/iceberg/pull/15470#pullrequestreview)).
This adds a short comment near the broadcast site documenting the scaling assumption: the map holds one entry per distinct rewritten delete-file location, so its footprint scales with the number of distinct delete files being rewritten rather than the table's total file count. No behavior change.
The comment is added identically to the v3.5, v4.0, and v4.1 Spark modules.
## How are these changes tested?
Comment-only change; no functional change, so no new tests.
---
**AI Disclosure**
- Model: Claude Opus 4.8
- Platform/Tool: Cursor
- Human Oversight: fully reviewed
- Prompt Summary: Add a comment documenting the broadcast size-map scaling assumption raised in review of #15470.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Document broadcast size-map scaling in rewrite table action [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #17023:
URL: https://github.com/apache/iceberg/pull/17023#issuecomment-4846821825
sorry spurious pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Document broadcast size-map scaling in rewrite table action [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho closed pull request #17023: Spark: Document broadcast size-map scaling in rewrite table action
URL: https://github.com/apache/iceberg/pull/17023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org