You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "abbywh (via GitHub)" <gi...@apache.org> on 2026/10/05 22:48:16 UTC
[PR] Spark: Allow distributed planning with unlimited driver results [iceberg]
abbywh opened a new pull request, #18389:
URL: https://github.com/apache/iceberg/pull/18389
Spark treats `spark.driver.maxResultSize=0` as unlimited, but Iceberg's 256 MiB planning check treats it as too small and forces local manifest planning. Allow distributed planning for zero while keeping the local path for positive limits below the threshold.
Test plan: Verify data and delete planning with zero, 128 MiB, and 256 MiB limits. The focused Spark 3.5 tests, formatting, checkstyle, and module build passed.
---
**AI Disclosure**
- Model: GPT-6
- Platform/Tool: Codex
- Human Oversight: [unknown - human to fill in]
- Prompt Summary: Fix distributed planning when Spark's driver result size limit is unlimited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Allow distributed planning with unlimited driver results [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on PR #18389:
URL: https://github.com/apache/iceberg/pull/18389#issuecomment-6161976930
Should we run some benchmark to understand the perf implication for this PR? i.e. some small table, if changed to distributed may see some regression while large table may have perf gain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Allow distributed planning with unlimited driver results [iceberg]
Posted by "abbywh (via GitHub)" <gi...@apache.org>.
abbywh commented on PR #18389:
URL: https://github.com/apache/iceberg/pull/18389#issuecomment-6085938165
@gerashegalov can do. I haven't checked this on 4.2 but will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org