You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "abnobdoss (via GitHub)" <gi...@apache.org> on 2026/05/27 00:59:00 UTC
[PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
abnobdoss opened a new pull request, #3420:
URL: https://github.com/apache/iceberg-python/pull/3420
# Rationale for this change
This is part 1 of 2 for fixing upsert segfault / recursion failures caused by large composite-key match filters. It should also improve planning performance because the current exact PyIceberg expression tree is expensive to construct, bind, and evaluate for large source batches.
For composite join keys, the initial upsert scan currently builds one `And(EqualTo(...), EqualTo(...))` predicate per unique source key and reduces them into a left-leaning `Or` tree. Large source batches therefore create deep recursive expression trees before the target scan starts; expression visitors then walk `Or.left` / `Or.right` recursively, so stack usage scales with the number of unique key combinations.
This change adds a bounded `create_file_match_filter` for the initial target file-pruning scan. It uses per-column min/max bounds, plus `IsNull` where needed, so the initial scan predicate size is tied to the number of join columns rather than the number of source rows. The exact `create_match_filter` is still used later for overwrite and insert filtering, so row-level correctness is unchanged.
The tradeoff is that the bounded file-pruning predicate is conservative: for composite keys it can scan cross-product false positives, which downstream exact matching culls before update/insert decisions. This can technically increase the number of files scanned, but the current failure mode makes upsert impossible for affected large composite-key inputs. The intent of this PR is to remove that crash first; if there is appetite to improve pruning further, a follow-up can use a more selective bounding-box or more sophisticated algorithm while preserving the bounded expression shape.
Match-filter timings below include expression construction, schema binding, PyArrow expression conversion, and applying the resulting Arrow filter to an in-memory table. These do not measure end-to-end upsert time, which depends on table layout, file pruning, IO, and downstream exact matching.
Benchmark script: https://gist.github.com/abnobdoss/824698ea74c29c02ead8da6410b89b6e
Current exact match-filter path:
| Unique source keys | 1 join column | 2 join columns | 3 join columns |
| --- | --- | --- | --- |
| 100 | 1.5 ms | 7.3 ms | 11 ms |
| 1k | 4.7 ms | 87 ms | 147 ms |
| 10k | 33 ms | SIGSEGV | SIGSEGV |
| 100k | 650 ms | SIGSEGV | SIGSEGV |
| 1m | 6.59 s | >30 s | >30 s |
Bounded file-match filter path after this PR:
| Unique source keys | 1 join column | 2 join columns | 3 join columns |
| --- | --- | --- | --- |
| 100 | 1.0 ms | 1.1 ms | 1.2 ms |
| 1k | 1.0 ms | 1.1 ms | 1.2 ms |
| 10k | 1.1 ms | 1.1 ms | 1.3 ms |
| 100k | 1.3 ms | 1.6 ms | 1.9 ms |
| 1m | 3.7 ms | 6.8 ms | 9.0 ms |
Follow-up work should address the remaining exact match-filter recursion paths used later during overwrite and insert filtering. Those predicates need exact semantics, so they should be fixed separately.
Related upsert performance / recursion reports and prior work: #2159, #2675, #3129, #2943, #3387.
## Are these changes tested?
Yes.
Added regression coverage for the large composite-key initial scan path, plus focused coverage for bounded file-match filter construction and the conservative false-positive culling path.
Ran:
- `uv run python -m pytest tests/table/test_upsert.py -k "large_composite_key_initial_scan or file_match_filter or create_match_filter"`
- `uv run python -m pytest tests/table/test_upsert.py`
- commit hooks
The added `test_upsert_large_composite_key_initial_scan_does_not_recurse` regression test runs the large composite-key upsert in a subprocess so runtimes that would segfault report a pytest failure instead of terminating the pytest worker; the equivalent base-branch path exits `139` under Python 3.10, while this branch exits successfully.
## Are there any user-facing changes?
No API changes. Upserts with large composite-key source batches should avoid the initial scan-planning recursion failure and may scan a conservative superset of candidate files before exact row matching.
---
AI assistance was used during development to help explore the failure mode, implement the change, and add tests. I reviewed the outputs throughout and validated the final changes locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-5210825502
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.
abnobdoss commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-5210883852
Still active - happy to address any feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.
abnobdoss commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-5573526444
Still active - happy to address any feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "azwanzuharimi (via GitHub)" <gi...@apache.org>.
azwanzuharimi commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-5886556406
Thank you for this PR. I tested it as part of #4027. The bounded scan filter works, but with 20,000 keys the scan returns about 100,000 rows and the insert path then builds an exact Or over them and crashes (exit 138 on macOS, pyarrow 25.0.1). #4027 uses the same idea, a superset filter for the scan, and adds an exact anti join in Arrow for the insert path and a re-append of removed rows for the overwrite path. The result table is in that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "ericyuanhui (via GitHub)" <gi...@apache.org>.
ericyuanhui commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-4900602403
Are there any updates on this? Has the issue been resolved? I noticed there are two PRs, but neither has been merged yet. Could someone provide some clarification on the current status?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-4814414809
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.
abnobdoss commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-6514199462
Hi @azwanzuharimi, thanks for testing this! I agree yours covers all three paths, this one was intentionally scoped to just the initial scan.
That scan is the step that decides which files get read, so this PR keeps its filter simple on purpose: per-column min/max bounds, which always prune by file stats no matter how many keys there are, and the exact matching happens after. A flat In doesn't have that property. Over 200 values it skips stats-based pruning (IN_PREDICATE_LIMIT), so once every key column has over 200 unique keys, which is pretty low for a real upsert, you end up scanning every file in the matched partitions or the whole table.
So I'd rather not close this yet. ANDing these bounds onto the Ins keeps pruning without making the filter deep. If #4027 implements these bounds specifically for the file filtering, I'm happy to close this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.
abnobdoss commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-4814428275
Still active - happy to address any feedback when reviewers have bandwidth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-5563401485
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(upsert): bound file match filter to avoid segfault [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3420:
URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-6511760181
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org