You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/09/28 07:48:20 UTC
[PR] [core] Reject a merge fan-in below two up front [paimon]
LuciferYang opened a new pull request, #10293:
URL: https://github.com/apache/paimon/pull/10293
### Purpose
`local-sort.max-num-file-handles` (the external-merge fan-in, default 128) has no lower-bound validation, so it can be set to 1. With fan-in = 1, `AbstractBinaryExternalMerger.mergeChannelList` divides by `maxFanIn - 1 = 0` once a second spill file forms, producing `Integer.MAX_VALUE` merges and a negative `subList` argument, so the sort crashes with `IllegalArgumentException`. Because it only triggers on the second spill, a fan-in of 1 passes small-data testing and then crash-loops in production as data grows (the job restarts on the same config and fails again).
This adds a `checkArgument(maxFanIn >= 2, ...)` in the `AbstractBinaryExternalMerger` constructor naming `local-sort.max-num-file-handles`, converting the latent, data-volume-dependent crash into a deterministic, actionable configuration error at construction time. The merger constructor is the single chokepoint through which every fan-in call path passes. A fan-in of 1 is degenerate for a merge sort (it can never converge N runs), so this rejects no legitimate configuration.
This closes #10292.
### Tests
- `BinaryExternalSortBufferTest#testFanInBelowTwoFailsFast` pins that constructing with a fan-in below two fails fast. Without the guard, construction succeeds and the crash only surfaces later on the second spill.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject a merge fan-in below two up front [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #10293:
URL: https://github.com/apache/paimon/pull/10293#issuecomment-5944767088
Thank you @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject a merge fan-in below two up front [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10293:
URL: https://github.com/apache/paimon/pull/10293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject a merge fan-in below two up front [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10293:
URL: https://github.com/apache/paimon/pull/10293#issuecomment-5944036564
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org